This adds support for the WIP (at the time of this commit)
draft/chathistory extension, based on the draft at [1] and the
additional comments at [2].
This gets the history by parsing the chat logs, and is therefore only
enabled when the logs are enabled and the log path is configured.
Getting the history only from the logs adds some restrictions:
- we cannot get history by msgid (those are not logged)
- we cannot get the users masks (maybe they could be inferred from the
JOIN etc, but it is not worth the effort and would not work every
time)
The regular soju network history is not sent to clients that support
draft/chathistory, so that they can fetch what they need by manually
calling CHATHISTORY.
The only supported command is BEFORE for now, because that is the only
required command for an app that offers an "infinite history scrollback"
feature.
Regarding implementation, rather than reading the file from the end in
reverse, we simply start from the beginning of each log file, store each
PRIVMSG into a ring, then add the last lines of that ring into the
history we'll return later. The message parsing implementation must be
kept somewhat fast because an app could potentially request thousands of
messages in several files. Here we are using simple sscanf and indexOf
rather than regexps.
In case some log files do not contain any message (for example because
the user had not joined a channel at that time), we try up to a 100 days
of empty log files before giving up.
[1]: https://github.com/prawnsalad/ircv3-specifications/pull/3/files
[2]: https://github.com/ircv3/ircv3-specifications/pull/393/files#r350210018
Previously we dropped all TAGMSG as well as any client message tag sent
from downstream.
This adds support for properly forwarding TAGMSG and client message tags
from downstreams and upstreams.
TAGMSG messages are intentionally not logged, because they are currently
typically used for +typing, which can generate a lot of traffic and is
only useful for a few seconds after it is sent.
This is preparatory work for forwarding errors of downstream-initiated
messages to their sender, as well as any other unknown message.
Preivously, we only sent labels (for labeled-response) for specific
downstream messages, such as WHO, where we knew the reply should only be
sent to that specific downstream.
However, in the case of an error of a message that is not labeled, the
error reply is not be tagged with a downstream id label and we can't
forward it to a specific downstream. It is not a good solution either to
forward this error to all downstreams.
This adds labels to all downstream-initiated messages (provided the
upstream supports it).
Many IRC clients use the query `WHOIS nick nick` rather than
`WHOIS nick` when querying a nick. The former command means to
specifically query the WHOIS on the server to which `nick` is connected,
which is useful to get information that is sometimes not propagated
between servers, such as idle time.
In the case where a downstream sends WHOIS nick/network nick/network in
multi-server mode, we need to unmarshal both fields.
Previously, we did not unmarshal those fields, and upstreams would
receive `WHOIS nick/network nick`, which is incorrect.
This adds support for unmarshaling the target field if it is the same as
the mask field, by simply using the unmarshaled nick that is already
computed from the mask.
Sometimes, doing a LIST on a single upstream can be useful: if a user is
already connected to Rizon and freenode, sending a LIST will contain
tens of thousands of LIST replies that may not be useful if the user is
interested in another upstream.
This adds support for sending `LIST */network`, which follows the ELIST
M mask extension, that will only send LIST to that specific network. No
other masks are supported by this commit.
Users often have different nicks on different upstreams, and we should
support changing the user nick on a single upstream.
This adds support for a new trivial extension, `NICK nick/network`,
which will change the nick on the specified network, and do nothing for
the other networks.
Previously, the downstream nick was never changed, even when the
downstream sent a NICK message or was in single-server mode with a
different nick.
This adds support for updating the downstream nick in the following
cases:
- when a downstream sends NICK
- additionally, in single-server mode:
- when a downstream connects and its single network is connected
- when an upstream connects
- when an upstream sends NICK
Previously, we only considered channel modes in the modes of a MODE
messages, which means channel membership changes were ignored. This
resulted in bugs where users channel memberships would not be properly
updated and cached with wrong values. Further, mode arguments
representing entities were not properly marshaled.
This adds support for correctly parsing and updating channel memberships
when processing MODE messages. Mode arguments corresponding to channel
memberships updates are now also properly marshaled.
MODE messages can't be easily sent from history because marshaling these
messages require knowing about the upstream available channel types and
channel membership types, which is currently only possible when
connected. For now this is not an issue since we do not send MODE
messages in history.
User channel memberships are actually a set of memberships, not a single
value. This introduces memberships, a type representing a set of
memberships, stored as an array of memberships ordered by descending
rank.
This also adds multi-prefix to the permanent downstream and upstream
capabilities, so that we try to get all possible channel memberships.
Channels can now be detached by leaving them with the reason "detach",
and re-attached by joining them again. Upon detaching the channel is
no longer forwarded to downstream connections. Upon re-attaching the
history buffer is sent.
This makes use of cap-notify to dynamically advertise support for
away-notify. away-notify is advertised to downstream connections if all
upstreams support it.
When writing a PRIVMSG or NOTICE on a channel, it is very common to use
autocompletion to mention other users on that channel. When using soju
in multi-network mode, all users will have their nicked suffixed by
`/network`. This suffix should be removed before sending it upstream.
This adds support for removing all `/network` suffixes in messages sent
to a channel of that network.
Instead of having one ring buffer per network, each network has one ring
buffer per entity (channel or nick). This allows history to be more
fair: if there's a lot of activity in a channel, it won't prune activity
in other channels.
We now track history sequence numbers per client and per network in
networkHistory. The overall list of offline clients is still tracked in
network.offlineClients.
When all clients have received history, the ring buffer can be released.
In the future, we should get rid of too-old offline clients to avoid
having to maintain history for them forever. We should also add a
per-user limit on the number of ring buffers.
Any SendMessage call after Close could potentially block forever if the
outgoing channel was filled up. Now the channel is drained before the
writer goroutine exits.
This commit adds support for downstream LIST messages from multiple
concurrent downstreams to multiple concurrent upstreams, including
support for multiple pending LIST requests from the same downstream.
Because a unique RPL_LISTEND message must be sent to the requesting
downstream, and that there might be multiple upstreams, each sending
their own RPL_LISTEND, a cache of RPL_LISTEND replies of some sort is
required to match RPL_LISTEND together in order to only send one back
downstream.
This commit adds a list of "pending LIST" structs, which each contain a
map of all upstreams that yet need to send a RPL_LISTEND, and the
corresponding LIST request associated with that response. This list of
pending LISTs is sorted according to the order that the requesting
downstreams sent the LIST messages in. Each pending set also stores the
id of the requesting downstream, in order to only forward the replies to
it and no other downstream. (This is important because LIST replies can
typically amount to several thousands messages on large servers.)
When a single downstream makes multiple LIST requests, only the first
one will be immediately sent to the upstream servers. The next ones will
be buffered until the first one is completed. Distinct downstreams can
make concurrent LIST requests without any request buffering.
Each RPL_LIST message is forwarded to the downstream of the first
matching pending LIST struct.
When an upstream sends an RPL_LISTEND message, the upstream is removed
from the first matching pending LIST struct, but that message is not
immediately forwarded downstream. If there are no remaining pending LIST
requests in that struct is then empty, that means all upstreams have
sent back all their RPL_LISTEND replies (which means they also sent all
their RPL_LIST replies); so a unique RPL_LISTEND is sent to downstream
and that pending LIST set is removed from the cache.
Upstreams are removed from the pending LIST structs in two other cases:
- when they are closed (to avoid stalling because of a disconnected
upstream that will never reply to the LIST message): they are removed
from all pending LIST structs
- when they reply with an ERR_UNKNOWNCOMMAND or RPL_TRYAGAIN LIST reply,
which is typically used when a user is not allowed to LIST because they
just joined the server: they are removed from the first pending LIST
struct, as if an RPL_LISTEND message was received
Split user.register into two functions, one to make sure the user is
authenticated, the other to send our current state. This allows to get
rid of data races by doing the second part in the user goroutine.
Closes: https://todo.sr.ht/~emersion/soju/22
In a later commit, we'll be able to move part of downstreamConn.register
into the user goroutine to prevent races.
References: https://todo.sr.ht/~emersion/soju/22
Downstream and upstream message handling are slightly different because
downstreams can send KICK messages with multiple channels or users,
while upstreams can only send KICK messages with one channel and one
user (according to the RFC).
downstreamConnection.unmarshalEntity already returns an ircError of
command ERR_NOSUCHCHANNEL, so there's no need to explicitly return
another ircError of that type.
Using labeled-response, the replies to several commands such as NAMES,
WHO, WHOIS can be routed back to a specific downstream, rather than
being broadcast to all downstreams.
For example, after this commit, if the server supports labeled-response,
if a downstream requests the NAMES or WHO or WHOIS of a channel, the
replies of the upstream will only be sent back to that downstream, and
the other downstreams won't receive these messages.
- Add RPL_ISUPPORT support with CHANMODES, CHANTYPES, PREFIX parsing
- Add support for channel mode state with mode arguments
- Add upstream support for RPL_UMODEIS, RPL_CHANNELMODEIS
- Request channel MODE on upstream channel JOIN
- Use sane default channel mode and channel mode types
This allows message handlers to read upstream/downstream connection
information without causing any race condition.
References: https://todo.sr.ht/~emersion/soju/1
We now store SASL credentials in the database and automatically populate
them on NickServ REGISTER/IDENTIFY.
References: https://todo.sr.ht/~emersion/jounce/10
- RingConsumer is now used directly in the goroutine responsible for
writing downstream messages. This allows the ring buffer not to be
consumed on write error.
- RingConsumer now has a channel attached. This allows PRIVMSG messages
to always use RingConsumer, instead of also directly pushing messages
to all downstream connections.
- Multiple clients with the same history name are now supported.
- Ring is now protected by a mutex
