This commit takes insert query compilation and transaction creation out
of the critical loop for migrating message logs. I have tested with
the sqlite backend, and a speedup of approximately 40x has been achieved
for log migration.
Previously, we would pass SQLite FTS5 MATCH queries as is from the
SEARCH parameter, when searching for messages.
By default FTS5 queries follow a specific syntax, e.g. AND, OR, ...
can be used to filter searches logically.
Passing special characters in these unescaped queries, like `.`,
would be rejected by SQLite, failing the SEARCH request.
Here, in order to replicate PostgreSQL and FS behavior, we simply
want to search for the string as is, and not give it glob or logic
operators semantics.
To do that, we quote each token of the string, skipping over
separators.
No need to re-check that a Web Push subscription is valid every
time a downstream connects. Mobile devices may reconnect pretty
frequently.
Check at most once a day.
PostgreSQL tests use pg_temp only. pg_temp is never searched for FTS
objects, so creating then altering an FTS configuration will not work
because PostgreSQL will not be able to find the FTS configuration it
just created.
Instead, we explicitly refer to the FTS objects with their full name
including their prefix, which makes PostgreSQL able to find the object.
This is only needed for tests.
See: https://stackoverflow.com/a/31095452/2347617
See: https://www.postgresql.org/message-id/15191.1208975632@sss.pgh.pa.us
This adds a new config option, `logs db`, which enables storing chat
logs in the soju database.
Regular store options, CHATHISTORY options, and SEARCH operations are
supported, like the fs logs backend.
Messages are stored in a new table, Message. In order to track the list
of targets we have messages for in an optimized manner, another database
is used: MessageTarget.
All new requests are backend by indexes so should be fast even with
hundreds of thousands of messages.
A contrib script is provided for migrating existing logs fs chat logs to
the database. It can be run with eg:
go run ./contrib/migrate-logs/ logs/ sqlite3:soju.db
Co-authored-by: Simon Ser <contact@emersion.fr>
We need to use sql.NullTime, otherwise we get errors like these:
sql: Scan error on column index 7, name "downstream_interacted_at": unsupported Scan, storing driver.Value type <nil> into type *time.Time
Add a new flag to disable users. This can be useful to temporarily
deactivate an account without erasing data.
The user goroutine is kept alive for simplicity's sake. Most of the
infrastructure assumes that each user always has a running goroutine.
A disabled user's goroutine is responsible for sending back an error
to downstream connections, and listening for potential events to
re-enable the account.
This implements sql.Scanner and sql/driver.Valuer, so that we can
load/store time values into SQLite with the format we want, and
properly handle NULL (which the go-sqlite3 package doesn't do
correctly).
The soju username is immutable. Add a separate nickname setting so
that users can change their nickname for all networks.
References: https://todo.sr.ht/~emersion/soju/110
Some WebPushSubscription entries aren't tried to a network, in
which case the "network" column is NULL. But then all users share
the same row. Oops.
Fortunately network-less subscriptions aren't used for anything
yet, they're just stored. So the impact should be minimal.
Simon Ser
sentriz
Calvin Lee
delthas
rj1
zsrv
Łukasz Margiela
gildarts