This commit takes insert query compilation and transaction creation out
of the critical loop for migrating message logs. I have tested with
the sqlite backend, and a speedup of approximately 40x has been achieved
for log migration.
Currently, if times sent to `CHATHISTORY TARGETS` are reversed (e.g.
current time occurs first, and initial time occurs second) then no
targets will be returned from the database backend.
Changes have been tested with the sqlite backend.
This adds a new config option, `logs db`, which enables storing chat
logs in the soju database.
Regular store options, CHATHISTORY options, and SEARCH operations are
supported, like the fs logs backend.
Messages are stored in a new table, Message. In order to track the list
of targets we have messages for in an optimized manner, another database
is used: MessageTarget.
All new requests are backend by indexes so should be fast even with
hundreds of thousands of messages.
A contrib script is provided for migrating existing logs fs chat logs to
the database. It can be run with eg:
go run ./contrib/migrate-logs/ logs/ sqlite3:soju.db
Co-authored-by: Simon Ser <contact@emersion.fr>
The FS message store truncates message times to the second.
This means that a message sent out as 2020-01-01T00:00:00.123Z could be
sent later as part of a CHATHISTORY batch as 2020-01-01T00:00:00.000Z,
which could cause issues in clients.
One such issue is a client sending a MARKREAD for
2020-01-01T00:00:00.000Z, with another client considering the
2020-01-01T00:00:00.123Z message it has as unread.
This fixes the issue by truncating all message times to the second when
using the FS message store.
Calvin Lee
Simon Ser
delthas