Commit Graph

109 Commits

Author SHA1 Message Date
Eugen Rochko b6d7726ecb
Remove language detection through cld3 (#17478)
* Remove language detection through cld3

* Update app/helpers/languages_helper.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2022-02-08 02:41:17 +01:00
Claire c8b1e72a4f
Fix compacted JSON-LD possibly causing compatibility issues on forwarding (#17428) 2022-02-03 14:09:04 +01:00
Claire d1ecc323e7
Compact JSON-LD signed incoming activities (#17426)
Co-authored-by: Puck Meerburg <puck@puck.moe>
2022-02-03 14:07:29 +01:00
Claire 03d59340da
Fix Sidekiq warnings about JSON serialization (#17381)
* Fix Sidekiq warnings about JSON serialization

This occurs on every symbol argument we pass, and every symbol key in hashes,
because Sidekiq expects strings instead.

See https://github.com/mperham/sidekiq/pull/5071

We do not need to change how workers parse their arguments because this has
not changed and we were already converting to symbols adequately or using
`with_indifferent_access`.

* Set Sidekiq to raise on unsafe arguments in test mode

In order to more easily catch issues that would produce warnings in production
code.
2022-01-28 00:43:56 +01:00
Eugen Rochko 6505b39e5d
Fix poll updates being saved as status edits (#17373)
Fix #17344
2022-01-26 18:05:39 +01:00
Eugen Rochko d412a8d1f2
Fix error when processing poll updates (#17333)
Regression from #16697
2022-01-19 22:50:01 +01:00
Eugen Rochko 1060666c58
Add support for editing for published statuses (#16697)
* Add support for editing for published statuses

* Fix references to stripped-out code

* Various fixes and improvements

* Further fixes and improvements

* Fix updates being potentially sent to unauthorized recipients

* Various fixes and improvements

* Fix wrong words in test

* Fix notifying accounts that were tagged but were not in the audience

* Fix mistake
2022-01-19 22:37:27 +01:00
Jeong Arm 2d1f082bb6
Fix NameError on ActivityPub::FetchFeaturedCollectionService (#17326)
Related: #16954
2022-01-19 04:08:46 +01:00
Claire d5c9feb7b7
Add support for private pinned posts (#16954)
* Add support for private pinned toots

* Allow local user to pin private toots

* Change wording to avoid "direct message"
2022-01-17 00:49:55 +01:00
Eugen Rochko 7de0ee7aba
Remove Keybase integration (#17045) 2021-11-26 05:58:18 +01:00
Claire 3f9b28ce26
Add support for fetching Create and Announce activities by URI (#16383)
* Add support for fetching Create and Announce activities by URI

This should improve compatibility with ZAP and offer a way to fetch boosts,
which is currently not possible.

* Add tests
2021-10-21 01:14:04 +02:00
Claire 9a19227f17
Fix some RedisLocks auto-releasing too fast (#16276)
* Fix Delete and Create-related locks expiring too fast

Fixes #16238

By default, RedisLock expires after 10 seconds, which may not be enough to
process statuses, especially when those have attached media files.

This commit extends those 10 seconds to 15 minutes, which should be plenty
enough to handle any status, while being short enough to not waste many
sidekiq job retries in the exceedingly rare case in which a sidekiq process
would crash when processing a `Create` or `Delete`.

* Fix other RedisLock autorelease durations

Fixes #15645

- things that only perform a few simple database queries (e.g. finding and
  saving a record) have been left unchanged, so they'll still use the default
  10s duration
- things that perform significantly more complex database queries have been
  changed to a 5 minutes timeout
- things that perform multiple HTTP queries have been changed to a 15 minutes
  timeout
2021-05-19 23:52:08 +02:00
Takeshi Umeda c403c3695b
Fix to be able to redownload avatar and header (#16190)
* Fix to reset if header and avatar download fails

* Add RedownloadAvatarWorker and RedownloadHeaderWorker
2021-05-11 14:19:22 +02:00
Eugen Rochko 2c77d97e0d
Add joined date to profiles in web UI (#16169) 2021-05-07 14:33:19 +02:00
Eugen Rochko f627d2eb93
Fix trying to fetch key from empty URI when verifying HTTP signature (#16100) 2021-05-01 23:18:59 +02:00
luigi 7ea9588520
Use Enumerable#filter_map in more places (#15527) 2021-01-22 16:28:15 +01:00
luigi 087ed84367
Optimize map { ... }.compact calls (#15513)
* Optimize map { ... }.compact

using Enumerable#filter_map, supported since Ruby 2.7

* Add poyfill for Enumerable#filter_map
2021-01-10 00:32:01 +01:00
ThibG a60d9335d8
Fix resolving accounts sometimes creating duplicate records for a given AP id (#15364)
* Fix ResolveAccountService accepting mismatching acct: URI

* Set attributes that should be updated regardless of suspension

* Fix key fetching

* Automatically merge remote accounts with duplicate `uri`

* Add tests

* Add "tootctl accounts fix-duplicates"

Finds duplicate accounts sharing a same ActivityPub `id`, re-fetch them and
merge them under the canonical `acct:` URI.

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-18 23:26:26 +01:00
Eugen Rochko 3134691948
Add support for reversible suspensions through ActivityPub (#14989) 2020-11-08 00:28:39 +01:00
ThibG fb586584f2
Fix account processing failing because of large collections (#15027)
Fixes #15025
2020-10-23 14:21:31 +02:00
ThibG ca56527140
Add follower synchronization mechanism (#14510)
* Add support for followers synchronization on the receiving end

Check the `collectionSynchronization` attribute on `Create` and `Announce`
activities and synchronize followers from provided collection if possible.

* Add tests for followers synchronization on the receiving end

* Add support for follower synchronization on the sender's end

* Add tests for the sending end

* Switch from AS attributes to HTTP header

Replace the custom `collectionSynchronization` ActivityStreams attribute by
an HTTP header (`X-AS-Collection-Synchronization`) with the same syntax as
the `Signature` header and the following fields:
- `collectionId` to specify which collection to synchronize
- `digest` for the SHA256 hex-digest of the list of followers known on the
   receiving instance (where “receiving instance” is determined by accounts
   sharing the same host name for their ActivityPub actor `id`)
- `url` of a collection that should be fetched by the instance actor

Internally, move away from the webfinger-based `domain` attribute and use
account `uri` prefix to group accounts.

* Add environment variable to disable followers synchronization

Since the whole mechanism relies on some new preconditions that, in some
extremely rare cases, might not be met, add an environment variable
(DISABLE_FOLLOWERS_SYNCHRONIZATION) to disable the mechanism altogether and
avoid followers being incorrectly removed.

The current conditions are:
1. all managed accounts' actor `id` and inbox URL have the same URI scheme and
   netloc.
2. all accounts whose actor `id` or inbox URL share the same URI scheme and
   netloc as a managed account must be managed by the same Mastodon instance
   as well.

As far as Mastodon is concerned, breaking those preconditions require extensive
configuration changes in the reverse proxy and might also cause other issues.

Therefore, this environment variable provides a way out for people with highly
unusual configurations, and can be safely ignored for the overwhelming majority
of Mastodon administrators.

* Only set follower synchronization header on non-public statuses

This is to avoid unnecessary computations and allow Follow-related
activities to be handled by the usual codepath instead of going through
the synchronization mechanism (otherwise, any Follow/Undo/Accept activity
would trigger the synchronization mechanism even if processing the activity
itself would be enough to re-introduce synchronization)

* Change how ActivityPub::SynchronizeFollowersService handles follow requests

If the remote lists a local follower which we only know has sent a follow
request, consider the follow request as accepted instead of sending an Undo.

* Integrate review feeback

- rename X-AS-Collection-Synchronization to Collection-Synchronization
- various minor refactoring and code style changes

* Only select required fields when computing followers_hash

* Use actor URI rather than webfinger domain in synchronization endpoint

* Change hash computation to be a XOR of individual hashes

Makes it much easier to be memory-efficient, and avoid sorting discrepancy issues.

* Marginally improve followers_hash computation speed

* Further improve hash computation performances by using pluck_each
2020-10-21 18:04:09 +02:00
Eugen Rochko 7d985f2aac
Remove dependency on goldfinger gem (#14919)
There are edge cases where requests to certain hosts timeout when
using the vanilla HTTP.rb gem, which the goldfinger gem uses. Now
that we no longer need to support OStatus servers, webfinger logic
is so simple that there is no point encapsulating it in a gem, so
we can just use our own Request class. With that, we benefit from
more robust timeout code and IPv4/IPv6 resolution.

Fix #14091
2020-10-08 00:34:57 +02:00
Eugen Rochko 64aac30733
Add customizable thumbnails for audio and video attachments (#14145)
- Change audio files to not be stripped of metadata
- Automatically extract cover art from audio if it exists
- Add `thumbnail` parameter to `POST /api/v1/media`, `POST /api/v2/media` and `PUT /api/v1/media/:id`
- Add `icon` to represent it in attachments in ActivityPub
- Fix `preview_url` containing URL of missing missing image when there is no thumbnail instead of null
- Fix duration of audio not being displayed on public pages until the file is loaded
2020-06-29 13:56:55 +02:00
Eugen Rochko 5d8398c8b8
Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
Eugen Rochko 4b766f9846
Refactor monkey-patching of Goldfinger (#12561) 2020-05-10 11:41:43 +02:00
ThibG b154428e14
Add federation support for the "hide network" preference (#11673)
* Change ActivityPub follower/following collections to not link first page

* Add support for hiding followers and following of remote users

* Switch to using a single `hide_collections` column

* Address code style remarks
2020-03-09 00:10:29 +01:00
ThibG 3704402dcc
Fix duplicate accounts being created when fetching an account for its key only (#13147)
Fixes #13136

When a user's canonical acct domain is different from its id's domain
(WEB_DOMAIN ≠ LOCAL_DOMAIN), two webfinger queries are required to find the
canonical domain from the URI. However, we skip webfinger queries when
updating only the key of a remote user, which led to the creation of a
duplicate account, using the URI's domain instead of the canonical acct: one.
2020-02-24 21:15:53 +01:00
Daigo 3 Dango 206dfd7dad Clarify keyword arguments with ** (#12769)
This change is to suppress the warning below on on ruby-2.7.0:

- warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call

https://www.ruby-lang.org/en/news/2019/12/12/separation-of-positional-and-keyword-arguments-in-ruby-3-0/
2020-01-11 05:57:05 +09:00
ThibG 00c219aa45 Fix empty poll options not being filtered on remote poll update (#12484)
If a poll contains empty options (which is apparently possible on Pleroma),
it is created without them. However, the poll update code doesn't filter
empty options, and thus:
1. Clear known votes, as it assumes the set of options has changed
2. Errors out because it tries adding empty options, which fails validation

This commit fixes that by filtering them out the same way they are filtered
out at poll creation time.
2019-11-26 22:46:31 +01:00
ThibG 3babf8464b Add voters count support (#11917)
* Add voters count to polls

* Add ActivityPub serialization and parsing of voters count

* Add support for voters count in WebUI

* Move incrementation of voters count out of redis lock

* Reword “voters” to “people”
2019-09-29 22:58:01 +02:00
Eugen Rochko cb447b28c4
Add profile directory to web UI (#11688)
* Add profile directory to web UI

* Add a line of bio to the directory
2019-08-30 00:14:36 +02:00
Eugen Rochko 5d3feed191
Refactor fetching of remote resources (#11251) 2019-07-10 18:59:28 +02:00
Eugen Rochko 4e92183227
Refactor domain block checks (#11268) 2019-07-09 03:27:35 +02:00
Eugen Rochko 2cfa427ea7
Change domain block behaviour to prevent creation of accounts from suspended domains (#11219) 2019-07-01 19:13:30 +02:00
Eugen Rochko 707ddf7808
Change domain blocks to automatically support subdomains (#11138)
* Change domain blocks to automatically support subdomains

If a more authoritative domain is blocked (example.com), then the
same block will be applied to a subdomain (foo.example.com)

* Match subdomains of existing accounts when blocking/unblocking domains

* Improve code style
2019-06-22 00:13:10 +02:00
ThibG 14f6ce2885 Record account suspend/silence time and keep track of domain blocks (#10660)
* Record account suspend/silence time and keep track of domain blocks

* Also unblock users who were suspended/silenced before dates were recorded

* Add tests

* Keep track of suspending date for users suspended through the CLI

* Show accurate number of accounts that would be affected by unsuspending an instance

* Change migration to set silenced_at and suspended_at

* Revert "Also unblock users who were suspended/silenced before dates were recorded"

This reverts commit a015c65d2d1e28c7b7cfab8b3f8cd5fb48b8b71c.

* Switch from using suspended and silenced to suspended_at and silenced_at

* Add post-deployment migration script to remove `suspended` and `silenced` columns

* Use Account#silence! and Account#suspend! instead of updating the underlying property

* Add silenced_at and suspended_at migration to post-migration

* Change account fabricator to translate suspended and silenced attributes

* Minor fixes

* Make unblocking domains always retroactive
2019-05-14 19:05:02 +02:00
Eugen Rochko 1714ea5978
Add ActivityPub representation for identity proofs (#10414)
* Add ActivityPub representation for identity proofs

* Add tests
2019-03-30 02:12:06 +01:00
ThibG 62dd0b7f9c Fix processing poll Updates (#10333)
ActivityPub::ProcessPollService was checking the JSON-LD context although
it was passed only the `Question` object embedded in the `Update` activity.
2019-03-21 14:24:09 +01:00
Eugen Rochko 9f5b55ad4f
Fix poll update handler calling method was that was not available (#10246)
* Fix poll update handler calling method was that was not available

Fix regression from #10209

* Refactor VoteService

* Refactor ActivityPub::DistributePollUpdateWorker and optimize it

* Fix typo

* Fix typo
2019-03-12 22:58:59 +01:00
ThibG 3a92885a86 Support pushing and receiving updates to poll tallies (#10209)
* Process incoming poll tallies update

* Send Update on poll vote

* Do not send Updates for a poll more often than once every 3 minutes

* Include voters in people to notify of results update

* Schedule closing poll worker on poll creation

* Add new notification type for ending polls

* Add front-end support for ended poll notifications

* Fix UpdatePollSerializer

* Fix Updates not being triggered by local votes

* Fix tests failure

* Fix web push notifications for closing polls

* Minor cleanup

* Notify voters of both remote and local polls when those close

* Fix delivery of poll updates to mentioned accounts and voters
2019-03-11 00:49:31 +01:00
ThibG 09c042aa10 Handle StaleObjectError when retrieving polls (#10208) 2019-03-07 15:52:38 +01:00
ThibG df5924a1db Do not error out on unsalvageable errors in FetchRepliesService (#10175)
* Do not error out on unsalvageable errors in FetchRepliesService

Fixes #10152

* Fix FetchRepliesWorker erroring out on deleted statuses
2019-03-05 15:21:14 +01:00
Eugen Rochko 4037b5eb1e
Fix last_fetched_at not being set on polls (#10170) 2019-03-05 04:10:01 +01:00
Eugen Rochko e6900b167b
Fix another typo in ActivityPub::FetchRemotePollService (#10146) 2019-03-04 00:52:18 +01:00
Eugen Rochko 878a75ba21
Fix typo in ActivityPub::FetchRemotePollService (#10145) 2019-03-04 00:50:56 +01:00
Eugen Rochko ae1b9cf70a
Fix remote poll expiration time (#10144) 2019-03-04 00:44:34 +01:00
ThibG e13d3792f3 Make sure the poll is created before storing its id (#10142)
* Make sure the poll is created before storing its id

* Fix updating poll results

* Support fetching Question activities from the search bar
2019-03-04 00:39:06 +01:00
Eugen Rochko 230a012f00
Add polls (#10111)
* Add polls

Fix #1629

* Add tests

* Fixes

* Change API for creating polls

* Use name instead of content for votes

* Remove poll validation for remote polls

* Add polls to public pages

* When updating the poll, update options just in case they were changed

* Fix public pages showing both poll and other media
2019-03-03 22:18:23 +01:00
ThibG 9d3c6f1849 Improved remote thread fetching (#10106)
* Fetch up to 5 replies when discovering a new remote status

This is used for resolving threads downwards. The originating
server must add a “replies” attributes with such replies for it to
be useful.

* Add some tests for ActivityPub::FetchRepliesWorker

* Add specs for ActivityPub::FetchRepliesService

* Serialize up to 5 public self-replies for ActivityPub notes

* Add specs for ActivityPub::NoteSerializer

* Move exponential backoff logic to a worker concern

* Fetch first page of paginated collections when fetching thread replies

* Add specs for paginated collections in replies

* Move Note replies serialization to a first CollectionPage

The collection isn't actually paginable yet as it has no id nor
a `next` field. This may come in another PR.

* Use pluck(:uri) instead of map(&:uri) to improve performances

* Fix fetching replies when they are in a CollectionPage
2019-02-28 15:22:21 +01:00
Eugen Rochko dad339da6d
Filter incoming Create activities by relation to local activity (#10005)
Reject those from accounts with no local followers, from relays
that are not enabled, which do not address local accounts and are
not replies to accounts that do have local followers
2019-02-13 18:42:47 +01:00