Commit Graph

1296 Commits

Author SHA1 Message Date
Claire 28329ba62f
Add /api/v1/admin/domain_blocks (#18247)
* Add /api/v1/admin/domain_blocks

Fixes #18140

- `GET /api/v1/admin/domain_blocks` lists domain blocks
- `GET /api/v1/admin/domain_blocks/:id` shows one by ID
- `DELETE /api/v1/admin/domain_blocks/:id` deletes a given domain block
- `POST /api/v1/admin/domain_blocks` to create a new domain block:
  if it conflicts with an existing one, returns an error with
  an attribute `existing_domain_block` with the rendered domain block

* Simplify conflict handling as suggested in review
2022-06-01 17:31:36 +02:00
Eugen Rochko 9f81b9f29a
Fix suspended users being able to access APIs that don't require a user (#18524) 2022-05-26 22:04:05 +02:00
Eugen Rochko 96129c2f10
Fix confirmation redirect to app without `Location` header (#18523) 2022-05-26 22:03:54 +02:00
Claire 440eb71310
Change unapproved and unconfirmed account to not be accessible in the REST API (#17530)
* Change unapproved and unconfirmed account to not be accessible in the REST API

* Change Account#searchable? to reject unconfirmed and unapproved users

* Disable search for unapproved and unconfirmed users in Account.search_for

* Disable search for unapproved and unconfirmed users in Account.advanced_search_for

* Remove unconfirmed and unapproved accounts from Account.searchable scope

* Prevent mentions to unapproved/unconfirmed accounts

* Fix some old tests for Account.advanced_search_for

* Add some Account.advanced_search_for tests for existing behaviors

* Add some tests for Account.search_for

* Add Account.advanced_search_for tests unconfirmed and unapproved accounts

* Add Account.searchable tests

* Fix Account.without_unapproved scope potentially messing with previously-applied scopes

* Allow lookup of unconfirmed/unapproved accounts through /api/v1/accounts/lookup

This is so that the API can still be used to check whether an username is free
to use.
2022-05-26 15:50:33 +02:00
Eugen Rochko 6cf57c6765
Refactor how Redis locks are created (#18400)
* Refactor how Redis locks are created

* Fix autorelease duration on account deletion lock
2022-05-13 00:02:35 +02:00
Eugen Rochko 2b8dc58b7f
Change RSS feeds (#18356)
* Change RSS feeds

- Use date and time for titles instead of ellipsized text
- Use full content in body, even when there is a content warning
- Use media extensions

* Change feed icons and add width and height attributes to custom emojis

* Fix custom emoji animate on hover breaking

* Fix tests
2022-05-09 07:43:08 +02:00
Claire 9a3be0ad68
Fix error when looking handle with surrounding spaces (#18225) 2022-05-02 01:00:08 +02:00
Eugen Rochko 33f3818d66
Fix double render error when authorizing interaction (#18203) 2022-05-01 00:56:34 +02:00
Eugen Rochko 5a48bf1085
Fix error when trying to revoke OAuth token without supplying a token (#18205) 2022-05-01 00:51:32 +02:00
Claire 66a63d79ea
Fix 500 error when a bookmark or favorite has been reported and deleted (#18174) 2022-04-28 23:34:58 +02:00
Eugen Rochko 3917353645
Fix single Redis connection being used across all threads (#18135)
* Fix single Redis connection being used across all Sidekiq threads

* Fix tests
2022-04-28 17:47:34 +02:00
0x2019 012537452a
Fix error resposes for `from` search prefix (#17963)
* Fix error responses in `from` search prefix (addresses mastodon/mastodon#17941)

Using unsupported prefixes now reports a 422; searching for posts from an
account the instance is not aware of reports a 404. TODO: The UI for this
on the front end is abysmal.

Searching `from:username@domain` now succeeds when `domain` is the local
domain; searching `from:@username(@domain)?` now works as expected.

* Remove unused methods on new Error classes as they are not being used

Currently when `raise`d there are error messages being supplied, but
this is not actually being used. The associated `raise`s have been
edited accordingly.

* Remove needless comments

* Satisfy rubocop

* Try fixing tests being unable to find AccountFindingConcern methods

* Satisfy rubocop

* Simplify `from` prefix logic

This incorporates @ClearlyClaire's suggestion (see
https://github.com/mastodon/mastodon/pull/17963#pullrequestreview-933986737).

Accepctable account strings in `from:` clauses are more lenient than
before this commit; for example, `from:@user@example.org@asnteo +cat`
will not error, and return posts by @user@example.org containing the
word "cat". This is more consistent with how Mastodon matches mentions
in statuses. In addition, `from` clauses will not be checked for
syntatically invalid usernames or domain names, simply 404ing when
`Account.find_remote!` raises ActiveRecord::NotFound.

New code for this PR that is no longer used has been removed.
2022-04-08 21:21:49 +02:00
Claire 3906dd67ed
Fix extremely rare race condition when deleting a toot or account (#17994) 2022-04-08 19:17:37 +02:00
Eugen Rochko 8e20e16cf0
Change e-mail notifications to only be sent when recipient is offline (#17984)
* Change e-mail notifications to only be sent when recipient is offline

Change the default for follow and mention notifications back on

* Add preference to always send e-mail notifications

* Change wording
2022-04-08 18:03:31 +02:00
Eugen Rochko 465ee7792f
Fix pagination header on empty trends responses in REST API (#17986) 2022-04-07 18:06:15 +02:00
Eugen Rochko 6221b36b27
Remove sign-in token authentication, instead send e-mail about new sign-in (#17970) 2022-04-06 20:58:12 +02:00
Claire 62c6e12fa5
Fix admin API unconditionally requiring CSRF token (#17975)
Fixes #17898

Since #17204, the admin API has only been available through the web
application because of the unconditional requirement to provide a valid CSRF
token.

This commit changes it back to `null_session`, which should make it work
both with session-based authentication (provided a CSRF token) and with a
bearer token.
2022-04-06 20:57:18 +02:00
Eugen Rochko d116cb7733
Fix `GET /api/v1/trends/tags` missing `offset` param in REST API (#17973) 2022-04-06 20:56:57 +02:00
Claire 894956e20c
Fix /api/v1/admin/accounts (#17887)
* Fix /api/v1/admin/accounts

Compatibility was broken since #17009 which changed the underlying filter class
without changing the controller.

This commits restore support for the old parameters.

* Add /api/v2/admin/accounts with the new parameters

* Add tests

* Add missing filter for `silenced` status

Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>

Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
2022-03-28 23:57:38 +02:00
Eugen Rochko cefa526c6d
Refactor formatter (#17828)
* Refactor formatter

* Move custom emoji pre-rendering logic to view helpers

* Move more methods out of Formatter

* Fix code style issues

* Remove Formatter

* Add inline poll options to RSS feeds

* Remove unused helper method

* Fix code style issues

* Various fixes and improvements

* Fix test
2022-03-26 02:53:34 +01:00
Eugen Rochko e3a2203061
Add offset pagination to trends in REST API (#17872) 2022-03-26 00:26:50 +01:00
Eugen Rochko 67d550830b
Fix locale not being set in REST API (#17847) 2022-03-22 12:29:04 +01:00
Eugen Rochko e6ffbfb5e7
Add `types` param to `GET /api/v1/notifications` in REST API (#17767)
* Add `types` param to `GET /api/v1/notifications` in REST API

* Improve tests
2022-03-15 04:11:29 +01:00
Eugen Rochko bc320d6cec
Fix `POST /api/v1/emails/confirmations` not being available after sign-up (#17743) 2022-03-12 04:14:25 +01:00
Eugen Rochko b2cd34474b
Add rate limit for editing (#17728) 2022-03-09 20:06:51 +01:00
chandrn7 a6ed6845c9
Allow login through OpenID Connect (#16221)
* added OpenID Connect as an SSO option

* minor fixes

* added comments, removed an option that shouldn't be set

* fixed Gemfile.lock

* added newline to end of Gemfile.lock

* removed tab from Gemfile.lock

* remove chomp

* codeclimate changes and small name change to make function's purpose clearer

* codeclimate fix

* added SSO buttons to /about page

* minor refactor

* minor style change

* removed spurious change

* removed unecessary conditional from ensure_valid_username and added support for auth.info.name in user_params_from_auth

* minor changes
2022-03-09 12:07:35 +01:00
Eugen Rochko bd53dd5210
Change design of federation pages in admin UI (#17704)
* Change design of federation pages in admin UI

* Fix query performance in instance media attachments measure

* Fix reblogs being included in instance languages dimension
2022-03-09 08:52:32 +01:00
Eugen Rochko 8f6c67bfde
Fix performance of account timelines (#17709)
* Fix performance of account timelines

* Various fixes and improvements

* Fix duplicate results being returned

Co-authored-by: Claire <claire.github-309c@sitedethib.com>

* Fix grouping for pinned statuses scope

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-03-08 09:14:39 +01:00
Eugen Rochko edf09ec747
Add `/api/v1/accounts/familiar_followers` to REST API (#17700)
* Add `/api/v1/accounts/familiar_followers` to REST API

* Change hide network preference to be stored consistently for local and remote accounts

* Add dummy classes to migration

* Apply suggestions from code review

Co-authored-by: Claire <claire.github-309c@sitedethib.com>

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-03-07 09:36:47 +01:00
Eugen Rochko 2ea754b861
Fix duplicate notifications being possible after poll expiration (#17697) 2022-03-04 01:06:33 +01:00
Claire ff43e54a49
Allow editing media attachments for scheduled toots (#17690)
Fixes #17676
2022-03-03 16:13:58 +01:00
Eugen Rochko 631e495a79
Change `follow` scope to be covered by `read` and `write` scopes in REST API (#17678)
Deprecate `follow` scope
2022-03-03 16:13:40 +01:00
Eugen Rochko e24b14cc74
Fix leak of existence of otherwise inaccessible statuses in REST API (#17684) 2022-03-02 18:57:26 +01:00
Eugen Rochko 02b8d63fce
Fix report category not being saved in REST API (#17682) 2022-03-02 18:57:08 +01:00
Eugen Rochko 25d3dc4373
Add ability to mark statuses as sensitive from reports in admin UI (#17668)
* Add ability to mark statuses as sensitive from reports in admin UI

* Allow mark as sensitive action on statuses with preview cards
2022-03-01 22:20:29 +01:00
Claire 14919fe11e
Change old moderation strikes to be displayed in a separate page (#17566)
* Change old moderation strikes to be displayed in a separate page

Fixes #17552

This changes the moderation strikes displayed on `/auth/edit` to be those from
the past 3 months, and make all moderation strikes targeting the current user
available in `/disputes`.

* Add short description of what the strikes page is for

* Move link to list of strikes to “Account status” instead of navigation item

* Normalize i18n file

* Fix layout and styling of strikes link

* Revert highlights_on regexp

* Reintroduce account status summary

- this way, “Account status” is never empty
- account status is not necessarily bound to strikes, or recent strikes
2022-03-01 19:37:47 +01:00
Eugen Rochko 50ea54b3ed
Change authorized applications page (#17656)
* Change authorized applications page

* Hide revoke button for superapps and suspended accounts

* Clean up db/schema.rb
2022-03-01 16:48:58 +01:00
Claire 6aef76b5cd
Fix error when a MX is shared across blocked domains (#17650) 2022-02-26 17:26:28 +01:00
Eugen Rochko 27965ce5ed
Add trending statuses (#17431)
* Add trending statuses

* Fix dangling items with stale scores in localized sets

* Various fixes and improvements

- Change approve_all/reject_all to approve_accounts/reject_accounts
- Change Trends::Query methods to not mutate the original query
- Change Trends::Query#skip to offset
- Change follow recommendations to be refreshed in a transaction

* Add tests for trending statuses filtering behaviour

* Fix not applying filtering scope in controller
2022-02-25 00:34:14 +01:00
Eugen Rochko a29a982eaa
Change e-mail domain blocks to block IPs dynamically (#17635)
* Change e-mail domain blocks to block IPs dynamically

* Update app/workers/scheduler/email_domain_block_refresh_scheduler.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

* Update app/workers/scheduler/email_domain_block_refresh_scheduler.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2022-02-24 17:28:23 +01:00
Eugen Rochko 7b816eb5ae
Add notifications for new sign-ups (#16953) 2022-02-23 16:45:22 +01:00
dependabot[bot] 2e89067d21
Bump pundit from 2.1.1 to 2.2.0 (#17543)
* Bump pundit from 2.1.1 to 2.2.0

Bumps [pundit](https://github.com/varvet/pundit) from 2.1.1 to 2.2.0.
- [Release notes](https://github.com/varvet/pundit/releases)
- [Changelog](https://github.com/varvet/pundit/blob/main/CHANGELOG.md)
- [Commits](https://github.com/varvet/pundit/compare/v2.1.1...v2.2.0)

---
updated-dependencies:
- dependency-name: pundit
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* `include Pundit` is deprecated

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2022-02-19 13:30:06 +09:00
Eugen Rochko de448ab1b5
Change global `locale` param to `lang` to avoid conflicts (#17592) 2022-02-19 03:12:28 +01:00
Claire ac99f586bb
Fix issues when attempting to appeal an old strike (#17554)
* Display an error when an appeal could not be submitted

* Do not offer users to appeal old strikes

* Fix 500 error when trying to appeal a strike that is too old

* Avoid using an extra translatable string
2022-02-16 22:29:48 +01:00
Eugen Rochko 564efd0651
Add appeals (#17364)
* Add appeals

* Add ability to reject appeals and ability to browse pending appeals in admin UI

* Add strikes to account page in settings

* Various fixes and improvements

- Add separate notification setting for appeals, separate from reports
- Fix style of links in report/strike header
- Change approving an appeal to not restore statuses (due to federation complexities)
- Change style of successfully appealed strikes on account settings page
- Change account settings page to only show unappealed or recently appealed strikes

* Change appealed_at to overruled_at

* Fix missing method error
2022-02-14 21:27:53 +01:00
Claire 7396a254c1
Fix empty batch statuses selection causing a 500 error (#17532)
* Fix empty batch statuses selection causing a 500 error

* Simplify current_params
2022-02-12 19:52:17 +01:00
Eugen Rochko 8f03b7a2fb
Add notifications when a reblogged status has been updated (#17404)
* Add notifications when a reblogged status has been updated

* Change wording to say "edit" instead of "update" and add missing controls

* Replace previous update notifications with the most up-to-date one
2022-02-11 22:20:19 +01:00
Eugen Rochko d0fcf07436
Change actions in reports to require only one click (#17487) 2022-02-11 21:51:57 +01:00
Eugen Rochko 63002cde03
Add editing for published statuses (#17320)
* Add editing for published statuses

* Fix change of multiple-choice boolean in poll not resetting votes

* Remove the ability to update existing media attachments for now
2022-02-10 00:15:30 +01:00
Eugen Rochko 2f8159baad
Add `category` and `rule_ids` params to `POST /api/v1/reports` (#17492) 2022-02-10 00:10:16 +01:00
Eugen Rochko fd3a45e348
Add edit history to web UI (#17390)
* Add edit history to web UI

* Change history reducer to store items per status

* Fix missing loading prop
2022-02-09 01:17:07 +01:00
Eugen Rochko 85b86fe28c
Add global `locale` param (#17464)
- Remove the session-based locale stickyness
2022-02-08 02:34:56 +01:00
Claire 73a782391c
Fix replies collection incorrectly looping (#17462)
* Refactor tests

* Add tests

* Fix replies collection incorrectly looping
2022-02-07 17:06:43 +01:00
Claire 92658f0fb0
Fix instance actor not being dereferenceable (#17457)
* Add tests

* Fix instance actor not being dereferenceable

* Fix tests

* Fix tests for real
2022-02-06 15:31:03 +01:00
Eugen Rochko e03e7ac290
Fix error on account relationships page in admin UI (#17444) 2022-02-05 05:06:34 +01:00
Claire f5639e1cbe
Change public profile pages to be disabled for unconfirmed users (#17385)
Fixes #17382

Note that unconfirmed and unapproved accounts can still be searched for
and their (empty) account retrieved using the REST API.
2022-01-28 14:24:37 +01:00
Claire 03d59340da
Fix Sidekiq warnings about JSON serialization (#17381)
* Fix Sidekiq warnings about JSON serialization

This occurs on every symbol argument we pass, and every symbol key in hashes,
because Sidekiq expects strings instead.

See https://github.com/mperham/sidekiq/pull/5071

We do not need to change how workers parse their arguments because this has
not changed and we were already converting to symbols adequately or using
`with_indifferent_access`.

* Set Sidekiq to raise on unsafe arguments in test mode

In order to more easily catch issues that would produce warnings in production
code.
2022-01-28 00:43:56 +01:00
Claire bddd9ba36d
Add OMNIAUTH_ONLY environment variable to enforce externa log-in (#17288)
* Remove support for OAUTH_REDIRECT_AT_SIGN_IN

Fixes #15959

Introduced in #6540, OAUTH_REDIRECT_AT_SIGN_IN allowed skipping the log-in form
to instead redirect to the external OmniAuth login provider.

However, it did not prevent the log-in form on /about introduced by #10232 from
appearing, and completely broke with the introduction of #15228.

As I restoring that previous log-in flow without introducing a security
vulnerability may require extensive care and knowledge of how OmniAuth works,
this commit removes support for OAUTH_REDIRECT_AT_SIGN_IN instead for the time
being.

* Add OMNIAUTH_ONLY environment variable to enforce external log-in only

* Disable user registration when OMNIAUTH_ONLY is set to true

* Replace log-in links When OMNIAUTH_ONLY is set with exactly one OmniAuth provider
2022-01-23 15:52:58 +01:00
Claire cfa583fa71
Remove support for OAUTH_REDIRECT_AT_SIGN_IN (#17287)
Fixes #15959

Introduced in #6540, OAUTH_REDIRECT_AT_SIGN_IN allowed skipping the log-in form
to instead redirect to the external OmniAuth login provider.

However, it did not prevent the log-in form on /about introduced by #10232 from
appearing, and completely broke with the introduction of #15228.

As I restoring that previous log-in flow without introducing a security
vulnerability may require extensive care and knowledge of how OmniAuth works,
this commit removes support for OAUTH_REDIRECT_AT_SIGN_IN instead for the time
being.
2022-01-23 15:50:41 +01:00
Eugen Rochko 1060666c58
Add support for editing for published statuses (#16697)
* Add support for editing for published statuses

* Fix references to stripped-out code

* Various fixes and improvements

* Further fixes and improvements

* Fix updates being potentially sent to unauthorized recipients

* Various fixes and improvements

* Fix wrong words in test

* Fix notifying accounts that were tagged but were not in the audience

* Fix mistake
2022-01-19 22:37:27 +01:00
Eugen Rochko 14f436c457
Add notifications for statuses deleted by moderators (#17204) 2022-01-17 09:41:33 +01:00
Claire d5c9feb7b7
Add support for private pinned posts (#16954)
* Add support for private pinned toots

* Allow local user to pin private toots

* Change wording to avoid "direct message"
2022-01-17 00:49:55 +01:00
Eugen Rochko 8e84ebf0cb
Remove IP tracking columns from users table (#16409) 2022-01-16 13:23:50 +01:00
Claire 76761d5fc0
Add ability for admins to delete canonical email blocks (#16644)
* Add admin option to remove canonical email blocks from a deleted account

* Add tootctl canonical_email_blocks to inspect and remove canonical email blocks
2021-12-17 23:02:14 +01:00
Claire 7f803c41e2
Add ability to purge undeliverable domains from admin interface (#16686)
* Add ability to purge undeliverable domains from admin interface

* Add tests
2021-12-17 23:01:21 +01:00
heguro 41503507ec
Fix redirection when succeeded WebAuthn (#17098) 2021-12-05 21:50:12 +01:00
Eugen Rochko 0fb9536d38
Add batch suspend for accounts in admin UI (#17009) 2021-12-05 21:48:39 +01:00
Jeong Arm 1c826471e7
Fix admin statuses order(#16937) (#16969)
* Fix #16937

* Add test for statuses order
2021-11-26 22:12:27 +01:00
Eugen Rochko 7de0ee7aba
Remove Keybase integration (#17045) 2021-11-26 05:58:18 +01:00
Eugen Rochko 6e50134a42
Add trending links (#16917)
* Add trending links

* Add overriding specific links trendability

* Add link type to preview cards and only trend articles

Change trends review notifications from being sent every 5 minutes to being sent every 2 hours

Change threshold from 5 unique accounts to 15 unique accounts

* Fix tests
2021-11-25 13:07:38 +01:00
Claire 6da135a493
Fix reviving revoked sessions and invalidating login (#16943)
Up until now, we have used Devise's Rememberable mechanism to re-log users
after the end of their browser sessions. This mechanism relies on a signed
cookie containing a token. That token was stored on the user's record,
meaning it was shared across all logged in browsers, meaning truly revoking
a browser's ability to auto-log-in involves revoking the token itself, and
revoking access from *all* logged-in browsers.

We had a session mechanism that dynamically checks whether a user's session
has been disabled, and would log out the user if so. However, this would only
clear a session being actively used, and a new one could be respawned with
the `remember_user_token` cookie.

In practice, this caused two issues:
- sessions could be revived after being closed from /auth/edit (security issue)
- auto-log-in would be disabled for *all* browsers after logging out from one
  of them

This PR removes the `remember_token` mechanism and treats the `_session_id`
cookie/token as a browser-specific `remember_token`, fixing both issues.
2021-11-06 00:13:58 +01:00
Jeong Arm 458830ee7c
Fix statuses order in account's statuses admin page (#16937) 2021-11-04 15:49:35 +01:00
Takeshi Umeda 17f4e457b3
Add remove from followers api (#16864)
* Add followed_by? to account_interactions

* Add RemoveFromFollowersService

* Fix AccountBatch to use RemoveFromFollowersService

* Add remove from followers API
2021-10-18 12:02:35 +02:00
Eugen Rochko 07341e7aa6
Add graphs and retention metrics to admin dashboard (#16829) 2021-10-14 20:44:59 +02:00
Claire 5159ba26e4
Fix error when rendering public pages with media attachments (#16763)
* Add tests

* Fix error when rendering public pages with media attachments

* Add tests

* Fix tests

* Please CodeClimate
2021-10-13 15:27:19 +02:00
Claire 24f9ea7818
Fix webauthn secure key authentication (#16792)
* Add tests

* Fix webauthn secure key authentication

Fixes #16769
2021-09-30 05:26:29 +02:00
Eugen Rochko 52e5c07948
Change routing paths to use usernames in web UI (#16171) 2021-09-26 05:46:13 +02:00
Claire e0af97164a
Fix followers synchronization mechanism not working when URI has empty path (#16744)
Follow-up to #16510, forgot the controller exposing the actual followers…
2021-09-15 18:51:16 +02:00
Truong Nguyen 7283a5d3b9
Explicitly set userVerification to discoraged (#16545) 2021-08-26 09:51:22 -05:00
Claire 94bcf45321
Fix authentication failures after going halfway through a sign-in attempt (#16607)
* Add tests

* Add security-related tests

My first (unpublished) attempt at fixing the issues introduced (extremely
hard-to-exploit) security vulnerabilities, addressing them in a test.

* Fix authentication failures after going halfway through a sign-in attempt

* Refactor `authenticate_with_sign_in_token` and `authenticate_with_two_factor` to make the two authentication steps more obvious
2021-08-25 22:52:41 +02:00
Daniel 5c21021176
Fix undefined variable for Auth::OmniauthCallbacksController (#16654)
The addition of authentication history broke the omniauth login with
the following error:

  method=GET path=/auth/auth/cas/callback format=html
  controller=Auth::OmniauthCallbacksController action=cas status=500
  error='NameError: undefined local variable or method `user' for
  #<Auth::OmniauthCallbacksController:0x00000000036290>
  Did you mean?  @user' duration=435.93 view=0.00 db=36.19

* app/controllers/auth/omniauth_callbacks_controller.rb: fix variable
  name to `@user`
2021-08-25 17:40:56 +02:00
Claire 4ac78e2a06
Add feature to automatically delete old toots (#16529)
* Add account statuses cleanup policy model

* Record last inspected toot to delete to speed up successive calls to statuses_to_delete

* Add service to cleanup a given account's statuses within a budget

* Add worker to go through account policies and delete old toots

* Fix last inspected status id logic

All existing statuses older or equal to last inspected status id must be
kept by the current policy. This is an invariant that must be kept so that
resuming deletion from the last inspected status remains sound.

* Add tests

* Refactor scheduler and add tests

* Add user interface

* Add support for discriminating based on boosts/favs

* Add UI support for min_reblogs and min_favs, rework UI

* Address first round of review comments

* Replace Snowflake#id_at_start with with_random parameter

* Add tests

* Add tests for StatusesCleanupController

* Rework settings page

* Adjust load-avoiding mechanisms

* Please CodeClimate
2021-08-09 23:11:50 +02:00
Claire d8629e7b86
Add logging of S3-related errors (#16381) 2021-07-21 18:34:39 +02:00
Claire 30ce6e395c
Fix user email address being banned on self-deletion (#16503)
* Add tests

* Fix user email address being banned on self-deletion

Fixes #16498
2021-07-14 05:35:49 +02:00
Eugen Rochko 771c9d4ba8
Add ability to skip sign-in token authentication for specific users (#16427)
Remove "active within last two weeks" exception for sign in token requirement

Change admin reset password to lock access until the password is reset
2021-07-08 05:31:28 +02:00
Claire 49219508bc
Fix anonymous access to outbox not being cached by the reverse proxy (#16458)
* Fix anonymous access to outbox not being cached by the reverse proxy

Up until now, anonymous access to outbox was marked as public, but with a
0 duration for caching, which means remote proxies would only serve from cache
when the server was completely overwhelmed.

Changed that cache duration to one minute, so that repeated anonymous access
to one account's outbox can be appropriately cached.

Also added `Signature` to the `Vary` header in case a page is requested, so
that authenticated fetches are never served from cache (which only contains
public toots).

* Remove Vary: Accept header from webfinger controller

Indeed, we have stopped returning xrd, and only ever return jrd, so the
Accept request header does not matter anymore.

* Cache negative webfinger hits for 3 minutes
2021-07-03 21:13:47 +02:00
Claire 63b807cffc
Fix serialization of followers/following counts when user hides their network (#16418)
* Add tests

* Fix serialization of followers/following counts when user hides their network

Fixes #16382

Signed-off-by: Claire <claire.github-309c@sitedethib.com>
2021-06-21 20:14:47 +02:00
Eugen Rochko d174d12c83
Add authentication history (#16408) 2021-06-21 17:07:30 +02:00
Eugen Rochko 1410dffdf4
Fix e-mail confirmations API not working correctly (#16348)
* Fix e-mail confirmations API not working correctly

* Fix typo
2021-06-02 21:07:50 +02:00
Claire 3b27b09acb
Fix some IDs in instance actor outbox (#16343) 2021-05-31 22:59:30 +02:00
Jeong Arm 5ef216d032
Remove set-cookie header on custom.css (#16314)
* Remove set-cookie header on custom.css

* Additional fix for set-cookie
2021-05-30 17:57:47 +02:00
Claire 12f8f39e25
Fix media proxy RedisLocks auto-releasing too fast (#16291)
Follow-up to #16276
2021-05-22 15:00:33 +02:00
Eugen Rochko 74081433d0
Change trending hashtags to be affected be reblogs (#16164)
If a status with a hashtag becomes very popular, it stands to
reason that the hashtag should have a chance at trending

Fix no stats being recorded for hashtags that are not allowed
to trend, and stop ignoring bots

Remove references to hashtags in profile directory from the code
and the admin UI
2021-05-07 14:33:43 +02:00
Claire 566fc90913
Add Ruby 3.0 support (#16046)
* Fix issues with POSIX::Spawn, Terrapin and Ruby 3.0

Also improve the Terrapin monkey-patch for the stderr/stdout issue.

* Fix keyword argument handling throughout the codebase

* Monkey-patch Paperclip to fix keyword arguments handling in validators

* Change validation_extensions to please CodeClimate

* Bump microformats from 4.2.1 to 4.3.1

* Allow Ruby 3.0

* Add Ruby 3.0 test target to CircleCI

* Add test for admin dashboard warnings

* Fix admin dashboard warnings on Ruby 3.0
2021-05-06 14:22:54 +02:00
Takeshi Umeda 7cb34b32f8
Add management of delivery availability in Federation settings (#15771)
* Add management of delivery availavility in Federation settings

* fix translate

* Remove useless object creation

* Fix DeepSource issue

* Add shortcut for all

* Fix DeepSource(skipcq)

* Change 'remove' to 'clear'

* Fix style

* Change class method name (exhausted_deliveries_key_by)
2021-05-05 23:39:02 +02:00
Eugen Rochko 351c744590
Fix error when trying to render component for media without meta (#16112) 2021-05-05 21:16:55 +02:00
Eugen Rochko 059df83d1d
Fix database serialization failure returning HTTP 500 (#16101)
Database serialization failure occurs when a read-replica is used
and a query takes long enough that rows on the primary database
become unavailable. It should return HTTP 503 as it is temporary.

Re-order rescue definitions according to their status codes
2021-05-05 19:44:35 +02:00
Claire 8c44b723bb
Change confirmations controller to redirect to / for approved users (#16151)
Clicking the confirmation link multiple times currently leads to entering
account settings, which can be confusing. This commit changes that so that
it redirects to the root path, so it behaves the same way as clicking only
once in most cases.
2021-05-03 15:45:19 +02:00
abcang d0fc69d721
Further improve the media attached status query for accounts (#16106) 2021-04-26 18:57:46 +02:00
abcang 1f47511023
Improve media attached status query (#16105) 2021-04-25 06:34:48 +02:00
Eugen Rochko daccc07dc1
Change auto-following admin-selected accounts, show in recommendations (#16078) 2021-04-24 17:01:43 +02:00
Eugen Rochko ce2148c571
Add `policy` param to `POST /api/v1/push/subscriptions` (#16040)
With possible values `all`, `followed`, `follower`, and `none`,
control from whom notifications will generate a Web Push alert
2021-04-15 05:00:25 +02:00
Eugen Rochko f7117646af
Add cold-start follow recommendations (#15945) 2021-04-12 12:37:14 +02:00
Eugen Rochko 619fad6cf8
Remove spam check and dependency on nilsimsa gem (#16011) 2021-04-11 11:22:50 +02:00
Eugen Rochko 487e37d6d4
Add system checks to dashboard in admin UI (#15989) 2021-04-03 14:12:30 +02:00
Eugen Rochko 82cce18227
Change health check (#15988) 2021-04-03 02:39:04 +02:00
Claire a650a1157d
Fix /admin/tags/:id crashing since Rails 6.1 update (#15953)
Raw SQL passed to `pluck` now has to be explicitly marked as SQL via
Arel.sql, see https://github.com/rails/rails/pull/27947
2021-03-26 18:36:16 +01:00
Claire 59f94593d0
Add warning in admin dashboard if some required queues are not handled (#15954) 2021-03-26 18:22:54 +01:00
Eugen Rochko dd1eb9918a
Add `email` param to `POST /api/v1/emails/confirmations` (#15949)
Allow changing e-mail as long as the account is unconfirmed
2021-03-25 02:46:13 +01:00
Claire cbd0ee1d07
Update Mastodon to Rails 6.1 (#15910)
* Update devise-two-factor to unreleased fork for Rails 6 support

Update tests to match new `rotp` version.

* Update nsa gem to unreleased fork for Rails 6 support

* Update rails to 6.1.3 and rails-i18n to 6.0

* Update to unreleased fork of pluck_each for Ruby 6 support

* Run "rails app:update"

* Add missing ActiveStorage config file

* Use config.ssl_options instead of removed ApplicationController#force_ssl

Disabled force_ssl-related tests as they do not seem to be easily testable
anymore.

* Fix nonce directives by removing Rails 5 specific monkey-patching

* Fix fixture_file_upload deprecation warning

* Fix yield-based test failing with Rails 6

* Use Rails 6's index_with when possible

* Use ActiveRecord::Cache::Store#delete_multi from Rails 6

This will yield better performances when deleting an account

* Disable Rails 6.1's automatic preload link headers

Since Rails 6.1, ActionView adds preload links for javascript files
in the Links header per default.

In our case, that will bloat headers too much and potentially cause
issues with reverse proxies. Furhermore, we don't need those links,
as we already output them as HTML link tags.

* Switch to Rails 6.0 default config

* Switch to Rails 6.1 default config

* Do not include autoload paths in the load path
2021-03-24 10:44:31 +01:00
Claire b358229834
Further preparation for Rails 6 (#15916)
* Use ActiveRecord::Result#to_ary instead of deprecated to_hash

They do the same thing, and to_hash has been removed from Rails 6.1

* Explicitly name polymorphic indexes to workaround a bug in Rails 6.1

cf. https://github.com/rails/rails/issues/41693

* Fix incorrect usage of “foreign_key” in migration script

* Use `ActiveModel::Errors#delete` instead of deprecated clear method

* Fix link headers tests on Rails 6.1

Rails 6.1 adds values to the Link header by default, thus it is not a
LinkHeader object anymore. Fix the test to parse the Link header instead
of assuming it is a LinkHeader.
2021-03-19 02:45:34 +01:00
Claire 5027abecd1
Fix cache_collection crashing when given an empty collection (#15921)
* Fix cache_collection crashing when given an empty collection

* Add tests
2021-03-18 00:41:32 +01:00
Eugen Rochko ee1119208c
Add `POST /api/v1/emails/confirmations` to REST API (#15816)
Only available to the application the user originally signed-up with
2021-03-01 18:39:47 +01:00
Eugen Rochko 9aa37b32c3
Add `details` to error response for `POST /api/v1/accounts` in REST API (#15803) 2021-03-01 04:59:13 +01:00
Claire 75189af528
Fix crash on receiving requests with missing Digest header (#15782)
* Fix crash on receiving requests with missing Digest header

Return an error pointing out that Digest is missing, instead of crashing.

Fixes #15743

* Fix from review feedback
2021-02-26 17:40:27 +01:00
Eugen Rochko 8331fdf7e0
Add server rules (#15769) 2021-02-21 19:50:12 +01:00
Justin Tracey c9e8e1739c
replace all instances of "ends_with?" with "end_with?" (#15745)
The "ends_with?" method is just a Rails alias of Ruby's "end_with?" method.
Using the latter makes the code less brittle.
2021-02-19 09:56:14 +01:00
Eugen Rochko e31ed27485
Add `GET /api/v1/accounts/lookup` REST API (#15740) 2021-02-16 15:28:32 +01:00
Eugen Rochko 15ced8728f
Refactor Api::Web::SettingsController (#15717) 2021-02-12 07:19:15 +01:00
Cecylia Bocovich e79f8dd85c
Onion service related changes to HTTPS handling (#15560)
* Enable secure cookie flag for https only

* Disable force_ssl for .onion hosts only

Co-authored-by: Aiden McClelland <me@drbonez.dev>
2021-02-11 04:40:13 +01:00
Claire acdeb162b8
Create instance actor if it hasn't been properly seeded (#15693)
An uncommon but somewhat difficult to digagnose issue is dealing with
improperly-seeded databases. In such cases, instance-signed fetches will
fail with a ActiveRecord::RecordNotFound error, usually caught and handled
as generic 404, leading people to think the remote resource itself has not
been found, while it's the local instance actor that does not exist.

This commit changes the code so that failure to find the instance actor
automatically creates a new one, so that improperly-seeded databases do
not cause any issue.
2021-02-09 18:12:54 +01:00
abcang 7ab53f221a
Improved performance of notification preloading (#15640)
* Improved performance of notification preloading

* Remove Cacheable from Notification

* Fix test
2021-01-31 21:24:57 +01:00
ThibG 2ff01f78f7
Fix /activity endpoint not require signature in authorized fetch mode (#15592)
Fixes #15589

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2021-01-19 06:47:36 +01:00
ThibG 54d4e5252b
Use Rails' index_by where it makes sense (#15542)
* Use Rails' index_by where it makes sense

* Fix tests

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2021-01-12 09:27:38 +01:00
Levi Bard 11d603101a
Fix muting users with duration via the REST api (#15516) 2021-01-10 12:47:21 +01:00
luigi 087ed84367
Optimize map { ... }.compact calls (#15513)
* Optimize map { ... }.compact

using Enumerable#filter_map, supported since Ruby 2.7

* Add poyfill for Enumerable#filter_map
2021-01-10 00:32:01 +01:00
ThibG a1a8aa4a08
Skip processing Update activities on unknown accounts (#15514)
This also skips fetching the actor completely.

This will be useful if we end up distributing Update activities linked to
account suspensions more widely (they are currently only delivered to
the suspended account's followers), as currently, instances not knowing
about the suspended account would fetch it to then process the suspension.

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2021-01-08 12:26:29 +01:00
ThibG 4552449c26
Fix profile update not being sent on profile/header picture deletion (#15461)
Fixes #15460

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-30 23:19:18 +01:00
trwnh 4950e59cdc
Use existing FeaturedTag serializer and delete AccountFeaturedTag serializer (#15415)
* Update featured_tags_controller.rb

* Update featured_tag_serializer.rb

* Update featured_tag_serializer.rb

* Delete account_featured_tag_serializer.rb

* please codeclimate

* please codeclimate
2020-12-23 16:43:38 +01:00
Eugen Rochko 8a95867693
Add option to obfuscate domain name in public list of domain blocks (#15355)
- Replace the middle of the domain with * characters (except for periods)
- Add SHA-256 digest of the domain name in tooltip
2020-12-18 08:30:41 +01:00
Eugen Rochko 1045549f85
Add stoplight for object storage failures, return HTTP 503 (#13043) 2020-12-15 12:55:29 +01:00
Eugen Rochko 216b85b053
Fix performance on instances list in admin UI (#15282)
- Reduce duplicate queries
- Remove n+1 queries
- Add accounts count to detailed view
- Add separate action log entry for updating existing domain blocks
2020-12-14 09:06:34 +01:00
ThibG 49eb4d4ddf
Add honeypot fields and minimum fill-out time for sign-up form (#15276)
* Add honeypot fields to limit non-specialized spam

Add two honeypot fields: a fake website input and a fake password confirmation
one. The label/placeholder/aria-label tells not to fill them, and they are
hidden in CSS, so legitimate users should not fall into these.

This should cut down on some non-Mastodon-specific spambots.

* Require a 3 seconds delay before submitting the registration form

* Fix tests

* Move registration form time check to model validation

* Give people a chance to clear the honeypot fields

* Refactor honeypot translation strings

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-10 06:27:26 +01:00
Eugen Rochko 48bef17cc9
Fix slow distinct queries where grouped queries are faster (#15287)
About 2x speed-up on inboxes query
2020-12-07 12:08:30 +01:00
Takeshi Umeda 44d5c6bc8f
Fix incorrect conditions for suspended accounts in Get API for account featured tags (#15270) 2020-12-04 04:22:35 +01:00
ThibG c43f4cd3bb
Fix not being able to unfavorite toots one has lost access to (#15192)
Fixes #15191
2020-11-21 06:18:09 +01:00
ThibG 96c1e71329
Add import/export feature for bookmarks (#14956)
* Add ability to export bookmarks

* Add support for importing bookmarks

* Add bookmark import tests

* Add bookmarks export test
2020-11-19 17:48:13 +01:00
Eugen Rochko 8532429af7
Fix 2FA/sign-in token sessions being valid after password change (#14802)
If someone tries logging in to an account and is prompted for a 2FA
code or sign-in token, even if the account's password or e-mail is
updated in the meantime, the session will show the prompt and allow
the login process to complete with a valid 2FA code or sign-in token
2020-11-12 23:05:01 +01:00
Takeshi Umeda 148ce97e21
Add interrelationship icon (#15149)
* Add interrelationship icon

* Fix arrow for rtl

* Fix to predefined color
2020-11-12 17:43:12 +01:00
Takeshi Umeda 2b1a6e734f
Add follow selected followers button (#15148)
* Add follow selected followers button

* Fix unused variable

* Fix i18n normalize
2020-11-12 16:58:00 +01:00
Eugen Rochko 3134691948
Add support for reversible suspensions through ActivityPub (#14989) 2020-11-08 00:28:39 +01:00
ThibG b29defb851
Change order of announcements in admin page to sort them newest-first (#15091)
* Change order of announcements in admin page to sort show newly-created first

Fixes #15090

* Use reverse-chronological rather than creation date only
2020-11-04 21:15:22 +01:00
Takeshi Umeda d6fe0c94ca
Add account sensitized (#14361)
* Add account sensitized

* Fix i18n normalize

* Fix description and spec

* Fix spec

* Fix wording
2020-11-04 20:45:01 +01:00
ThibG fa929d8b81
Tweak signature verification (#15069)
* Add more specific error message when request body digest is invalid

This may help other implementors debug their implementation.

* Relax Host parameter requirement to GET requests

The only POST requests processed by Mastodon need objects/actors (including
their host) to be explicitly mentioned in the request's body, so replaying
a legitimate request to another host should not be a security issue.

* Support Digest headers using multiple algorithms or lowercase alogirthm names
2020-11-01 23:38:31 +01:00
Mélanie Chauvel 0a8ab822e2
Sort filters by “keyword or phrase” in Settings (#15050) 2020-10-27 03:00:06 +01:00
ThibG ca56527140
Add follower synchronization mechanism (#14510)
* Add support for followers synchronization on the receiving end

Check the `collectionSynchronization` attribute on `Create` and `Announce`
activities and synchronize followers from provided collection if possible.

* Add tests for followers synchronization on the receiving end

* Add support for follower synchronization on the sender's end

* Add tests for the sending end

* Switch from AS attributes to HTTP header

Replace the custom `collectionSynchronization` ActivityStreams attribute by
an HTTP header (`X-AS-Collection-Synchronization`) with the same syntax as
the `Signature` header and the following fields:
- `collectionId` to specify which collection to synchronize
- `digest` for the SHA256 hex-digest of the list of followers known on the
   receiving instance (where “receiving instance” is determined by accounts
   sharing the same host name for their ActivityPub actor `id`)
- `url` of a collection that should be fetched by the instance actor

Internally, move away from the webfinger-based `domain` attribute and use
account `uri` prefix to group accounts.

* Add environment variable to disable followers synchronization

Since the whole mechanism relies on some new preconditions that, in some
extremely rare cases, might not be met, add an environment variable
(DISABLE_FOLLOWERS_SYNCHRONIZATION) to disable the mechanism altogether and
avoid followers being incorrectly removed.

The current conditions are:
1. all managed accounts' actor `id` and inbox URL have the same URI scheme and
   netloc.
2. all accounts whose actor `id` or inbox URL share the same URI scheme and
   netloc as a managed account must be managed by the same Mastodon instance
   as well.

As far as Mastodon is concerned, breaking those preconditions require extensive
configuration changes in the reverse proxy and might also cause other issues.

Therefore, this environment variable provides a way out for people with highly
unusual configurations, and can be safely ignored for the overwhelming majority
of Mastodon administrators.

* Only set follower synchronization header on non-public statuses

This is to avoid unnecessary computations and allow Follow-related
activities to be handled by the usual codepath instead of going through
the synchronization mechanism (otherwise, any Follow/Undo/Accept activity
would trigger the synchronization mechanism even if processing the activity
itself would be enough to re-introduce synchronization)

* Change how ActivityPub::SynchronizeFollowersService handles follow requests

If the remote lists a local follower which we only know has sent a follow
request, consider the follow request as accepted instead of sending an Undo.

* Integrate review feeback

- rename X-AS-Collection-Synchronization to Collection-Synchronization
- various minor refactoring and code style changes

* Only select required fields when computing followers_hash

* Use actor URI rather than webfinger domain in synchronization endpoint

* Change hash computation to be a XOR of individual hashes

Makes it much easier to be memory-efficient, and avoid sorting discrepancy issues.

* Marginally improve followers_hash computation speed

* Further improve hash computation performances by using pluck_each
2020-10-21 18:04:09 +02:00
OSAMU SATO 96761752ec
Add duration parameter to muting. (#13831)
* Adding duration to muting.

* Remove useless checks
2020-10-13 01:01:14 +02:00
Eugen Rochko 5e1364c448
Add IP-based rules (#14963) 2020-10-12 16:33:49 +02:00
ThibG 78e45a5285
Add option to disable swiping motions across the WebUI (#13885)
Fixes #13882
2020-09-30 19:31:03 +02:00
Eugen Rochko 8c8bf0289f
Fix not being able to enable status notifications in Web Push API (#14822) 2020-09-21 17:48:55 +02:00