Awesome

Compress Synapse State Tables

This workspace contains experimental tools that attempt to reduce the number of rows in the state_groups_state table inside of a Synapse Postgresql database.

Automated tool: synapse_auto_compressor

Introduction:

This tool is significantly more simple to use than the manual tool (described below). It scans through all of the rows in the state_groups database table from the start. When it finds a group that hasn't been compressed, it runs the compressor for a while on that group's room, saving where it got up to. After compressing a number of these chunks it stops, saving where it got up to for the next run of the synapse_auto_compressor.

It creates three extra tables in the database: state_compressor_state which stores the information needed to stop and start the compressor for each room, state_compressor_progress which stores the most recently compressed state group for each room and state_compressor_total_progress which stores how far through the state_groups table the compressor has scanned.

The tool can be run manually when you are running out of space, or be scheduled to run periodically.

Building

This tool requires cargo to be installed. See https://www.rust-lang.org/tools/install for instructions on how to do this.

This project follows the deprecation policy of Synapse on Rust and will assume a recent stable version of Rust and the ability to fetch a more recent one if necessary.

To build synapse_auto_compressor, clone this repository and navigate to the synapse_auto_compressor/ subdirectory. Then execute cargo build.

This will create an executable and store it in synapse_auto_compressor/target/debug/synapse_auto_compressor.

Example usage

Compress 100 chunks of size 500 in a remote PostgreSQL database:

$ synapse_auto_compressor -p postgresql://user:pass@localhost/synapse -c 500 -n 100

Compress 100 chunks of size 500 using local PostgreSQL socket:

$ sudo -u postgres synapse_auto_compressor -p "user=postgres dbname=matrix-synapse host=/var/run/postgresql" -c 500 -n 100

Running Options

-p [POSTGRES_LOCATION] Required The configuration for connecting to the Postgres database. This should be of the form "postgresql://username:password@mydomain.com/database" or a key-value pair string: "user=username password=password dbname=database host=mydomain.com" See https://docs.rs/tokio-postgres/0.7.2/tokio_postgres/config/struct.Config.html for the full details.
-c [CHUNK_SIZE] Required The number of state groups to work on at once. All of the entries from state_groups_state are requested from the database for state groups that are worked on. Therefore small chunk sizes may be needed on machines with low memory. Note: if the compressor fails to find space savings on the chunk as a whole (which may well happen in rooms with lots of backfill in) then the entire chunk is skipped.
-n [CHUNKS_TO_COMPRESS] Required CHUNKS_TO_COMPRESS chunks of size CHUNK_SIZE will be compressed. The higher this number is set to, the longer the compressor will run for.
-l [LEVELS] Sizes of each new level in the compression algorithm, as a comma-separated list. The first entry in the list is for the lowest, most granular level, with each subsequent entry being for the next highest level. The number of entries in the list determines the number of levels that will be used. The sum of the sizes of the levels affects the performance of fetching the state from the database, as the sum of the sizes is the upper bound on the number of iterations needed to fetch a given set of state. [defaults to "100,50,25"]

Scheduling the compressor

The automatic tool may put some strain on the database, so it might be best to schedule it to run at a quiet time for the server. This could be done by creating an executable script and scheduling it with something like cron.

Manual tool: synapse_compress_state

Introduction

A manual tool that reads in the rows from state_groups_state and state_group_edges tables for a specified room and calculates the changes that could be made that (hopefully) will significantly reduce the number of rows.

This tool currently does not write to the database by default, so should be safe to run. If the -o option is specified then SQL will be written to the given file that would change the tables to match the calculated state. (Note that if -t is given then each change to a particular state group is wrapped in a transaction). If you do wish to send the changes to the database automatically then the -c flag can be set.

The SQL generated is safe to apply against the database with Synapse running. This is because the state_groups and state_groups_state tables are append-only: once written to the database, they are never modified. There is therefore no danger of a modification racing against a running Synapse. Further, this script makes its changes within atomic transactions, and each transaction should not affect the results from any of the queries that Synapse performs.

The tool will also ensure that the generated state deltas do give the same state as the existing state deltas before generating any SQL.

Building

This tool requires cargo to be installed. See https://www.rust-lang.org/tools/install for instructions on how to do this.

To build synapse_compress_state, clone this repository and then execute cargo build.

This will create an executable and store it in target/debug/synapse_compress_state.

Example usage

$ synapse_compress_state -p "postgresql://localhost/synapse" -r '!some_room:example.com' -o out.sql -t
Fetching state from DB for room '!some_room:example.com'...
Got initial state from database. Checking for any missing state groups...
Number of state groups: 73904
Number of rows in current table: 2240043
Number of rows after compression: 165754 (7.40%)
Compression Statistics:
  Number of forced resets due to lacking prev: 34
  Number of compressed rows caused by the above: 17092
  Number of state groups changed: 2748
New state map matches old one

# It's finished, so we can now go and rewrite the DB
$ psql synapse < out.data

Running Options

-p [POSTGRES_LOCATION] Required The configuration for connecting to the Postgres database. This should be of the form "postgresql://username:password@mydomain.com/database" or a key-value pair string: "user=username password=password dbname=database host=mydomain.com" See https://docs.rs/tokio-postgres/0.7.2/tokio_postgres/config/struct.Config.html for the full details.
-r [ROOM_ID] Required The room to process (this is the value found in the rooms table of the database not the common name for the room - it should look like: "!wOlkWNmgkAZFxbTaqj:matrix.org".
-b [MIN_STATE_GROUP] The state group to start processing from (non-inclusive).
-n [GROUPS_TO_COMPRESS] How many groups to load into memory to compress (starting from the 1st group in the room or the group specified by -b).
-l [LEVELS] Sizes of each new level in the compression algorithm, as a comma-separated list. The first entry in the list is for the lowest, most granular level, with each subsequent entry being for the next highest level. The number of entries in the list determines the number of levels that will be used. The sum of the sizes of the levels affects the performance of fetching the state from the database, as the sum of the sizes is the upper bound on the number of iterations needed to fetch a given set of state. [defaults to "100,50,25"]
-m [COUNT] If the compressor cannot save this many rows from the database then it will stop early.
-s [MAX_STATE_GROUP] If a max_state_group is specified then only state groups with id's lower than this number can be compressed.
-o [FILE] File to output the SQL transactions to (for later running on the database).
-t If this flag is set then each change to a particular state group is wrapped in a transaction. This should be done if you wish to apply the changes while synapse is still running.
-c If this flag is set then the changes the compressor makes will be committed to the database. This should be safe to use while synapse is running as it wraps the changes to every state group in it's own transaction (as if the transaction flag was set).
-g If this flag is set then output the node and edge information for the state_group directed graph built up from the predecessor state_group links. These can be looked at in something like Gephi (https://gephi.org).

Running tests

There are integration tests for these tools stored in compressor_integration_tests/.

To run the integration tests, you first need to start up a Postgres database for the library to talk to. There is a docker-compose file that sets one up with all of the correct tables. The tests can therefore be run as follows:

$ cd compressor_integration_tests/
$ docker-compose up -d
$ cargo test --workspace
$ docker-compose down

Using the synapse_compress_state library

If you want to use the compressor in another project, it is recomended that you use jemalloc https://github.com/tikv/jemallocator.

To prevent the progress bars from being shown, use the no-progress-bars feature. (See synapse_auto_compressor/Cargo.toml for an example)

Troubleshooting

Connecting to database

From local machine

If you setup Synapse using the instructions on https://matrix-org.github.io/synapse/latest/postgres.html you should have a username and password to use to login to the postgres database. To run the compressor from the machine where Postgres is running, the url will be the following:

postgresql://synapse_user:synapse_password@localhost/synapse

From remote machine

If you wish to connect from a different machine, you'll need to edit your Postgres settings to allow remote connections. This requires updating the pg_hba.conf and the listen_addresses setting in postgresql.conf

Printing debugging logs

The amount of output the tools produce can be altered by setting the RUST_LOG environment variable to something.

To get more logs when running the synapse_auto_compressor tool try the following:

$ RUST_LOG=debug synapse_auto_compressor -p postgresql://user:pass@localhost/synapse -c 50 -n 100

If you want to suppress all the debugging info you are getting from the Postgres client then try:

RUST_LOG=synapse_auto_compressor=debug,synapse_compress_state=debug synapse_auto_compressor [etc.]

This will only print the debugging information from those two packages. For more info see https://docs.rs/env_logger/0.9.0/env_logger/.

Building difficulties

Building the openssl-sys dependency crate requires OpenSSL development tools to be installed, and building on Linux will also require pkg-config

This can be done on Ubuntu with: $ apt-get install libssl-dev pkg-config

Note that building requires quite a lot of memory and out-of-memory errors might not be obvious. It's recomended you only build these tools on machines with at least 2GB of RAM.

Auto Compressor skips chunks when running on already compressed room

If you have used the compressor before, with certain config options, the automatic tool will produce lots of warnings of the form: The compressor tried to increase the number of rows in ...

To fix this, ensure that the chunk_size is set to at least the L1 level size (so if the level sizes are "100,50,25" then the chunk_size should be at least 100).

Note: if the level sizes being used when rerunning are different to when run previously this might lead to less efficient compression and thus chunks being skipped, but this shouldn't be a large problem.

Compressor is trying to increase the number of rows

Backfilling can lead to issues with compression. The synapse_auto_compressor will skip chunks it can't reduce the size of and so this should help jump over the backfilled state_groups. Lots of state resolution might also impact the ability to use the compressor.

To examine the state_group hierarchy run the manual tool on a room with the -g option and look at the graphs.