#1851: Swarmica Settings for Better Performance

In this article, recommendations for setting up a Swarmica installation (~1000 users total and ~100-150 simultaneously online) on a server with 16 CPU and 32Gb RAM are presented.

The main load in Swarmica falls on the services: Postgres, Django, and Celeryworker.

Postgres

Recommended settings:

max_connections=500
shared_buffers=8GB
effective_cache_size=24GB
work_mem=16MB
maintinance_work_mem=512MB
random_page_cost=1.1
effective_io_concurrency=200
temp_buffers=32MB
track_activity_query_size=16384
wal_keep_size=512MB

Directly affect performance:

max_connections - Maximum number of simultaneous client connections to the server.
shared_buffers - PostgreSQL's main buffer for working with tables and indexes. It is recommended to set 20–25% of the server's RAM.
effective_cache_size - The total amount of cache available for data, including shared_buffers and the operating system cache. It is recommended to set 70-75% of the server's RAM.
work_mem - Memory for a single sort operation or hash table within a query.
maintenance_work_mem - Memory for maintenance operations: VACUUM, CREATE INDEX, REINDEX, ALTER TABLE, etc. A large value speeds up index creation and cleanup.
random_page_cost - Cost estimate for random page reads relative to sequential ones. A value of 1.1 is usually set for fast SSD/NVMe to prevent the planner from avoiding index scans too aggressively.

Optional:

effective_io_concurrency - How many simultaneous disk I/O operations PostgreSQL can perform in parallel.
track_activity_query_size - Maximum length of the query text that is saved in pg_stat_activity.
temp_buffers - Local buffers for temporary tables within a session. Memory is allocated as needed, not at once for everyone.
wal_keep_size - Minimum amount of old WAL files that PostgreSQL will keep in the pg_wal directory. Useful for replication.

How to apply

To apply these settings, you need to change the command to start the postgres service in the docker-compose.yml file:

...
postgres:
 image: postgres:14.2-bullseye
 container_name: swarmica-postgres
 command: postgres -N 500 -c shared_buffers=8GB -c effective_cache_size=24GB -c work_mem=16MB -с maintenance_work_mem=512MB -с random_page_cost=1.1 -с effective_io_concurrency=200 -c temp_buffers=32MB -с track_activity_query_size=16384 -с wal_keep_size=512MB
 restart: always
...

and restart Swarmica:

docker compose down && docker compose up -d

Django

Recommended settings:

threads=4
workers=16
max-requests=2000
max-requests-jitter=100
graceful-timeout=60

Directly affect performance:

threads - Number of threads per worker. Recommended 2-8 per worker.
workers - Number of workers. It is recommended to set 1 worker per 1 CPU unit.

It is highly not recommended to set too high values for workers and threads, as this can lead to the opposite effect - reduced performance.

For efficient memory usage:

max-requests - Maximum number of requests for one worker, after which it restarts.
max-requests-jitter - Error margin for max-requests, so that all workers do not restart simultaneously.
graceful-timeout - (Optional) Time given to complete request processing before restart. Default is 30 sec.

These parameters are needed to avoid memory leaks. When a worker restarts, all memory occupied by it is released.

How to apply:

To apply these settings, you need to change the GUNICORN_CMD_ARGS line in the .env file:

GUNICORN_CMD_ARGS="--threads=4 --workers=16 --max-requests=2000 --max-requests-jitter=100 --graceful-timeout=60"

and restart Swarmica:

docker compose down && docker compose up -d

Celeryworker

Recommended settings:

max-tasks-per-child=1000

For efficient memory usage:

max-tasks-per-child - Number of tasks after which the process restarts. Necessary to avoid memory leaks.

How to apply

To apply these settings, you need to change the CELERY_CMD_ARGS line in the .env file:

CELERY_CMD_ARGS="--max-tasks-per-child=1000"

and restart Swarmica:

docker compose down && docker compose up -d