#1671: How to configure Swarmica for high availability

Symptoms

You need to deploy all systems with high availability and are considering whether to:

• run the Swarmica database on a separate server;
• use a highly available database cluster;
• run Swarmica as an active-active cluster.

Solution

Recommended architecture

For most on-premises deployments, the best balance of reliability, performance, operational simplicity and recovery objectives (RTO and RPO) is to run Swarmica in one virtual machine while placing its virtual disk on replicated, highly available network storage.

This approach minimizes the number of moving parts and potential failure points while allowing the virtualization and storage platform to provide failover and replication.

Separate database server with replication

Important: This architecture is not recommended or officially supported. It is not regularly tested by the Swarmica team. If the primary database server fails, the replica does not become primary automatically; an administrator must promote it manually. A network connection between Swarmica and PostgreSQL can also reduce performance and introduces an additional failure point.

This topology requires three Ubuntu 24 servers:

• Swarmica
• PostgreSQL-1 — primary
• PostgreSQL-2 — replica

1. Install Swarmica by following the Swarmica installation guide.
2. Create a database backup as described in Backup and restore of Swarmica on your own equipment.
3. Stop Swarmica and back up the Compose file:

   cd /root/swarmica
   docker compose down
   cp /root/swarmica/docker-compose.yml{,.orig}

4. Comment out the built-in postgres service and its depends_on entries in /root/swarmica/docker-compose.yml.
5. In /root/swarmica/.env, replace POSTGRES_HOST=postgres with POSTGRES_HOST=<IP_ADDRESS>, then start Swarmica:

   docker compose pull && docker compose up -d

6. On both PostgreSQL servers, install PostgreSQL 14:

   sudo apt install curl ca-certificates -y
   sudo install -d /usr/share/postgresql-common/pgdg
   sudo curl -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc --fail https://www.postgresql.org/media/keys/ACCC4CF8.asc
   sudo sh -c 'echo "deb [arch=amd64 signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
   sudo apt update
   sudo apt install postgresql-14 postgresql-contrib-14 -y

7. Create the Swarmica database and user on both servers. Use the database password from the Swarmica .env file:

   su - postgres
   psql
   CREATE USER swarmica_user WITH PASSWORD '<PASSWORD>';
   CREATE DATABASE swarmica_app OWNER swarmica_user;
   GRANT ALL PRIVILEGES ON DATABASE swarmica_app TO swarmica_user;
   quit
   exit

8. On PostgreSQL-1, restore the backup:

   pg_restore --if-exists --clean -U swarmica_user -d swarmica_app <PATH-TO-BACKUP.sql>

9. On PostgreSQL-1, configure external access and replication in /etc/postgresql/14/main/postgresql.conf:

   listen_addresses = '*'
   wal_level = replica
   max_wal_senders = 10
   max_replication_slots = 5
   hot_standby = on

10. Add the Swarmica and replica access rules to /etc/postgresql/14/main/pg_hba.conf:

    host    swarmica_app    swarmica_user    <IP>/<MASK>                 md5
    host    replication     repuser          <REPLICA_IP>/<REPLICA_MASK> md5

11. Create a replication user and restart PostgreSQL:

    sudo -u postgres psql -c "CREATE ROLE repuser REPLICATION LOGIN PASSWORD 'your_secure_password';"
    systemctl restart postgresql@14-main

12. On PostgreSQL-2, stop PostgreSQL, clear its data directory, and clone the primary:

    systemctl stop postgresql@14-main
    sudo rm -rf /var/lib/postgresql/14/main/*
    sudo -u postgres pg_basebackup -h <PRIMARY_IP> -U repuser -D /var/lib/postgresql/14/main/ -P -R -v
    systemctl start postgresql@14-main

Separate database server with high availability

Important: This architecture is also not recommended or officially supported. It is not regularly tested by the Swarmica team. It adds network latency, increases operational complexity and introduces additional failure points. Use it only when your infrastructure requirements justify the extra complexity and you have the expertise to operate and test the resulting stack.

The example topology uses four Ubuntu 24 servers:

• swarmica-master-1
• swarmica-postgres-1
• swarmica-postgres-2
• swarmica-pgbouncer-1

swarmica-master-1

Follow steps 1–6 from the previous section. In the .env file, use the PgBouncer address and port:

POSTGRES_HOST=<PGBOUNCER_IP>
POSTGRES_PORT=6432

swarmica-pgbouncer-1

PgBouncer and HAProxy provide a stable connection endpoint and route traffic to the PostgreSQL node currently acting as primary.

1. Install Docker:

   apt install docker.io docker-compose-v2

2. Create the working directory:

   mkdir /root/swarmica-pgbouncer
   cd /root/swarmica-pgbouncer

3. Create docker-compose.yml with HAProxy and PgBouncer. Expose HAProxy on port 5433 and PgBouncer on port 6432.
4. Configure HAProxy health checks against the Patroni /master endpoint and list both PostgreSQL servers as backends.
5. Configure PgBouncer to connect to HAProxy on port 5433, using transaction pooling and the Swarmica database credentials.
6. Start the services:

   docker compose pull && docker compose up -d

swarmica-postgres-1 and swarmica-postgres-2

Install PostgreSQL 14, etcd and Patroni on both servers. etcd stores cluster state and Patroni performs PostgreSQL failover: if the primary becomes unavailable, the replica is promoted and the recovered node returns as a replica.

1. Install PostgreSQL 14 and prepare the database on the first node. Do not restore the backup on the second node.
2. Stop and disable the system PostgreSQL service on both servers because Patroni will manage PostgreSQL:

   systemctl stop postgresql@14-main && systemctl disable postgresql@14-main

3. Install etcd:

   apt install etcd-server etcd-client

4. Configure a two-member etcd cluster with the private IP addresses of both PostgreSQL servers, then start it:

   systemctl daemon-reload
   systemctl enable etcd --now
   etcdctl member list

5. Install Patroni and its dependencies:

   apt install python3 python3-pip python3-dev pipx -y
   su - postgres
   pipx ensurepath
   source ~/.bashrc
   pipx install patroni[psycopg2-binary,etcd]
   exit

6. Create /etc/systemd/system/patroni.service and /etc/patroni/patroni.yml on both servers. Configure the node name, REST API address, etcd peers, PostgreSQL address, replication credentials and the appropriate network and subnet mask.
7. Start Patroni on the first node, wait 5–10 seconds, then start it on the second:

   systemctl daemon-reload
   systemctl enable patroni --now

8. Verify the cluster:

   sudo -u postgres -i patronictl -c /etc/patroni/patroni.yml list

   The output should show one Leader in running state and one Replica in streaming state.

Active-active Swarmica cluster

Active-active Swarmica deployments are not supported. If you need high availability, use the recommended single-VM architecture with replicated storage, or discuss your requirements with the Swarmica team before designing a custom deployment.