SCM

Deploy the self-hosted DevOps server: source code hosting, artifact delivery (container images, Python packages, binaries, release assets), and CI/CD pipelines. It runs as Docker containers with the internal Root CA baked into every certificate, generates its own secrets, creates the admin service user, and registers CI/CD runners against itself.

Ansible hosts group: `scm`

Variables

Option	Type	Description	Default
`ds_postgres_app_database`	string	Application database name on the shared database server, and the Vault credential name	`devops-server-scm`
`ds_cicd_runner_new_registration`	bool	Delete every runner's existing registration so the runners register again	`false`
`ds_cicd_runner_container_count`	int	Number of runner containers started in Docker	`3`
`ds_cicd_runner_container_cache_port`	int	Base host port for the runner containers' Docker cache; runner N publishes base + N	`44100`
`ds_cicd_runner_systemd_cache_port`	int	Cache port for the systemd runner	`44200`

The application container's extra DNS resolver is fixed in group_vars/all/all.yml (see Prerequisites), not read from Vault.

Constraints the prerequisites stage enforces:

The SSH port must not be the standard SSH port, 22.
The HTTP port must not be a standard HTTP(S) port, 80 or 443.
The root URL must start with https.
The Docker network, app container name, and install directory are pinned to devops-server-scm, devops-server-scm, and /app/devops-server-scm.

Database

The shared PostgreSQL server is deployed by the Database service. This service connects to it only as a client, never deploying or administering the server itself. List ds_postgres_app_database in the database service's ds_db_downstream_databases and run --tags database_downstream_db_credentials before the first SCM run, so the application's generated credential exists in Vault.

Vault configurations

Every field below is mandatory: a missing field fails the run instead of falling back to a default.

key: {{ cs_project_code }}/devops-server/hosts/{{ inventory_hostname }}/apps/postgres/config: connection details for the shared database server (see Database | Vault configurations)

{
    "maintenance_db": "Maintenance database, used only to create the application user/database/schema and fix permissions",
    "maintenance_user": "Maintenance login user",
    "maintenance_password": "Maintenance login password"
}

key: {{ cs_project_code }}/devops-server/hosts/{{ inventory_hostname }}/apps/postgres/generated-{{ ds_postgres_app_database }}-db-credential: written by the database service's database_downstream_db_credentials tag

{
    "host": "Postgres host",
    "port": "(int) Postgres port",
    "user": "Application's own Postgres role, created by the `scm_postgres` tag in the application database owned by that role",
    "password": "Password for that role"
}

key: {{ cs_project_code }}/devops-server/hosts/{{ inventory_hostname }}/apps/scm/config

{
    "backup_password": "Restic repository password",
    "http_port": "(int) HTTP port; must not be 80 or 443",
    "ssh_port": "(int) SSH port; must not be 22",
    "ssh_domain": "Domain used for SSH access",
    "root_url": "Root URL; must start with https",
    "admin_service_user_username": "Persistent admin service user's username",
    "admin_service_user_password": "Persistent admin service user's password",
    "admin_service_user_email": "Persistent admin service user's email",
    "redis_master_password": "Redis `default` ACL user password",
    "redis_app_password": "Password for the `devops_server` Redis ACL user, used for the cache connection"
}

key: {{ cs_project_code }}/devops-server/hosts/{{ inventory_hostname }}/apps/scm/generated: written by the scm_post_install tag, read back on every later run

{
    "devops_server_admin_api_token": "Admin API token",
    "devops_server_global_runner_token": "CI runner registration token",
    "devops_server_local_url": "Local instance URL",
    "devops_server_ssh_url": "SSH clone URL",
    "devops_server_url": "Public instance URL"
}

The shared Root CA and SMTP paths are listed in Prerequisites.

Tags

scm runs every stage below in order except scm_backup, scm_restore, and dangerously_cleanup_scm, which must be requested by name. The prerequisites stage runs with every SCM tag.

`prerequisites`: `tasks/scm/prerequisites.yml`

Runs a local and remote sudo id check; gathers facts; verifies restic, docker, and rsync on the remote host; asserts required variables are set; enforces the pinned network/container/directory names and port constraints. This stage is validation-only and leaves the host unchanged.

`scm_prepare`: `tasks/scm/prepare.yml`

Installs APT dependencies, and creates the devops-server-scm Docker network and the devops-server-scm group and user that own the container's files on the host. Also runs alongside every stage that depends on those resources: scm_postgres, scm_redis, scm_secrets, scm_install, scm_post_install, scm_cicd_runner_container, scm_cicd_runner_systemd, scm_cicd_runner, scm_backup, and scm_restore. It does not run with dangerously_cleanup_scm.

`scm_postgres`: `tasks/scm/postgres.yml`

Installs the Postgres client role; issues a short-lived maintenance client certificate (signed by the Root CA, clientcert=verify-full against the shared server); connects with the maintenance credentials to create the application user and database, and hands ownership of the database to that user. The devops-server-scm-schema schema itself, and the application user's privileges over it, are created by the scm_install stage.

`scm_redis`: `tasks/scm/redis.yml`

Stops the devops-server-scm container; creates the data/cert directories; issues a server key/cert (signed by the Root CA); writes an ACL file with a full-privilege default user and a scoped devops_server user; opens the Redis UFW port; starts the devops-server-scm-redis container over TLS (--tls-auth-clients optional), with its port published on all interfaces (0.0.0.0); restarts the devops-server-scm container. The application's own Redis client cannot present a client certificate, so the connection is TLS-encrypted and server-authenticated against the Root CA, with the devops_server ACL user/password authenticating the client. The server stays capable of accepting a client certificate from a future client that supports it.

`scm_secrets`: `tasks/scm/secrets.yml`

Generates the application's SECRET_KEY, INTERNAL_TOKEN, JWT_SECRET, and LFS_JWT_SECRET via a throwaway token-generator container; generates an ed25519 commit-signing key; issues the HTTPS server certificate/chain and a Postgres client certificate, all signed by the Root CA.

`scm_install`: `tasks/scm/main.yml`

Builds the application image from files/scm/devops-server/. Connects to the shared database server with the application user's own credentials to create the devops-server-scm-schema schema and grant that user full privileges over it, then starts the devops-server-scm container against that database over TLS, with HTTPS, SSH, LFS, OAuth2, actions/CI, cron cleanup jobs, commit signing, audit events recorded in the shared database (kept for 30 days), and Redis for both the cache (db 13) and the session store (db 14), both over TLS; started with restart_policy: "no".

`scm_service_user`: `tasks/scm/service_user.yml`

Creates a temporary admin user/token, uses the server's REST API to find-or-create the persistent admin service user, patches it to the desired state, then deletes the temporary admin.

`scm_post_install`: `tasks/scm/post_install.yml`

Restarts devops-server-scm with restart_policy: always; generates a fresh admin API token and a CI runner registration token; writes them, plus the instance URLs, back to the apps/scm/generated Vault path over mTLS.

`scm_fail2ban`: `tasks/scm/fail2ban.yml`

Installs fail2ban; writes a filter matching failed auth attempts; creates host and Docker-forward jails (nftables-allports); enables and restarts the service.

`scm_cicd_runner_container`: `tasks/scm/cicd_runner/container.yml`

Removes existing runner containers; builds the runner image from files/scm/devops-server-cicd-runner/, trusting the Root CA; starts ds_cicd_runner_container_count runner containers registered against the server with the generated global runner token; opens each runner's Docker-cache UFW port.

`scm_cicd_runner_systemd`: `tasks/scm/cicd_runner/systemd.yml`

Creates the devops-server-scm-cicd-runner user and group; trusts the Root CA on the host and writes it into the runner's home directory; installs UV, Node.js, JDK, Gitleaks, and Terraform for that user via arpanrec.nebula; downloads the runner binary, registers it against the local instance with the generated global runner token, and installs/starts it as a systemd service; opens the runner's cache UFW port.

`scm_cicd_runner`

Not a stage of its own; a group tag carried by both scm_cicd_runner_container and scm_cicd_runner_systemd. --tags scm_cicd_runner runs both.

Both runner stages need devops_server_global_runner_token from the apps/scm/generated Vault path, which scm_post_install writes. The stage order runs scm_post_install before both runner stages, so a --tags scm run satisfies this on a fresh host.

`scm_backup`: `tasks/scm/backup.yml`

Opt-in only. Installs the Postgres client role; stops/disables any existing backup timer; ensures the restic repository exists and is initialized; deploys the backup script and systemd service/timer (daily at 03:00); runs a backup immediately, then enables the timer. The backup script dumps the application database with the host's pg_dump over the Postgres client certificate, to /app/devops-server-scm/pg-dump.sql; stops the devops-server-scm container; snapshots the install directory (including the dump) with restic; deletes the dump; restarts the container.

`scm_restore`: `tasks/scm/restore.yml`

Opt-in only. Stops and disables the backup timer/service and the CICD runner systemd service if present; stops the devops-server-scm, devops-server-scm-redis, and CI runner containers, so no lingering session blocks the database drop below; runs restic restore latest into a temporary directory under /app (kept off /tmp since a restore can be far larger than it) and checks that the restored pg-dump.sql exists. It then drops the application database on the shared database server, rsyncs the restored files into the install directory, and removes the temporary directory. Next it re-runs the scm_postgres stage to recreate the application user, database, and ownership, and regenerates secrets (including the application user's Postgres client certificate). Finally it imports the restored pg-dump.sql into the fresh database and deletes the dump file. The schema and its objects come from the restored dump; this stage does not create the schema separately.

`dangerously_cleanup_scm`: `tasks/scm/cleanup.yml`

Destructive. Opt-in only. Stops the backup timer and the CICD runner systemd service; deletes the CICD runner systemd user and group and the devops-server-scm user and group; force-removes the app, Redis, and CI runner containers and the devops-server-scm Docker network; drops the application database on the shared database server if it exists (the server itself is left untouched, and an unreachable server only produces a warning); deletes the install, Redis, CI runner, and CICD runner systemd data directories, the CICD runner shared data directory (/devops-server-cicd-runner-shared-data), the restic cache directory, the backup systemd units, the CICD runner systemd unit, and the trusted Root CA certificate. The CICD runner systemd binary and config go with its home directory. The backup repository itself is left in place.

Downstream

Vault apps/scm/generated: devops_server_admin_api_token, devops_server_global_runner_token, devops_server_local_url, devops_server_ssh_url, devops_server_url, written by scm_post_install and read back on later runs (for example to resolve the runner token for the runner stages). Other platform repos read the same path to reach the server's registries.
Admin service user: a persistent, always-admin account, named by the apps/scm/config secret's admin_service_user_username, created by scm_service_user for downstream automation rather than interactive login.
CI runners: the runner containers self-register against the server using the global runner token from Vault, expose the devops-server, any, in-docker, has-docker labels, and cache uv/Ansible/Docker build state under /data/.cache and /devops-server-cicd-runner-shared-data. The systemd runner self-registers the same way, exposing the devops-server, any, metal, {{ ansible_facts.fqdn }}:host, has-docker labels, and caches the same build state under its home directory's .cache (/app/devops-server-scm-cicd-runner/systemd-runner-home/.cache) and /devops-server-cicd-runner-shared-data.
restic repository (/app/devops-server-scm-backup): populated nightly by the backup timer; consumed by the scm_restore tag and inspectable with restic snapshots using the same RESTIC_REPOSITORY/RESTIC_PASSWORD pair.
fail2ban: tails the application's log for failed logins and bans offending IPs via nftables-allports.

Deployment

uv sync --all-extras --all-packages --no-progress
uv --offline run --no-sync --no-progress ansible-galaxy install -r requirements.yml
uv --offline run --no-sync --no-progress ansible-playbook playbook.yml --tags scm

Ansible hosts group: scm​

Variables​

Database​

Vault configurations​

Tags​

prerequisites: tasks/scm/prerequisites.yml​

scm_prepare: tasks/scm/prepare.yml​

scm_postgres: tasks/scm/postgres.yml​

scm_redis: tasks/scm/redis.yml​

scm_secrets: tasks/scm/secrets.yml​

scm_install: tasks/scm/main.yml​

scm_service_user: tasks/scm/service_user.yml​

scm_post_install: tasks/scm/post_install.yml​

scm_fail2ban: tasks/scm/fail2ban.yml​

scm_cicd_runner_container: tasks/scm/cicd_runner/container.yml​

scm_cicd_runner_systemd: tasks/scm/cicd_runner/systemd.yml​

scm_cicd_runner​

scm_backup: tasks/scm/backup.yml​

scm_restore: tasks/scm/restore.yml​

dangerously_cleanup_scm: tasks/scm/cleanup.yml​

Downstream​

Deployment​