Skip to main content

DevOps Server Onboarding

  • Users are created and updated on the DevOps server, with SSH keys, from resources/users.json.
  • Devpi (pypi mirror) users are created and updated from the same resources/users.json.
  • Generic binary artifacts are downloaded from upstream and uploaded to the <project_code> package registry.
  • Container images are built with Docker buildx bake and pushed to the <project_code> container registry.
  • OpenAPI Python clients are generated from OpenAPI schemas and published to the <project_code> PyPI registry.
  • GitHub repositories are mirrored as pull mirrors onto a dedicated organization, from resources/public-mirros.json.

Prerequisites​

  • uv
  • Docker with buildx (container builds)
  • Java (OpenAPI generator)
  • CS_PROJECT_CODE environment variable holding the project code (in CI it is provided by the CS_PROJECT_CODE Actions variable on the <project_code> organization)
  • DEVOPS_SERVER_INVENTORY_HOSTNAME environment variable holding the DevOps server host's inventory hostname, used to build every per-host Vault path. Every command fails immediately when it is missing or empty, before logging or Vault is set up
  • Vault access via environment variables: VAULT_API_ENDPOINT, VAULT_API_KEY, VAULT_CA_CERT_BASE64, VAULT_CLIENT_CERT_BASE64, VAULT_CLIENT_KEY_BASE64 (the *_BASE64 variables hold base64-encoded PEM contents; devops_server_onboarding decodes them to temporary files)
  • The <project_code> organization must already exist on the DevOps server (importing devops_server_onboarding checks this and fails if it's missing)

Setup​

uv sync
uv --offline run --no-sync --no-progress update-devops-server-org

devops-server-onboarding is an installable package (src/devops_server_onboarding/) whose commands are exposed as console scripts ([project.scripts] in pyproject.toml). Importing the package reads the project code from the CS_PROJECT_CODE environment variable, the DevOps server host from the DEVOPS_SERVER_INVENTORY_HOSTNAME environment variable, and the VAULT_* environment variables, fetches the DevOps server details from Vault (bucket <project_code>, key devops-server/hosts/<DEVOPS_SERVER_INVENTORY_HOSTNAME>/apps/scm/generated) and the root CA (bucket <project_code>, key certificate-authority/root-ca), checks that the <project_code> organization already exists on the DevOps server, and exposes CS_PROJECT_CODE, DEVOPS_SERVER_INVENTORY_HOSTNAME, DEVOPS_SERVER_LOCAL_URL, DEVOPS_SERVER_DOMAIN, DEVOPS_SERVER_PORT, DEVOPS_SERVER_ADMIN_USER, DEVOPS_SERVER_ADMIN_API_TOKEN, ROOT_CA_CERT_FILE (a temporary file holding the root CA), and VAULT_* values as package constants. A .env file in the working directory is loaded when present; no .env file is written. Every command triggers this bootstrap as a side effect of its first import, so a broken Vault connection or a missing organization surfaces immediately.

update-devops-server-org reads the resolved config plus the raw VAULT_* environment variables, then pushes the Vault credentials as Actions secrets on the <project_code> organization (VAULT_API_ENDPOINT, VAULT_API_KEY, VAULT_CA_CERT_BASE64, VAULT_CLIENT_CERT_BASE64, VAULT_CLIENT_KEY_BASE64), so Actions workflows running in that organization can reach Vault.

It also sets the project code as the CS_PROJECT_CODE Actions variable and the DevOps server host's inventory hostname as the DEVOPS_SERVER_INVENTORY_HOSTNAME Actions variable (neither is a secret) on the <project_code> organization: each variable is created if absent, updated if its stored value differs, and left untouched if it already matches. Workflows read them as ${{ vars.CS_PROJECT_CODE }} and ${{ vars.DEVOPS_SERVER_INVENTORY_HOSTNAME }}.

Publish artifacts​

All commands are run from the repo root.

The users, devpi-users, generic-artifacts, containers, openapi-client, and github-public-mirror commands process each declared item independently. A failing item (a user, artifact, build target, package, or mirror) logs a warning and the run continues with the next one. At the end the command logs a table with one row per item and its status (SUCCESS, SKIPPED, or FAILED). If any item failed, the command exits with a non-zero code after the table.

Each JSON declaration under resources/ is validated against a Pydantic model when the command starts, so an unknown or misspelled field, a missing required field (such as a user's email), or an invalid visibility stops the command before anything is changed on a server.

Users​

uv --offline run --no-sync --no-progress users

Users are declared in resources/users.json as "<username>": {settings}. A user missing on the server is created with a freshly generated random password (a password is mandatory at creation time). Every field present in the JSON entry is then compared against the user's current value on the server, and an edit call is only made, and printed, for fields that actually differ; fields absent from the JSON entry are left untouched. Any declared SSH key not already on the account is added.

{
    "admin": {
        "email": "admin@localhost",
        "full_name": "Admin",
        "admin": true,
        "active": true,
        "allow_create_organization": false,
        "allow_git_hook": false,
        "allow_import_local": false,
        "prohibit_login": false,
        "visibility": "private",
        "ssh_keys": []
    }
}

Only email is required; every other key is optional. send_notification (default false) controls whether Gitea emails the user on creation.

Devpi (pypi mirror) users​

uv --offline run --no-sync --no-progress devpi-users

Reads the same resources/users.json declarations as users, but only the email field applies; every Gitea-specific field (admin, full_name, ssh_keys, visibility, ...) is ignored. A user missing on the mirror is created with a freshly generated random password (a password is mandatory at creation time, and devpi has no way to reset one without knowing the old one). An existing user's email is patched only if it differs from the declared value. Implemented by calling devpi-client's own Python API (devpi.user.user_create/user_modify, driven through a devpi.main.Hub) directly, never the devpi CLI executable.

Beyond the package-level Vault bootstrap, this command makes its own additional read of {project_code}/devops-server/hosts/{DEVOPS_SERVER_INVENTORY_HOSTNAME}/apps/pypi-mirror/generated for endpoint and root_password (shared with devops-server-deployer, whose pypi_mirror_* tags provision the mirror and write that secret on every run), then logs into the mirror as root at that endpoint.

Generic binary artifacts​

uv --offline run --no-sync --no-progress generic-artifacts

Artifacts are declared in resources/generic-artifacts.json as "<package>/<version>/<filename>": {"uri": ..., "checksum": "<algorithm>:<hexdigest>"}. Downloads are cached in generic_artifacts/ (override with DEVOPS_SERVER_BINARY_ARTIFACTS_DIRECTORY), checksum-verified, and uploaded to the <project_code> organization. Already published files are skipped.

Container images​

uv --offline run --no-sync --no-progress containers

Installs the internal root CA for the Docker daemon (restarting docker through systemctl when it isn't running in CI), logs into the registry, registers the qemu binfmt handlers for cross-platform builds, and creates a buildx builder trusting the root CA if one doesn't exist yet. It then builds and pushes every target in the default group of resources/containers/docker-bake.hcl (Dockerfiles under resources/containers/dockerfiles/) to the <project_code> organization. Image versions are pinned in the bake file, most of them as *_VERSION variables. Every image is pushed under its pinned version tag and one floating alias tag: latest for most images, trixie for the redis, node, rust, python, httpd, postgres, and uv images, and latest-jre or latest-jdk for the two eclipse-temurin targets, which share one image name. Requires docker with buildx on PATH and password-less sudo for the root CA installation.

Set CONTAINERS_NO_CACHE=1 (also true or yes) to pass --no-cache to every bake target; it defaults to false. The Onboard workflow exposes it as the no_cache input.

OpenAPI Python clients​

uv --offline run --no-sync --no-progress openapi-client

Schemas live in resources/openapi-schemas/ named <package>-<version>.json; the file name is the source of truth for the package name and version. For each schema not yet published, a Python client is generated with openapi-generator-cli (jar downloaded on demand) and published with Poetry to the <project_code> organization. Already published versions are skipped.

GitHub public mirrors​

uv --offline run --no-sync --no-progress github-public-mirror

Mirrors are declared in resources/public-mirros.json:

{
    "github": {
        "organization": "<organization>",
        "mirrors": {
            "<repo in Gitea>": {
                "source": "<github owner>/<github repository>",
                "visibility": "public"
            }
        }
    }
}

Only source is required; visibility ("public" or "private") is optional per mirror.

Before writing anything, the declared GitHub API token (managed-services/github Vault secret, key GH_PROD_API_TOKEN) is validated against https://api.github.com/user, and every declared source repo is checked to exist and be readable with that token; the command stops before touching Gitea if any source repo is missing or inaccessible. The visibility applied to each mirror is its declared visibility when present, else the source repo's actual current visibility on GitHub.

The declared organization is created (public) on the DevOps server if it doesn't exist yet, else made public if it isn't already, regardless of any individual mirror's visibility. A mirror repo missing on the server is created as a pull mirror of its source with an 8-hour sync interval and the resolved visibility above. An existing mirror repo has its Clone From URL checked against the declared source first; since Gitea's API has no way to repoint an existing mirror's remote, a mismatch is fixed by deleting and recreating the repo against the declared source. A matching mirror instead has its sync interval, source token, and resolved visibility re-applied on every run, so a rotated GitHub token, a visibility change in the JSON, or a visibility change on the source repo propagates automatically. Actions (CI/CD) are always disabled on every mirror repo.

Registry package cleanup​

cleanup-artifacts deletes packages from the <project_code> package registry. It connects to Vault for credentials and the server's REST API to list and remove packages. It is interactive and requires a TTY: it prompts with an arrow-key select for the package type to delete, then asks for an optional comma-separated list of package names to restrict deletion to (leave empty to remove every package of the selected type).

Passing the package type as an argument (container, generic, pypi or terraform) skips the type prompt and needs no TTY for it; any other value fails with a usage error.

uv --offline run --no-sync --no-progress cleanup-artifacts

With the package type given, the type prompt is skipped:

uv --offline run --no-sync --no-progress cleanup-artifacts container
uv --offline run --no-sync --no-progress cleanup-artifacts generic
uv --offline run --no-sync --no-progress cleanup-artifacts pypi
uv --offline run --no-sync --no-progress cleanup-artifacts terraform

--filter <name> restricts deletion to the named packages and skips the names prompt. Repeat it for several names. With both the type and --filter given, the command runs without any prompt:

uv --offline run --no-sync --no-progress cleanup-artifacts pypi --filter my-client
uv --offline run --no-sync --no-progress cleanup-artifacts container --filter app-api --filter app-worker

--all removes every package of the type without asking for names, so <type> --all also runs without any prompt. It cannot be combined with --filter:

uv --offline run --no-sync --no-progress cleanup-artifacts generic --all

CI​

  • .gitea/workflows/onboard.yml: manually triggered (workflow_dispatch) deployment pipeline with parallel jobs, one per command: users, devpi users, generic binaries, container images, OpenAPI Python clients, and GitHub public mirrors. Each job authenticates to Vault via the VAULT_* Actions secrets and reads the CS_PROJECT_CODE and DEVOPS_SERVER_INVENTORY_HOSTNAME organization Actions variables.
  • .gitea/workflows/lint.yml: runs the Python linters and type checkers, yamllint, and prettier.
  • .gitea/workflows/gitleaks.yml: secret scanning on push, pull request, manual trigger (workflow_dispatch), and a daily schedule.

Development​

See DEVELOPMENT.md for git hooks setup and the lint commands.