Build & CI/CD

The build, CI, and deploy artifacts all live under build/. The pipeline is driven by GitLab CI (.gitlab-ci.yml at the repo root) and deploys a Helm chart to AWS EKS.

Layout

.gitlab-ci.yml                       parent pipeline (build → tests → deploy)
docker-bake.hcl                      docker buildx targets for all 4 images
backend/docker/Dockerfile            multi-stage: build-base → test → build → runtime
build/
├── ci/
│   ├── staging.yaml                 child pipeline (env trigger)
│   ├── uat.yaml                     child pipeline (env trigger)
│   ├── prod.yaml                    child pipeline (env trigger)
│   └── templates/
│       ├── stages.yaml              child pipeline skeleton + job wiring
│       ├── common-variables.yaml    paths + helm timeouts
│       ├── migration-template.yaml  Django migrations (separate helm release)
│       ├── deploy-template.yaml     `helm upgrade --install` the app chart
│       ├── e2e-tests-template.yaml  placeholder (manual, allow_failure)
│       └── stop-template.yaml       teardown of both helm releases (manual)
└── helm/
    ├── package-and-publish.sh       packages + pushes both charts to OCI ECR
    ├── study-setup/                 main app chart
    └── study-setup-migrate/         migration-only chart (separate release)

Pipeline overview

parent (.gitlab-ci.yml)
  stage: build
    ├─ build-prod-image    docker buildx bake (backend / frontend / docs)
    └─ build-test-image    docker buildx bake test  (backend test stage)

  stage: tests
    └─ unit-tests          pytest tests/ --ignore=tests/e2e   (against test image)

  stage: deploy   (all manual)
    ├─ 1-staging   → triggers build/ci/staging.yaml
    ├─ 2-UAT       → triggers build/ci/uat.yaml
    └─ 3-prod      → triggers build/ci/prod.yaml

child (build/ci/<env>.yaml)
  stage: django-migrations
    └─ migrate             scale app to 0, helm install study-setup-migrate

  stage: deployment
    ├─ deploy              helm upgrade --install study-setup (needs: migrate)
    └─ stop                helm uninstall both releases (manual, allow_failure)

  stage: tests
    └─ e2e-tests           placeholder (manual, allow_failure)

Stages run in order; needs: allows jobs to start as soon as their explicit dependencies are ready. The numeric prefixes on the deploy jobs force the visual ordering staging → UAT → prod (GitLab orders alphabetically within a stage).

Build stage

Images are built with docker buildx bake driven by docker-bake.hcl. Four targets:

Target	Context	Dockerfile	Stage	Pushed to
backend	backend/	docker/Dockerfile	final (runtime)	saxecap/study-setup:<slug>-backend-<sha>
frontend	frontend/	docker/Dockerfile	final (runtime)	saxecap/study-setup:<slug>-frontend-<sha>
docs	docs/	docker/Dockerfile	final (runtime)	saxecap/study-setup:<slug>-docs-<sha>
test-image	backend/	docker/Dockerfile	test	saxecap/study-setup-tests:<slug>-test-<sha>

The default group default = [backend, frontend, docs] is what build-prod-image runs; build-test-image runs the test group separately so unit tests don't block on a frontend or docs build.

Notable bake configuration:

• Platforms: linux/arm64 only (the runners are Graviton).
• Cache: Optional S3-backed BuildKit cache (type=s3,...) keyed per target. Enabled when CACHE_BUCKET is set; otherwise the host's local cache is used.
• Docs context: The docs build needs vendored library READMEs (@includes in VitePress). Bake supplies them via a named context: libs = "backend/lib".

The docs and frontend Dockerfiles share the same base → development → build → runtime shape. The backend Dockerfile adds a test stage (between development and build) that installs runtime + dev deps and copies the app source, with CMD ["pytest"].

Tests stage

unit-tests runs against the test image built in the build stage. It starts a postgres:16 service alongside, runs:

cd /app
pytest tests/ --ignore=tests/e2e --strict-markers --strict-config -x -v --tb=short

tests/e2e/ is excluded — those tests need a running stack (see local/.justfile test-e2e). The CI pytest is for the layer-level unit suite only.

The job is configured with placeholder env vars that satisfy UNCOUPLE_REQUIRE_SECURE=1 in the test image and point Django at the postgres service.

Deploy stage

Each env trigger (staging.yaml / uat.yaml / prod.yaml) sets a small variable block (env name, namespace, cluster tag, certificate ARN, ingress group name) and include:s templates/stages.yaml, which composes the child pipeline from the template fragments.

The deploy job composition:

# build/ci/templates/stages.yaml (excerpt)
stages: [django-migrations, deployment, tests]
include:
  - build/ci/templates/common-variables.yaml
  - build/ci/templates/migration-template.yaml
  - build/ci/templates/deploy-template.yaml
  - build/ci/templates/e2e-tests-template.yaml
  - build/ci/templates/stop-template.yaml

migrate:    { extends: .migration_template }
deploy:     { extends: .deploy_template }
e2e-tests:  { extends: .e2e_tests_template }
stop:       { extends: .stop_template }

Migration (build/ci/templates/migration-template.yaml)

The migration runs as its own Helm release (study-setup-migrate) so a slow or failed migration is decoupled from the app's deploy lifecycle:

1. Assume the GitLab OIDC → STS role to log into ECR (cross-account).
2. Scale backend, worker, beat to 0 so nothing speaks to a half-migrated schema. Service objects stay up — clients get 503s for the duration.
3. helm uninstall the previous migrate release (K8s Jobs are immutable; helm upgrade against an existing django-migrate Job fails on image mutation), then helm upgrade --install the migrate chart fresh from OCI ECR, --wait --wait-for-jobs.
4. A background heartbeat keeps GitLab from killing the job during long helm --waits; migration logs are surfaced into the GitLab job output once the Job pod terminates.

Deploy (build/ci/templates/deploy-template.yaml)

After needs: [migrate], runs helm upgrade --install --force study-setup against the study-setup chart from OCI ECR. The image tags are passed via --set so the chart pins to the SHA-tagged images produced by the parent pipeline's build stage.

Stop (build/ci/templates/stop-template.yaml)

Manual job that uninstalls both releases (study-setup and study-setup-migrate). allow_failure: true — used when tearing an env down.

E2E tests (build/ci/templates/e2e-tests-template.yaml)

Currently a placeholder — manual, allow_failure, prints "not yet implemented". The hook is in place for when the e2e suite is wired up.

Helm charts

Two independent releases per env namespace.

study-setup (main app chart)

Deploys backend / worker / beat / flower / redis / docs / frontend, an ingress that joins the cluster's shared ALB, and a fluent-bit ConfigMap for CloudWatch Container Insights.

build/helm/study-setup/
├── Chart.yaml                    name: study-setup, version: 0.1.0
├── values.yaml                   defaults (all components enabled)
├── values-staging.yaml           env overrides (ingress group, cert ARN, ...)
├── values-uat.yaml
├── values-prod.yaml
└── templates/
    ├── _helpers.tpl              labels, selectors, SA name, chart name
    ├── serviceaccount.yaml       study-setup-backend SA (Pod Identity target)
    ├── backend-deployment.yaml   + service + HPA
    ├── frontend-deployment.yaml  + service + HPA
    ├── docs-deployment.yaml      + service (VitePress static, served on :5173)
    ├── redis-deployment.yaml     + service (Celery broker)
    ├── worker-deployment.yaml    Celery worker
    ├── beat-deployment.yaml      Celery scheduler
    ├── flower-deployment.yaml    Celery monitoring UI (mounted at /-flower/)
    ├── ingress.yaml              ALB ingress joining the cluster anchor group
    ├── fluent-bit-configmap.yaml CloudWatch Container Insights parsers + outputs
    └── NOTES.txt

study-setup-migrate (migration chart)

Single Job/django-migrate plus its own ServiceAccount. Lives in a separate release so its lifecycle (retry, rollback, inspect) is decoupled from the app.

build/helm/study-setup-migrate/
├── Chart.yaml                    name: study-setup-migrate
├── values.yaml
├── values-{staging,uat,prod}.yaml
└── templates/
    ├── _helpers.tpl
    ├── serviceaccount.yaml       study-setup-migrate SA (own Pod Identity target)
    └── migration-job.yaml        runs `python manage.py migrate --noinput`

Secrets and extraEnv

The chart does not mount any Kubernetes Secret via envFrom: secretRef:. The app reads its secrets from AWS Secrets Manager at runtime via the Pod Identity-bound IAM role on its service account.

Each Django pod (backend / worker / beat / flower) and the migrate Job accept an extraEnv: [] list that is appended to the container's hardcoded env: block. Use it for non-secret pointers (e.g. SECRETS_BUNDLE_ARN) and for local-dev overrides where you want to inject DJANGO_SECRET_KEY / DJANGO_DATABASE_URL directly.

# values-local.yaml
backend:
  extraEnv:
    - name: DJANGO_SECRET_KEY
      value: dev-secret
    - name: DJANGO_DATABASE_URL
      value: sqlite:////tmp/local.sqlite3

AWS coordinates (production targets)

Resource	Value
AWS account (BerEpiCode)	920687946667
ECR registry	920687946667.dkr.ecr.eu-central-1.amazonaws.com
ECR repos	saxecap/study-setup, saxecap/study-setup-tests
GitLab runner role	arn:aws:iam::920687946667:role/SaxeCap-ECR-runner-Role
Runner tag	BerLimsDev-blue (build + test)
Cluster tag (staging+uat)	SaxeCapDev-blue
Cluster tag (prod)	SaxeCapProd-blue
Namespaces	study-setup-{staging,uat,prod}
Ingress group (dev cluster)	saxecapdev-blue
Ingress group (prod cluster)	saxecapprod-blue
CloudWatch region	us-east-1
CloudWatch log group prefix	/aws/containerinsights/study-setup

AWS_ROLE_ARN is consumed by all jobs that talk to AWS via GitLab OIDC (id_tokens.GITLAB_TOKEN) → STS AssumeRoleWithWebIdentity. The trust policy on SaxeCap-ECR-runner-Role must accept this project's GitLab JWT.

The study-setup-backend and study-setup-migrate service accounts each need a Pod Identity Association binding them to an IAM role with secretsmanager:GetSecretValue (and whatever else the app needs at runtime). The bindings are created out-of-band against the cluster + namespace + SA name.

Publishing helm charts

The chart version isn't auto-bumped. Charts are packaged + pushed manually:

# from repo root
./build/helm/package-and-publish.sh             # uses version from each Chart.yaml
./build/helm/package-and-publish.sh 0.2.0       # overrides version for both charts

The script helm pushes to oci://${REGISTRY}/saxecap/<chart-name>:<ver>. CHART_VERSION is supplied to the GitLab pipeline as a project- or environment-level variable so deploys pin to whatever release is current.

Local testing

The pipeline can be exercised almost end-to-end against an OrbStack Kubernetes cluster. What you can't exercise locally: GitLab OIDC → STS, ECR push/pull, ALB ingress controller, CloudWatch ingestion, Pod Identity Association wiring. Everything else has a local equivalent.

Prerequisites:

brew install helm gitlab-ci-local   # docker comes with OrbStack

Enable Kubernetes in OrbStack and confirm kubectl config current-context returns orbstack.

Just recipes

The build/ layer has its own .justfile with the shortcuts below. Run from the repo root:

Recipe	What it does
just build bake	docker buildx bake --load for all 4 images (backend / frontend / docs / test) and re-tag the test image to the CI-shape tag.
just build lint	helm lint --strict on both charts.
just build template [ENV]	Render the main chart and pipe it through kubectl apply --dry-run=server against the current kube context. Defaults to staging.
just build ci-local	Full CI sim: bake + lint + run the unit-tests job via gitlab-ci-local (postgres service + pytest). The canonical "test the pipeline locally" entry point.
just build deploy-local	Install both charts into study-setup-local namespace on the current kube context, using local-built images and injecting DJANGO_SECRET_KEY / DJANGO_DATABASE_URL via extraEnv.
just build down-local	helm uninstall both releases and delete the local namespace.

What ci-local does

just build ci-local is the single command for confirming the pipeline still works after a change. It runs the bake stage, lints both charts, then runs the unit-tests job in a docker container exactly as a real GitLab runner would (with postgres:16 brought up as a service alongside). Variables are overridden so the templated image: resolves to the locally-built test image.

A green ci-local covers everything in the pipeline except the AWS-specific bits called out under What's not exercised locally.

Verifying the chart on a real cluster

just build deploy-local installs both releases against your current kube context (typically OrbStack). It tears down with just build down-local. Once the pods are up:

kubectl get pods -n study-setup-local                                # all 1/1 Running
kubectl port-forward -n study-setup-local svc/backend 18080:8000 &
curl -s http://127.0.0.1:18080/-/_health/                            # {"status": "ok"}

Parsing the pipeline YAML directly

If you just want a quick YAML sanity check without going through ci-local:

gitlab-ci-local --list

Lists the parent pipeline jobs and validates the JSON schema. Child pipelines use local: includes that resolve from the repo root — if you point gitlab-ci-local at a child file directly, copy it to a scratch repo with the template paths preserved.

What's not exercised locally

These only run against real AWS, and are the things to verify first when the pipeline is wired up against the GitLab project:

• GitLab OIDC → STS AssumeRoleWithWebIdentity. Requires the trust policy on SaxeCap-ECR-runner-Role to accept the project's JWT (audience https://gitlab.com).
• ECR repos exist (saxecap/study-setup, saxecap/study-setup-tests).
• ECR push and pull of both images and helm OCI charts.
• ALB ingress controller picking up the chart's Ingress, merging into the shared ALB via group.name, attaching the env's certificate ARN as an SNI alt cert. ACM cert ARNs need to be filled into each env's values-*.yaml (currently "" # FIXME).
• Pod Identity Associations binding the study-setup-backend and study-setup-migrate service accounts to IAM roles that can read from AWS Secrets Manager.
• CloudWatch Container Insights ingestion via the cluster's aws-for-fluent-bit DaemonSet, driven by the chart's fluent-bit-config ConfigMap.
• The runtime SecretsManager fetcher inside the app — until that lands, Django pods will CrashLoopBackOff with InsecureConfigError unless DJANGO_SECRET_KEY / DJANGO_DATABASE_URL are supplied via extraEnv.