Keycloak Setup

This guide covers running Keycloak locally for development, the operator checklist for a new environment, and — in detail — the real problems encountered deploying Keycloak to ECS Fargate in this project, since none of them are obvious from the AWS or Keycloak documentation alone.

Read this before your first production-style deploy

The "Deployment postmortem" section below documents 10 distinct failures hit deploying this exact template, several of which produce misleading symptoms (e.g. a 403 HTTPS required that has nothing to do with TLS, or an ECS circuit breaker that fires because of a database problem two layers away). Skim the symptom table first if something is broken.

Local development (Docker)

For local development you don't need a full ECS deployment. Run Keycloak in dev mode with an in-memory H2 database — sufficient for testing the OIDC login flow and Admin API:

docker run --rm -p 8080:8080 \
  -e KC_BOOTSTRAP_ADMIN_USERNAME=admin \
  -e KC_BOOTSTRAP_ADMIN_PASSWORD=admin \
  quay.io/keycloak/keycloak:26.2.5 start-dev

Admin console: http://localhost:8080/admin — credentials admin / admin.

Set the local env var so the payment service and portal point to it:

export KEYCLOAK_URL=http://localhost:8080
export NEXT_PUBLIC_KEYCLOAK_URL=http://localhost:8080

warning

start-dev uses an embedded H2 database and is not suitable for production. The ECS deployment uses start with KC_DB=postgres pointing to RDS — see the postmortem below for why this distinction caused real problems.

Creating a test realm locally

Either use the Admin Console UI or call the Admin API directly:

TOKEN=$(curl -s -X POST http://localhost:8080/realms/master/protocol/openid-connect/token \
  -d "grant_type=password&client_id=admin-cli&username=admin&password=admin" \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['access_token'])")

curl -s -X POST http://localhost:8080/admin/realms \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"realm":"test-corp","enabled":true,"displayName":"Test Corp"}'

Or use the portal self-registration flow (requires the payment service running with KEYCLOAK_ADMIN_USER and KEYCLOAK_ADMIN_PASSWORD set):

curl -X POST http://localhost:8081/api/v1/portal/register \
  -H "Content-Type: application/json" \
  -d '{"company_name":"Test Corp","trading_name":"Test Corp","admin_email":"admin@test.com"}'

---

Deployment postmortem: what actually went wrong

This section documents, in the order encountered, every failure hit deploying Keycloak on ECS Fargate against the shared RDS instance in this environment. Each entry has the exact symptom, the real root cause, and the fix — because in almost every case the visible error pointed somewhere unhelpful.

Symptom quick-reference

Symptom	Root cause	Jump to
CREATE_FAILED on KeycloakAlbSecurityGroup — "Character sets beyond ASCII are not supported"	Em dash (—) in a GroupDescription string	#1
Container log: Invalid value for option 'KC_PROXY_HEADERS': x-forwarded	Keycloak 26 renamed the enum value	#2
ECS Deployment Circuit Breaker triggers before Keycloak ever logs a real error	keycloak_core schema / keycloak_user role didn't exist yet — chicken-and-egg with RUN_MIGRATIONS	#3
ALB target stuck unhealthy even though container logs show Keycloak ... started	Health check hit port 8080; Keycloak 26 moved /health/ready to the management port 9000	#4
"To use the HostedRotationLambda property, you must use the AWS::SecretsManager transform"	Missing Transform: header + wrong deploy command (deploy needs a packaged template + CAPABILITY_AUTO_EXPAND)	#5
{"error":"invalid_request","error_description":"HTTPS required"} on the token endpoint, over what is otherwise a working HTTP deployment	Keycloak enforces sslRequired per-realm; the enum value is case-sensitive (NONE, not none)	#6
A manual aws ec2 authorize-security-group-ingress fix "disappears"	A later cloudformation deploy (even a no-op parameter change) rolled the stack back and reverted the manual fix	#7
Two ECS deployments shown IN_PROGRESS at once, confusing task/log correlation	Calling force-new-deployment repeatedly without waiting for the prior deployment to resolve	#8
New API routes return a bare 404 with no JSON body	The running container image predates the source code — routes were written but never built/pushed	#9
db error: TLS handshake: invalid peer certificate: UnknownIssuer	An empty placeholder cert file (created for local cargo check) got baked into the real Docker image	#10
column "cognito_client_id" does not exist on a migration that ran fine before	Migrations replay on every container boot (no version-tracking table); a later rename breaks an earlier migration's unconditional CREATE INDEX	#11
null value in column "legal_name" violates not-null constraint on an INSERT ... ON CONFLICT DO UPDATE — even though the conflicting row already exists	PostgreSQL validates NOT NULL constraints on the candidate tuple before checking for a conflict	#12
{"message": "realm-admin role in acme-corp not found"} aborting registration entirely	Keycloak does not create a plain realm-level realm-admin role in new realms; the assumption in the original code was wrong	#13

---

#1. Non-ASCII characters in CloudFormation resource descriptions

Symptom:

KeycloakAlbSecurityGroup CREATE_FAILED
Value (bopen-dev Keycloak ALB — allow HTTPS/HTTP inbound) for parameter
GroupDescription is invalid. Character sets beyond ASCII are not supported.

Root cause: the template's GroupDescription used a typographic em dash (—) instead of a plain hyphen. The EC2 API rejects it outright — this has nothing to do with CloudFormation itself, it's an EC2 SecurityGroup field validation rule.

Fix: replace all em dashes with - in any string that becomes an EC2 API parameter (GroupDescription, tag values passed to EC2 resources, etc.). Comments and descriptions elsewhere in the YAML are unaffected — only fields that get passed literally to an AWS API with ASCII-only validation.

Lesson: if you write CloudFormation templates with an LLM or a text editor that auto-converts -- to —, grep for non-ASCII characters in any field that maps to a cloud API parameter before deploying:

grep -nP '[^\x00-\x7F]' infra/cloudformation/templates/keycloak.yaml

---

#2. KC_PROXY_HEADERS value renamed in Keycloak 26

Symptom:

Invalid value for option 'KC_PROXY_HEADERS': x-forwarded.
Expected values are: forwarded, xforwarded

Root cause: Keycloak's proxy-header option historically accepted x-forwarded; Keycloak 26 renamed the enum value to xforwarded (no hyphen). Templates or examples copied from older Keycloak docs will silently fail startup.

Fix:

- Name: KC_PROXY_HEADERS
  Value: xforwarded   # NOT x-forwarded

---

#3. Chicken-and-egg: DB schema vs. first boot

Symptom: deploying the full stack in one shot (DesiredCount=2, RUN_MIGRATIONS equivalent behaviour baked into the container) trips the ECS Deployment Circuit Breaker almost immediately, with no useful Keycloak log output — just repeated task starts and stops.

Root cause: Keycloak needs the keycloak_core schema and keycloak_user PostgreSQL role to exist before it can connect (KC_DB_USERNAME=keycloak_user). Those are created by 008_keycloak_setup.sql, which must be run once, manually, as the RDS admin user — it is deliberately excluded from the payment-service's automatic migration runner (see migrate.rs) because it needs elevated privileges the app role doesn't have. If you deploy Keycloak with tasks trying to start before this script has run, every task fails to authenticate to Postgres, and enough consecutive failures trip the circuit breaker, which then rolls the whole stack back — undoing the ECS service, target group, and any manual fixes applied in between.

Fix — deploy in two phases:

1. Deploy the CloudFormation stack with DesiredCount=0. This creates the ALB, target group, security groups, task definition, and Secrets Manager secrets, but launches no tasks — nothing can fail yet.
2. Retrieve the auto-generated keycloak_user password from Secrets Manager and run 008_keycloak_setup.sql against RDS as the admin user (see checklist step 2).
3. Only then scale the service to DesiredCount=1 (or 2) via aws ecs update-service --desired-count 1.

Lesson: any deployment that combines "create infrastructure" and "run first-boot migrations that need elevated DB privileges" in one atomic step should be split. The two-phase approach here isn't just a workaround — it's the actual, most reliable sequence for this class of problem, and deploy-keycloak.sh should be treated as "phase 1 only" until this is automated.

---

#4. Health check on the wrong port

Symptom: ECS shows the task RUNNING, container logs clearly show Keycloak 26.2.5 ... started in 46s. Listening on: http://0.0.0.0:8080. Management interface listening on http://0.0.0.0:9000. — yet the ALB target group keeps reporting the target unhealthy, and after UnhealthyThresholdCount failures ECS kills the task and starts over, forever.

Root cause: the target group's HealthCheckPort defaulted to traffic-port (8080). Keycloak 26 moved /health/ready off the main port and onto the dedicated management port (9000) — a request to :8080/health/ready gets nothing useful back. This is a genuine behavioural change from earlier Keycloak versions and isn't obvious unless you read the Quarkus management-interface changelog.

Fix:

1. Set the target group's health check port explicitly:

   aws elbv2 modify-target-group \
     --target-group-arn "$TG_ARN" \
     --health-check-port 9000 \
     --health-check-path /health/ready

2. Open port 9000 (not just 8080) from the ALB security group to the Keycloak tasks security group — this is easy to forget since it's a second port on the same container:

   aws ec2 authorize-security-group-ingress \
     --group-id "$KC_TASKS_SG" --protocol tcp --port 9000 \
     --source-group "$KC_ALB_SG"

3. keycloak.yaml now hard-codes HealthCheckPort: '9000' on the target group and includes an explicit port-9000 ingress rule on KeycloakTasksSecurityGroup — this is fixed in the template going forward, but is worth knowing about if you ever hand-edit the target group or notice it drift back to traffic-port after a stack update (see #7).

Also worth knowing: the container-level Docker HealthCheck (using curl) turned out to be unreliable in Keycloak's minimal runtime image (curl behaved inconsistently inside the container even though the ALB's own HTTP health check against the same port succeeded every time). The container health check was removed entirely — the ALB target group health check is the sole health arbiter. This is a reasonable simplification: ECS doesn't need a redundant, less-reliable check duplicating what the load balancer already verifies more accurately from outside the container.

---

#5. Secrets Manager rotation requires a SAM transform

Symptom:

KeycloakDbSecretRotation CREATE_FAILED
Resource handler returned message: "To use the HostedRotationLambda property,
you must use the AWS::SecretsManager transform"

Root cause: AWS::SecretsManager::RotationSchedule with the convenience HostedRotationLambda property (rather than a hand-rolled Lambda) is implemented via a macro that CloudFormation expands using the AWS::SecretsManager-2020-07-23 transform — this requires (a) declaring the transform at the top of the template and (b) deploying via aws cloudformation package (to resolve the macro reference) followed by deploy --capabilities CAPABILITY_AUTO_EXPAND (not just CAPABILITY_NAMED_IAM).

Fix:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::SecretsManager-2020-07-23   # required for HostedRotationLambda
Description: ...

aws cloudformation package \
  --template-file keycloak.yaml \
  --s3-bucket "$BUCKET" --s3-prefix cloudformation/packaged \
  --output-template-file /tmp/keycloak-packaged.yaml

aws cloudformation deploy \
  --template-file /tmp/keycloak-packaged.yaml \
  --stack-name bopen-dev-keycloak \
  --capabilities CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND \
  --parameter-overrides ...

Deploying the raw (unpackaged) template, or omitting CAPABILITY_AUTO_EXPAND, fails at the same point every time.

---

#6. HTTPS required error on a deliberately HTTP-only dev deployment

Symptom: Keycloak is up, the ALB target is healthy, GET /realms/master returns a valid response — but POST /realms/master/protocol/openid-connect/token (the admin-cli password grant, needed for every subsequent Admin API call) returns:

{"error":"invalid_request","error_description":"HTTPS required"}

This has nothing to do with the ALB, DNS, or an actual missing certificate — it's a realm setting.

Root cause: every Keycloak realm has an sslRequired setting (ALL / EXTERNAL / NONE) that defaults to EXTERNAL — meaning "HTTPS required for any request coming from outside the local network." Since this dev environment deliberately has no ACM certificate on the Keycloak ALB (plain HTTP, AcmCertificateArn=""), every request looks "external + non-HTTPS" to Keycloak and gets rejected at the token endpoint, independent of the actual ALB/network configuration.

The first fix attempt didn't work and revealed a second issue: updating the realm directly in the keycloak_core.realm table —

UPDATE keycloak_core.realm SET ssl_required = 'none';

— still failed with the same error. The reason: ssl_required is stored and compared as an uppercase enum-like string internally (NONE, EXTERNAL, ALL); Keycloak's Java code does not normalise the case. 'none' is silently accepted by Postgres (it's just a varchar column, no CHECK constraint) but never matches Keycloak's internal comparison.

Also required: setting KC_HOSTNAME_URL=http://dev-auth.bopenbanking.com on the container so Keycloak's own hostname resolution doesn't independently assume HTTPS.

Fix — full sequence:

-- Must be uppercase. lowercase 'none' silently fails to take effect.
UPDATE keycloak_core.realm SET ssl_required = 'NONE';

# Task definition environment
- Name: KC_HOSTNAME_URL
  Value: http://dev-auth.bopenbanking.com

Then force a new ECS deployment — Keycloak caches realm configuration at startup; a database-only change does not take effect on already-running tasks.

Lesson: in production, this whole problem doesn't arise — you'd have a real ACM certificate on the Keycloak ALB and leave sslRequired at its secure default. This fix is specifically for dev/test environments that intentionally run HTTP-only, and should not be replicated in an environment with a real certificate.

---

#7. CloudFormation rollback silently reverts manual fixes

Symptom: a manual fix applied directly via the CLI (a security group rule, a target group's health check port) works — then, after a subsequent cloudformation deploy for an unrelated parameter change, the same problem comes back with no obvious cause.

Root cause: CloudFormation stacks reconcile their entire managed resource state on every deploy, including a rollback. If an earlier deploy attempt fails and rolls back, or a later deploy re-asserts the template's declared state, any out-of-band manual change to a CloudFormation-managed resource (security group rules, target group attributes) gets silently reverted back to whatever the template says — even if the template itself doesn't yet reflect a fix you've only applied by hand.

Fix: treat manual CLI fixes as temporary and disposable during troubleshooting — the moment a fix is validated, port it back into the actual CloudFormation template (as was done for the port-9000 health check and security group rule in keycloak.yaml). Don't rely on a manual fix surviving the next stack operation, and don't spend time debugging "why did my fix disappear" — assume a rollback or redeploy reverted it and check the template's current declared state first.

---

#8. Overlapping force-new-deployment calls

Symptom: aws ecs describe-services shows two deployments both IN_PROGRESS at once; task logs and target-group health become difficult to correlate to "which attempt is this."

Root cause: ECS allows multiple deployments to be queued; calling update-service --force-new-deployment again before the previous deployment has resolved (COMPLETED or rolled back) doesn't cancel the earlier one — it queues alongside it. During active troubleshooting (rebuilding an image, fixing an env var, retrying) it's very easy to issue a second force-deployment while the first is still cycling through failed task attempts, especially since each Keycloak boot takes 45-90 seconds before you learn whether it worked.

Fix: poll aws ecs describe-services ... --query 'services[0].deployments[0].rolloutState' until it reaches COMPLETED or FAILED before issuing another deployment command. If you must intervene urgently, note the deployment ID (ecs-svc/...) you're tracking and don't assume the newest stopped task belongs to the newest deployment — check createdAt timestamps directly:

aws ecs describe-tasks --cluster ... --tasks "$TASK_ARN" \
  --query 'tasks[0].[taskArn,createdAt,stoppedAt]'

---

#9. The running image was a week old

Symptom: a newly written API route (POST /api/v1/portal/register) returns a bare {"message": "Not Found"} — not the application's own JSON error format, which would have included a proper error field — with HTTP 404, even after confirming the API Gateway route and ALB listener rule were both configured correctly.

Root cause: the {"message": "Not Found"} body is Axum's own default 404 response, not an error surfaced by application code — meaning the route genuinely doesn't exist in the binary that's running. The ECR image tagged latest had been pushed before the portal registration, webhook engine, and refund endpoint code was written; git log confirmed the source was committed, but nobody had rebuilt and pushed a new container image since.

Fix:

./infra/cloudformation/scripts/build-and-push.sh payment-service bopen-dev latest eu-west-2
aws ecs update-service --cluster bopen-dev-cluster --service bopen-dev-payment-service --force-new-deployment

Lesson: a 404 with a generic framework body (rather than your application's own error envelope) is a strong signal the route isn't compiled into the running binary at all — check the image push timestamp (aws ecr describe-images ... --query 'imageDetails[*].[imageTags,imagePushedAt]') against your latest relevant commit before debugging routing config.

---

#10. Empty cert placeholder shipped to production

Symptom:

database migration failed error=Error occurred while creating a new object:
error performing TLS handshake: invalid peer certificate: UnknownIssuer

Root cause: state.rs bundles the RDS CA certificate at compile time via include_bytes!("../certs/rds-global-bundle.pem"). Earlier, to get a local cargo check to compile (the macro requires the file to exist, even if unused at check-time), an empty placeholder file was created at that path. That placeholder was never replaced with the real certificate bundle — and because there's no .dockerignore, Docker's build context included it exactly as-is. The resulting image's rustls TLS client had zero trusted CA certificates, so every RDS connection attempt failed certificate validation.

Fix:

curl -sf -o services/payment-service/certs/rds-global-bundle.pem \
  https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem

Then rebuild and push the image. Note: this file is covered by a root *.pem gitignore rule (a reasonable default to avoid accidentally committing real secrets), so it will never show up in git status — anyone who clones the repo fresh must fetch it themselves before building the Docker image. Consider this a standing gap: either document the fetch step prominently in the build instructions, or special-case this specific public CA bundle in the gitignore rules and commit it (it contains no secrets — it's a public trust store).

---

#11. Non-idempotent migration replay

Symptom: a migration that worked perfectly the first time (003_b2b_payments.sql) fails on every subsequent container boot with:

db error: ERROR: column "cognito_client_id" does not exist

— on a migration file that hasn't been touched.

Root cause: this project has no schema_migrations version-tracking table — migrate.rs batch-executes every file in its list, unconditionally, on every single container start (see RUN_MIGRATIONS=true). This "replay everything, every time" design requires every statement in every migration file to remain valid forever, even after later migrations change the schema out from under it. Migration 003_b2b_payments.sql unconditionally creates CREATE INDEX idx_merchants_cognito_client ON bopen.merchants(cognito_client_id). Migration 009_rename_client_id.sql later renames that column to client_id. The first time 009 runs, this is fine. The second time the container boots (any redeploy, any restart), migration 003 replays and tries to index a column that no longer exists — permanent failure from that point on, for every future boot, on both old and new container images.

The exact same class of bug existed in 009 itself: its own ALTER TABLE ... RENAME COLUMN cognito_client_id TO client_id fails on the second replay, once the column has already been renamed once.

Fix — guard every non-idempotent DDL statement with an existence check:

-- 003_b2b_payments.sql
DO $$
BEGIN
  IF EXISTS (
    SELECT 1 FROM information_schema.columns
    WHERE table_schema = 'bopen' AND table_name = 'merchants' AND column_name = 'cognito_client_id'
  ) THEN
    EXECUTE 'CREATE INDEX IF NOT EXISTS idx_merchants_cognito_client ON bopen.merchants(cognito_client_id)';
  END IF;
END
$$;

-- 009_rename_client_id.sql
DO $$
BEGIN
  IF EXISTS (
    SELECT 1 FROM information_schema.columns
    WHERE table_schema = 'bopen' AND table_name = 'merchants' AND column_name = 'cognito_client_id'
  ) THEN
    ALTER TABLE bopen.merchants RENAME COLUMN cognito_client_id TO client_id;
  END IF;
END
$$;

Lesson for every future migration: because there is no version tracking, any migration that does something the schema will no longer look like after a later migration (renames, drops, type changes) must be written defensively from day one. CREATE ... IF NOT EXISTS / DROP ... IF EXISTS are not sufficient on their own if the statement references a column/table name that a different migration might remove — you specifically need an information_schema existence check, as above. Before adding a new migration, grep for RENAME|DROP COLUMN|ALTER COLUMN.*TYPE across the whole migrations directory to make sure you're not the next victim of this pattern:

grep -ln "RENAME COLUMN\|RENAME TO\|ALTER COLUMN.*TYPE\|DROP COLUMN" database/scripts/migrations/*.sql

---

#12. ON CONFLICT DO UPDATE does not skip NOT NULL validation

Symptom:

db error: ERROR: null value in column "legal_name" of relation "merchants"
violates not-null constraint
DETAIL: Failing row contains (<new-random-uuid>, dev-acquirer-client-0001, ..., null, null, ...).

— on a seed file's INSERT ... ON CONFLICT (client_id) DO UPDATE, even after confirming via direct query that a row with that exact client_id already exists in the table, with a valid, non-null legal_name.

Root cause: this is counter-intuitive enough to be worth explaining precisely. PostgreSQL's INSERT ... ON CONFLICT executor first fully constructs the candidate row from the VALUES clause (filling any omitted column with its default or NULL), and validates NOT NULL/CHECK constraints against that candidate tuple — before it ever probes the unique index to check whether a conflict exists. Conflict detection happens via a "speculative insertion" step that only runs after the candidate tuple has already passed basic validity checks. So if a migration adds a NOT NULL column with no DEFAULT (legal_name, trading_name via migration 005; webhook_signing_secret via migration 012) after a seed file was originally written, and that seed file's INSERT column list doesn't mention the new column at all, the statement fails at tuple-construction time — regardless of whether the row already exists and the DO UPDATE branch would have fired and never touched that column anyway.

Fix: every column in the table that is NOT NULL with no DEFAULT must appear in the INSERT column list of any ON CONFLICT DO UPDATE statement, even if the DO UPDATE SET clause never references it and even if a later seed/migration will immediately overwrite the value:

INSERT INTO bopen.merchants (
  client_id, name, legal_name, trading_name, webhook_url, is_active, webhook_signing_secret
)
VALUES (
  'dev-acquirer-client-0001', 'Demo Acquirer Ltd',
  'Demo Acquirer Ltd',        -- legal_name: placeholder, seed 003 overwrites it later
  'Demo Acquirer Ltd',        -- trading_name: same
  'https://webhook.example.com/bopen/payments', TRUE,
  encode(sha256('dev-acquirer-client-0001-seed-secret'::bytea), 'hex')
)
ON CONFLICT (client_id) DO UPDATE
  SET name = EXCLUDED.name, webhook_url = EXCLUDED.webhook_url,
      is_active = EXCLUDED.is_active, updated_at = NOW();

Lesson: whenever you add a NOT NULL column with no DEFAULT to an existing table, grep the seed files for ON CONFLICT statements against that table and update their column lists — this bug will not show up until the next time that seed file actually executes against a database where the row already exists (i.e. every redeploy after the first), which is precisely the scenario that combines badly with #11's "replay everything forever" design.

---

#13. Assumed a role that Keycloak doesn't create

Symptom: merchant self-registration (POST /api/v1/portal/register) got as far as creating the Keycloak realm, the OIDC client, and the admin user — then failed the entire request with:

{"error": "realm-admin role in acme-corp-limited"}

leaving a half-provisioned realm behind (with no automatic cleanup).

Root cause: the original registration code assumed every newly created realm would contain a plain realm-level role literally named realm-admin, and tried to assign it to the new user. Keycloak does not create such a role automatically — its actual tenant-admin permission model works differently (via roles on a {realm}-realm client registered in the master realm, not a same-named role inside the target realm itself). Since this application never actually needs the registered user to have native Keycloak admin rights — the payment service always calls the Keycloak Admin API using its own master-realm service credentials (KEYCLOAK_ADMIN_USER/KEYCLOAK_ADMIN_PASSWORD), never the merchant's own token — this step was solving a problem that didn't exist, and its failure was blocking registrations that would otherwise have succeeded completely.

Fix: make the role assignment best-effort — log a warning and continue if it fails, rather than aborting the whole registration:

if let Err(e) = admin.assign_realm_admin_role(&realm, &user_id).await {
    tracing::warn!(realm = realm, user_id = user_id, error = %e,
        "could not assign realm-admin role — continuing registration anyway");
}

Known follow-up issue (not yet fixed): the registration handler currently sets bopen.merchants.client_id = "bopen-portal" — the same literal string — for every self-registered merchant, because that's the shared public client ID used for browser login in every realm. Since client_id has a UNIQUE constraint, the second successful self-registration will fail with a unique-constraint violation. This needs a genuinely per-merchant value before self-registration is used beyond one-off testing; it hasn't been fixed yet because it surfaced only after this postmortem's fixes finally let a registration complete successfully for the first time.

---

Production deployment checklist

This reflects the sequence validated by the postmortem above, not the originally-assumed sequence.

1. Deploy infrastructure with zero tasks

aws cloudformation package \
  --template-file infra/cloudformation/templates/keycloak.yaml \
  --s3-bucket "$TEMPLATES_BUCKET" --s3-prefix cloudformation/packaged \
  --output-template-file /tmp/keycloak-packaged.yaml

aws cloudformation deploy \
  --template-file /tmp/keycloak-packaged.yaml \
  --stack-name bopen-dev-keycloak \
  --capabilities CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND \
  --parameter-overrides \
    NamePrefix=bopen-dev \
    ... \
    DesiredCount=0

See #5 for why package + CAPABILITY_AUTO_EXPAND are both required.

2. Initialise the database schema

KC_DB_SECRET_ARN=$(aws cloudformation describe-stacks --stack-name bopen-dev-keycloak \
  --query "Stacks[0].Outputs[?OutputKey=='KeycloakDbSecretArn'].OutputValue" --output text)

KC_DB_PASS=$(aws secretsmanager get-secret-value --secret-id "$KC_DB_SECRET_ARN" \
  --query 'SecretString' --output text \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['password'])")

psql "$DATABASE_URL" -v kc_pass="$KC_DB_PASS" \
  -f database/scripts/migrations/008_keycloak_setup.sql

This must be run before any Keycloak task starts — see #3.

3. Verify the target group health check port

aws elbv2 describe-target-groups --target-group-arns "$TG_ARN" \
  --query 'TargetGroups[0].[HealthCheckPort,HealthCheckPath]'
# Expect: ["9000", "/health/ready"] — see #4

4. Scale up and watch for a healthy target

aws ecs update-service --cluster bopen-dev-cluster --service bopen-dev-keycloak \
  --desired-count 1

# Poll until healthy — Keycloak takes 45-90s to boot
aws elbv2 describe-target-health --target-group-arn "$TG_ARN" \
  --query 'TargetHealthDescriptions[0].TargetHealth.State'

5. Add DNS CNAME for the Keycloak domain

aws cloudformation describe-stacks --stack-name bopen-dev-keycloak \
  --query "Stacks[0].Outputs[?OutputKey=='KeycloakAlbDnsName'].OutputValue" --output text

Add a CNAME: auth.bopenbanking.com → <AlbDnsName> (or dev-auth.bopenbanking.com for dev).

6. Dev-only: disable HTTPS enforcement if there's no ACM certificate

Only if you're deliberately running HTTP-only (see #6 for why this is dev-specific and should not be done in an environment with a real ACM cert):

UPDATE keycloak_core.realm SET ssl_required = 'NONE';  -- uppercase; case-sensitive

Set KC_HOSTNAME_URL=http://<your-dev-hostname> on the task definition, then force a new deployment (realm config is cached at boot).

7. Set up SES email (for email invites)

./infra/cloudformation/scripts/deploy.sh bopen-cfn-templates-... bopen-dev eu-west-2 EnableSes=true
# Add DKIM CNAME records to DNS (from outputs: SesDkimRecord*Name/Value)
./infra/cloudformation/scripts/generate-ses-smtp-credentials.sh bopen-dev eu-west-2
# If SES is in sandbox mode, request production access in the AWS console

Not actually done in bopen-dev

This step has not been run against the current bopen-dev environment — aws secretsmanager list-secrets shows no SES secrets and no SES stack exists. Email invites currently fail silently; see IAM — Email invites. Run the commands above for real if you need working email invites.

8. Log in to Keycloak admin console

aws secretsmanager get-secret-value \
  --secret-id bopen-dev/keycloak/admin-password \
  --query 'SecretString' --output text \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['password'])"

Visit http(s)://<keycloak-hostname>/admin → log in as admin → change the admin password immediately.

9. Rebuild and push the payment-service image before testing anything new

Do this every time, not just once — see #9:

./infra/cloudformation/scripts/build-and-push.sh payment-service bopen-dev latest eu-west-2
aws ecs update-service --cluster bopen-dev-cluster --service bopen-dev-payment-service --force-new-deployment

Realm provisioning (via self-registration API)

New acquirers are onboarded via POST /api/v1/portal/register. The payment service calls the Keycloak Admin API to:

1. Create a realm with FAPI defaults (PS256, 15-min tokens, single-use refresh)
2. Create a bopen-portal public PKCE client
3. Create the admin user and attempt to send a set-password email invite via SES — currently fails silently since SES isn't deployed (see step 7 above)
4. Best-effort attempt to assign a realm-admin role (non-fatal if it fails — see #13)

For manual realm provisioning (operator-created merchant accounts), use the Keycloak Admin Console or the Admin REST API directly with the master realm admin token.

FAPI realm configuration checklist

When creating realms manually, verify these settings in the Keycloak realm settings:

Setting	Required value
Default signature algorithm	PS256
Access token lifespan	900 s (15 min)
Refresh token: revoke	✓
Brute force protection	✓ Enabled
Self-registration	✗ Disabled
SSL required	EXTERNAL or ALL in any environment with a real TLS certificate (only NONE in HTTP-only dev — see #6)
bopen-portal client: PKCE challenge method	S256
bopen-portal client: Direct access grants	✗ Disabled

Troubleshooting

Keycloak tasks failing health check (target group shows unhealthy)

• Confirm the target group's HealthCheckPort is 9000, not 8080 or traffic-port — see #4
• Confirm port 9000 is open from the ALB security group to the Keycloak tasks security group
• Check CloudWatch logs: /ecs/bopen-dev-keycloak — look for Keycloak ... started to confirm the app itself is healthy before assuming the health check config is the problem

ECS deployment circuit breaker fires with no clear Keycloak error

• Confirm keycloak_core schema and keycloak_user role exist in RDS — see #3
• Confirm KC_DB_PASSWORD in the running task actually matches the current value in Secrets Manager (a rotation can silently invalidate an already-created user's password — re-run the DB init script to sync)

403 HTTPS required on the token endpoint

• This is a realm setting, not a TLS/network problem — see #6
• If you already set ssl_required = 'NONE' and it's still failing, check case sensitivity — Postgres will happily store 'none' but Keycloak won't honour it
• Realm config is cached at boot — a DB change requires a fresh ECS deployment to take effect

A route that should exist returns a bare {"message": "Not Found"}

• That's Axum's default 404, not your application's error format — the route isn't compiled into the running image. Rebuild and redeploy — see #9

A previously-working migration suddenly fails on redeploy

• Check whether a later migration renamed or dropped something the failing migration references unconditionally — see #11

Seed script fails with a NOT NULL violation on a column the ON CONFLICT branch never touches

• The candidate row is validated before conflict detection — every NOT NULL column with no default must be in the INSERT list, full stop. See #12

Email invites not arriving

• First, confirm SES is actually deployed in this environment (aws secretsmanager list-secrets | grep ses) — in bopen-dev it currently is not, which is the whole reason invites aren't arriving there. The steps below assume you've deployed the SES stack (step 7 above).
• Verify SES domain is verified (not sandbox mode)
• Check SES sending statistics in AWS console
• Re-run generate-ses-smtp-credentials.sh and force a new ECS deployment
• Confirm realm SMTP settings match the Secrets Manager ses/smtp-credentials values

realm already exists on registration

The realm slug is derived from the company name. If acme-corp-limited already exists (including from a previous failed attempt that got as far as creating the realm before failing later — see #13), the API returns HTTP 409. Delete the partial realm from the Keycloak console, or register with a different company name.