Secrets
Vault and External Secrets Operator
The OpenShift clusters now have a validated smoke path to external Vault. This proves the integration pattern while keeping Vault tokens, TokenReview JWTs, and application secret values out of Git.
Status
Current integration state
| Area | State | Notes |
|---|---|---|
| Vault placement | External HA Vault | vault-rke2.apps.sub.comptech-lab.com is the OCP-facing endpoint. |
| ESO operator | Installed on all four OCP clusters | Red Hat External Secrets Operator plus ExternalSecretsConfig/cluster. |
| Vault auth | One Kubernetes auth mount per cluster | Mounts are kubernetes-hub-dc, kubernetes-hub-dr, kubernetes-spoke-dc, and kubernetes-spoke-dr. |
| GitOps component | components/platform/external-secrets-vault | Creates only non-secret OpenShift resources. |
| Smoke validation | Passing on all four clusters | SecretStore/rke2-vault is valid and ExternalSecret/eso-vault-smoke is synced. |
| Application onboarding | Not started | Real apps still need namespace-specific Vault policies, roles, and ExternalSecrets. |
Design
Boundary model
- Keep Vault outside OpenShift as the shared HA secrets control plane for this lab.
- Use a separate Vault Kubernetes auth mount per OCP cluster so each cluster can have its own reviewer credential, policy boundary, and role mapping.
- Use namespace-scoped
SecretStoreobjects for application namespaces unless there is a specific platform reason to expose a cluster-wide store. - Bind ESO login to
external-secrets/eso-vault-authand keep TokenReview onexternal-secrets/vault-token-reviewer. - Do not commit Vault tokens, TokenReview JWTs, pull secrets, app secret values, or full Secret manifests.
GitOps
Commits and rendered resources
5a972baadds Vault-backed ESO smoke wiring to all four OCP overlays.e672ee5declares ESO default remoteRef strategies to avoid Argo drift after operator defaulting.- Rendered resources include
ServiceAccount/eso-vault-auth,ServiceAccount/vault-token-reviewer,ClusterRoleBinding/vault-token-reviewer-external-secrets,SecretStore/rke2-vault, andExternalSecret/eso-vault-smoke. - Each cluster overlay patches the Vault auth mount and smoke KV path to its own cluster name.
External Vault
Scoped export token hardening
The earlier Vault export repair used current recovery material to restore the broken export job quickly. That has now been replaced by a scoped periodic export token.
- The external Vault export automation now renews its scoped token at job start.
- The DC
vault/vault-replicatorSecret now uses a scoped periodic token from policyvault-replicator-export. - Manual export validation completed with the scoped token and uploaded fresh MinIO snapshots.
- The Vault replication automation was validated healthy after the scoped-token change.
Validation
Read-only checks
for c in hub-dc hub-dr spoke-dc spoke-dr; do
K=/home/ze/codex-opp-agent/ocp-clusters/$c/auth/kubeconfig
echo "## $c"
oc --kubeconfig "$K" -n external-secrets get serviceaccount eso-vault-auth vault-token-reviewer
oc --kubeconfig "$K" -n external-secrets get secretstore rke2-vault
oc --kubeconfig "$K" -n external-secrets get externalsecret eso-vault-smoke
oc --kubeconfig "$K" -n external-secrets get secret eso-vault-smoke \
-o json | jq '{name:.metadata.name, keys:(.data|keys)}'
doneThe last command lists only Secret keys. Do not print Secret data values.
Next
Before real app secrets
- Create workload-specific Vault policy: grant only the exact KV paths the namespace needs.
- Create workload-specific role: bind the policy to the workload namespace and service account.
- Add GitOps references only: commit SecretStore or ExternalSecret resources, not secret values.
- Validate without leaking data: confirm Ready conditions and target Secret key names only.
- Plan reviewer-token rotation: rotate the live TokenReview JWTs before expiry or replace them with a durable rotation process.