CI/CD runbook

Manual Jenkins to Nexus to OpenShift delivery

This guide documents the manually repeatable CI/CD path proven in the lab: create or update a workload, build its container image with Jenkins, push the image to Nexus, publish desired state to Git, and let hub Argo CD apply the application to spoke-dc through the registered destination cluster. The example uses the Open Liberty Operator smoke app, but the same control flow applies to a plain Deployment workload.

Scope

What this process does

Stage	System	Output
Source and manifests	`lab-workloads`	Application source, Containerfile, namespace, service, route, and workload CRs.
Image build	Jenkins on RKE2	Short-lived Buildah pod builds the image.
Image registry	Nexus	Application image under `nexus-docker.apps.sub.comptech-lab.com/opp/...`.
Deployment trigger	`lab-gitops-full`	Argo CD Application points the spoke to the workload path.
Runtime	OpenShift `spoke-dc`	Operator-managed app, pod, service, route, health endpoints, and GitOps status.

Safety

Non-negotiable rules

Do not print or commit kubeconfigs, kubeadmin passwords, Jenkins passwords, PAT values, Nexus passwords, Vault tokens, pull secrets, or full Secret manifests.
Use lab-workloads for application source and workload manifests. Use lab-gitops-full only to register the Argo CD Application and platform dependencies.
Prefer GitOps for durable deployment state. Live oc apply is acceptable for dry-run validation and short bootstrap steps, but reconcile desired state back to Git.
Use a unique image tag per build, normally build-YYYYMMDD-HHMMSS. Avoid mutable latest in OpenShift manifests.
Use app-specific Vault and External Secrets Operator wiring for production. A live-copied nexus-docker-pull Secret is only a bootstrap bridge.

Prerequisites

Confirm the path is ready

K=/home/ze/codex-opp-agent/ocp-clusters/spoke-dc/auth/kubeconfig

# OpenShift GitOps is running on the spoke.
oc --kubeconfig "$K" -n openshift-gitops get pods,argocd,app

# Open Liberty Operator is present when using OpenLibertyApplication.
oc --kubeconfig "$K" get crd | grep openliberty
oc --kubeconfig "$K" get csv -A | grep -i "open.*liberty"

# Jenkins and Nexus routes should be reachable from the operator workstation.
curl -skI https://jenkins.apps.sub.comptech-lab.com
curl -skI https://nexus.apps.sub.comptech-lab.com
curl -skI https://nexus-docker.apps.sub.comptech-lab.com/v2/

A 401 Unauthorized response from the Nexus Docker /v2/ endpoint is normal; it proves the registry endpoint is reachable and asking for authentication.

Step 1

Choose application coordinates

Pick names once and use them consistently through Jenkins, Nexus, GitOps, and OpenShift.

APP_NAME=openliberty-smoke-app
APP_NAMESPACE=openliberty-smoke-app
IMAGE_REPOSITORY=opp/openliberty-smoke-app
IMAGE_REGISTRY=nexus-docker.apps.sub.comptech-lab.com
IMAGE_PUSH_REGISTRY=nexus.nexus.svc.cluster.local:5000
IMAGE_TAG=build-$(date -u +%Y%m%d-%H%M%S)
ROUTE_HOST=openliberty-smoke-app.apps.spoke-dc.ocp.comptech-lab.com

Step 2

Create the workload source

Keep the workload in lab-workloads/components/apps/<app>. For Open Liberty, a very small app can be an expanded WAR directory plus server.xml.

cd /home/ze/codex-opp-agent/lab-workloads
mkdir -p components/apps/$APP_NAME/source/app/WEB-INF
mkdir -p components/apps/$APP_NAME/source/src/main/liberty/config

Containerfile for the validated Open Liberty smoke path:

FROM icr.io/appcafe/open-liberty:25.0.0.6-kernel-slim-java17-openj9-ubi-minimal
COPY --chown=1001:0 src/main/liberty/config/server.xml /config/server.xml
RUN features.sh
COPY --chown=1001:0 app /config/apps/openliberty-smoke.war
ENV APP_NAME=openliberty-smoke-app
EXPOSE 9080

Minimal Liberty server config:

<server description="Open Liberty smoke app">
  <featureManager>
    <feature>pages-3.1</feature>
    <feature>mpHealth-4.0</feature>
  </featureManager>

  <httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="9080"/>
  <webApplication location="openliberty-smoke.war" contextRoot="/"/>
</server>

Use pages-3.1 for JSP pages on this Liberty version. The feature name jsp-3.1 is not valid and makes features.sh fail.

Step 3

Create or update the Jenkins job

The validated jobs are freestyle Jenkins jobs represented as XML files under /home/ze/codex-opp-agent/reports/. The job should do four things: prepare source, obtain the Nexus push credential without printing it, create a short-lived privileged Buildah pod, then push the final image to Nexus.

JENKINS_URL=https://jenkins.apps.sub.comptech-lab.com
JOB=openliberty-smoke-app-image-build

# Read credentials from an approved local-only source. Never echo this value.
JENKINS_USER=admin
JENKINS_PASSWORD_FILE=/path/to/local-only/jenkins-password-file
AUTH="$JENKINS_USER:$(tr -d '\r\n' < "$JENKINS_PASSWORD_FILE")"
COOKIE=$(mktemp)

crumb_json=$(curl -sk -c "$COOKIE" --user "$AUTH" \
  "$JENKINS_URL/crumbIssuer/api/json")
crumb_header=$(printf '%s' "$crumb_json" \
  | jq -r '.crumbRequestField + ": " + .crumb')

curl -sk -b "$COOKIE" --user "$AUTH" \
  -H "$crumb_header" \
  -H 'Content-Type: application/xml' \
  --data-binary @/home/ze/codex-opp-agent/reports/openliberty-smoke-app-jenkins-job.xml \
  "$JENKINS_URL/job/$JOB/config.xml"

If the job does not exist yet, create it first with $JENKINS_URL/createItem?name=$JOB and the same XML body. If using the Jenkins UI, create a freestyle job and paste the shell body from the XML into the build step.

Step 4

Run the Jenkins build

IMAGE_TAG=build-$(date -u +%Y%m%d-%H%M%S)
NEXT=$(curl -sk -b "$COOKIE" --user "$AUTH" \
  "$JENKINS_URL/job/$JOB/api/json?tree=nextBuildNumber" | jq -r '.nextBuildNumber')

curl -sk -b "$COOKIE" --user "$AUTH" \
  -H "$crumb_header" \
  --data-urlencode "IMAGE_TAG=$IMAGE_TAG" \
  --data-urlencode 'APP_MESSAGE=Open Liberty app built by Jenkins, pushed to Nexus, and deployed by the Open Liberty Operator' \
  "$JENKINS_URL/job/$JOB/buildWithParameters"

echo "Started $JOB build #$NEXT with tag $IMAGE_TAG"

Watch the build without exposing credentials:

curl -sk -b "$COOKIE" --user "$AUTH" \
  "$JENKINS_URL/job/$JOB/$NEXT/consoleText" \
  | sed -E 's#(X-Vault-Token: )[A-Za-z0-9._-]+#\1[REDACTED]#g; s#(Authorization: Bearer )[A-Za-z0-9._-]+#\1[REDACTED]#g; s#(auth": ")[^"]+#\1[REDACTED]#g'

The successful Open Liberty reference run was Jenkins build #6, tag build-20260507-215919.

Step 5

Confirm the image in Nexus

Jenkins prints the image and digest at the end of a successful build. For independent verification, inspect the manifest using an auth file generated from a local secret source. Do not print the auth file.

IMAGE="$IMAGE_REGISTRY/$IMAGE_REPOSITORY:$IMAGE_TAG"
AUTHFILE=$(mktemp)

# Populate $AUTHFILE from a local-only approved credential source.
# Example output should not be committed or pasted into tickets.

podman manifest inspect --authfile "$AUTHFILE" --tls-verify=false "$IMAGE" \
  | jq '{schemaVersion, mediaType, config: .config.digest, compressedBytes: (.layers | map(.size) | add)}'

rm -f "$AUTHFILE"

The validated Open Liberty smoke image digest is sha256:f4e759125a2eb562a5047b625364d0f1bbcd8e705bca26a56e077827116889bf.

Step 6

Create the OpenShift workload manifests

For an Open Liberty Operator deployment, use an OpenLibertyApplication. Pin the exact Nexus tag and use manageTLS: false when the app listens on HTTP behind an edge-terminated OpenShift Route.

apiVersion: apps.openliberty.io/v1
kind: OpenLibertyApplication
metadata:
  name: openliberty-smoke-app
  namespace: openliberty-smoke-app
spec:
  applicationName: openliberty-smoke-app
  applicationVersion: build-20260507-215919
  applicationImage: nexus-docker.apps.sub.comptech-lab.com/opp/openliberty-smoke-app:build-20260507-215919
  pullSecret: nexus-docker-pull
  pullPolicy: IfNotPresent
  replicas: 1
  manageTLS: false
  expose: true
  route:
    host: openliberty-smoke-app.apps.spoke-dc.ocp.comptech-lab.com
    termination: edge
    insecureEdgeTerminationPolicy: Redirect
  service:
    type: ClusterIP
    port: 9080
    targetPort: 9080
    portName: http
  probes:
    readiness:
      httpGet:
        path: /health/ready
        port: 9080
      initialDelaySeconds: 20
      periodSeconds: 10
      timeoutSeconds: 3
      failureThreshold: 12
    liveness:
      httpGet:
        path: /health/live
        port: 9080
      initialDelaySeconds: 40
      periodSeconds: 20
      timeoutSeconds: 3
      failureThreshold: 6

Step 7

Prepare image pull credentials

Preferred production pattern: create an app-specific Vault policy and Kubernetes auth role, then use ESO to materialize Secret/nexus-docker-pull in the app namespace.

Temporary bootstrap pattern: copy an existing pull secret into the app namespace without printing its contents. This is a bridge only and must be tracked as cleanup.

K=/home/ze/codex-opp-agent/ocp-clusters/spoke-dc/auth/kubeconfig

oc --kubeconfig "$K" create namespace "$APP_NAMESPACE" \
  --dry-run=client -o yaml | oc --kubeconfig "$K" apply -f -

oc --kubeconfig "$K" -n wso2-demo-app get secret nexus-docker-pull -o json \
  | jq 'del(.metadata.uid,.metadata.resourceVersion,.metadata.creationTimestamp,.metadata.managedFields,.metadata.ownerReferences,.metadata.annotations["kubectl.kubernetes.io/last-applied-configuration"])
        | .metadata.name="nexus-docker-pull"
        | .metadata.namespace="openliberty-smoke-app"
        | .metadata.labels={"app.kubernetes.io/name":"openliberty-smoke-app","app.kubernetes.io/component":"image-pull","gitops.lab/bootstrap":"live-secret-copy"}' \
  | oc --kubeconfig "$K" apply -f - -o name

Step 8

Render and dry-run before pushing

cd /home/ze/codex-opp-agent

oc kustomize lab-workloads/components/apps/openliberty-smoke-app \
  > /tmp/openliberty-smoke-app-render.yaml

oc --kubeconfig "$K" apply --dry-run=client \
  -f /tmp/openliberty-smoke-app-render.yaml -o name

oc --kubeconfig "$K" apply --dry-run=server \
  -f /tmp/openliberty-smoke-app-render.yaml -o name

Server-side dry-run requires the CRD and target namespace to exist. If the namespace is new, create it first or rely on Argo CD CreateNamespace=true after the client-side render passes.

Step 9

Commit and push the workload repo

cd /home/ze/codex-opp-agent/lab-workloads
git status --short
git add components/apps/openliberty-smoke-app
git commit -m "Add Open Liberty Jenkins Nexus smoke app"
GIT_ASKPASS=/home/ze/codex-opp-agent/secrets/git-askpass.sh git push origin main

Step 10

Register the Argo CD Application

Add this to lab-gitops-full/clusters/spoke-dc/ and include it in the spoke overlay kustomization.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: openliberty-smoke-app
  namespace: openshift-gitops
  annotations:
    argocd.argoproj.io/sync-wave: "20"
spec:
  project: default
  source:
    repoURL: http://30.30.30.5/gitops/lab-workloads.git
    targetRevision: main
    path: components/apps/openliberty-smoke-app
  destination:
    server: https://kubernetes.default.svc
    namespace: openliberty-smoke-app
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ApplyOutOfSyncOnly=true
      - SkipDryRunOnMissingResource=true
      - ServerSideApply=true

cd /home/ze/codex-opp-agent/lab-gitops-full
oc kustomize clusters/spoke-dc > /tmp/spoke-dc-render.yaml
git add CHANGELOG.md clusters/spoke-dc/kustomization.yaml clusters/spoke-dc/openliberty-smoke-app-application.yaml
git commit -m "Add Open Liberty smoke app Argo application"
GIT_ASKPASS=/home/ze/codex-opp-agent/secrets/git-askpass.sh git push origin main

Step 11

Refresh Argo CD and wait for sync

K=/home/ze/codex-opp-agent/ocp-clusters/spoke-dc/auth/kubeconfig

oc --kubeconfig "$K" -n openshift-gitops annotate app spoke-dc-cluster-config \
  argocd.argoproj.io/refresh=hard --overwrite

oc --kubeconfig "$K" -n openshift-gitops get app openliberty-smoke-app

oc --kubeconfig "$K" -n openshift-gitops annotate app openliberty-smoke-app \
  argocd.argoproj.io/refresh=hard --overwrite

oc --kubeconfig "$K" -n openshift-gitops get app openliberty-smoke-app \
  -o custom-columns=NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status,REV:.status.sync.revision

Expected final state: Synced and Healthy at the pushed workload commit.

Step 12

Validate the deployed app

oc --kubeconfig "$K" -n openliberty-smoke-app get openlibertyapplication,pod,svc,route -o wide

oc --kubeconfig "$K" -n openliberty-smoke-app get pod \
  -l app.kubernetes.io/instance=openliberty-smoke-app \
  -o jsonpath='{range .items[*]}{.metadata.name}{" image="}{.spec.containers[0].image}{" imageID="}{.status.containerStatuses[0].imageID}{" ready="}{.status.containerStatuses[0].ready}{" phase="}{.status.phase}{"\n"}{end}'

curl -sk https://openliberty-smoke-app.apps.spoke-dc.ocp.comptech-lab.com/health/live
curl -sk https://openliberty-smoke-app.apps.spoke-dc.ocp.comptech-lab.com/health/ready
curl -sk https://openliberty-smoke-app.apps.spoke-dc.ocp.comptech-lab.com/metadata

Expected reference output: app openliberty-smoke-app, runtime Open Liberty, image tag build-20260507-215919, namespace openliberty-smoke-app, and HTTP 200 from both health endpoints.

Step 13

Record the delivery

Update CURRENT_STATE.md with image tag, digest, route, Argo revision, and validation result.
Update TODO.md for any residual cleanup, especially bootstrap pull secrets.
Append SESSION_LOG.md and create a detailed file under reports/sessions/.
If lab-gitops-full changed, update lab-gitops-full/CHANGELOG.md.
If the process itself changed, update RUNBOOK.md and this wiki page.

Troubleshooting

Common failure modes

Symptom	Likely cause	Action
Build pod stuck on base image pull	Slow external registry or vfs storage unpack	Use the IBM Open Liberty mirror, prefer the kernel-slim base, and use Buildah overlay storage on the Jenkins cache PVC.
`CWWKF1402E` from `features.sh`	Invalid Liberty feature name	Use `pages-3.1` for JSP and `mpHealth-4.0` for health endpoints on the validated Liberty image.
Pod stays unready with HTTPS probe errors	Open Liberty Operator defaulted TLS management for a HTTP service	Set `manageTLS: false` for HTTP-on-9080 behind an edge-terminated Route.
Image pull fails on OpenShift	Missing or invalid `nexus-docker-pull`	Verify the pull secret exists in the app namespace. Replace bootstrap copies with Vault/ESO as soon as possible.
Argo CD stays OutOfSync	Operator or Kubernetes defaulted a field	Compare desired and live specs. Make harmless defaults explicit, such as `service.type: ClusterIP`.
Route returns 503	Pod not ready or Service selector mismatch	Check pod readiness, service endpoints, and the operator-managed labels on the Deployment and Service.

Reference state

Validated Open Liberty smoke deployment

Jenkins job: openliberty-smoke-app-image-build
Successful build: #6
Nexus image: nexus-docker.apps.sub.comptech-lab.com/opp/openliberty-smoke-app:build-20260507-215919
Digest: sha256:f4e759125a2eb562a5047b625364d0f1bbcd8e705bca26a56e077827116889bf
GitOps app: Application/openliberty-smoke-app in openshift-gitops
Runtime CR: OpenLibertyApplication/openliberty-smoke-app
Route: https://openliberty-smoke-app.apps.spoke-dc.ocp.comptech-lab.com
Residual cleanup: replace the bootstrap pull secret with app-specific Vault/ESO wiring.

Next: production readiness