Production Deployment — Interrupted Mid-Rollout

A backend engineer was performing a staged deployment of a new service version when an unrelated page forced them to hand over to a colleague. The deployment is partially complete and in an inconsistent state. You need to produce a document that allows the colleague to safely continue.

What was being deployed

Version 2.4.0 of the inventory-service, replacing 2.3.1 in production.

Deployment steps completed

• Updated the Docker image tag to inventory-service:2.4.0 in the Kubernetes deployment manifest
• Applied the manifest to the prod namespace: kubectl apply -f k8s/inventory-service.yaml
• 3 of 6 replicas have been updated to 2.4.0; the remaining 3 are still running 2.3.1
• New replicas are passing readiness checks and serving traffic

What has NOT been done

• The database migration script (migrations/V14__add_index_on_sku.sql) has not been run — it needs to run before the old replicas are removed, as 2.3.1 pods will fail without the index once traffic reaches a threshold
• The old replicas have not been drained or terminated
• Feature flag inventory.new_batch_processing has not been enabled in the config service; 2.4.0 falls back to legacy behaviour without it

Known pitfalls encountered during this deployment

• Running kubectl rollout restart at this point would wipe all 2.4.0 progress and restart from 2.3.1; do NOT do this
• The migration script is idempotent but must not be run while the old replicas are handling write traffic; wait until a low-traffic window
• A previous attempt to enable the feature flag before the migration was complete caused 500 errors on the /batch endpoint for 8 minutes; always run the migration first

Prepare the handoff document for the colleague taking over the deployment.