Why We Run Staging in Production

The safest environment in the world is an empty one. Zero noise, infinite resources, perfect latency. It is also completely useless.

Every startup I’ve worked in eventually runs into the same tension: fidelity (how real is this test?) versus safety (what happens when it breaks?). Note the “when”. Most teams settle it by standing up a “staging” environment, and what they get is a low-fidelity ghost town — a sterile lab bearing no resemblance whatsoever to the chaotic, noisy, actually-under-load reality of production.

For years we followed the industry playbook: absolute separation, distinct GCP or AWS projects, no connectivity between them. It looks great on paper. There’s simply no way a staging release can touch production, not even under load testing. The environments are configured identically, so as long as our deployment artifacts are immutable, watching it work in staging means it’ll work in production.

Right?

In reality it’s a nightmare of duplication. We aren’t just mirroring code and artifacts; we’re mirroring infrastructure, and therefore operational cost. We pay for CloudSQL instances that sit idle for 20 hours a day. We spin up redundant VMs, Spark clusters, and CloudRun deployments that serve almost no traffic. We build “prod-like” datasets in BigQuery, and before long we’ve got a sprawling estate of resources that all need patching, upgrading, and monitoring. And after all that, we STILL can’t be sure the code running in staging will hold up against the volume and shape of the data sitting in production.

So we paper over the mismatch with workarounds: exporting and anonymising production data on a schedule, tweaking configuration in ways we’ve decided are “safe enough”, bolting on scale-down rules to claw back some money overnight. None of it is optimal, all of it takes real engineering effort to build and then keep alive, and each piece is a fresh home for bugs (sanitisation failures, startup race conditions, the usual). Meanwhile nobody is shipping anything a customer would notice.

We’re taking a different approach and architecting isolation: running staging and production in the same Kubernetes cluster and giving staging a clone of production data.

The Submarine Philosophy

Naval architects don’t design submarines on the assumption the hull will never breach. They design them with bulkheads: watertight compartments, so a flood in one section doesn’t drag the whole vessel to the bottom.

In our architecture the GKE cluster is the hull — shared nodes, ingress, monitoring, all of it. Namespaces are the bulkheads.

Sharing the hull is where the leverage comes from. One cluster to patch. One control plane to watch. And as long as we hold the bulkhead boundaries properly, a memory leak in some staging experiment can’t flood the engine room where the production database lives. A few other things fall out of it too:

• BigQuery zero-copy clones: we can provision a staging dataset that’s byte-for-byte identical to production, instantly. Combined with time travel, that means testing a gnarly transformation against real data — the actual volume, the actual weird edge cases — without duplicating a byte of storage or going anywhere near the live dataset.
• Logical isolation: no more paying for idle infrastructure. Rather than a dedicated Cloud SQL instance babysitting a staging environment that sees almost no traffic, we use separate schemas and users inside the shared instance. Same story for Redis and the other managed services.
• The “shared service” reality: here’s the part that made me feel a bit silly. We were already doing this. Our authentication provider (Clerk), our payment gateways, our email providers — none of them hand out physically air-gapped instances per environment. We’d been trusting external vendors to segregate our data logically for years. So either that’s an acceptable pattern or it isn’t, and if it is, we should be competent enough to run it ourselves.
• Forced discipline: this one is deliberate. When a bad staging deploy could theoretically reach production, configuration stops being an afterthought. You learn Resource Quotas, NetworkPolicies, and admission controllers properly, because you have to.

None of this is really about saving money — though it does. It’s about high-fidelity simulation. We want staging workloads fighting for resources. We want them living with the same network topography and the same noisy neighbours as the live services, because that’s the only way to find out what breaks. We just want to do it without sinking the ship, and that means bulkheads and logical isolation rather than duplicates.

Isolated Neighbours

We treat the staging namespace as hostile. Not because we think our developers are hostile, but because it’s a much easier assumption to reason about: any pod in there might be compromised, or wedged, or just doing something extremely stupid at 3am. So: default deny.

• Silence by default: no pod in the cluster can talk to any other pod. Full stop.
• The one-way mirror: NetworkPolicy rules then allow the specific internal chatter staging needs, while explicitly blocking egress to prod. Staging can see itself. It cannot see production at all.

This manifest is about as boring as configuration gets, and it’s the cornerstone of the whole thing — unless traffic is explicitly allowed, it’s forbidden.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: staging
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Identity Firewall

The data layer is the bulkhead that actually matters, and Kubernetes makes it uncomfortably easy to mount a god-mode secret without noticing. Workload identity is the fix: tie permissions to the pod, not the node.

• Production identity: a pod in prod carries a Google Cloud IAM identity with Cloud SQL Client on the live database.
• Staging identity: a pod in staging, running the exact same binary, carries a different identity — one with no permissions on the production instance whatsoever.

The nice property here is what happens when someone inevitably hardcodes the production connection string into a staging config. Nothing. The connection gets rejected at the IAM level, long before it becomes an incident report.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: backend-service
  namespace: staging
  annotations:
    # This binds the K8s ServiceAccount to a specific Google Cloud IAM Service Account
    iam.gke.io/gcp-service-account: staging-sa@my-project.iam.gserviceaccount.com

Resource Quotas & Priority Classes

In a shared hull the danger isn’t only security, it’s weight. A runaway process in staging chewing through every available core will starve production just as effectively as an attack would — the noisy neighbour problem, and it’s a lot more likely to happen than a breach.

• Priority classes: production workloads get PriorityClass: High. When the cluster runs short, the scheduler plays captain and evicts staging pods without ceremony or apology. Which is exactly what you want it to do.
• Resource quotas: hard caps on the staging namespace, in CPU and memory. It cannot balloon past its bulkhead no matter how much load it tries to pull. (Per-pod limits too, if you need the finer grain.)

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: production-high-priority
value: 1000000
globalDefault: false
description: "This priority class should be used for production service pods only."
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: staging-quota
  namespace: staging
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi

A Safer Place to Fail … and to Succeed

The biggest killer of developer velocity isn’t bad code. It’s deployment anxiety.

When an engineer isn’t quite sure whether their experiment might wipe a production table, they hesitate. They double-check. They ask someone. Then they ask someone else, just to be sure. Multiply that by a team and you’ve accidentally built a culture of gatekeeping, without anyone ever deciding to.

Isolation flips it. Codify the boundaries as guardrails and you stop leaning on human vigilance — which is finite, and worst at 5pm on a Friday — and start leaning on platform guarantees. When a developer knows the network policy makes it impossible for their pod to reach the production database, they stop worrying and start shipping.

We didn’t build this because we don’t trust our developers. We built it because we do trust them to experiment, push at things, and move fast. Isolation isn’t about restricting access. It’s about granting the freedom to break things, safely.

So: cake, had and eaten. The fidelity of a shared production environment with something close to the safety of an air-gapped lab. And production stops being the scary place you tiptoe around, and becomes what it should have been all along — a compartmentalised vessel that can take a hit and keep going.

---

This article was written with drafting and grammar assistance provided by AI. I take full responsibility for the content, including any and all errors. And I’m human, I think!