From NGINX to Istio: Lessons from a Zero-Downtime Migration at Scale

Ingress NGINX was retired in March 2026 with no further releases, fixes, or patches. We didn't wait for the deadline.

At Deel, we process payroll and manage employment for millions of workers across 150+ countries. Our infrastructure sits at the center of thousands of companies' critical HR operations - any downtime ripples through real paychecks, real compliance requirements, real people's lives. So when Ingress NGINX entered its sunset phase, we faced a decision that went beyond just "swap out a controller." It was about moving millions of end users without them noticing.

We migrated to Istio ahead of schedule, executed a zero-downtime cutover, and didn't miss a beat. Here's how we pulled it off, and what we learned in the process.

Why This Mattered: The Clock Was Ticking

Kubernetes SIG Network and the Security Response Committee announced the transition out of Ingress NGINX in November 2025, with March 2026 as the hard deadline. Waiting until the last moment wasn't an option for us. At our scale and with our operational surface area, a migration this large needed:

• Time to test thoroughly - Any misconfiguration could break routing for critical services
• Gradual rollout capacity - We needed to shift traffic incrementally, watch metrics, catch issues before they hit all users
• Runbook hardening - Our on-call engineers needed to practice failure scenarios

The deadline gave us urgency, but we treated it as a planning constraint, not a panic button.

Our Stack Before Istio

At Deel, we're managing global infrastructure that processes payroll, compliance, and hiring across 150+ countries. As our platform grew, so did the complexity of managing traffic patterns, enforcing policies, and ensuring reliability at scale. Our architecture looked like what you'd expect from a modern Kubernetes-based platform:

Internet → Cloudflare → Nginx Ingress → Services → Internal resources

Cloudflare handled DDoS protection and SSL termination at the edge. Nginx Ingress Controller routed traffic into our Kubernetes cluster. Individual services handled business logic. Our nginx setup was solid. It handled load balancing, TLS termination, and basic routing reliably. Nginx is battle-tested, well-documented, and has a massive community. We could configure routing rules, add basic rate limiting, and handle SSL termination without much friction.

As Nginx reached EOL, we needed to rework our internal solution. The good news: this became a catalyst for modernizing our networking foundation in ways we hadn't anticipated.

NGINX Pain Points

• Limited traffic control (no built-in fault injection or retries)
• Complex TLS or header handling (requiring additional NGINX configuration)
• Lack of deep observability
• No built-in service-to-service security (Istio provides mutual TLS or mTLS)

Must-Have Requirements

• Traffic Mirroring: Send a copy of live production traffic to a shadow deployment without affecting real traffic flow, allowing us to validate at production scale before committing to a rollout.
• TCP Traffic Exposure: Route raw TCP traffic (not just HTTP) so a single gateway can expose multiple database instances on different ports behind VPN, without creating expensive LoadBalancers or NodePorts for each one.
• Gateway Controller Sharding: Run multiple isolated gateway controllers in the same cluster, each responsible for different namespaces or responsibilities, so one team's misconfiguration doesn't take down critical paths for everyone else.
• AWS ALB Rule Limit: AWS ALBs have a hard limit of 100 rules per Application Load Balancer (200 with support request). With thousands of routes, this becomes a blocker. Istio's Gateway runs its own Envoy proxies inside the cluster. The ALB only needs a single rule pointing to those pods. All routing logic (path rewriting, header matching, traffic splitting) happens in Kubernetes via HTTPRoute resources, not in the ALB.

What Istio Gives Us That NGINX Can't

Why Istio Is Better Than NGINX (Beyond Feature Parity)

Security

Every service-to-service connection is encrypted automatically with mutual TLS (mTLS). NGINX only handles external traffic. Everything between services inside the cluster remains unencrypted today.

Services are identified by cryptographic certificates tied to their Kubernetes service account. No one can impersonate a service by spoofing an IP address.

You can restrict which services can talk to each other using AuthorizationPolicy. In a proof of concept, a "rogue" pod with a valid mesh certificate still couldn't call any service it wasn't explicitly permitted to. All 4 call paths returned "RBAC: access denied."

Operations

Config changes apply live without restarting anything. At our scale (thousands of services, hundreds of nodes), NGINX reloads cause connection drops.

The same configuration works identically on AWS EKS, Azure AKS, and GCP GKE. NGINX ingress annotations behave differently across versions and forks.

HTTPRoute is a Kubernetes standard (Gateway API). If we ever move away from Istio, the routing config is portable to Envoy Gateway, Cilium, Traefik, and others.
Traffic splitting is built in. Send X% of traffic to a new version and gradually shift it. Integrates with tools like Argo Rollouts and Flagger for automated canary analysis.

Observability

Access logs on every proxy (gateway, waypoint, ztunnel) with no extra setup. Just apply a Telemetry resource.
Prometheus metrics are automatically exported for every connection: request count, latency histograms, error rates broken down by source and destination service. No code changes needed.

The Migration Strategy

Phase 1: Planning and Testing

We validated that Istio covered everything we were using: NGINX annotations, OAuth2/token auth, URL rewrites, header manipulation, and request timeouts.

Once confirmed, we ran load tests using the gateway-api-bench project, which independently benchmarks 7 Gateway API implementations. This is not our own testing. It's a third-party benchmark run by an Istio maintainer - John Howard, using standardised tests across all controllers. The results are directly relevant to the scenarios we care about: large route counts, zero-downtime updates, and raw throughput.
Controllers tested: Cilium, Envoy Gateway, Istio Envoy, Kgateway, Kong, Traefik, and NGINX Gateway Fabric.

Results:

• Istio had zero errors during zero-downtime route changes (others dropped traffic or crashed)
• At scale (5,000 routes across 50 namespaces), Istio had the lowest control plane CPU usage of any controller tested

This gave us the confidence to proceed.

Phase 2: Shadow Deployment

We deployed Istio in observe-only mode alongside our existing NGINX Ingress controller. All traffic still flowed through NGINX. Istio's controller captured metrics and mirrored requests for analysis, but made no forwarding decisions.

This let us:

• Verify Istio's performance characteristics under real load
• Build confidence in our observability setup (alerts, dashboards, SLO definitions)
• Identify Ingress NGINX rules that didn't have direct Istio equivalents
• Train the team on debugging Istio-specific issues

By the end of this phase, we had hard data: Istio was handling traffic correctly, our observability caught issues quickly, and our team felt comfortable with the operational surface.

Phase 3: Gradual Shift

We shifted services to Istio in cohorts based on criticality and complexity:

Tier 1 - Internal services - non-customer-facing.
Tier 2 - Customer APIs - read-heavy, high-traffic.
Tier 3 - Payment and payroll services - highest criticality: cut over at low-traffic windows, with full rollback plan ready.

For each cohort, we:

• Ran synthetic monitoring (continuous small requests to verify response times, error rates)
• Watched custom SLO dashboards showing latency percentiles, error budgets, and saturation metrics
• Kept Ingress NGINX running in parallel, ready to absorb traffic if needed
• Maintained runbooks for fast rollback: DNS record flip and re-route traffic

This approach let us migrate at our own pace, we never needed the rollback. But knowing it was there kept everyone calm and focused.

The operation took about a month to migrate all resources in lower environments and an additional 2 weeks for production.

Phase 4: Validation and Decommissioning

Once all services were running on Istio, the real work began: proving we could turn off NGINX and never look back.

We built a comprehensive monitoring dashboard in Prometheus that tracked:

• Traffic volume and patterns - Request rate, connection count, throughput by service
• HTTP status codes - 2xx, 4xx, 5xx distributions; any deviation from baseline was an instant alert
• Latency percentiles - p50, p95, p99; we'd accept a small latency increase, but not degradation
• Error rates - Absolute count and as a percentage of total traffic
• Pods health - Memory and CPU usage, restart counts, proxy sync status

For two weeks straight, we ran on-call rotation dedicated to the migration. The team watched dashboards, spot-checked logs, and validated that everything looked right. We didn't have a specific incident, but we caught several near-misses - small configuration issues that would've become problems at scale - and fixed them proactively.

Only after 14 days of green dashboards, no customer reports, and full confidence in both the system and our monitoring setup did we move to the final step.

Technical Decisions and Tradeoffs

GeoIP and Location-Based Traffic

NGINX injects country codes into request headers using a built-in GeoIP module. Istio and Envoy have no equivalent.

Our solution was a Golang service running as an Envoy ext_proc backend that reproduces NGINX's X-geoip-* headers.

The service fetches MaxMind's GeoLite2-City database from S3 on startup and periodically hot-swaps it (no restart). A Lambda refreshes the database daily after verifying its signature against MaxMind's published checksum. Each request passing through the gateway is enriched with geographic headers before reaching the backend service.

Results and Impact

The migration improved our security posture significantly by enabling stricter access rules and managing internal resources under unified governance. We also migrated related infrastructure to Istio, including AWS Load Balancer Controller, cross-account access, and internal APIs. This unified approach gave us a single control plane for all networking resources.

Resource consumption dropped dramatically. Our largest cluster's NGINX controller consumed ~90GB of RAM. Istio consumes ~3GB at peak. This freed up resources and reduced operational overhead.

Most importantly, the migration unlocked future initiatives: canary deployments using Argo Rollouts, service-to-service encryption with mutual TLS, and traffic mirroring for production validation.

Recommendations for Teams Considering This Migration

Preparation is essential. Networking is the backbone of every resource in your platform. This migration touched infrastructure fundamentals, and you can't do it without proper planning and testing. We spent months validating before we started, and that confidence was warranted. It's not shameful to return to the drawing board. In our case, the GeoIP solution took time to get right, but once solved, it worked flawlessly.

Monitoring is phase zero. Don't skip observability. You need visibility into your traffic, error rates, and resource consumption from day one. This isn't something to add later. It's foundational to safe migration.

Go slowly. Migrate one service at a time. Test thoroughly. Keep rollback paths open. The flexibility to move at your own pace and revert quickly is what prevents a migration from becoming a crisis.

Conclusion

NGINX Ingress Controller served us well until it reached end-of-support. Istio has become the industry standard for a reason: it balances security, networking capabilities, and operational maturity. While other tools exist, Istio's feature set and community support make it the practical choice for large-scale deployments.

The migration isn't trivial, but it's worth it. You're not just replacing an ingress controller; you're modernizing your networking foundation. That foundation will support the security, observability, and traffic management features your platform will need as it scales.