Staff Backend Engineer - K8 (Envoy)

Job Summary

Staff Backend Engineer - K8 (Envoy), Bengaluru

Responsibilities

• As a Staff Backend Engineer, you will work closely with platform and product leaders to design and deliver solutions for complex infrastructure problems.
• You will drive the development of highly scalable, reliable, and efficient platform services while providing technical direction across teams working with Java, AWS, Kafka, Kubernetes, Kubeflow, Argo CD, and gRPC.
• Architect and build Coupang's next-generation AI Inference Gateway platform that serves mission-critical machine learning and generative AI workloads at scale.
• Design and develop high-performance request routing, model endpoint abstraction, traffic shaping, load balancing, failover, caching, and policy enforcement mechanisms for inference services.
• Drive the technical vision and roadmap for scalable, secure, and reliable AI inference infrastructure across cloud and on-prem environments.
• Develop critical infrastructure components in Go, Java, or Python with a strong focus on performance, resiliency, and operational excellence.
• Design multi-tenant platform capabilities including authentication, authorization, quota management, cost attribution, rate limiting, and governance controls.
• Partner closely with ML Platform, Model Serving, Data, and Product Engineering teams to enable seamless deployment and operation of AI workloads.
• Lead architecture and design reviews, raise engineering standards, and mentor senior engineers across multiple teams.
• Optimize system performance, latency, throughput, and infrastructure efficiency for large-scale inference workloads.
• Define observability standards through metrics, tracing, logging, and SLO-based operations for business-critical AI services.
• Investigate complex production issues, drive root-cause analysis, and implement long-term architectural solutions.
• Collaborate with engineering leaders across Coupang to establish common platform standards and unlock AI innovation across the organization.

Requirements

• 8+ years of professional software development experience.
• 5+ years of experience designing and operating large-scale distributed systems in production.
• Strong hands-on programming expertise in one or more of Go, Java, or Python.
• Proven track record of building highly available, mission-critical platform or infrastructure services.
• Experience designing API platforms, service gateways, service mesh, or large-scale networking infrastructure.
• Deep understanding of microservices architecture, distributed systems design, and cloud-native technologies.
• Experience with Kubernetes and containerized workloads in production environments.
• Experience operating services on AWS, Azure, or GCP.
• Strong understanding of observability, reliability engineering, capacity planning, and production operations.

Qualifications

• Experience building AI/ML inference platforms, LLM gateways, model serving infrastructure, or GPU-accelerated workloads.
• Deep expertise in Kubernetes ecosystem technologies such as Gateway API, Ingress Controllers, Service Mesh (Istio, Linkerd, Envoy), and platform networking.
• Experience with inference serving frameworks such as vLLM, Triton Inference Server, TensorRT-LLM, Ray Serve, KServe, SGLang, or similar technologies.
• Strong understanding of high-performance networking, gRPC, HTTP/2, streaming protocols, API gateways, and service proxy architectures.
• Experience designing large-scale traffic management systems including routing, retries, circuit breaking, rate limiting, and request prioritization.
• Experience optimizing latency, throughput, and resource utilization for CPU and GPU workloads.
• Familiarity with GenAI and LLM ecosystems including model deployment, prompt routing, RAG systems, model observability, and AI governance.
• Experience with distributed data systems such as Kafka, Cassandra, Redis, MongoDB, or similar technologies.
• Strong understanding of concurrency, synchronization, asynchronous programming, and non-blocking I/O.

Job Type

Hybrid / Onsite / Remote working

• Our Hybrid work model: Coupang hybrid work model is designed to enable a culture of collaboration that acts a catalyst to enrich the experience of employees. Employees are required to work at least 3 days in the office per week, with the flexibility to work from home 2 days a week, depending on the role requirement. Some businesses may require more time in office due to nature of work.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.