Senior Principal Engineer - Kubernetes

Role Summary

As a Senior Principal Engineer, you will design, build, and operate Lillys Kubernetes-based hosting platform across multiple clouds. You will own platform capabilities end to end: writing Kubernetes controllers in Go and platform automation in Go or Python, defining infrastructure as code, evolving CI/CD and GitOps delivery, and raising the reliability and security of the platform. You will also lead a small team of engineers, guiding them technically and driving delivery.

What You'll Do

• Build, deploy and operate Kubernetes clusters end to end, from provisioning and configuration through upgrades, networking, scaling and decommissioning, on a managed offering such as EKS or ROSA, or an equivalent platform.
• Design, operate, and troubleshoot Kubernetes networking, including CNI plugins (Cilium, AWS VPC CNI), Services and kube-proxy replacement, CoreDNS, Ingress and Gateway API, network policies, pod-to-pod and cross-cluster connectivity, and egress control.
• Develop and maintain custom Kubernetes controllers and operators in Go (controller-runtime, Kubebuilder, or Operator SDK), including CRD design, reconciliation logic, finalizers, and admission webhooks.
• Operate and extend controller-based platform tooling such as AWS Controllers for Kubernetes (ACK), Crossplane, kro, Argo CD, and cert-manager.
• Deploy and operate the Istio service mesh (sidecar and ambient modes), including mTLS, traffic management (Gateway, VirtualService, DestinationRule, ServiceEntry), authorization policies, mesh observability, and version upgrades.
• Author and maintain automation, tooling, and services in Go or Python that streamline platform operations, onboarding, and review processes.
• Build and manage infrastructure as code using Terraform, Helm, and Kustomize, with strong module design, state management, and policy-as-code practices.
• Own and evolve CI/CD and GitOps pipelines using GitHub, GitHub Actions, Argo CD, Artifactory, and Checkmarx, with a security-first approach to software supply chain.
• Implement node and workload autoscaling with Karpenter, Cluster Autoscaler, HPA/VPA, and KEDA, including node image and upgrade strategies.
• Review and approve pull requests for platform hosting, deployment configurations, and infrastructure changes, ensuring they are secure, reliable, and aligned with platform standards.
• Lead a small team of platform engineers, providing technical direction, architecture and design guidance, and mentoring to grow engineers skills and independence.
• Own team delivery: break down roadmap items into sprint-sized work, plan and track sprints, manage dependencies and risks, and communicate status to stakeholders and leadership.
• Partner with application teams during onboarding, migration, and go-live, providing hands-on technical guidance and clear documentation.
• Participate in on-call and incident response for platform issues, including root cause analysis (RCA).

How You Will Succeed

• Demonstrate deep, hands-on Kubernetes expertise: scheduling, networking, storage, RBAC, API machinery, and troubleshooting production issues.
• Write production-quality Go (and Python where needed), including tests, and apply good software engineering practices to platform code.
• Apply strong knowledge of AWS/Azure, including compute, networking (VPC/VNet, routing, private endpoints), IAM (IRSA, Pod Identity, workload identity), and managed Kubernetes services.
• Lead and mentor the team, setting engineering standards and technical direction while keeping delivery predictable, with clear priorities, estimates, and outcomes.
• Use Istio to secure and manage service-to-service traffic, and set clear guidance on where mesh, CNI, and network policy responsibilities begin and end.
• Operate CI/CD and GitOps toolchains effectively, with sound judgment on release strategies such as progressive delivery, canary, and rollback.
• Use observability tooling to diagnose platform and application issues quickly, and drive improvements from the findings.

Required Skills

• Bachelors degree in Computer Science, Engineering, or a related field (or equivalent practical experience).
• 9+ years of experience in Platform or DevOps engineering, including running Kubernetes in production.
• Experience technically leading or mentoring a small team of engineers, with a track record of guiding design decisions and delivering platform work on schedule.
• Strong, hands-on experience with at least one managed Kubernetes offering (EKS, AKS, ROSA, or ARO); exposure to multiple is strongly preferred.
• Strong programming experience in Go, including hands-on experience building Kubernetes controllers and operators; Python for tooling and automation is a plus.
• Hands-on experience with Kubernetes manifests, rollouts and Helm charts (deployments, rollbacks, upgrades), along with a solid understanding of core Kubernetes concepts such as storage (PV/PVC, StorageClasses, CSI), networking (Services, Ingress, DNS, network policies) and scheduling.
• Strong Kubernetes networking knowledge: CNI plugins, Service and kube-proxy modes, CoreDNS, Ingress and Gateway API, network policies, and troubleshooting pod, node, and VPC connectivity.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.