Site Reliability Engineer

Responsibilities:

• Resolve escalated incidents across Kubernetes, API Proxy, WAF, DBs, and infra platforms.
• Design and improve runbooks, automating manual steps wherever possible.
• Lead and contribute to building self-healing systems and self-service tooling for users.
• Analyze incident trends and propose improvements in monitoring, capacity, and reliability.
• Collaborate with engineering teams on deployment, upgrades, and performance optimization.
• Conduct postmortems, document RCA, and ensure learning is captured.
• Mentor and coach engineer(s).

Requirements:

• 7+ years in SRE/DevOps/Systems Engineering as a senior or principal engineer.
• Strong hands-on experience with Kubernetes, container orchestration, and API management.
• Working knowledge of WAFs, network security, and database technologies (SQL/NoSQL).
• Proficient in automation and scripting (Python, Go, Ansible, Terraform, etc. ).
• Strong observability/monitoring experience.
• Experience with CI/CD pipelines, GitOps, and infrastructure as code.
• Solid problem-solving and collaboration skills.

The core must-have skills for the job include the following:

Advanced Incident Troubleshooting and Resolution:

• Expectation: Diagnose and resolve escalated incidents that engineers cannot handle, often across multiple layers (infrastructure, application, network).
• Example: For an API outage, identify if the root cause is in Kubernetes pod networking, API gateway misconfiguration, or backend DB latency and apply fixes.

Kubernetes and Container Orchestration Expertise:

• Expectation: Comfortable with deployments, scaling, networking, and debugging cluster-level issues.
• Example: Troubleshoot why pods are pending by checking node capacity, taints/tolerations, and cluster autoscaler logs.

Automation and Scripting (Python, Go, Bash, Ansible, Terraform)

• Expectation: Write scripts and automation to reduce manual toil, enhance monitoring, and improve incident resolution speed.
• Example: Develop a Python script to automatically collect pod and system logs when a service crashes.

Observability and Monitoring Tooling:

• Expectation: Deep understanding of monitoring, alerting, tracing, and logging systems.
• Example: Build Prometheus alert rules to detect DB query spikes; configure Grafana dashboards for API latency.

CI/CD and Infrastructure as Code (IaC):

• Expectation: Familiarity with GitOps workflows, CI/CD pipelines, and infrastructure provisioning.
• Example: Enhance Jenkins pipeline to add automated smoke tests before promoting Kubernetes deployments.

Database Troubleshooting (SQL and NoSQL):

• Expectation: Identify performance bottlenecks, connection issues, and basic tuning opportunities.
• Example: Run queries to detect slow-running SQL statements causing latency in an application.

Incident Management and RCA:

• Expectation: Act as incident commander for escalated issues, lead bridge calls, and produce root cause analyses.
• Example: After a WAF misconfiguration causes downtime, lead the investigation, document the timeline, and propose preventive actions.

Mentorship and Runbook Improvement:

• Expectation: Coach Engineer(s), refine runbooks, and introduce new automated workflows.
• Example: Update a runbook to add automated Kubernetes log collection instead of manual steps.

The core is nice to have for the job, including the following:

Cloud Platform Engineering (AWS, Azure, GCP):

• Expectation: Hands-on skills in provisioning, scaling, and securing cloud workloads.
• Example: Diagnose why an AWS ALB is misrouting traffic after a deployment.

Security and WAF Management:

• Expectation: Understand WAF rules, common attacks (SQL injection, XSS), and how to apply fixes.
• Example: Investigate false positives in WAF logs and adjust rule sets with security teams.

Capacity and Performance Engineering:

• Expectation: Anticipate scaling needs, tune resource utilization, and propose optimizations.
• Example: Identify that a Kubernetes deployment is CPU-throttled and adjust HPA (Horizontal Pod Autoscaler) configs.

Automation Platform Integration (AIOps, ChatOps):

• Expectation: Integrate AI/ML-powered tools for anomaly detection and auto-remediation.
• Example: Implement a ChatOps bot that runs predefined Kubernetes troubleshooting commands in Slack.

Cross-Platform Expertise (Hybrid Infra):

• Expectation: Experience supporting both on-prem and cloud environments seamlessly.
• Example: Compare latency patterns between on-prem DBs and cloud-hosted APIs to identify bottlenecks.