L3 Observability

Job Description for L3 Managed Service Engineer - Observability
Role Summary
Level 3 Managed Service Engineer serves as the senior escalation point and technical authority for end-to-end observability across the environment. This role is responsible for resolving or driving resolution of highly complex incidents spanning application, infrastructure, network, and third-party dependencies, while owning observability strategy, Datadog architecture, monitoring standards, and continuous improvement. The engineer is expected to guide L1 and L2 engineers, lead root cause analysis and major incident reviews, and partner with cross-functional teams to improve detection, reduce noise, and strengthen operational resilience for both 3-tier and cloud-native applications.
Key Responsibilities

• Serve as the final escalation and technical authority for complex, cross-domain incidents involving application, infrastructure, network, and third-party dependencies.
• Own the end-to-end observability architecture and strategy in Datadog, including instrumentation standards, tagging strategy, dashboard governance, and monitor design principles.
• Lead root cause analysis for major/critical incidents and drive post-incident reviews, ensuring permanent fixes and preventive monitoring are implemented.
• Define and continuously improve monitoring coverage for new and existing services, including 3-tier applications, microservices, and cloud-native workloads.
• Design and govern SLOs/SLIs aligned to business-critical services, and advise application and platform teams on instrumentation best practices.
• Drive automation initiatives: Datadog-as-code (Terraform/API-based configuration management), auto-remediation workflows, and integration with CI/CD pipelines.
• Partner with application, infrastructure, network, and security teams to close monitoring gaps and reduce mean time to detect (MTTD) and mean time to resolve (MTTR).
• Own capacity planning and performance engineering insights derived from long-term observability data trends.
• Manage Datadog platform health: cost optimization, license/usage governance, data retention, and integration management.
• Mentor L1 and L2 engineers, review their RCA quality, and build enablement material (runbooks, training, best practices).
• Act as a key stakeholder in client governance calls, presenting trends, risks, and improvement roadmaps.
• Evaluate and pilot new Datadog features or complementary observability tools, and lead their rollout.
• Define escalation matrices and coordinate cross-team response during major outages.

Required Skills and Experience

• 6+ years of experience in Managed Services, SRE, or platform engineering roles, with at least 3 years of hands-on ownership of a Datadog (or equivalent) observability platform at scale.
• Expert-level knowledge of Datadog, including:
• Platform architecture: organization/team structuring, RBAC, tagging strategy, and multi-environment setups.
• APM: distributed tracing at scale, service dependency mapping, and instrumentation strategy across polyglot environments.
• Synthetics and RUM: designing comprehensive digital experience monitoring strategies tied to business KPIs.
• Log Management: pipeline architecture, log-based metrics, and cost/volume governance.
• SLOs, error budgets, and anomaly/forecast-based alerting at an advanced level.
• Deep understanding of infrastructure and cloud architecture: compute, containers, orchestration (Kubernetes), auto-scaling, and multi-cloud/hybrid environments.
• Deep understanding of networking: routing, load balancing, CDN, DNS, and complex connectivity troubleshooting across hybrid environments.
• Strong incident and problem management leadership experience, including major incident command and post-incident review facilitation.
• Experience defining and governing observability standards across multiple teams or business units.

Application & Architecture Understanding (technology-agnostic)

• Expert-level understanding of 3-tier and N-tier application architectures, including how failures cascade across presentation, application, and data layers.
• Expert-level understanding of cloud-native and distributed architectures: microservices, event-driven systems, service meshes, API gateways, and asynchronous messaging.
• Ability to advise application teams on observability best practices (instrumentation, tracing context propagation, structured logging) without needing to write code in a specific language.
• Strong grasp of resiliency and failure patterns (retries, circuit breakers, backpressure, caching, graceful degradation) and how to detect their failure modes through observability data.

Good to Have

• Datadog advanced/expert certifications, or equivalent recognized observability certifications.
• Strong scripting/automation background (Python, Terraform, or similar) for Datadog-as-code and custom tooling.
• Experience with FinOps or cost governance for observability tooling at scale.
• Exposure to multiple observability ecosystems (New Relic, Dynatrace, Splunk, Grafana/Prometheus, OpenTelemetry).
• Experience presenting to senior stakeholders or clients on observability maturity and roadmaps.

Soft Skills

• Strategic thinking with the ability to balance immediate incident needs against long-term platform health.
• Strong leadership and mentoring capability across L1/L2 engineers.
• Excellent stakeholder management and executive communication skills.
• Calm, decisive leadership during major incidents.
• Strong documentation and process-governance discipline.

Education

• Bachelor's or Master's degree in Computer Science, Information Technology, Electronics, or a related field (or equivalent practical experience).

Work Details

• Level: L3 (Senior/Lead, Managed Services)
• Shift: Primarily general shift, with on-call responsibility for major incident escalations
• Reporting To: Observability Practice Lead / Delivery Manager

Page 1