Observability (O11y) Engineer (Enterprise Observability Platform)

Key Responsibilities:

Observability Architecture & Implementation

• Design and implement observability solutions using Splunk Observability Cloud (APM, Infrastructure Monitoring, RUM, Log Observer).
• Build and optimize OpenTelemetry (OTel) collector pipelines for metrics, traces, and logs.
• Define instrumentation strategies for applications (auto-instrumentation, SDKs, agents).
• Develop service maps, dependency graphs, and distributed tracing configurations.

Application & Service Onboarding

• Partner with application teams to onboard services into O11y and ITSI.
• Define SLIs/SLOs, KPIs, and service health indicators.
• Build dashboards, detectors, alerts, and anomaly detection rules tailored to each service.
• Ensure consistent tagging, metadata, and observability standards across teams.

ITSI & Splunk Core Integration

• Implement ITSI service models, KPIs, correlation searches, and glass tables.
• Integrate O11y telemetry with Splunk Enterprise and ITSI for unified service health.
• Collaborate with Splunk Core SMEs to align ingestion, retention, and data modeling.

Telemetry Pipeline Engineering

• Deploy and configure OTel collectors across VMs, containers, Kubernetes, and cloud services.
• Implement processors, exporters, and routing logic for efficient telemetry flow.
• Optimize pipelines for performance, cost, and reliability.

Enablement & Best Practices

• Develop onboarding guides, instrumentation patterns, and observability standards.
• Conduct workshops, demos, and training sessions for engineering teams.
• Provide architectural guidance on observability adoption across the enterprise.

Cross-Functional Collaboration

• Work with SRE, DevOps, Cloud, and Application teams to embed observability into CI/CD and deployment workflows.
• Support modernization initiatives such as microservices, Kubernetes, and cloud migrations.
• Influence platform roadmap and observability strategy.

Required Skills & Experience:

Technical Skills

• Hands-on experience implementing Splunk Observability Cloud (APM, Infra Monitoring, RUM, Log Observer).
• Strong understanding of OpenTelemetry (collectors, processors, exporters, instrumentation).
• Experience with distributed tracing, span analysis, and service dependency mapping.
• Familiarity with Splunk Enterprise and ITSI for integrated observability use cases.
• Strong Linux/Unix skills and scripting (Python, Bash).
• Experience with Kubernetes, Docker, and cloud platforms (AWS, and Azure).

Implementation & Architecture Skills

• Experience designing observability solutions for microservices, APIs, and distributed systems.
• Ability to guide teams through instrumentation and telemetry onboarding.
• Strong understanding of SLIs/SLOs, service health scoring, and reliability engineering.

Soft Skills

• Excellent communication and stakeholder-facing skills.
• Ability to lead workshops and influence engineering teams.
• Strong documentation and architectural thinking.

Preferred Qualifications

• Splunk Observability or OTel certifications.
• Experience with Prometheus, Grafana, Datadog, New Relic, or similar tools.
• Familiarity with Terraform, Ansible, GitOps, or CI/CD pipelines.
• Experience in large enterprise, utilities, financial, or public sector environments.