Lead Systems Operations Engineer

Wells Fargo is seeking a Lead Systems Operations Engineer.

• Lead complex, broad impact initiatives including provision of high level systems consultation for the technology teams
• Work as key participant in large scale planning of computer systems and network infrastructure for Systems Operations functional area
• Review and analyze complex technical challenges, as well as escalated support issues related to core business solutions that require in depth evaluation of multiple factors, such as alternatives, enhancements, periodic systems reviews, or improvements to existing systems
• Make decisions on technical changes and enhancements
• Consult with engineering team on change design requiring solid understanding of technical process controls or standards that influence and drive new initiatives
• Collaborate and consult with technical peers, colleagues, and mid to more experienced level managers to resolve systems support issues and achieve goals

• 5+ years of Systems Engineering, Technology Architecture experience, or equivalent demonstrated through one or a combination of the following: work experience, training, military experience, education

• Experience in Software Engineering, SRE, DevOps, or Platform Engineering.
• Strong proficiency in Python for automation and tooling.
• Hands-on experience with Grafana, Prometheus, and Splunk in production environments.
• Solid understanding of SLIs, SLOs, dashboards, alerting, and observability best practices.
• Experience applying AI/ML concepts to monitoring, alerting, or operational analytics.
• Strong knowledge of Linux, networking, and distributed systems.
• Experience with Cloud platforms and Kubernetes/OpenShift.
• Proven experience leading incidents, RCAs, and reliability initiatives
• Experience building custom Prometheus exporters or advanced Grafana dashboards.
• Strong Splunk expertise (search, dashboards, alerts, log pipelines).
• Experience operationalizing ML models for observability (AIOps).
• Familiarity with CI/CD, Terraform, Ansible, and enterprise automation platforms.
• Experience supporting large-scale, regulated, or globally distributed systems.
• Improved reliability and performance against defined SLOs.
• Reduced alert noise and faster detection and recovery of incidents.
• Increased automation and self-healing adoption using Python and observability signals.
• Strong observability maturity across platforms and applications.
• Improved MTTD and MTTR through effective use of Grafana, Prometheus, and Splunk.

• Own and improve availability, performance, scalability, and resilience of production systems.
• Define, monitor, and manage SLIs/SLOs and error budgets to guide reliability investments.
• Lead capacity planning, performance testing, failover readiness, and disaster-recovery design.

• Design and operate a comprehensive observability stack using:
  • Prometheus for metrics collection and alerting
  • Grafana for dashboards, visualization, and SLO tracking
  • Splunk for log aggregation, troubleshooting, and incident forensics
• Build and maintain golden dashboards and actionable alerts aligned to business impact.
• Reduce alert fatigue through signal-based monitoring and correlation of metrics, logs, and traces.
• Partner with application teams to define instrumentation standards for metrics and logging.
• Use observability data to improve MTTD, MTTR, and reliability outcomes.

• Develop Python-based automation for monitoring, alert remediation, deployments, scaling, and recovery.
• Build self-healing workflows integrated with Prometheus alerts and Splunk signals.
• Create reusable automation frameworks and internal SRE tooling.
• Embed automation into CI/CD pipelines to improve deployment safety and reliability.

• Apply AI/ML techniques to observability and operations use cases, including:
  • Anomaly detection on Prometheus metrics
  • Log pattern analysis and correlation in Splunk
  • Predictive capacity and trend forecasting
  • Noise reduction and intelligent alerting
• Partner with data and platform teams to operationalize ML models in production.
• Evaluate and integrate AIOps capabilities into the observability ecosystem.

• Serve as incident commander and senior escalation point for P1/P2 incidents.
• Lead blameless post-incident reviews (PIRs) backed by Grafana metrics and Splunk evidence.
• Drive corrective and preventive actions to completion.

• Collaborate with platform, application, cloud, and SRE teams to embed reliability and observability by design.
• Influence architectural decisions to ensure systems are observable, scalable, and operable.
• Provide SRE guidance during major releases, migrations, and modernization initiatives.

• Ensure observability and automation comply with enterprise security and audit requirements.
• Support resilience validation, failover drills, and business continuity testing.

• Mentor and guide SRE and software engineers.
• Define standards for observability, automation, reliability, and incident response.
• Act as the technical authority for complex production and platform issues.