Support Lead -Data Production & Platform Operations

About the role

India delivery centre Day-peak ET shift + on-call Own the production side of the Data Ops Platform cockpit. Your stream answers did the workflow run and deliver? you lead L1/L2 operations over the Workflow Monitor and System Health console for standard Databricks pipelines that load vendor market data, run curve-calibration DAGs and publish to trading and risk systems. You own the operational side of Gate 1 (the config-based technical QC engine running every ~10 minutes across 50+ catalog tables), manage production incidents end-to-end, and lead the junior production engineers. You are the senior on-seat presence on the main trading-day shift and the escalation point on all others.

Key responsibilities

• Lead 246/247 L1/L2 operations for standard Databricks pipelines: live workflow status (queued / running / complete / failed), progress and timing across 50+ workflows, and the EoD batch window (~17:42 ET).

• Operate and evolve Gate 1 technical QC: feed arrival and latency, data freshness, row counts vs rolling averages, schema conformance and range checks including the planned move from warn-only alerts to enforced pipeline gating.

• Own incident & problem management for production failures (each linked to its workflow ID and affected entities): triage, drive verify/fix/escalate per SOP, route vendor-side feed issues to vendor management, run root-cause analysis and post-incident reviews, improve MTTR.

• Track System Health for every component the platform depends on Databricks clusters and jobs, Kafka, orchestration engines, databases and raise degradations before they become breaks.

• Own SOPs, runbooks and the knowledge base; run the standard-pipeline onboarding checklist as new use cases join the platform (the framework generalises beyond Phase 1).

• Build, coach and performance-manage the junior production team (3–5 engineers); run shift handoffs and quality reviews; mature the function toward 2027 scorecards, full intra-day L1 and continuous improvement.

• Optimise Databricks workloads: Spark job tuning, cluster sizing and job-cost management, DAG dependency health; champion automation and AI-assisted triage to cut manual toil.

Required skills & experience

• Strong hands-on experience operating production data platforms — Databricks/Spark strongly preferred (cluster management, job optimisation, Delta Lake, workflow orchestration).

• Proven L1/L2 operations leadership: incident/problem/change discipline (ITIL or equivalent), ticketing and observability tooling, on-call rota design; a record of improving operational metrics.

• Solid SQL and Python for diagnostics and automation; comfort reading pipeline code, configs and execution plans.

• Cloud exposure (Azure or AWS); understanding of data security, governance and auditability in a regulated financial environment; clear, proactive stakeholder communication during incidents.

Preferred

• Asset-management / hedge-fund / capital-markets data operations (market-data feeds, EoD batch cycles, vendor management); Kafka; experience leading teams in a GCC or managed-services setting.

Profile

B.E./B.Tech/MCA or equivalent. Calm, decisive, accountable — able to make a defensible deliver-or-hold call and communicate it to demanding front-office stakeholders