Associate Director - Data Engineering

Role Overview:

We are seeking an experienced data engineering leader to drive multiple tracks and complex data initiatives. As the Associate Director of Data Engineering, you will be responsible for driving strategic alignment, cross-functional collaboration, and overseeing the delivery of highly scalable data platforms and products. You will lead a dedicated team of data engineers, ensuring technical excellence, operational efficiency, and the successful execution of our data roadmap. This role requires up to 50% of hands-on contribution.

Key Responsibilities:

- Team Leadership: Manage and lead a high-performing team of data engineers, coordinating cross-functional dependencies and fostering a culture of continuous learning and technical excellence.

- Strategic Execution: Drive and execute the technical roadmap for key data infrastructure, pipelines, and products, balancing short-term delivery with long-term strategic goals.

- Technical Excellence: Drive operational efficiency across teams by enforcing engineering standards and SDLC best practices. Own the overall performance, reliability, observability, and accuracy of data pipelines. Continuously optimize highly scalable, fault-tolerant data pipelines.

- Mentorship & Collaboration: Mentor senior engineers and act as the first point of contact for external stakeholders. Identify and resolve technical bottlenecks and resource constraints. Help to recruit and onboard data engineering talents to scale the team effectively.

- Troubleshooting & Incident Response: Proactively identify, diagnose, and resolve critical data pipeline and platform issues, minimizing downtime and ensuring system stability.

Functional & Technical Requirements:

- Experience: 10 - 16 years of proven experience in data engineering, with a strong track record of managing complex technical programs.

- Architecture & System Design: Deep understanding of scalable data systems with strong system design and architecture review skills.

- Cloud & Data Ecosystems: Expert-level proficiency in data architecture, the Databricks ecosystem (Delta Lake, Unity Catalog, etc.), and the AWS data ecosystem.

- Databricks & Spark Expertise: Deep, hands-on expertise in Apache Spark, PySpark, and the Databricks ecosystem. Must have advanced knowledge of Delta Lake, Open Table Formats, Unity Catalog, and Databricks Asset Bundles (DAB).

- Big Data & Architecture: Expert understanding of scalable data systems, Medallion Architecture, and distributed data pipelining. Strong system design and architecture review skills.

- AWS Data Ecosystem: High proficiency in AWS cloud ingestion and storage (S3), IAM, compute optimizations, and Infrastructure as Code (Terraform).

- Scaling & Optimization: Proven experience in scaling data workloads, Data Ops, compute optimizations (Docker, Kubernetes, Databricks cluster policies), and performance tuning (e.g., query optimization, Z-ordering/Liquid clustering).

- AIML & Innovation: Familiarity with AI tools, MLOps, and techniques to enhance data engineering practice at scale, improve efficiency and quality of the pipelines.

- Data Domain Expertise: Prior knowledge of Finserv (especially Credit, Lending, or similar lines of businesses) and familiarity with regulatory data compliance is highly desirable.

Experience Range:

10 - 16 years

Educational Qualifications:

B.Tech/B.E, and M.Tech

Skills Required:

Data brick, Delta Lake, Apache Spark, Pyspark, Big Data, ETL, ELT, Performance Tuning, Data Pipelines, Data architecture, System Design, Docker, Kubernetes, AWS