Associate Director / Director Data Engineer

Role Overview:

We are seeking an experienced data engineering leader to drive multiple tracks and complex data initiatives. As the Associate Director of Data Engineering, you will be responsible for driving strategic alignment, cross-functional collaboration, and overseeing the delivery of highly scalable data platforms and products. You will lead a dedicated team of data engineers, ensuring technical excellence, operational efficiency, and the successful execution of our data roadmap. This role requires up to 50% of hands-on contribution.

Key Responsibilities:

 Team Leadership: Manage and lead a high-performing team of data engineers, coordinating cross-functional dependencies and fostering a culture of continuous learning and technical excellence.

Strategic Execution: Drive and execute the technical roadmap for key data infrastructure, pipelines, and products, balancing short-term delivery with long-term strategic goals.

Technical Excellence: Drive operational efficiency across teams by enforcing engineering standards and SDLC best practices. Own the overall performance, reliability, observability, and accuracy of data pipelines. Continuously optimize highly scalable, fault-tolerant data pipelines.

Mentorship & Collaboration: Mentor senior engineers and act as the first point of contact for external stakeholders. Identify and resolve technical bottlenecks and resource constraints. Help to recruit and onboard data engineering talents to scale the teameffectively.

Troubleshooting & Incident Response: Proactively identify, diagnose, and resolve critical

data pipeline and platform issues, minimizing downtime and ensuring system stability.

Functional & Technical Requirements:

Experience: 10+ years of proven experience in data engineering, with a strong track

record of managing complex technical programs.

Architecture & System Design: Deep understanding of scalable data systems with strong

system design and architecture review skills.

Cloud & Data Ecosystems: Expert-level proficiency in data architecture, the Databricks

ecosystem (Delta Lake, Unity Catalog, etc.), and the AWS data ecosystem.

Databricks & Spark Expertise: Deep, hands-on expertise in Apache Spark, PySpark, and

the Databricks ecosystem. Must have advanced knowledge of Delta Lake, Open Table

Formats, Unity Catalog, and Databricks Asset Bundles (DAB).

Big Data & Architecture: Expert understanding of scalable data systems, Medallion

Architecture, and distributed data pipelining. Strong system design and architecture

review skills.

AWS Data Ecosystem: High proficiency in AWS cloud ingestion and storage (S3), IAM,

compute optimizations, and Infrastructure as Code (Terraform).

Scaling & Optimization: Proven experience in scaling data workloads, Data Ops, compute

optimizations (Docker, Kubernetes, Databricks cluster policies), and performance tuning

(e.g., query optimization, Z-ordering/Liquid clustering).

AIML & Innovation: Familiarity with AI tools, MLOps, and techniques to enhance data

engineering practice at scale, improve efficiency and quality of the pipelines.

Data Domain Expertise: Prior knowledge of Finserv (especially Credit, Lending, or similar

lines of businesses) and familiarity with regulatory data compliance is highly desirabl