IN_Senior Associate_Data Engineer Databricks_Data and Analytics_Advisory_Hyderabad

Job Description & Summary:

We are seeking an experienced Data Engineer - Databricks to design, build, and operate scalable, high-performance data pipelines on the Databricks Lakehouse Platform. The role involves hands-on development using Apache Spark, Databricks notebooks, Delta Lake, and cloud-native services, along with close collaboration with analytics, AI/ML, and business teams

Responsibilities:

• Design, build, and maintain end-to-end data pipelines using Databricks (PySpark/ Spark SQL)

. Implement batch and incremental data processing using Delta Lake and multi-hop architecture

. Develop and optimize Databricks notebooks, jobs, and workflows

. Ingest, transform, and curate large-scale structured and semi-structured datasets

. Support analytics, reporting, and downstream data consumption use cases

. Ensure data quality, reliability, lineage, and governance

. Collaborate with data scientists, analysts, and architects on AI/ML workloads

. Optimize Spark jobs for performance and cost efficiency

. Adhere to enterprise security, access control, and compliance standards

. Provide production support and troubleshoot data pipeline issues

. Document technical designs, data flows, and operational runbooks

. Mentor junior engineers and contribute to best practices

Mandatory skill sets:

4+ years of experience as a Data Engineer with strong Databricks expertise

. Hands-on experience with Apache Spark,PySpark, and Spark SQL

. Strong knowledge of Delta Lake and Lakehouse architecture

. Advanced SQL skills

. Experience with ETL/ELT patterns and data warehousing concepts

. Exposure to at least one cloud platform (Azure / AWS / GCP)

. Understanding of distributed computing concepts

. Experience working in Agile teams

Preferred skill sets:

Experience with Unity Catalog and data governance

. Exposure to Auto Loader or streaming frameworks

. CI/CD for data pipelines

. Python for data engineering and automation

. Databricks certification (Associate / Professional)

Years of experience required:

5to8years

Education qualification:

Bachelor's orMaster's degree in Computer Science, Engineering, or related field(60% above)