Senior Data Engineer

Data Engineer – Azure Databricks

Experience: 6+ Years

Location: Bengaluru- Whitefield

Work Mode: Hybrid – 3 days/week

Interview Process: 2 Technical Round

Job Summary We are looking for an experienced

Data Engineer with 6+ years of hands-on experience in building and managing scalable data engineering solutions. The ideal candidate should have strong expertise in

Spark, Scala, Airflow, Hive, Kubernetes, Iceberg, and Azure Databricks . The candidate should also have a good understanding of modern data platforms, cloud technologies, CI/CD practices, and the ability to leverage

AI across the Software Development Lifecycle (SDLC) and Data Engineering to drive innovation and business value.

Key Responsibilities

• Design, develop, and maintain scalable and reliable data engineering pipelines.
• Build data processing solutions using

Apache Spark and Scala .

• Develop and manage workflows using

Apache Airflow .

• Work with

Hive, Iceberg, Kubernetes, and Docker in modern data platforms.

• Develop and maintain CI/CD pipelines using

GitHub Actions .

• Design and implement data solutions using

Azure Databricks .

• Work with

ADLS, Azure Data Factory (ADF), Delta Lake, Databricks Jobs and Pipelines .

• Utilize

Lakeflow Connect, Unity Catalog, and Databricks Compute (Classic & Serverless) .

• Implement data governance, security, and access controls using Unity Catalog.
• Optimize data pipelines and processing workloads for performance and scalability.
• Collaborate with data scientists, architects, product teams, and other engineering teams.
• Apply AI capabilities and tools across the SDLC and Data Engineering lifecycle to improve productivity, automation, and business outcomes.
• Follow engineering best practices around version control, testing, deployment, monitoring, and documentation.

Required Skills Must-Have – Priority 1

• Strong hands-on experience with

Apache Spark

• Scala
• Apache Airflow
• Hive
• Kubernetes
• Apache Iceberg
• Good knowledge of

Docker

• Good experience with

GitHub Actions / CI-CD Must-Have – Priority 2

• Strong experience with

Azure Databricks

• ADLS
• Databricks Jobs & Pipelines
• Azure Data Factory (ADF)
• Delta Lake
• Lakeflow Connect
• Unity Catalog
• Databricks Compute –

Classic & Serverless Must-Have – Priority 3

• Experience using

AI tools/capabilities within SDLC and Data Engineering

• Ability to identify opportunities to use AI for automation, development, optimization, and innovation
• Strong problem-solving and analytical skills

Additional Requirements

• 6+ years of relevant experience in Data Engineering.
• Strong understanding of data engineering concepts and distributed data processing.
• Excellent communication and collaboration skills.
• Candidates must be based in or willing to relocate to

Bengaluru .

• Candidates should be comfortable working from the office
• Willingness to attend

2 technical interview rounds .

Preferred Candidate Profile Candidates with strong hands-on experience across

Spark/Scala/Airflow and the

Azure Databricks ecosystem will be given priority. Experience in modern data lakehouse technologies, containerization, CI/CD, and practical application of AI in engineering environments will be an added advantage.