Azure Databricks Specialist

Area(s) of responsibility

Skills: Azure Databricks Engineer Experience: 4–7 Years

Location: Pune, Mumbai, Noida, Chennai, Bangalore

Role Summary We are looking for a highly skilled

Databricks Data Engineer with strong hands‑on expertise in

Databricks, PySpark, ADF, and SQL . The candidate will be responsible for architecting, developing, and optimizing modern data platforms and analytical solutions on Azure. The role requires strong experience building enterprise-grade data pipelines, enabling ingestion from diverse sources, implementing complex transformations, and supporting scalable analytics initiatives.

Key Responsibilities

• Data Engineering Execution
• Build and maintain ingestion frameworks (ADF / Databricks / Spark) Implement Bronze -> Silver transformations aligned to architecture Architect data quality checks, schema validation, and contract rules
• Build and optimize robust, high-throughput ELT/ETL pipelines, enabling ingestion, transformation, and curation of structured, semi‑structured, and unstructured data.
• Integrate data from multiple on‑premise and cloud‑based systems, APIs, and third-party sources.
• Implement complex transformations using PySpark, ensuring performance efficiency and code modularity.
• Build orchestration workflows in ADF, including pipelines, triggers, linked services, integration runtimes, and parameterized datasets.
• Familiar with using Databicks Genie.
• Databricks & PySpark Engineering
• Develop scalable transformation scripts using PySpark on Databricks, applying advanced optimizations like caching, partitioning, and Delta Lake capabilities.
• Implement Delta Lake features—ACID transactions, schema enforcement, schema evolution, and time travel—across the data lifecycle.
• Perform performance tuning, handling bottlenecks related to cluster configuration, shuffle operations, joins, and parallelization.
• Collaborate with platform teams to manage Databricks clusters, jobs, notebooks, and CI/CD integrations.
• Collaboration & Stakeholder Management
• Collaborate closely with customer IT and business teams to understand data requirements and deliver reliable, production-ready solutions.
• Work with architects, product owners, and cross-functional engineering teams to align technical delivery with business objectives.
• Provide guidance and mentoring to junior engineers when required.

Mandatory Skills & Experience

• Minimum 4+ years of experience in data engineering, with strong hands‑on exposure to Azure data ecosystem.
• At least 3 years of real project experience in Databricks (Azure Data Platform (ADF, ADLS, Azure SQL) & Databricks (DLT, Delta Lake, Spark, Workflows, Unity Catalog).
• At least 2 years of hands-on experience building data pipelines using Azure Data Factory (ADF).
• At least 2 years of experience developing PySpark-based transformations in Databricks.
• Strong SQL programming experience, including writing complex queries, performance tuning, and handling large datasets. Knowledge of CI/CD pipelines (Azure DevOps preferred) for automated deployment of ADF/Databricks artifacts