Senior Data Engineer

We are looking for an experienced Data Engineer with strong hands-on expertise in Databricks and PySpark to design, develop, and optimize scalable data pipelines. The role involves processing large datasets, implementing reliable ETL/ELT workflows, and delivering high-quality data for business analytics and reporting.

Responsibilities:

• Design, develop, and maintain data pipelines using Databricks, PySpark, and Spark SQL.
• Build reusable transformation logic for structured and semi-structured data from multiple sources.
• Develop and maintain Databricks notebooks, jobs, and scheduled workflows.
• Implement full and incremental data loads, including data cleansing, deduplication, validation, and reconciliation.
• Build and maintain Delta Lake tables and curated datasets using Lakehouse and Medallion architecture principles.
• Optimize Spark workloads through appropriate partitioning, join strategies, caching, and management of data skew.
• Troubleshoot pipeline failures, performance issues, and data-quality defects.
• Implement logging, error handling, monitoring, and recovery mechanisms.
• Collaborate with business analysts, architects, and downstream teams to translate requirements into technical solutions.
• Participate in code reviews, testing, documentation, and deployment activities.

Requirements:

• Strong hands-on experience developing production data-engineering solutions using Databricks and PySpark.
• Proficiency in Python, SQL, and Spark SQL.
• Experience with PySpark DataFrames, joins, aggregations, window functions, and complex transformations.
• Understanding of Apache Spark execution, distributed processing, and performance tuning.
• Experience building ETL/ELT pipelines and implementing incremental processing.
• Working knowledge of Delta Lake, data warehousing, and dimensional modeling.
• Ability to troubleshoot production workloads and independently own assigned deliverables.

Preferred Skills:

• Experience with Databricks on Azure, AWS, or GCP.
• Exposure to structured streaming, Auto Loader, and change data capture.
• Familiarity with Unity Catalog, access controls, and data governance.
• Experience with Git, CI/CD, and automated testing.
• Exposure to orchestration tools such as Azure Data Factory or Apache Airflow.
• Relevant Databricks certification.