Data Engineer and Senior Data Engineer

Key Responsibilities

• Design and develop scalable data pipelines using Databricks and PySpark.
• Build and optimize ETL/ELT workflows for large-volume data processing.
• Develop data transformations using PySpark and SQL.
• Work extensively with Delta Lake for reliable and scalable data storage.
• Develop and maintain data pipelines using Databricks Workflows/Lakeflow.
• Perform data integration from multiple structured and unstructured sources.
• Optimize Spark jobs, queries, and pipeline performance.
• Implement data quality, validation, monitoring, and error-handling mechanisms.
• Collaborate with Data Architects, BI teams, and business stakeholders.
• Follow best practices for CI/CD, version control, and deployment.

Required Skills

• Strong hands-on experience with Databricks
• Strong PySpark and SQL skills
• Good understanding of Delta Lake
• Strong experience in Data Engineering and ETL
• Understanding of Data Warehousing and Data Modelling
• Experience with cloud platforms such as Azure / AWS / GCP
• Knowledge of Git, CI/CD and DevOps
• Good analytical and problem-solving skills
• Strong communication and stakeholder management skills

Good to Have

• Experience with Unity Catalog
• Databricks certifications
• Experience with Kafka / Azure Data Factory / Event Hubs
• Knowledge of Lakehouse architecture
• Experience in migrating legacy data platforms to Databricks