Data Engineer

Job Summary:

We are seeking a highly motivated and experienced Azure Databricks Engineer with strong expertise in Azure Databricks and cloud-based data platforms. In this role, you will design, develop, and optimize scalable data pipelines, support the implementation of enterprise-grade data and analytics solutions, and leverage Databricks best practices to deliver reliable, high-quality data products. You will work closely with data engineers, architects, business stakeholders, and analytics teams to enable data-driven decision-making across the organization.

Key Responsibilities:

• Design, develop, and optimize data pipelines using Azure Databricks (Apache Spark).
• Build and support large-scale, cloud-native data processing solutions on Azure.
• Collaborate with data scientists, analysts, and business stakeholders to ensure data availability, reliability, and quality.
• Implement and support Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and MLflow solutions.
• Follow and promote best practices for CI/CD, version control, testing, and deployment in a Databricks environment.
• Work closely with project teams to deliver solutions aligned with business requirements and timelines.

Required Qualifications:

• Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.
• 5 + years of experience in data engineering.
• Hands-on expertise with Azure Databricks, including Spark SQL, Delta Lake, job orchestration, and performance tuning.
• Proficient in Python, SQL, and Apache Spark; knowledge of Scala is a plus.
• Experience working with Azure cloud services, including Databricks deployments.
• Strong understanding of data modeling, ETL/ELT, data warehousing, and modern data lake architectures.
• Advanced SQL skills for data analysis. Expertise on problem-solving skills with a focus on finding innovative solutions to complex data challenges. Ability to proactively identify and address potential issues and bottlenecks.
• Fluent in multiple coding languages, such as Python or Java and able to learn new ones quickly
• Technical skills in metadata capturing and management using Collibra to document, organize, and ensure the quality of metadata assets
• Strong Knowledge of cloud-based data, compute, and storage services, including AWS S3, EC2, RDS, EBS, Lambda, Airflow (MWAA), Containerization Services (ECS, EKS) and so on.
  

Preferred Qualifications:

• Databricks certification (e.g., Databricks Certified Data Engineer Professional).
• Experience with Unity Catalog, MLflow, and building data products on the Lakehouse.
• Experience with Azure services such as Azure Data Factory (ADF) and Azure Data Lake Storage (ADLS).