Data Engineer

Job Summary

Data Engineer - Azure / AWS / Databricks / Microsoft Fabric. About the Role: We are looking for a Data Engineer with 2-3 years of experience to join our Data Analytics team and contribute to the development of modern, cloud-based data platforms and analytics solutions. The role involves designing and developing scalable data pipelines, data lake/lakehouse solutions, data transformations and curated data layers that support Business Intelligence, Advanced Analytics, AI/ML and Decision Intelligence use cases. The engineer will work closely with Data Architects, Business Analysts, BI Developers, Data Scientists and Project Managers to convert business requirements into reliable and production-ready data solutions.

Key Responsibilities

• Design, develop and maintain scalable ETL/ELT data pipelines.
• Build data ingestion and transformation pipelines across databases, APIs, enterprise applications and file-based sources.
• Develop data processing solutions using SQL, Python and PySpark/Spark.
• Work with cloud data platforms, Data Lakes and Lakehouse architectures.
• Develop and maintain Bronze, Silver and Gold/curated data layers.
• Implement data transformations, cleansing, validation, reconciliation and data quality checks.
• Work with Databricks, Microsoft Fabric and/or Azure data services.
• Develop efficient and reusable data engineering frameworks and components.
• Optimize SQL queries, Spark jobs and data pipelines for performance and cost.
• Implement appropriate error handling, logging, monitoring and pipeline recovery mechanisms.
• Support data modelling for analytical and reporting use cases.
• Collaborate with BI teams to provide trusted and analytics-ready datasets.
• Support Data Scientists and AI teams with appropriately prepared datasets and feature-ready data.
• Follow engineering practices around Git, version control, CI/CD, testing and deployment.
• Contribute to technical documentation, data lineage and operational runbooks.
• Troubleshoot production data pipeline and data quality issues.
• Participate in code reviews and follow development standards.
• Work in an Agile project environment and collaborate with distributed teams and client stakeholders.

Typical Technology Environment

• Microsoft Azure
• Microsoft Fabric
• Databricks
• Apache Spark / PySpark
• SQL
• Python
• Delta Lake
• Data Lakes / Lakehouses
• Azure Data Factory / Fabric Data Factory
• Power BI
• Git / Azure DevOps
• Relational and NoSQL databases
• APIs and enterprise data sources

The exact technology stack may vary by project.

Requirements

Required

• Bachelors degree in Computer Science, Information Technology, Engineering, Data Engineering or a related technical discipline.
• 2-3 years of professional experience in Data Engineering.
• Strong hands-on experience with SQL.
• Good programming experience in Python.
• Experience developing and supporting ETL/ELT pipelines.
• Understanding of Data Lake, Lakehouse or modern data platform architectures.
• Experience working with structured and semi-structured data.
• Good understanding of data transformation, cleansing, validation and data quality concepts.
• Experience working with relational databases.
• Understanding of dimensional modelling and analytical data structures.
• Good understanding of data engineering best practices including logging, error handling, testing and monitoring.
• Experience working in an Agile development environment.
• Ability to understand business requirements and translate them into technical data solutions.

Preferred

• Hands-on experience with Microsoft Azure.
• Experience with Databricks.
• Experience with Apache Spark / PySpark.
• Experience with Microsoft Fabric.
• Experience with Azure Data Factory or Fabric Data Factory.
• Experience with Delta Lake / Delta Tables.
• Exposure to Apache Iceberg or other open table formats.
• Experience with Git and CI/CD.
• Experience with Power BI or other BI platforms.
• Experience integrating data from enterprise applications such as SAP, Salesforce, ERP, CRM, LIMS or similar systems.
• Exposure to REST APIs and API-based data ingestion.
• Exposure to streaming/event-driven data pipelines.
• Understanding of data governance, security, access control and PII handling.
• Exposure to AI/ML data pipelines or analytics platforms.

Education

Bachelors degree in Computer Science, Information Technology, Engineering or a related technical field.

Experience

2-3 years of relevant professional Data Engineering experience.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.