Senior Data Engineer

A Data Engineer for Enterprise Data and Analytics role focuses on unifying complex, siloed data from internal ERPs (like SAP) and Product Life Cycle Manager systems like Laboratory Information Systems with external retail signals (POS data, shipment tracking) to optimise every stage from production to the shelf.

Responsibilities:

• Pipeline Development: Design and maintain scalable ETL/ELT pipelines using Azure Databricks to ingest data into Azure Data Lake Storage (ADLS).
• Data Transformation: Utilise PySpark and Spark SQL to clean, normalise, and aggregate raw data into "analytics-ready" formats (Medallion Architecture: Bronze, Silver, Gold layers).
• Orchestration and Monitoring: Configure and monitor data workflows using Azure Data Factory or Databricks Workflows (Lakeflow Jobs) to ensure daily retail reports are delivered accurately and on time.
• Governance and Security: Implement data access controls and cataloguing using Unity Catalogue to ensure secure and compliant data handling across the enterprise.
• Quality Assurance: Develop automated data quality checks (e. g., using Delta Live Tables) to prevent inaccurate metrics reporting.

Requirements:

• Education: Bachelor's degree in Computer Science, Data Science, or a related engineering field.
• Programming: 2 - 4 years of work experience with proficiency in Python (PySpark) and SQL is essential for building distributed data processing logic.
• Databricks Ecosystem: Job clusters, Unity Catalogue, Databricks Notebooks.
• Familiarity with Delta Lake (ACID transactions, time travel) and basic knowledge of Spark core architecture (drivers, executors, tasks).
• Azure Proficiency: Exposure to core Azure services, specifically ADLS Gen2 Azure Data Factory (ADF) and Key Vault.
• Version Control: Experience using Git for collaborative code development and CI/CD processes.
• Performance Tuning: Basic ability to identify bottlenecks in Spark jobs, such as optimising joins for large-scale transaction tables.
• Agile Development: Experience working in a collaborative agile environment.
• Cloud Architecture: Understanding of cloud architecture principles and best practices.
• Experience in the design and build of end-to-end solutions that meet business requirements and adhere to scalability, reliability, and security standards.