Data Engineer

2.1 Data Platform Architecture & Engineering Leadership
•
Technically manage the end-to-end Azure Data Warehouse, Databricks, and Power BI solutions, ensuring high availability, scalability, and performance.
•
Design and own enterprise data models, schemas, and database structures (dimensional, normalized, Data Vault 2.0) to support analytical and operational use cases.
•
Lead the end-to-end implementation and continuous evolution of the Data Lake / Lakehouse architecture using the Medallion model (Bronze, Silver, Gold layers).
•
Define and enforce architectural standards, design patterns, and reference architectures for data ingestion, transformation, storage, and consumption.
•
Evaluate and recommend data storage solutions including relational databases, NoSQL databases, data lakes, lakehouse, and cloud storage services.
•
Identify alternate components within the existing Data Lake Architecture to improve performance and reduce cost through POCs and feasibility studies.
Job Description | Data Lake Architect
Confidential – Internal Use Only Page 2 of 5
2.2 Data Engineering, Pipelines & SAP Integration
•
Design, build, and optimize end-to-end ETL/ELT data pipelines for ingesting, processing, and transforming large volumes of structured, semi-structured, and unstructured data within the Medallion and Lakehouse architecture.
•
Engineer SAP data integration pipelines (ECC / S/4HANA / BW) leveraging tools such as SAP CDS Views, SLT, OData, BAPIs, and third-party replication tools.
•
Implement data validation, quality checks, reconciliation, and observability frameworks (e.g., Great Expectations, Delta Live Tables expectations) to ensure accuracy and consistency.
•
Optimize pipelines and Spark/Databricks workloads (cluster sizing, partitioning, Z-ordering, caching, photon engine) to reduce Azure compute / storage cost and improve latency.
•
Design and maintain CI/CD pipelines using Azure DevOps for automated code migration, testing, and deployment across Dev UAT Prod environments.
•
Manage routine incident handling, root-cause analysis, and service requests within the defined SLAs.
2.3 Analytics, BI & AI/ML Enablement
•
Define the semantic layer and enterprise data models in Power BI / Microsoft Fabric to enable self-service analytics for business users.
•
Govern Power BI workspaces, datasets, dataflows, row-level security (RLS), capacity (Fabric F-SKU), and DAX optimization.
•
Partner with Data Scientists and AI engineers to enable advanced analytics, Machine Learning, and Generative AI use cases on the platform (Azure ML, Databricks ML, Azure OpenAI, Mosaic AI).
•
Establish feature engineering pipelines, feature stores, and MLOps practices (MLflow, Azure ML Pipelines) for productionizing models.
•
Drive adoption of Generative AI / RAG solutions on enterprise data, including vector store design, prompt engineering oversight, and responsible AI guardrails.
2.4 Vendor & Developer Coordination
•
Act as the technical owner and single point of contact for external vendors, implementation partners, and internal development teams working on the data platform.
•
Review vendor deliverables, technical designs, code, and solution proposals to ensure adherence to architectural standards and best practices.
•
Define and track development milestones, conduct technical reviews, code reviews, and ensure timely delivery of vendor-managed work streams.
•
Drive estimation, planning, and prioritization of data initiatives in collaboration with business and IT leadership.
2.5 Cloud Cost Optimization & Budget Management
•
Prepare, present, and track the annual Azure cloud infrastructure budget for data platform components, including compute, storage, networking, and licensing.
•
Continuously monitor Azure consumption using tools such as Azure Cost Management, Azure Advisor, and custom dashboards; identify and act on cost-saving opportunities.
•
Implement FinOps best practices including right-sizing of clusters, reserved/savings plans, spot instances, auto-pause/auto-scaling policies, storage tiering, and lifecycle management.
•
Publish monthly cost reports and variance analysis to leadership with actionable recommendations.
•
Establish chargeback / showback mechanisms for business units consuming data platform services.
2.6 Data Security, Governance & Compliance
Job Description | Data Lake Architect
Confidential – Internal Use Only Page 3 of 5
•
Drive continuous enhancement of data platform security, including identity & access management (Entra ID), network isolation (VNet, Private Endpoints), encryption (at-rest and in-transit), Key Vault, and secure secrets handling.
•
Implement and enforce data governance policies covering data classification, lineage, cataloging, masking, and retention using Microsoft Purview and Databricks Unity Catalog.
•
Ensure compliance with applicable regulatory, audit, and internal information security requirements.
•
Conduct periodic security reviews, vulnerability assessments, and remediation tracking for the data estate.
2.7 Data Integration & API Development
•
Build and maintain integrations with internal and external data sources and APIs, including SAP, MES, IoT, CRM, and SaaS applications.
•
Design and implement RESTful APIs, web services, and event-driven interfaces for data access and consumption by downstream applications.
•
Ensure compatibility and interoperability between different systems and platforms.
2.8 Collaboration, Documentation & Stakeholder Management
•
Collaborate with business owners, analysts, data scientists, and other stakeholders to understand data requirements and deliver tailored solutions.
•
Prepare Business Requirement Documents (BRDs), High-Level / Low-Level Designs (HLD/LLD), data flow diagrams, and operational runbooks for all new ETL and data platform initiatives.
•
Document technical designs, workflows, and best practices to facilitate knowledge sharing and maintain a current system documentation repository.
•
Provide technical guidance and mentoring to team members and stakeholders as needed.
•
Communicate effectively with business users, the analytics team, data scientists, architects, and other internal and external stakeholders.
3. Mandatory Requirements
3.1 Core Technical Skills
•
Proven hands-on experience in data engineering, data architecture, or related roles on the Microsoft Azure stack.
•
Strong proficiency in Python, PySpark, and Spark-SQL; working knowledge of Scala is a plus.
•
Comprehensive expertise in database systems, data modelling methodologies (dimensional, normalized, Data Vault 2.0), and advanced SQL scripting & performance tuning.
•
Hands-on expertise with Azure Data Factory, Azure Databricks, Azure Data Lake Storage (ADLS Gen2), Delta Lake, Azure Synapse / Microsoft Fabric, and Databricks SQL Warehouse.
•
Strong working knowledge of Power BI semantic models, DAX, dataset optimization, RLS, dataflows, and workspace / capacity governance.
•
Sound understanding of SQL and NoSQL databases — preferably SAP HANA, MySQL, MS SQL Server, Azure Cosmos DB.
•
In-depth understanding of Azure DevOps for building CI/CD pipelines, branching strategies, and infrastructure-as-code deployments.
•
In-depth expertise across the Azure cloud platform, including networking, security, monitoring, and cost management services.
3.2 Functional & Behavioural Skills
•
Strong problem-solving skills with attention to detail.
Job Description | Data Lake Architect
Confidential – Internal Use Only Page 4 of 5
•
Excellent communication and stakeholder management skills.
•
Ability to manage multiple priorities, vendors, and project streams concurrently.
•
Ability to adapt to evolving technologies and changing business requirements.