Data Architect Engineer

Purpose/Objective

In this role, you will be instrumental in designing, developing, and optimizing data pipelines on the Microsoft Azure cloud platform, with a primary focus on Azure Databricks and Delta Lake.
You will work closely with cross-functional teams including Data Scientists, Analysts, and Software Engineers to deliver high-quality, reliable data solutions that drive business intelligence and machine learning initiatives.
The ideal candidate will play a key role in transforming raw enterprise data into trusted, analytics ready datasets that power reporting, advanced analytics, and AI/ML initiatives

Key Responsibilities of Role

- Data Engineering & Lakehouse Design- Design, implement, and optimize Azure Lakehouse architectures using Azure Data Lake Storage (ADLS Gen2) and Databricks (Delta Lake) - Implement medallion architecture (Bronze, Silver, Gold layers) for structured and unstructured data - Develop scalable batch and streaming data pipelines using Databricks - Databricks Development- Build and maintain Databricks notebooks using PySpark / Spark SQL - Optimize Spark jobs for performance and cost (partitioning, caching, Delta optimizations) - Implement Delta Lake features: ACID transactions, schema evolution, time travel, and data versioning - Data Integration & Processing Develop, test, and deploy highly scalable ETL/ELT data pipelines using Azure Databricks (PySpark/SQL) and Azure Data Factory (ADF).
- Develop ingestion pipelines from multiple sources: - Relational databases - APIs - Files (CSV, JSON, Parquet) - Streaming sources (Event Hubs / Kafka, if applicable) - Handle incremental loads, CDC, and data quality validations - Azure Ecosystem Integration- Integrate Databricks with: - Azure Data Factory / Synapse Pipelines - Azure Functions & Logic Apps (where applicable) - Power BI and other BI tools - Implement secure access using Azure Active Directory, Key Vault, Managed Identities - Data Governance & Quality- Apply data validation, reconciliation, and quality checks - Support data cataloging, lineage, and governance (Unity Catalog preferred) - Ensure compliance with security and regulatory standards - Collaboration & Delivery- Work closely with business analysts, data scientists, and stakeholders - Participate in solution design, estimation, and code reviews - Contribute to documentation, best practices, and reusable frameworks -- Documenting architectural designs, configurations, and implementation guidelines for reference and knowledge sharing.

Technical Competencies

Experience with Delta Lake and Lakehouse architecture,Strong experience with Azure Databricks,Proficiency with Azure Data Lake Storage (ADLS Gen2),Experience with Azure Data Factory or equivalent orchestration tools

Qualifications and Experience

- Bachelor's and/or master’s degree in computer science, IT Management, Information Technology, Engineering.
(Math / Statistics background is added advantage).
- Good communications and listening skills are critical components of this job.
Must be able to interact with development and infrastructure teams, business and vendors including the training and mentoring of team members.
- A proven track record for the successful delivery of cloud data warehouse solutions and related components