Lead Data Engineer (Azure & Fabric)
Whitewhale Software Solutions · Remote · remote · Posted 8d ago
Undisclosed9–12 yrsfulltimeSourced from
About a minute. You'll make an account on the way through.
About this role
Job Summary
Responsibilities
Requirements
Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
We are seeking a highly skilled and experienced Azure Cloud Lead / Data Architect to evaluate, optimize, and elevate our existing data platform built on Microsoft Fabric. In this role, you will conduct a comprehensive audit of our current production architecture, capacity configurations, Spark notebooks, orchestration pipelines, and storage layer. Your primary mission will be to identify performance bottlenecks, optimize cost efficiency, ensure architectural best practices, and deliver a robust roadmap for scalability and governance.
Responsibilities
- Architectural Assessment Optimization: Audit end-to-end Microsoft Fabric architecture, OneLake / Delta Lake storage layer, and capacity settings (F-SKUs).
- Code Pipeline Engineering Audit: Review PySpark / Spark SQL notebooks to optimize execution plans, caching, and memory. Assess ADF pipelines and Dataflows Gen2.
- CI/CD Governance: Define guidelines for CI/CD automation, version control (Git integration), and deployment pipelines within Microsoft Fabric.
- Mentorship Leadership: Mentor the engineering team on advanced Spark tuning and Fabric-specific optimization techniques.
Requirements
- 9 to 12 Years of experience, with minimum 8+ years engineering and architecting enterprise data platforms using ADF, Synapse, Databricks, and ADLS Gen2.
- Deep, hands-on expertise in PySpark, Spark optimization, performance tuning, and debugging complex Spark jobs.
- Strong understanding of Delta Lake storage formats, ACID transactions, and optimization (Z-ordering, vacuuming).
- Practical understanding of Fabric: OneLake, Lakehouse, Data Warehouse, Data Factory, and Capacity management.
- Experience setting up CI/CD pipelines for data workloads using Azure DevOps or GitHub.
Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
Sourced from naukri
Skills
AutomationAzure DevopsData EngineerCI/CD Pipelinesdata warehouseData Architectcachingdata platformsCapacity managementAdls Gen2AWS CodePipelineADLSdebuggingdeploymentDatabrickscloudArchitectureMicrosoft Azure