Senior Data Engineer- Azure

Role Summary

We are hiring a Senior Data Engineer to drive the development of scalable, high-performance data pipelines for a migration project to a modern Azure Native architecture. This role will be instrumental in building an AI-ready data platform using Microsoft Fabric (Fabric IQ) and enabling seamless integration with AI/ML workflows (Foundry IQ).

Key Responsibilities

• Design and build robust, scalable data pipelines (batch + near real-time) using Microsoft Fabric (Data Pipelines, Lakehouse)
• Lead migration of existing pipelines (AKS/custom ETL) to Fabric-native architecture
• Implement Medallion Architecture (Bronze Silver Gold) for structured and unstructured healthcare data
• Develop and optimize data models and transformations for analytics and AI use cases
• Build ingestion frameworks for:
• APIs (FHIR, EHR systems)
• Files (JSON, HL7, CSV)
• Ensure data quality, validation (FHIR/claims), and observability across pipelines
• Enable AI/ML workflows by preparing datasets for:
• RAG pipelines
• Feature engineering and model consumption
• Optimize pipeline performance, cost, and scalability
• Collaborate with Architects, AI/ML engineers, and Product teams to translate requirements into data solutions
• Mentor junior engineers and drive engineering best practices

Required Experience

• 4-6 years of experience (3+ years in data engineering)
• Strong hands-on experience with Azure Data Platform
• Proven experience in building and optimizing large-scale data pipelines
• Experience in data platform modernization or migration projects

Technical Skills

• Languages: Python, PySpark, SQL (advanced)
• Azure / Fabric:
• Microsoft Fabric (preferred) OR Azure Data Factory, Synapse, Data Lake
• Data Architecture:
• Lakehouse, Medallion Architecture
• Batch + streaming data processing
• Data Handling:
• JSON, Parquet, CSV, API integrations
• Performance Optimization:
• Partitioning, indexing, query tuning

Nice to Have

• Experience with healthcare data (FHIR, claims, EHR)
• Exposure to RAG pipelines / vector databases / AI data prep
• Familiarity with event-driven architectures
• Understanding of data security & governance in regulated environments

Success Metrics

• High-quality, scalable pipelines delivered on time
• Seamless migration with minimal business disruption
• Improved data performance, reliability, and cost efficiency
• Strong enablement of AI/ML and analytics use cases

What Were Looking For

• Strong hands-on engineer with ownership mindset
• Ability to operate in fast-paced, evolving architecture environment
• Problem-solver with focus on performance, scalability, and data quality
• Mentor and team player

Why This Role Matters

This role is critical to building the foundation of AI-first platform, enabling:

• Scalable data processing via Fabric IQ
• AI-driven intelligence via Foundry IQ
• Future-ready healthcare data platform