Microsoft Fabric Data Engineer

Role summary
Build and operationalize the Fabric pipeline designed by the architect: ingestion from multiple sources, Spark-based transformation logic, Gold layer tables (both MLV and DirectQuery-served), and ongoing maintenance/optimization through go-live and stabilization. This is a sustained build-and-run role, following an existing architecture rather than designing it from scratch.

Key responsibilities
- Build and maintain Spark notebooks for BronzeSilver transformations, including parsing semi-structured (JSON) and NoSQL-sourced data
- Implement entity-resolution/join logic across sources (structured relational + semi-structured + NoSQL)
- Build and schedule Materialized Lake Views and/or DirectQuery views per the architecture's real-time/analytical split
- Set up and maintain Data Pipeline orchestration (ingestion, transform, refresh scheduling, dependency chains)
- Monitor and tune performance: table optimization/vacuuming, small-file compaction, Direct Lake troubleshooting
- Maintain and extend the mirroring/ingestion connections (Oracle mirroring, OFSC API polling, MongoDB connector) as data volume or requirements evolve
- Support integration testing with the AI agent layer (Fabric Data Agent and/or direct SQL/MCP tool queries)
- Document pipeline logic and hand over runbooks for BAU support

Must-have skills
- Strong hands-on PySpark and Spark SQL comfortable writing production transformation logic, not just tutorials
- Working experience with Microsoft Fabric: Lakehouse, notebooks, Data Pipelines, Delta tables
- Experience with Delta Lake concepts: schema evolution, optimize/vacuum, change data feed
- SQL proficiency (T-SQL for Warehouse/SQL analytics endpoint work)
- Experience with API-based data ingestion (REST APIs, JSON parsing, pagination/polling patterns)
- Comfortable working from an architecture document and translating design decisions into working code

Nice-to-have
- Materialized Lake Views experience specifically
- Experience with MongoDB or other NoSQL Spark connectors
- Power BI / semantic model exposure (helpful for testing Gold layer outputs, not core to the role)
- DP-203 or DP-700 (Fabric Data Engineer) certification.