Senior Data Engineer

To design and deliver AI-powered Business Intelligence solutions by integrating Databricks Genie, PowerBI Co-Pilot Semantic Models, and Metric Views on top of a robust end-to-end data engineering stack. The role focuses on enabling self-service analytics using natural language (GenAI), ensuring business-friendly data consumption, and building governed, scalable, and high-performance data ecosystems across batch and real-time pipelines.

• Translate business problems into AI-driven BI solutions
• Own end-to-end delivery: ingestion transformation semantic layer AI consumption
• Design scalable data architecture aligned with lakehouse principles
• Ensure data quality, governance, and metric consistency
• Collaborate with business teams to onboard them onto Genie-based analytics
• Optimize performance of queries, pipelines, and AI responses
• Establish best practices for AI in BI (prompting, semantic tuning, governance)
• Drive adoption of self-service BI with minimal dependency on tech teams

• Define semantic layer design and metric definitions
• Decide GenAI prompting strategies and instruction frameworks
• Prioritize data vs AI optimization trade-offs
• Handle production issues with RCA and long-term fixes
• Drive architectural decisions for lakehouse + AI integration

• Ensuring accuracy and trust in AI-generated insights
• Driving adoption of GenAI-based BI over traditional dashboards
• Maintaining semantic consistency across multiple tools
• Balancing performance, cost, and scalability
• Managing dependencies across data engineering, AI, and business teams

• Azure Databricks - PySpark, SQL, Delta Lake
• PowerBI - CoPilot, Semantic Model
• Strong experience in Semantic Modeling Metrics Layer design
• Hands-on with Databricks Genie / GenAI-based BI workflows
• Databricks One (Unified BI Experience)
• Prompt Engineering / Instruction tuning for AI systems
• Python (Pandas, PySpark, FastAPI)
• Azure Data Factory (ADF) for ETL pipelines
• Strong SQL and data modeling skills
• Good to Have
• Cosmos DB / MongoDB (NoSQL concepts)
• Azure Data Explorer (KQL)

• Databricks Lakehouse - PySpark, SQL, Delta Lake
• AI for BI - Databricks Genie, Genie Rooms, Instructions, Agents, PowerBI CoPilot
• Semantic Layer - Semantic Models, Metric Views
• ETL Orchestration - Azure Data Factory
• Programming - Python, C#/.NET
• Cloud Platform - Azure (Preferred)
• DevOps - CI/CD Pipelines, Git