Data Engineering Lead

Role & responsibilities

• Data Modeling & Analytics Engineering
• Design, develop, and maintain scalable dimensional data models, including Star and Snowflake schemas
• Build curated data products that support reporting, analytics, AI, and machine learning use cases
• Define and standardize business logic, KPIs, metrics, and calculations across enterprise domains
• Create reusable datasets, semantic models, and data assets to improve consistency and reduce duplication
• Collaborate with business stakeholders to translate requirements into analytical and reporting solutions
• Databricks Platform Engineering
• Design and implement modern data architectures leveraging Databricks Lakehouse principles
• Develop and maintain Bronze, Silver, and Gold data layers
• Build scalable ETL/ELT pipelines using Spark, PySpark, SQL, and Databricks Workflows
• Optimize Delta Lake implementations using partitioning, Z-Ordering, data skipping, Change Data Feed (CDF)
• Delta optimization techniques
• Implement CI/CD pipelines and DevOps best practices for data engineering and analytics solutions
• Context Layer & Semantic Layer Development
• Design and manage enterprise semantic and context layers to provide a single source of truth for reporting and analytics
• Standardize business metrics, dimensions, hierarchies, and definitions across reporting platforms
• Enable self-service analytics through Power BI, Tableau, and AI-driven applications
• Manage semantic models and governance processes to ensure reporting consistency
• Establish metric certification, lineage tracking, and data governance standards
• Performance Optimization & Cost Management
• Analyze query execution plans and identify performance bottlenecks
• Optimize Spark workloads for scalability, reliability, and cost efficiency
• Implement cluster sizing, autoscaling, caching, broadcast joins, and Adaptive Query Execution (AQE) strategies
• Tune Delta Lake tables through file compaction, partition optimization, and storage management
• Monitor warehouse and cluster utilization to improve performance and control cloud expenses
• Enhance dashboard and reporting performance through optimized data models and query design
• Data Governance & Quality Management
• Develop and implement data quality frameworks, validation processes, and automated monitoring
• Ensure compliance with enterprise security, governance, and regulatory requirements
• Support metadata management, data cataloging, and lineage initiatives
• Establish monitoring and alerting for data pipelines, processing failures, and data anomalies
• Partner with governance teams to drive data stewardship and certification processes
• Technical Leadership & Collaboration
• Collaborate with data scientists, analysts, engineers, product owners, and business leaders
• Mentor and guide junior analytics engineers and data engineers
• Conduct architecture reviews and recommend technology best practices
• Drive adoption of enterprise data standards, frameworks, and governance processes
• Lead technical design discussions and influence data platform strategy and roadmap
• Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements

Preferred candidate profile

• Master's degree in Computer Science, Data Science, Analytics, or a related discipline
• Databricks certifications (Data Engineer Associate, Professional, or equivalent)
• Experience with Databricks Unity Catalog and enterprise data governance implementations
• Experience supporting AI/ML, Generative AI, or advanced analytics use cases
• Experience designing enterprise semantic models and metric layers
• Healthcare, insurance, or highly regulated industry experience
• Demonstrated experience leading technical teams, mentoring engineers, and driving enterprise data strategy
• Knowledge of data catalog and governance tools such as Purview, Collibra, or Alation

Key Competencies:

• Data Architecture & Modeling
• Analytics Engineering
• Databricks & Delta Lake
• Spark/PySpark Development
• Data Governance & Quality
• Performance Optimization
• Semantic Layer Design
• Cloud Data Platforms
• Technical Leadership
• Stakeholder Management