AI Lead Data Engineer

Role Summary

Technical leader responsible for the modernization and future-state architecture of the Sales Validation platform.

Responsibilities

• Lead Databricks architecture and delivery
• Drive Synapse-to-Databricks migration
• Establish engineering standards and reusable frameworks
• Mentor engineers and conduct design reviews
• Own technical delivery across major workstreams
• Drive performance optimization and scalability

Required Skills

Databricks (Highest Priority)

• Databricks
• PySpark
• Spark SQL
• Delta Lake
• Unity Catalog
• Databricks Workflows
• Databricks Asset Bundles
• Lakehouse Architecture

Azure Platform

• Azure Data Factory (ADF)
• ADLS Gen2
• Azure SQL
• Azure Security Patterns

Migration Experience

• Synapse Pipelines
• Synapse Spark
• Synapse SQL Pools
• Synapse Notebooks
• Synapse Data Flows
• Modernization of legacy ETL workloads

Engineering

• Advanced SQL
• Data Modeling
• GitLab CI/CD
• Performance Tuning
• Code Reviews

GenAI

• Hands-on experience with GenAI coding assistants (Claude Code, GitHub Copilot, or similar) for AI-assisted development.
• Prompt engineering skills writing clear instructions and feedback to guide AI toward desired output.
• Ability to critically review and validate AI-generated code and content before use.
• Basic understanding of LLM concepts — RAG, context windows, hallucinations, and responsible AI practices.

Preferred

• Apache Airflow
• Snowflake
• Data Governance
• Enterprise-scale Azure platform implementation