Data Engineer - Lead

Job Summary Job Description

• Databricks Workflows
• PySpark
• Amazon Kinesis
• Delta Lake on Databricks

• CI/CD Source Control (CI/CD + Jenkins + Git)

Key Responsibilities

• Define and drive enterprise data engineering strategy aligned with organizational objectives and data modernization initiatives.
• Establish data engineering standards, governance frameworks, and best practices across teams.
• Lead the design of enterprise-scale data processing architectures using PySpark and modern data platform technologies.
• Define enterprise standards for Snowflake and Delta Lake-based data platforms supporting analytical and operational workloads.
• Drive real-time and event-driven data architecture initiatives using Apache Kafka or Amazon Kinesis.
• Establish governance standards for data ingestion, transformation, streaming, and processing frameworks.
• Define workflow orchestration, scheduling, and operational governance standards using Apache Airflow or Databricks Workflows.
• Establish data quality, validation, monitoring, and operational excellence frameworks across data engineering ecosystems.
• Define enterprise standards for data products, data quality ownership, metadata management, discoverability, and trusted business data consumption across the organization.
• Establish architecture standards for modern Lakehouse platforms, data observability, platform engineering, and scalable cloud-native data ecosystems supporting enterprise analytics and AI initiatives.
• Define AI-ready data foundation strategies supporting structured and unstructured data processing, vector-enabled architectures, retrieval patterns, and future GenAI and Agentic AI initiatives.
• Partner with business stakeholders to translate business objectives into scalable data platform capabilities, data products, and enterprise data architecture decisions.
• Drive adoption of AI-assisted engineering practices across data engineering teams to improve developer productivity, code quality, documentation, testing, and delivery effectiveness while maintaining governance standards.
• Lead architecture reviews and ensure data solutions meet scalability, reliability, maintainability, and performance objectives.
• Guide teams on distributed data processing, streaming architectures, modern data platforms, and engineering best practices.
• Identify platform risks, scalability bottlenecks, operational gaps, and architectural challenges while defining mitigation strategies.
• Collaborate with various teams and leadership stakeholders to align data initiatives with organizational objectives.
• Drive continuous improvement initiatives focused on platform maturity, engineering excellence, scalability, reliability, and delivery effectiveness.

Mandatory Competencies

• Data Science and Machine Learning - Data Science and Machine Learning - Apache Spark
• Data AI - Data Engineering - Data Quality Validation
• Big Data - Big Data - Pyspark
• Data Science and Machine Learning - Data Science and Machine Learning - Python
• Database - Database Programming - SQL
• Data Science and Machine Learning - Data Science and Machine Learning - Databricks
• Beh - Communication and collaboration