AWS Data Bricks Py spark

Job Description: AWS Databricks Data Engineer

Role Overview

We are looking for an experienced AWS Databricks Data Engineer to design, develop and optimize scalable Lakehouse data solutions. The ideal candidate should have strong hands-on experience with Databricks AWS cloud services, Pyspark, Spark SQL and Delta Lake with the ability to build reliable production-grade data pipelines and translate business requirements into scalable data engineering solutions.

Experience Level

• Senior Data Engineer / Lead Engineer: 7-12 years of experience with strong expertise in Databricks architecture, data pipeline design, performance optimization, production deployments and leading enterprise-scale data engineering solutions.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, Delta Lake, Auto Loader and Delta Live Tables (DLT). 
• Design and implement metadata-driven data ingestion and transformation frameworks to enable scalable, reusable and configuration-based pipeline development across multiple data sources. 
• Develop framework components for managing source configurations, ingestion rules, schema evolution, data quality validations, audit logging and pipeline execution monitoring.
• Implement Medallion Architecture (Bronze, Silver, Gold) with appropriate data quality checks, validation frameworks, testing, and monitoring. 
• Develop ingestion frameworks using Databricks Auto Loader, including checkpointing, schema evolution and handling structured/semi-structured data.

Preferred candidate profile