AWS Databricks, Pyspark

Role- AWS Databricks, Pyspark

Desired Experience Range- 7-11 Years

Location- Chennai, Pan India

Roles & Responsibility -

• Strong hands-on experience with AWS Databricks, Pyspark, Delta Lake, Unity Catalog, Workflows/Jobs and Lakehouse architecture
• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, Delta Lake, Auto Loader and Delta Live Tables (DLT). 
• Design and implement metadata-driven data ingestion and transformation frameworks to enable scalable, reusable and configuration-based pipeline development across multiple data sources. 
• Develop framework components for managing source configurations, ingestion rules, schema evolution, data quality validations, audit logging and pipeline execution monitoring.
• Implement Medallion Architecture (Bronze, Silver, Gold) with appropriate data quality checks, validation frameworks, testing, and monitoring. 
• Develop ingestion frameworks using Databricks Auto Loader, including checkpointing, schema evolution and handling structured/semi-structured data. 
• Configure and manage Unity Catalog including catalogs, schemas, access controls, audit logging, data lineage and security policies. 
• Optimize Spark workloads by tuning Spark jobs, cluster configurations, partitioning strategies, and Delta Lake storage layouts for performance and cost efficiency. 
• Build and manage workflows using Databricks Jobs/Lakeflow Jobs for batch and streaming data processing. 
• Implement CI/CD practices and integrate with Git-based DevOps processes. 
• Integrate Databricks solutions with AWS services such as S3, IAM, Glue, Step Functions and other AWS data services. 
• Enable analytics consumption through BI tools such as Power BI, Tableau or Looker using optimized connectivity patterns. 
• Collaborate with data architects, analysts, data scientists and business stakeholders to deliver enterprise data platform solutions. 
• Troubleshoot pipeline failures, performance issues and operational challenges while maintaining technical documentation.

Nice to Have-

• Experience with Databricks advanced capabilities such as MLflow, Feature Store, Vector Search and GenAI workloads. 
• Knowledge of Delta Sharing and Databricks Marketplace. 
• Experience migrating workloads from platforms such as Snowflake, Azure Databricks or traditional data warehouses. 
• Exposure to enterprise-scale data platform implementations.