Azure Databricks (Anywhere in India)

Must-Have Skills:

• Strong hands-on experience with Azure Databricks
• Expertise in Apache Spark (PySpark) and Spark SQL
• Strong programming skills in Python and SQL
• Experience with Delta Lake including ACID transactions and schema evolution
• Hands-on experience in ETL/ELT pipeline development
• Strong understanding of Lakehouse Architecture
• Experience building and maintaining Medallion Architecture (Bronze, Silver, Gold) data layers
• Experience with batch and streaming data processing
• Experience with Azure Data Factory (ADF) and Azure Data Engineering ecosystem
• Strong understanding of Data Modeling and Analytics data structures
• Experience with Unity Catalog or Data Governance frameworks
• Experience in performance tuning, query optimization, partitioning, and caching
• Data Engineering experience
• Azure Databricks experience

Good-to-Have Skills:

• Git, Azure DevOps, GitHub Actions
• CI/CD implementation and deployment automation
• Terraform (Infrastructure as Code)
• Docker and Kubernetes
• Airflow
• Kafka or Azure Event Hub
• Power BI or Tableau
• Databricks Workflows and Job Orchestration
• Cluster Management and Optimization
• Experience with Delta Table features such as MERGE, UPSERT, and Time Travel
• BFSI domain experience

Key Responsibilities:

• Design, develop, and optimize scalable batch and streaming data pipelines using Azure Databricks.
• Develop data processing frameworks using PySpark and Spark SQL.
• Implement Delta Lake-based storage solutions to ensure high reliability and data consistency.
• Build and maintain Bronze, Silver, and Gold data layers within a Lakehouse architecture.
• Ingest data from APIs, databases, Azure storage, and streaming platforms.
• Develop scalable data models supporting analytics and reporting requirements.
• Implement data quality, validation, and reconciliation processes.
• Monitor and optimize Spark workloads for performance and cost efficiency.
• Establish and support data governance practices through Unity Catalog and related solutions.
• Orchestrate and integrate data workflows using Azure Data Factory, Airflow, and Databricks Workflows.
• Collaborate with business and technical stakeholders to gather requirements and deliver enterprise-grade data solutions.