Azure Databricks Engineer

Job Title: Azure Databricks Engineer

Experience: 8+ Years Location: Hyderabad

Employment Type: Full-Time/ Onsite Domain: Insurance

Job Summary: We are seeking an experienced Azure Databricks Engineer with 8+ years of IT experience and strong expertise in building scalable data platforms on Azure. The ideal candidate should possess hands-on experience with Lakehouse architecture, Delta Lake, Azure Event Hub, and Databricks-based data engineering solutions. Experience in the Insurance domain is highly preferred. Databricks certification is mandatory.

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines using Azure Databricks.
• Implement and optimize Lakehouse architecture leveraging Delta Lake for high-performance analytics.
• Build batch and streaming data ingestion solutions using Azure Event Hub.
• Develop ETL/ELT workflows using PySpark, Spark SQL, and Databricks Workflows.
• Integrate data from multiple sources and ensure data quality, reliability, and governance.
• Optimize cluster performance, partitioning strategies, and Delta tables for efficient processing.
• Collaborate with data architects, analysts, and business stakeholders to deliver data solutions aligned with Insurance business requirements.
• Implement CI/CD processes and automate deployment pipelines.
• Monitor and troubleshoot production issues to ensure platform stability and performance.
• Follow best practices for security, scalability, and cost optimization within the Azure ecosystem.

Required Skills:

• 8+ years of overall IT experience with strong Data Engineering expertise.
• Hands-on experience with Azure Databricks and Apache Spark.
• Strong knowledge of Lakehouse Architecture and Delta Lake.
• Experience with Azure Event Hub for real-time data ingestion.
• Proficiency in PySpark, Spark SQL, Python, and SQL.
• Experience with Azure services such as ADLS Gen2, Azure Data Factory, Key Vault, and Azure Monitor.
• Knowledge of batch and streaming data processing concepts.
• Experience in performance tuning and optimization of Databricks workloads.
• Familiarity with Git, CI/CD pipelines, and DevOps practices.
• Excellent analytical, troubleshooting, and communication skills.

Preferred Qualifications:

• Experience in the Insurance domain.
• Exposure to real-time analytics and event-driven architectures.
• Knowledge of data governance and security frameworks.
• Experience with Agile/Scrum methodologies.

Mandatory Certification

• Databricks Certified Data Engineer Associate or Professional Certification (Mandatory).