Databricks Lead

Job Title: Databricks Lead / Senior Developer

Job Description:

We are seeking a highly skilled and experienced Databricks Lead/Senior Developer to join our team. The ideal candidate will possess strong expertise in Databricks, PySpark, and orchestration using Azure services such as Azure Event Hubs and Azure Event Grid. As a Databricks Lead, you will be responsible for designing, implementing, and optimizing data processing pipelines and solutions on the Databricks platform.

Key Responsibilities:

• Design and architect scalable data processing solutions using Databricks and PySpark (Spark Streaming not DLT).
• Develop and maintain data pipelines, ensuring data quality and reliability.
• Implement data orchestration workflows using Azure Event Hubs and Azure Event Grid.
• Collaborate with data engineers, data scientists, and other stakeholders to understand data requirements and deliver solutions.
• Optimize and tune Spark queries for performance and efficiency.
• Ensure data security and compliance with industry standards and best practices.
• Provide technical guidance and mentorship to junior team members.
• Stay updated with the latest trends and advancements in Databricks, PySpark, and Azure services.

Qualifications:

• Proven experience as a Databricks Lead or similar role.
• Strong expertise in Databricks, PySpark, and Spark SQL.
• Experience with Azure services, including Azure Event Hubs and Azure Event Grid.
• Proficiency in designing and implementing data processing pipelines.
• Knowledge of data warehousing concepts and ETL processes.
• Familiarity with data security and compliance best practices.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.

Nicety To Have:

• Experience with other Azure services such as Azure Data Factory
• Certification in Databricks & Azure.
• Experience with containerization and orchestration tools like Docker and Kubernetes will add additional benefit.