Data and Analytics - Lead

Key Responsibilities

• Lead the design, architecture, and implementation of Azure Databricks solutions for data processing and analytics.
• Develop and optimize ETL/ELT pipelines using Apache Spark, Python (PySpark), Scala, and SQL.
• Work closely with data engineers, data scientists, and business stakeholders to ensure efficient data workflows.
• Implement data governance, security, and compliance best practices in Databricks and Azure Data Lake.
• Optimize performance tuning, cost management, and scalability of Databricks workloads.
• Automate CI/CD pipelines for Databricks notebooks using DevOps tools like Azure DevOps, Terraform, or GitHub Actions.
• Troubleshoot and resolve issues related to big data processing, Spark jobs, and cloud storage.
• Stay updated with Azure and Databricks best practices and mentor junior team members.
• Required Skills & Experience:
• 
  
• 
  

10+ years of experience in data engineering, big data, or cloud-based data solutions.

• Strong hands-on experience with Azure Databricks, Apache Spark, and Delta Lake.
• Expertise in Python (PySpark), Scala, SQL, and data transformation techniques.
• Deep understanding of Azure Data Services (Azure Data Lake, Azure Synapse, Azure Data Factory, etc.).
• Experience with data modeling, ETL/ELT design, and distributed data processing.
• Proficiency in DevOps practices, including CI/CD pipelines, Infrastructure as Code (IaC), and version control (Git).
• Familiarity with MLflow, Unity Catalog, and Databricks SQL is a plus.
• Strong analytical, problem-solving, and communication skills.
• 
  
• 
  

Preferred Qualifications

• Databricks certifications (e.g., Databricks Certified Data Engineer Associate/Professional).
• Experience with containerization (Docker, Kubernetes) and orchestration tools.
• Knowledge of real-time data streaming using Kafka, Event Hubs, or similar technologies.
• Exposure to Machine Learning (ML) workflows in Databricks is a plus.

• Competitive Salary
• Performance based Variable Pay
• Health, Vision, Dental, and Life Insurance
• Comprehensive Employee Wellness Plan
• Retirement plan
• Paid Holidays and Vacation Time Accrual
• Encouraging and highly energetic environment
• Personal and Professional Growth Opportunities