LEAD ENGINEER - Databricks

Responsibilities:

• Design, develop, and maintain scalable data pipelines and architectures using Databricks.
• Collaborate with data scientists and analysts to understand data requirements and deliver high-quality data solutions.
• Implement data ingestion processes from various sources, ensuring data quality and integrity.
• Optimize data storage and retrieval processes for performance and cost efficiency.
• Monitor and troubleshoot data pipeline performance and reliability issues.
• Document data engineering processes, architectures, and best practices.
• Stay updated with the latest trends and technologies in data engineering and analytics.

Mandatory Skills:

• Strong expertise in Databricks and its ecosystem.
• Proficient in programming languages such as Python, Scala, or Java.
• Experience with data modeling, ETL processes, and data warehousing concepts.
• Familiarity with cloud platforms (e.g., AWS, Azure, GCP) and their data services.
• Solid understanding of SQL and experience with relational databases.
• Knowledge of big data technologies (e.g., Spark, Hadoop) is a plus.

Preferred Skills:

• Experience with data visualization tools (e.g., Tableau, Power BI).
• Familiarity with machine learning concepts and frameworks.
• Knowledge of data governance and data security best practices.
• Experience with CI/CD pipelines and version control systems (e.g., Git).