Senior Specialist - Data Engineering

Job Description :Databricks Engineer We are seeking a highly skilled and experienced Databricks Engineer to join our growing team and play a crucial role in enhancing our data processing and analytics capabilities As a Databricks Engineer you will be responsible for dealing with the most complex situations related to development design implementation and maintenance of our Databricks environment to ensure optimal performance scalability and reliability while aligning with our audit and security standards You will work closely with crossfunctional teams including data scientists analysts and software engineers to build robust data pipelines and deliver actionable insights Key Responsibilities 1Databricks Environment Management Handson experience with API Set up and configure Databricks clusters to meet specific project requirements Optimize Databricks configurations for performance and costeffectiveness Ensure the security and integrity of the Databricks environment 2Data Pipeline Development Collaborate with data scientists and analysts to understand data requirements Design and implement scalable and efficient data processing pipelines using Databricks Troubleshoot and optimize existing data pipelines for improved performance 3Integration and Collaboration Work closely with crossfunctional teams to integrate Databricks solutions into the overall data architecture Collaborate with software engineers to incorporate Databricks functionality into custom applications 4Monitoring and Maintenance Implement robust monitoring and logging solutions for Databricks clusters Conduct regular maintenance tasks including updates and patches to ensure system stability Generate review consumption reports identify abnormal consumption and take corrective action 5Documentation and Training Create and maintain comprehensive documentation for Databricks configurations and processes document best practices and standards to be followed

Provide training and support to internal teams on using Databricks effectively

Qualifications

Bachelors degree in Computer Science Engineering or a related field Masters Degree preferred

Proven experience as a Databricks Engineer or similar role

Strong proficiency in Apache Spark and Databricks

Experience with cloud platforms such as AWS Azure or Google Cloud

Proficient in programming languages such as Python Scala or Java

Solid understanding of data architecture and ETL processes

Preferred Qualifications

Databricks Certification

Experience with big data technologies and distributed computing

Familiarity with machine learning frameworks