Databricks Developer - ADF

Job Description

SN Required Information Details

1 Role Senior Databricks Engineer

2 Required Technical Skill Set Azure Databricks + Apache Spark

4 Desired Experience Range 4-10 years

5 Location of Requirement Pune

Desired Competencies (Technical/Behavioral Competency)

Must-Have • Design, build, and maintain scalable and efficient data pipelines using Databricks. Integrate data from various sources (for eg. Azure storage account, SFTP, Fabric) using Apache Spark within Databricks. • Implement processes and checks to ensure the accuracy and completeness of data. • Work closely with data owners, analysts, and other stakeholders to understand data requirements and deliver solutions. • Monitor data pipelines for issues and resolve them promptly to maintain data flow integrity. • Experience with Databricks-specific tools and features. • Proficiency in programming languages such as Python or Scala for data processing on Databricks • Experience with Azure cloud platform. • Strong understanding of SQL. • Experience with ETL and processes, Unity Catalog.

Good-to-Have Certifications in Databricks or related cloud technologies

SN Responsibility of / Expectations from the Role

1 Design, build, and maintain scalable and efficient data pipelines using Databricks. Integrate data from various sources (for eg. Azure storage account, SFTP, Fabric) using Apache Spark within Databricks.

2 Implement processes and checks to ensure the accuracy and completeness of data.

3 Work closely with data owners, analysts, and other stakeholders to understand data requirements and deliver solutions

4 Work closely with data owners, analysts, and other stakeholders to understand data requirements and deliver solutions.

5 Monitor data pipelines for issues and resolve them promptly to maintain data flow integrity.