Data Engineer-Information Technology

Responsible for designing, developing, and maintaining data pipelines and ETL processes using Databricks and Azure Data Factory.

• Good experience in using SQL, Python and Spark within Databricks notebooks for data processing and transformations.
• Integrate and orchestrate data from various sources, ensuring data quality and accuracy.
• Build reusable rules for Data Quality and required transformations.
• Optimize, monitor data pipelines for performance and scalability and debug in case of issues.
• Proficiency in Databricks, Azure Data Factory, Python, SQL is must.
• Familiarity with ETL principles, data warehousing, and big data technologies.
• Evaluate and define technical specs using BRD for BI and DW solutions.
• Build conceptual and logical models based on the functional flow of business in scalable way.
• Work with Business functions/ leadership and Application SMEs to understand the requirement and analysing the source to fulfil the requirements.
• Be compliant to data quality, governance, compliance, and other legal requirements.
• Act as key P.O.C between data analyst, data scientists, and the business/application teams.
• Must have Azure Databricks experience & Data Integrations.
• Good work experience on Data bricks, ADF, and other Azure and Databricks objects.
• Experience on creating frameworks towards building the optimized data pipelines is a must.
• Working knowledge of different self-hosted and other runtimes along with SSIS.
• Strong work experience of SQL, Scala, Python and Spark.
• Performance Tuning of Spark SQL running on DELTA LAKE storage and Strong Knowledge on Databricks and Cluster Configurations.
• Working knowledge of Data Warehousing, Modelling, Data Validation, Data Migration.
• Documentation of all project resources and follow best practices for development.
• Knowledge of front-end tools like Qlik Sense, MS PBI
• Minimum bachelors degree in computer science or related field is required.

Strong problem-solving skills, good communication abilities and go-getter attitude.