Azure Data Lake Specialist

Summary

The successful candidate is an experienced Data Lake developer with expertise in SQL and Python for Jupyter Notebooks, and has the ability to transform large volumes of data following the principles of ETL. The ideal candidate is a motivated self-starter with excellent programming / debugging skills, strong communication skills with the ability to collaborate with stakeholders to refine business requirements and translate into a detailed technical specification.

This role requires strong programming skills, particularly in Python, and loads data from different sources including Azure Storage Accounts, Flat Files, API, SQL DB, etc.

Detailed Description

Performs tasks such as, but not limited to, the following:

• Advanced Microsoft SQL, Azure Synapse or Azure Data Factory skills, including proficiency with: Stored procedure and trigger development, schema management, and execution plan optimization.
• Capable of constructing resilient data integration processes with an emphasis on data quality management and failure-response planning.
• Expertise in ETL principles.
• Experience extracting data from heterogeneous sources, including databases, CRM systems, flat file, web services, storage accounts, etc.
• Experience transforming the extracted data involving cleaning, aggregating, mapping, and converting it to ensure it meets the business requirements and is in a suitable format for analysis.
• Experience loading the transformed data into a target database, data warehouse, or data lake, making it accessible for querying and analysis.
• Experience integrating Azure Data Factory patterns, processes, components, security, linked services, datasets, activities and pipelines, and integration runtimes.
• Maintain documentation of software architecture, processes, and code.
• Strong communication skills. Able to work with stakeholders and customers to come up with requirements, sizing.

Knowledge/Skills/Competencies

• Python: Write clean, scalable, and maintainable code following best practices.
• Knowledge of scripting, Forms and web services is an added advantage
• Pandas, NumPy: Data manipulation and analysis.
• SQL: Ability to extract, transform, and load (ETL) data from relational databases.
• Other: Familiarity with Azure Datalake (Synapse Workspace)
• Experience with TFS, GIT, Azure DevOps or any other source control system.
• Excellent communication skills for explaining technical complexities.
• Ability to work in a collaborative, cross-functional environment.
• Skill in managing data projects independently.
• Abstract Thinking: Advanced problem-solving abilities
• Problem-Solving: Proven experience in troubleshooting and optimizing code
• Punctuality: Reliable and disciplined in managing project timelines.
• Communication Skills: Clear in explaining Code and their business impact.
• International Experience: Ability to work effectively in international and diverse teams across different geographies. This is a multinational company, and it may require having late night meetings to connect with team members or business users from the Asia geography
• Production Support: Weekend on-call support is required.

Physical Demands

• Duties of this position are performed in a normal office environment.
• Duties may require extended periods of sitting and sustained visual concentration on a computer monitor or on numbers and other detailed data. Repetitive manual movements (e.g., data entry, using a computer mouse, using a calculator, etc.) are frequently required.

Typical Experience

• 5+ years of practical experience with Azure Data Lake / Data Engineering or similar role

Typical Education

• Bachelor's degree in Computer Engineering, Computer Science, Data Engineering
• Educational Requirements may vary by Geography

Notes

This job description is not intended to be an exhaustive list of all duties and responsibilities of the position. Employees are held accountable for all duties of the job. Job duties and the % of time identified for any function are subject to change at any time.