Azure Data Engineer -Data Load Tool (DLT)

Role & responsibilities

• Design, build and deploy ELT pipelines and data management processes.
• Operationalise ELT job workflow with reliable error/exception handling and rollback framework.
• Build and oversee data architecture & pipelines platforms and systems, managing the flow of data from internal and external sources by leveraging both distributed and local structures.
• Participate in developing, designing, configuring, and documenting database architecture.
• Monitor and improve data loading performance.
• Working with CI/CD pipelines to manage and execute releases
• Clearly communicate technical terms to non-technical people and help them understand why change might be required to achieve a specific goal or to complete a project
• Conform to agile development practices Evolutionary design, refactoring, continuous integration/delivery, test-driven development.
• Work collaboratively with Business Partner Team and Business Stakeholder during projects scoping and feasibility phases as a SME for concept investigation/viability and technical impact; Identify and communicate technical risks and mitigations.
• Provide production support for data load jobs.
• Write customized query to generate automatic periodic reports.
• Maintain or upgrade existing applications.

Attend key design meetings and provide support.

Preferred candidate profile

• Experience in managing Azure Data Lake structure and security
• Proven experience with data modeling, and working with star schema, snowflake schema
• Experience with the full development life cycle of an application stack - from architecture through test and deployment.
• Experience in Master Data management solutions.
• Experience in data warehouse analysis and design, with full knowledge of data warehouse methodologies and data modelling.
• Unit testing low level components SQL scripts, Stored Procedures, ETL/ELT modules; Integration testing data transition points; Performance, load and stress testing.
• Knowledge and experience in software development methodology and toolset for implementing, lifecycle and management – Agile methodology.
• GIT code repository and branching strategies; Microsoft Release Management and any deployment tools; Azure DevOps knowledge.
• Experience with Azure cloud solutions; Knowledge of Azure Data Factory, Azure Automation, Azure Data Lake, Azure Synapse, Azure Purview, Azure KeyVault, and Azure Batch.
• Experience with Google Cloud Platform solutions; knowledge of Google BigQuery, Google Storage, and Google CloudRun
• Experience with APIs.