Azure Data Engineer

Roles and Responsibilities

• Design, develop, and maintain large-scale data pipelines using Azure Data Factory (ADF) to extract, transform, and load data from various sources into Azure Storage solutions such as Azure Blobs, Files, and SQL DW.
  
• Develop complex data workflows using ADF's logical components like Triggers, Pipelines, and Linked Services to integrate with on-premises systems through Virtual Network Taps.
  
• Collaborate with cross-functional teams to gather requirements for designing scalable and efficient data architectures on Azure Cloud Platform.
  
• Implement machine learning models using Python libraries like scikit-learn and TensorFlow within an ADF pipeline for advanced analytics capabilities.
  
• Ensure high availability of deployed solutions by implementing monitoring tools like Logic Apps, Application Insights, and Service Bus.