Azure Databricks Consultant

Job Description:

• Infosys is seeking a highly skilled Azure Databricks Engineer with 6 to 8 years of experience in designing developing and optimizing cloud based data engineering solutions
• The ideal candidate will possess strong expertise in Azure Databricks PySpark Azure Data Factory ADF Azure Data Lake Storage ADLS SQL and Data Engineering practices

Key Responsibilities:

• Design develop and maintain scalable Azure Databricks solutions for enterprise data platforms
• Build and optimize end to end ETL ELT pipelines using PySpark Spark SQL and Azure Data Factory ADF
• Develop robust data ingestion frameworks from multiple structured and unstructured data sources
• Implement and manage Delta Lake and Lakehouse Architecture for high volume data processing
• Create and optimize data models transformations and workflows to support analytical and business requirements
• Perform performance tuning and troubleshooting of Databricks workloads and Apache Spark jobs
• Work with Azure services including ADLS Gen2 Azure Synapse Analytics Azure Key Vault and Azure Data Factory
• Collaborate with business stakeholders architects and cross functional teams to deliver data solutions
• Ensure adherence to data governance security and best practices
• Participate in code reviews CI CD implementation testing deployment and production support activities

Technical Requirements:

• 6 to 8 years of experience in Data Engineering and Big Data technologies
• Strong hands on experience with Azure Databricks and PySpark
• Expertise in Apache Spark Spark SQL and distributed data processing
• Experience with Azure Data Factory ADF and building enterprise scale data pipelines
• Strong knowledge of Azure Data Lake Storage ADLS Gen2

Preferred Skills:

Technology->Cloud Integration->Azure Data Factory (ADF),Technology->Cloud Platform->Azure Core services->Azure Virtual Machines,Technology->Cloud Platform->Azure Networking Services->Azure Application Gateway,Technology->Data Engineering->Databricks