Custom Software Engineer

Project Role : Custom Software Engineer

Project Role Description : Develop custom software solutions to design, code, and enhance components across systems or applications. Use modern frameworks and agile practices to deliver scalable, high-performing solutions tailored to specific business needs.

Must have skills : Data Engineering

Good to have skills : NA

Minimum 7.5 Year(s) Of Experience Is Required

Educational Qualification : 15 years full time education

Summary:

As a Data Engineer, a typical day involves designing, developing, and maintaining scalable data solutions on the Azure platform. The role requires building and optimizing ETL/ELT pipelines, implementing data processing frameworks, and ensuring data availability for business and analytical needs. Collaboration with cross-functional teams, troubleshooting data workflows, and following agile development practices are key aspects of the role to deliver reliable and high-performing data solutions.

Roles & Responsibilities:

• Expected to be an SME, collaborate and manage the team to perform.
• Engage with multiple teams and contribute to key technical decisions.
• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks, Python, and PySpark.
• Optimize Spark jobs, cluster configurations, and data processing workflows for performance and scalability.
• Develop and support data solutions leveraging Azure Storage Accounts, ADLS Gen2, and Delta Lake.
• Integrate Databricks with orchestration tools such as Apache Airflow for scheduling and monitoring data pipelines.
• Deploy and manage Databricks notebooks, jobs, workflows, and libraries across Development, QA, UAT, and Production environments.
• Troubleshoot data pipeline failures and ensure reliable data processing and delivery.
• Collaborate with stakeholders and development teams to deliver business-driven data solutions.
  
  

Professional & Technical Skills:

Must Have Skills:

• Strong hands-on experience in Azure Databricks with Python and PySpark.
• Good understanding of Apache Spark architecture, Spark configurations, performance tuning, and cluster management.
• Experience with Azure Storage Accounts, Azure Data Lake Storage (ADLS Gen2), and Delta Lake.
• Knowledge of authentication and authorization mechanisms, including Service Principals and Managed Identity.
• Experience integrating Databricks with Apache Airflow for job scheduling and monitoring.
• Understanding of Microservices architecture, REST APIs, and API integrations.
• Experience deploying Databricks notebooks, workflows, and libraries across multiple environments.
• Strong knowledge of Databricks Jobs, Workflows, and troubleshooting data pipeline failures.
• Ability to develop scalable ETL/ELT pipelines using PySpark and Python.
• Knowledge of ITSM processes and Production Support activities.
  
  

Additional Information:

The candidate should have a minimum of 11 years of experience in Data Engineering.

Experience In Azure Cloud-based Data Engineering Solutions Is Preferred.

This position is based at our Bengaluru office.

A 15 years full-time education is required.