Custom Software Engineer

Project Role : Custom Software Engineer


Project Role Description : Develop custom software solutions to design, code, and enhance components across systems or applications. Use modern frameworks and agile practices to deliver scalable, high-performing solutions tailored to specific business needs.


Must have skills : Data Engineering


Good to have skills : NA
Minimum
7.5 year(s) of experience is required


Educational Qualification : 15 years full time education

Summary:

As a Data Engineer, a typical day involves designing, developing, and maintaining scalable data solutions on the Azure platform. The role requires building and optimizing ETL/ELT pipelines, implementing data processing frameworks, and ensuring data availability for business and analytical needs. Collaboration with cross-functional teams, troubleshooting data workflows, and following agile development practices are key aspects of the role to deliver reliable and high-performing data solutions.

Roles Responsibilities:

Expected to be an SME, collaborate and manage the team to perform.

Engage with multiple teams and contribute to key technical decisions.

Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks, Python, and PySpark.

Optimize Spark jobs, cluster configurations, and data processing workflows for performance and scalability.

Develop and support data solutions leveraging Azure Storage Accounts, ADLS Gen2, and Delta Lake.

Integrate Databricks with orchestration tools such as Apache Airflow for scheduling and monitoring data pipelines.

Deploy and manage Databricks notebooks, jobs, workflows, and libraries across Development, QA, UAT, and Production environments.

Troubleshoot data pipeline failures and ensure reliable data processing and delivery.

Collaborate with stakeholders and development teams to deliver business-driven data solutions.

Professional Technical Skills:
Must Have Skills:

Strong hands-on experience in Azure Databricks with Python and PySpark.

Good understanding of Apache Spark architecture, Spark configurations, performance tuning, and cluster management.

Experience with Azure Storage Accounts, Azure Data Lake Storage (ADLS Gen2), and Delta Lake.

Knowledge of authentication and authorization mechanisms, including Service Principals and Managed Identity.

Experience integrating Databricks with Apache Airflow for job scheduling and monitoring.

Understanding of Microservices architecture, REST APIs, and API integrations.

Experience deploying Databricks notebooks, workflows, and libraries across multiple environments.

Strong knowledge of Databricks Jobs, Workflows, and troubleshooting data pipeline failures.

Ability to develop scalable ETL/ELT pipelines using PySpark and Python.

Knowledge of ITSM processes and Production Support activities.

Additional Information:
The candidate should have a minimum of 11 years of experience in Data Engineering.
Experience in Azure Cloud-based Data Engineering solutions is preferred.
This position is based at our Bengaluru office.
A 15 years full-time education is required.


Qualification 15 years full time education