Senior QA Data Engineer

Senior QA Data Engineer - Job Description

(Irrelevant profiles won't be considered)

Job Title: Senior QA Data Engineer 
Location: Mumbai
Experience Level: 5+ Years

Key Responsibilities

• Plan, design, and execute comprehensive testing strategies for new features, enhancements, and changes across existing systems.
• Take a hands-on role in creating, maintaining, and executing both manual and automated test cases, ensuring thorough test coverage.
• Champion test automation best practices, designing and delivering scalable and maintainable automation solutions that support continuous delivery.
• Develop and execute functional and non-functional test scenarios across various stages of the software development lifecycle to validate system quality, performance, and reliability.
• Identify, analyse, and troubleshoot critical defects, performing root cause analysis and working closely with development and cross-functional teams to ensure timely resolution.
• Produce clear, detailed, and well-structured defect reports that effectively demonstrate issues and support efficient remediation.
• Communicate testing progress, risks, and quality insights to stakeholders at different levels, tailoring communication to suit technical and non-technical audiences.

Knowledge & Experience Required:

Data Quality & Validation (Azure Data Platforms)

• Extensive experience validating data pipelines in Azure Data Factory (ADF) and Azure Databricks, ensuring data accuracy, reliability, and consistency across complex data workflows.
• Strong experience testing ETL/ELT processes, including data ingestion, transformations, data movement, and schema validation.
• Perform data completeness, consistency, and reconciliation checks between source and target systems.
• Conduct data profiling to identify anomalies, missing values, duplicates, and data integrity issues.
• Validate data lineage, auditing, and monitoring processes using logging and observability tools.

Test Automation for Data Platforms

• Experience designing and implementing automated data validation frameworks using Python (PyTest, PySpark, Pandas) or C#.
• Build reusable automation scripts to validate data pipelines, transformations, and integrations in Azure environments.
• Develop integration tests for SQL queries, data transformations, and data workflows to ensure reliability and scalability.

SQL & Database Testing

• Advanced proficiency in SQL for validating business rules, transformations, and complex data workflows.
• Perform source-to-target data validation, verifying data accuracy following ETL transformations.
• Validate database structures, indexes, constraints, stored procedures, and query performance for large-scale datasets.
• Business Intelligence & Reporting Testing
• Extensive experience testing BI solutions including Power BI, SAP Business Objects, and Crystal Reports.
• Validate BI dashboards, reports, datasets, and data models against underlying data sources.
• Experience working with Power BI data models, DAX queries, and tools such as DAX Studio.
• Experience testing Power Platform solutions, including Power Apps and Dataverse.

CI/CD & Data Pipeline Testing

• Integrate automated data validation tests into CI/CD pipelines using Azure DevOps or GitHub Actions.
• Implement automated data quality checks within ADF pipelines and Databricks workflows.
• Support deployment validation for ETL pipelines, data models, and transformation workflows.

Data Warehouse & Traditional ETL Tools

• Experience testing data warehouse solutions and traditional ETL tools, including SAP Business Objects Data Services, Informatica, and SSIS.
• Strong understanding of data warehouse architectures, data modelling, and enterprise data platforms.

Quality Engineering & Collaboration

• Extensive experience testing enterprise data platforms, software systems, and data warehouse solutions with a strong focus on test automation and quality engineering practices.
• Solid understanding of the testing pyramid and implementing testing strategies across different layers of the application stack.
• Experience working in Agile development environments, actively contributing to sprint ceremonies and quality practices.
• Proven ability to mentor QA engineers, promote automation best practices, and conduct code reviews.
• Strong collaboration with Business Analysts and Developers to refine requirements, define acceptance criteria, and participate in Three Amigos sessions.
• Excellent analytical, problem-solving, and innovative thinking skills when identifying and resolving complex issues.

Essential skills:

• Cloud & Data Platforms: Azure Data Factory, Azure Databricks, Azure SQL, Azure Synapse Analytics
• Databases & Storage: SQL Server, Delta Lake
• Programming & Scripting: Python (PySpark, Pandas), SQL, C#, REST APIs, DAX
• Automation & Testing Frameworks: PyTest, Great Expectations, DataDiffPy, Postman
• CI/CD & DevOps: Azure DevOps, GitHub Actions
• Monitoring & Observability: Azure Monitor, Log Analytics, Databricks Monitoring
• Business Intelligence & Power Platform: Power BI (dashboards, reports, datasets), Power Apps, Dataverse
• Database Testing: Advanced SQL for data validation, transformation testing, and performance analysis
• Collaboration & Soft Skills: Strong teamwork, communication, and collaboration within cross-functional Agile teams

Desired Experience:

• The ideal candidate will have financial services experience in the private equity, infrastructure & real assets, or private debt space. However, this is not a stringent requirement.

Desired skills:

• Experience working with AI-driven or intelligent data agents, including validating and testing agent-based workflows that interact with enterprise data platforms.
• Exposure to Databricks as a primary data source, including testing data pipelines, queries, and integrations that support AI or agent-based solutions.
• Familiarity with testing AI/ML-enabled systems, including validation of agent behaviour, data retrieval accuracy, and response reliability across data-driven environments.
• Experience working with private markets or financial services platforms, such as eFront or similar private equity / investment management systems.
• Exposure to AI/LLM enabled/accelerated testing and engineering practices.