Data Engineer

Role Overview

Looking for a hands-on Data Engineer to design, develop and maintain scalable data pipelines, data-processing services and integration solutions supporting enterprise applications, analytics and AI-enabled solutions.

The candidate should possess strong Python and SQL programming skills, along with practical experience in data ingestion, ETL/ELT, API integration, relational and NoSQL databases, data modeling and data-quality engineering.

The role requires someone who can independently develop production-quality data components while collaborating with application developers, architects, AI/ML engineers, business analysts and other engineering teams.

Key Responsibilities

• Design, develop and maintain scalable data ingestion, transformation and processing pipelines.
• Develop production-quality data engineering components using Python.
• Write complex and optimized SQL queries for data extraction, transformation, validation and analysis.
• Develop and maintain ETL/ELT pipelines for structured and semi-structured data.
• Integrate data from databases, enterprise applications, REST APIs, files and other data sources.
• Develop reusable Python modules for ingestion, transformation, validation, logging and exception handling.
• Design and implement relational database schemas and data models.
• Work with SQL and NoSQL databases based on application requirements.
• Implement data-quality controls including validation, reconciliation, deduplication and completeness checks.
• Develop and consume REST APIs for data integration.
• Work with enterprise search technologies such as Elasticsearch/OpenSearch or Apache Solr where required.
• Optimize database queries and pipelines for performance and scalability.
• Implement appropriate error handling, retry, logging and monitoring mechanisms.
• Develop automated unit, integration and data-quality tests.
• Troubleshoot data, database, API and pipeline-related issues across development and production environments.
• Follow Git-based development, code-review and CI/CD practices.
• Work collaboratively with architects and engineering teams to translate requirements into technical implementations.
• Create and maintain appropriate technical documentation.

Required Technical Skills

Skill Area Expected Proficiency

Python Strong Mandatory

SQL Strong/Advanced – Mandatory

Data Engineering ETL/ELT, ingestion, transformation, validation

Skill Area Expected Proficiency

Python Libraries Pandas, NumPy, PyArrow or equivalent

Relational Databases SQL Server / PostgreSQL / MySQL

NoSQL MongoDB / document databases or equivalent

API Integration REST APIs, JSON, authentication

Data Modeling Relational modeling, schema design, dimensional concepts

Data Formats JSON, CSV, XML, Parquet

Search Technologies Elasticsearch/OpenSearch, Apache Solr

Version Control Git / GitHub / GitLab

Testing PyTest, unit testing, integration and data-quality testing

Linux Shell commands, scripting and troubleshooting

Containerization Docker

CI/CD Basic understanding of automated build/test/deployment pipelines

SQL Skills

The candidate should have practical experience with:

Complex Joins • CTEs • Subqueries • Window Functions • Aggregations • Views • Stored Procedures • Transactions • Indexes • Query Execution Plans • Query Optimization

The engineer should understand how database design and indexing decisions affect application and pipeline performance.

Data Engineering Skills

The candidate should understand an end-to-end data flow such as:

Source Systems Ingestion Validation Transformation Data Quality Storage Search/Analytics/Application Consumption

Experience should include:

• Batch data processing
• Incremental data loading
• ETL/ELT
• Schema validation
• Data cleansing
• Deduplication
• Reconciliation
• Error handling
• Retry/recovery
• Data lineage fundamentals
• Metadata handling

AI/GenAI exposure – desirable

Educational Qualification

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, Data Science or a related technical discipline, or equivalent practical experience.

Professional Competencies

The candidate should demonstrate:

• Strong analytical and problem-solving skills
• Good programming discipline
• Ability to independently troubleshoot technical issues
• Understanding of software engineering best practices
• Ability to work within Agile development teams
• Effective communication and collaboration
• Ability to understand technical requirements and translate them into working solutions
• Willingness to learn emerging data and AI technologies