Data Engineer

Senior Data Engineer Developer

Job Summary

We are looking for a Senior Data Engineer with 5-10 years of experience to design, build, and optimize scalable data pipelines that power analytics, reporting, and application-level data needs. This role requires strong hands-on development skills combined with the ability to make sound architectural and design decisions, and collaborate closely with data architects, analysts, and business stakeholders to deliver reliable, high-performance data solutions.

Key Responsibilities

1. Data Pipeline Development

• Design, build, and maintain scalable ETL/ELT pipelines to ingest, transform, and load data from diverse sources (databases, APIs, files, streaming systems).
• Develop batch and real-time data processing workflows using modern data engineering frameworks.
• Optimize pipelines for performance, reliability, scalability, and cost-efficiency.
• Implement robust error handling, logging, retry logic, and monitoring for all data workflows.

2. Data Quality & Governance

• Implement data validation, cleansing, and reconciliation processes to ensure data accuracy and consistency.
• Establish and enforce data quality checks and monitoring/alerting frameworks.
• Support data governance initiatives including metadata management, lineage tracking, and documentation.

3. Integration & Automation

• Build integrations between internal systems, third-party APIs, and cloud services.
• Automate manual data processes to improve operational efficiency.
• Work with orchestration tools (Airflow, Azure Data Factory, dbt, etc.) to schedule and manage complex workflows.

4. Performance Optimization & Troubleshooting

• Tune SQL queries, pipeline jobs, and database performance for large-scale datasets.
• Troubleshoot and resolve data pipeline failures, latency issues, and data discrepancies.
• Conduct root cause analysis for recurring data issues and implement long-term fixes.

Required Qualifications

Experience

• 5-10 years of hands-on experience in data engineering, ETL development, or related backend/data roles.
• Proven experience designing and delivering production-grade data pipelines at scale.
• Experience working in Agile/Scrum development environments.

Technical Skills

• Strong proficiency in SQL (complex queries, performance tuning, stored procedures).
• Strong programming skills in Python, Scala, or Java for data processing.
• Hands-on experience with ETL/ELT tools (Informatica, Talend, SSIS, dbt, or similar).
• Experience with big data technologies (Spark, Hadoop, Kafka) is a strong plus.
• Proficiency with cloud data platforms (AWS Redshift/Glue, Azure Synapse/Data Factory, GCP BigQuery/Dataflow, Snowflake, or Databricks).
• Experience with orchestration/workflow tools (Airflow, Azure Data Factory, Prefect, Luigi).
• Solid understanding of relational and non-relational databases (PostgreSQL, MySQL, MongoDB, Cassandra).
• Familiarity with version control (Git) and CI/CD pipelines for data engineering workflows.
• Understanding of data modeling concepts (star schema, snowflake schema, normalization).