Senior Data Engineer

Full time opportunity with Innodata : Sr. Data Engineer

Experience:

• 6 to 10 years of hands-on experience in Data Engineering, ETL development, and data platform implementation.
• Proven experience designing, building, and maintaining scalable data pipelines for both batch and real-time data processing.

Technical Skills

• Strong proficiency in SQL and Python.
• Experience with distributed data processing frameworks such as Apache Spark and PySpark.
• Hands-on experience with cloud platforms, including Microsoft Azure, AWS, or Google Cloud Platform (GCP).
• Experience with cloud-native data services such as:

Azure Data Factory

Azure Synapse Analytics

AWS Glue

Amazon Redshift

Google BigQuery

Equivalent cloud-native data services

• Hands-on experience with Databricks for data engineering and analytics workloads.

Strong understanding of:

• Data warehousing concepts
• Dimensional modeling
• Data lake and lakehouse architectures

Experience with workflow orchestration tools such as:

• Apache Airflow
• Azure Data Factory
• Similar scheduling and orchestration platforms

Knowledge of messaging and streaming platforms, including:

• Apache Kafka
• Azure Event Hubs
• Amazon Kinesis
• Experience working with both relational and NoSQL databases.

Data Engineering & Architecture

• Design, develop, optimize, and maintain scalable ETL/ELT pipelines.
• Build reliable, high-performance, and scalable data processing solutions.
• Implement data quality frameworks, validation rules, monitoring, and logging mechanisms.
• Optimize query performance, partitioning strategies, and storage for large-scale datasets.
• Ensure data security, governance, privacy, and compliance with organizational and regulatory standards.

DevOps & Automation:

Experience with Git-based version control systems.

Knowledge of CI/CD pipelines using:

• Azure DevOps
• GitHub Actions
• Jenkins
• Similar automation platforms

Familiarity with Infrastructure as Code (IaC) tools such as:

• Terraform
• ARM Templates
• AWS CloudFormation

Experience with containerization and orchestration technologies, including:

Docker

Kubernetes (preferred)

Key Responsibilities:

End-to-End Data Architecture: Design, build, and continuously evolve all layers of the data ecosystem, from multi-source data ingestion and transformation to semantic layers and business-ready data marts.

Scalable Data Pipelines: Develop and optimize high-volume ETL/ELT pipelines capable of processing complex structured, semi-structured, and unstructured datasets.

Data Modeling: Design and maintain robust dimensional models, including star schemas, to support enterprise-wide analytics and reporting.

Cross-Functional Collaboration: Partner with Product Managers, Architects, Delivery Managers, and business stakeholders to translate business requirements into scalable technical solutions.

Workflow Orchestration: Build, schedule, and manage resilient data workflows using Apache Airflow or equivalent orchestration platforms, ensuring high availability and fault tolerance.

Streaming & Real-Time Processing: Design and implement real-time data pipelines using technologies such as Apache Kafka, Amazon Kinesis, or equivalent streaming platforms.

Data Quality & Engineering Excellence: Establish and enforce best practices for code reviews, CI/CD, automated data testing, observability, and data quality to ensure reliable and trusted data assets.

Performance Optimization: Continuously optimize data processing, storage, and query performance to improve scalability and cost efficiency.

Mentorship & Leadership: Mentor junior and mid-level engineers, promote engineering best practices, and contribute to a culture of technical excellence and continuous improvement.