Data Engineer

Job Summary

We are looking for a skilled Data Engineer with strong expertise in Apache Spark, Python, and cloud-based data engineering platforms. The ideal candidate will have hands-on experience developing scalable data pipelines, processing large datasets, and implementing data solutions on AWS, AWS Databricks, and Google Cloud Platform (GCP).

The candidate will be responsible for designing, developing, optimizing, and maintaining robust data processing workflows to support business intelligence, analytics, and data-driven applications.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Apache Spark and Python.
• Build and optimize batch and distributed data processing applications.
• Develop ETL/ELT workflows for data ingestion, transformation, cleansing, and integration.
• Implement data engineering solutions using AWS cloud services and Databricks.
• Develop and manage data processing workflows on Google Cloud Platform (GCP).
• Optimize Spark jobs for performance, scalability, resource utilization, and cost efficiency.
• Work with structured, semi-structured, and unstructured datasets.
• Implement data quality checks, error handling, logging, and monitoring.
• Collaborate with data scientists, analysts, architects, and business stakeholders.
• Follow best practices for data security, governance, access control, and cloud resource management.
• Troubleshoot production issues and ensure the reliability and availability of data pipelines.