Data Engineer

Job Summary:

We are looking for a skilled Data Engineer to design, develop, and optimize enterprise-grade data pipelines within an on-premises, containerized data platform environment. The ideal candidate should have strong expertise in Apache Spark, Python, SQL/PL-SQL, and container orchestration technologies to support scalable data ingestion, transformation, and analytics workloads.

The role involves building and maintaining robust data solutions that enable Business Intelligence, Analytics, AI/ML initiatives, and regulatory reporting across the organization.

Key Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines using Apache Spark (PySpark/Scala).
• Develop data ingestion frameworks for structured, semi-structured, and unstructured data sources.
• Deploy and manage containerized data workloads within OpenShift or Kubernetes environments.
• Build and maintain Lakehouse architectures using Medallion Architecture (Bronze, Silver, Gold layers).
• Implement data quality validation, monitoring, and alerting mechanisms.
• Optimize Spark workloads for performance, scalability, and resource efficiency.
• Develop and manage workflow orchestration using Apache Airflow.
• Collaborate with Business, Analytics, DevOps, and Data Governance teams to deliver enterprise data solutions.
• Ensure adherence to data governance, compliance, and security standards.
• Participate in code reviews, troubleshooting, and performance tuning activities.

Required Skills:

Technical Skills:

• Strong experience in Python programming.
• Hands-on experience with Apache Spark (PySpark preferred; Scala is an advantage).
• Expertise in SQL, PL/SQL, and relational databases such as Oracle and PostgreSQL.
• Experience with Red Hat OpenShift (OCP) or Kubernetes.
• Knowledge of Apache Airflow for workflow orchestration.
• Experience with Git and CI/CD pipelines.
• Understanding of Data Warehousing concepts and Lakehouse architecture.
• Familiarity with Delta Lake or equivalent open-source storage formats.
• Experience with Data Quality frameworks and monitoring solutions.

Preferred Skills:

• Exposure to Medallion Architecture (Bronze, Silver, Gold).
• Experience working in large-scale enterprise data environments.
• Certifications in Spark, Kubernetes/OpenShift, Python, or Data Engineering technologies will be an added advantage.

Qualifications:

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
• Relevant certifications in Data Engineering technologies are preferred.

Soft Skills:

• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management abilities.
• Ability to work independently as well as in cross-functional teams.
• Strong ownership mindset with a focus on quality and timely delivery