Data Engineer

Job Purpose
We are seeking a highly skilled Python & Cloud Platform Engineer to support, modernize, and migrate Python-based applications currently running on Linux servers with load balancers to containerized environments on Kubernetes, specifically Azure Kubernetes Service (AKS) or Amazon Elastic Kubernetes Service (EKS). Log extraction and RCA debug using Azure Data Explorer Duties and Responsibilities

Translate business requirements into technical solutions in collaboration with the PMO team.

Own end-to-end delivery of data projects, ensuring on-time execution and adherence to quality standards.

Design technical architecture and guide development efforts for enhancements and new projects.

Develop and maintain robust ETL pipelines and data integration modules across systems.

Ensure high data quality, platform stability, and resolution of critical process issues.

Monitor and resolve performance bottlenecks in data workflows and programs.

Establish best practices, standard operating procedures, and drive their implementation across teams.

Act as a liaison with business users and product managers to support daily data needs and strategic initiatives.

Coordinate with internal and external development teams to troubleshoot and resolve issues efficiently.

Manage workload through effective planning, prioritization, and progress tracking.

Key Decisions / Dimensions

Making critical decisions during production issues, including root cause analysis, quick fixes, and long-term resolutions.

Prioritizing and escalating support tasks effectively to minimize downtime and business impact.

Driving decisions around technical design, architecture, and optimization to ensure performance, scalability, and maintainability of solutions.

Major Challenges

Balancing development timelines with on-time delivery in a dynamic and evolving technical environment.

Coordinating with multiple internal and external stakeholders to align priorities, resolve dependencies, and ensure smooth execution.

Ensuring code quality, data integrity, and performance while scaling data solutions across diverse systems and platforms.

Required Qualifications and Experience

Strong experience with Python development (FastAPI, Pyspark) - Must Have

Strong Linux system administration skills. - Must Have

Hands-on experience with Docker and Kubernetes.

Proven experience with AKS and EKS.

Experience with load balancers and traffic routing.

Understanding of microservices architecture.

Experience with Git, CI/CD pipelines, and automation tools.

Knowledge of cloud networking concepts.

Azure Databricks - PySpark, SQL

Azure Data Explorer as Time Series Database with Kusto Query Language (KQL) As Programming Language - Good to Have