Senior Data Engineer

Data Engineer – Python & Pyspark (AWS)

Exp: 6+ yr sLocation : Chennai Client Place Guind

yWork Mode : All 5 day s

Role Overvi ewWe're seeking a Senior Data Engineer with strong hands-on expertise in Python, PySpark, and AWS to design, build, and operate scalable data pipelines for analytics and operational workloads. You will own end-to-end data engineering — from ingestion to delivery — across batch and near-real-time patterns, with a focus on pipeline reliability, data quality, and cloud-native engineerin g.This is an engineering role — you will build and own production systems, not just develop queries or reports. Individual contributor expected to influence architecture, drive technical decisions, and mentor peers without formal people management responsibilitie

s. Key Responsibilit

• iesDesign, build, and maintain scalable ETL/ELT pipelines using Python and PySpark to ingest data from files, APIs, RDBMS, NoSQL, and message queues into AWS data sto
• resDevelop and optimize PySpark jobs for large-scale data processing, transformation, and aggregation on AWS EMR or Databri
• cksBuild and manage AWS-native data workflows using S3, Glue, Lambda, Redshift, and Step Functions; manage scheduling, dependencies, retries, and S
• LAsWrite production-grade Python for data transformations, validations, orchestration, and API integrations — not notebook-level script
• ingImplement robust data quality controls: schema validation, deduplication, referential integrity checks, and anomaly detect
• ionCollaborate on data modeling (star/snowflake, data vault, medallion architecture) and define data contracts and line
• ageBuild CI/CD for data pipelines (tests, linting, packaging) using GitHub Actions or AWS CodePipeline; maintain documentation and runbo
• oksTroubleshoot and remediate production issues; drive continuous improvement in performance, reliability, and c

ost Required Qualificat

• ions6–8 years of hands-on data engineering experience with a strong focus on Python, PySpark, and
• AWSProduction-grade Python skills — modular code, logging, error handling, REST/GraphQL API integration; not scripting or notebook-only
• workStrong PySpark expertise — job optimization, partitioning, caching, broadcast joins, performance tuning at s
• caleHands-on AWS experience: S3, Glue, Lambda, Redshift, EMR, Step Functions,
• IAMAdvanced SQL skills — complex joins, window functions, query optimization; Oracle or PostgreSQL experience is a
• plusSolid understanding of ETL/ELT patterns, data modeling, and data quality framew
• orksExperience with workflow orchestration: Apache Airflow, AWS Step Functions, or equiva
• lentCI/CD experience: GitHub Actions, AWS CodePipeline, Azure DevOps, or GitLa
• b CIStrong engineering mindset — you build it, you own it, you operat

e itNice-to-

• Have
• Experience with Kafka, Kinesis, or Event Hubs for event stre
• amingFamiliarity with dbt for ELT and data te
• stingGreat Expectations or Deequ for data qu
• alityDocker / Kubernetes for containerized data ser
• vicesKnowledge of data security patterns: PII masking, tokenization, column/row-level sec
• urityInformatica, ODI, or SSIS exp

osureEducation & Certifica

• tions
• Bachelor's/Master's in Computer Science, Engineering, or equi
• valentAWS Certified Data Engineer, AWS Solutions Architect, Databricks, or Snowflake certifications are a plus