Lead Data Engineer

We are seeking a highly experienced Principal AWS Data Engineering Architect with deep expertise in AWS Data Engineering, Python Framework Development, Big Data Processing, Data Architecture, Automation, and Cloud Engineering. The ideal candidate will be a hands-on technical leader responsible for designing enterprise-scale data platforms, automation frameworks, and highly scalable cloud-native data solutions while driving innovation, governance, and engineering excellence.

This role requires a strong blend of Data Architecture, AWS Engineering, DevSecOps, Automation Framework Development, and technical leadership capabilities.

Key Responsibilities

Data Architecture & Engineering

• Design and architect large-scale enterprise data platforms and cloud-native data ecosystems.
• Build scalable, resilient, and high-performance data pipelines on AWS.
• Design modern Data Lakes, Data Warehouses, Data Assets, and Data Products.
• Drive enterprise Data Engineering standards and best practices.
• Own end-to-end data engineering lifecycle from requirements gathering through support and operations.

AWS Cloud Engineering

• Design and implement solutions leveraging AWS services including:
  
• S3
• Lambda
• Glue
• Athena
• EMR
• Redshift
• DynamoDB
• Kinesis
• EventBridge
• DMS
• Step Functions
• SQS
• SNS
• ECS
• EKS
• Fargate
• SageMaker
• CloudWatch
• Lead cloud modernization and automation initiatives.
  
• Establish scalable cloud architecture patterns and reusable engineering frameworks.
  

Automation & Framework Development

• Design and develop advanced Python-based automation and engineering frameworks.
  
• Build reusable components for:
  
• Data ingestion
• Data transformation
• Orchestration
• Monitoring
• Testing
• Governance
• Automate operational and engineering processes across platforms.
  
• Drive framework development using SDKs, CDKs, APIs, and automation tooling.
  

Big Data & Analytics

• Develop high-performance PySpark and Apache Spark solutions.
  
• Optimize big data applications for throughput, scalability, and cost efficiency.
  
• Work with data formats including:
  
• Parquet
• Avro
• ORC
• CSV
• JSON
• XML
• Implement data modeling, warehousing, and analytics architectures.
  

Leadership & Strategy

• Define technical strategy and roadmap for enterprise data engineering platforms.
• Lead Proof of Concepts (PoCs), innovation initiatives, and modernization programs.
• Mentor senior engineers and architects.
• Drive enterprise architecture decisions and technical governance.
• Partner with executive stakeholders and business leaders.

Security & Governance

• Implement enterprise security best practices including:
  
• Encryption at rest
• Encryption in transit
• SSL/TLS
• Tokenization
• Data masking
• IAM governance
• Establish metadata management and data governance standards.
  
• Ensure compliance with enterprise security and regulatory requirements.
  

Required Skills

• 8-15 years of experience in Data Engineering and Data Architecture.
  
• Extensive AWS Data Engineering experience.
  
• Expert-level Python development and framework engineering.
  
• Strong PySpark and Apache Spark expertise.
  
• Strong SQL and Data Modeling experience.
  
• Experience designing enterprise-scale Data Warehouses and Data Lakes.
  
• Strong AWS expertise across:
  
• Glue
• EMR
• Redshift
• Athena
• Lambda
• Kinesis
• DynamoDB
• SageMaker
• EventBridge
• DMS
• S3
• ECS/EKS
• CloudWatch
• Infrastructure as Code using:
  
• Terraform
• CloudFormation
• CDK
• Experience with:
  
• Airflow
• DBT
• Jenkins
• Artifactory
• Docker
• Kubernetes
• Strong DevSecOps and CI/CD experience.
  
• Expertise in Data Governance and Metadata Management.
  
• Linux/Unix administration experience.
  
• Experience with large-scale distributed systems.
  

Preferred Skills

• Apache Flink.
• AWS MSK (Kafka).
• Apache Iceberg.
• Agentic AI.
• Generative AI.
• Large Language Models (LLMs).
• Data Products Architecture.
• Enterprise Architecture experience.
• Erwin Data Modeling tools.
• Financial Services and Banking domain experience.

Desired Candidate Profile

• Strong technical leadership capabilities.
• Excellent communication and stakeholder management skills.
• Ability to influence enterprise-wide architectural decisions.
• Strong innovation mindset with focus on automation and modernization.
• Experience managing large-scale engineering initiatives.
• Passion for emerging technologies and cloud innovation.