NOIDA · FULLTIME
Lead Data Engineer
Iris Software
Noida · onsite · Posted 30d ago
Sourced from
Undisclosed10–12 yrsfulltimeNoida
About a minute. You'll make an account on the way through.
agent · working
- Read this posting0.2s
- Found the form on Naukri · 12 questions to answer1.1s
- Decided the answers for all of them2.7s
- Tailor your résumé to this role
- Waiting for you
Section · 01
About this role
Lead Data Engineer
Location: Noida, UP, India
Job Description - Must Have - Python & Advanced SQL
- Spark / distributed processing / PySpark
- Cloud platforms (AWS) Modules like Lambda, Glue, EMR, Dynamo DB, S3, Redshift, EC2, RDS, DMS, Step Functions, Cloud Watch, Systems Manager, Event Bridge, DMS, Boto3 APIs
- Data pipeline automation
- Data governance & quality. Preferred if worked on AWS Data Zone.
- Any orchestrator like Airflow, Step Functions
- Knowledge of any git repository like GitHub, GitLab, Bitbucket.
- NICE TO HAVE - Kafka/Kinesis streaming
- Basic knowledge of AI
- Power BI
- Postgre SQL
- DBT
- Experience with CI/CD implementation using Terraform or TypeScript.
- Job Summary - We are seeking a Lead Data Engineer who is able to develop and maintain ETL Pipelines preferably on AWS cloud. Should be able to lead a project that includes working with client and come up with suggestions as and when required. Oversee data pipelines, data lakes, data warehouses, and ETL/ELT processes. Establish data quality standards, monitor system health and drive continuous improvements. Should be proficient in Python, advanced SQL and PySpark. Sound understanding of Spark architecture. Basic understanding of AI is essential. Knowledge of DBT, Postgre database and Power BI would be an added advantage.
- Hands on in Python and advanced SQL.
- Data modelling with good understanding of dimensions and facts.
- Good understanding of Spark eco-system and PySpark programming.
- Able to convert existing SSIS workflows to ETL pipelines running on AWS. That includes knowledge of Lambda, Glue or EMR, Systems Manager, Event Bridge, EC2, DMS
- Knowledge of Dynamo DB, Redshift, RDS (Preferably Postgre)
- Sound knowledge of AWS Boto3 APIs
- Awareness of reporting tool like Power BI.
- Monitor and troubleshoot migration processes, resolving issues in a timely manner.
- Work closely with business stakeholders, developers, and administrators to understand data requirements.
- Good understanding of a Data Lakehouse
- Understanding of CI/CD process with exposure to TypeScript and any Git repository like GitHub, GitLab or Bitbucket
- Document migration procedures, processes, and best practices.
- Bachelors degree in Computer Science, Information Technology, or related field.
- 8+ years of experience in data migration or similar roles.
- Strong proficiency in SSIS.
- Proven experience in ETL tools like SSIS and data migration pipelines build on AWS.
- Familiarity with data modeling and schema design principles.
- Ability to analyze and resolve complex data-related issues.
- Strong analytical and problem-solving abilities.
- Excellent communication and collaboration skills.
- Ability to work independently and in a team-oriented environment.
Familiarity with Agile project management methodologies.
Mandatory Competencies - Data Science and Machine Learning - Data Science and Machine Learning - Python
- Data Science and Machine Learning - Data Science and Machine Learning - Apache Spark
- Data & AI - Data Engineering - Data Quality & Validation
- Big Data - Big Data - Pyspark
- Database - Database Programming - SQL
- Data Science and Machine Learning - Data Science and Machine Learning - Databricks
- Beh - Communication and collaboration
Sourced from naukri
Section · 02
Skills
AutomationAgileApache SparkData Engineercloud platformsapisApacheagile project managementadvanced sqlamazon web servicesapache airflowbitbucketAnalyticsbusiness automationData EngineeringcloudArchitectureData GovernanceBig DataArtificial Intelligence
Section · Company
About Iris Software
Iris Software
IT Services & Consulting
4.3k+
employees
1991
35 years old
Edison, New Jersey
United States (USA)
About
IT company for best tech-solutions to enhance business operations
Industries
IT Services & Consulting
Employee ratings
1,895 reviews
3.8/ 5
Culture
3.7
Career growth
3.3
Work-life
3.8
Employees rate it well for
Work Life Balance381Work Culture267Senior Management254Learning202Salary156
Find them on