AWS Data Engineer

Section III - Job Requirements And Responsibilities Section Details / Example Content Job Requirements* The successful candidate will be responsible for

• Design, build, and maintain data transformation pipelines using AWS services (Lambda Functions, Step Functions .. ) in Python.
• Design, build, and maintain data transformation pipelines using PySpark and Databricks. Implement and maintain complex calculation engine.
• Manage efficiently large dataset.
• Manage SQL database - Postgresql or MongoDB.
• Implement and maintain complex data calculation formula.
• Open to collaborate with Software Engineers.
• Participate in code reviews, design discussions, and software planning.
• Contribute to automation, testing, and deployment processes. Key Responsibilities*
• Design, build, and maintain data transformation pipelines using AWS services (Lambda Functions, Step Functions .. ) in Python.
• Design, build, and maintain data transformation pipelines using PySpark and Databricks. Implement and maintain complex calculation engine.
• Manage effeciently large dataset.
• Manage SQL database - Postgresql or MongoDB.
• Implement and maintain complex data calculation formula.
• Open to collaborate with Software Engineers.
• Participate in code reviews, design discussions, and software planning.
• Contribute to automation, testing, and deployment processes.

Section IV - Job Qualifications & Skills Section Details / Example Content Domain

• Any domain Soft Skills
• Excellent communication
• Team collaboration
• Documentation and knowledge sharing Education Requirements flexible Certifications Aws Certification(Optional) Section V - Sample Questions for Training Model

Technical: Python:

What’s the difference between a list and a generator? When would you use a generator in ETL?How do you read/process a large file in Python without loading it fully into memory?Explain mutable vs immutable objects with one example each.What is the difference between deep copy and shallow copy? When can shallow copy cause bugs?For I/O-bound vs CPU-bound tasks, which Python concurrency approach fits best and why? Pyspark:

What does lazy evaluation mean in Spark? Why is it useful?Explain the difference between repartition() and coalesce().What is data skew in Spark joins and one way to handle it?When would you use a broadcast join and what’s the risk?What’s the difference between cache() and persist()? Pandas:

Difference between merge() and concat() in pandas?How do you handle missing values (NaN) in pandas? Mention 2 methods.Why is vectorization faster than apply()/loops in pandas?How do you read a huge CSV efficiently in pandas (2 techniques)?What’s the difference between .loc[] and .iloc[]? Aws S3:

Why is S3 commonly used in data lakes? Mention 2 key benefits.What is a good S3 partition folder format for analytics datasets and why?How do you secure data in S3? Name two controls.What problem do many small files create in Spark/Athena, and what’s the fix?What is an S3 Lifecycle policy, and when would you use it? AWS Lambda Functions

Name two Lambda limits that impact data engineering workloads.What does idempotent processing mean for Lambda triggers from S3?How do you handle retries/failures in Lambda? Mention one AWS feature.When would you choose Step Functions over a single Lambda?How would you avoid throttling if 1000s of events arrive per minute? Sql:

What is a window function and one use-case in analytics?Explain the difference between INNER JOIN and LEFT JOIN.How do you remove duplicates using ROW_NUMBER() (high level)?What is the purpose of indexes, and when can they hurt performance?Difference between WHERE and HAVING? Section III - Job Requirements And Responsibilities Section Details / Example Content Job Requirements* The successful candidate will be responsible for

• Design, build, and maintain data transformation pipelines using AWS services (Lambda Functions, Step Functions .. ) in Python.
• Design, build, and maintain data transformation pipelines using PySpark and Databricks. Implement and maintain complex calculation engine.
• Manage efficiently large dataset.
• Manage SQL database - Postgresql or MongoDB.
• Implement and maintain complex data calculation formula.
• Open to collaborate with Software Engineers.
• Participate in code reviews, design discussions, and software planning.
• Contribute to automation, testing, and deployment processes. Key Responsibilities*
• Design, build, and maintain data transformation pipelines using AWS services (Lambda Functions, Step Functions .. ) in Python.
• Design, build, and maintain data transformation pipelines using PySpark and Databricks. Implement and maintain complex calculation engine.
• Manage effeciently large dataset.
• Manage SQL database - Postgresql or MongoDB.
• Implement and maintain complex data calculation formula.
• Open to collaborate with Software Engineers.
• Participate in code reviews, design discussions, and software planning.
• Contribute to automation, testing, and deployment processes.

Section IV - Job Qualifications & Skills Section Details / Example Content Domain

• Any domain Soft Skills
• Excellent communication
• Team collaboration
• Documentation and knowledge sharing Education Requirements flexible Certifications Aws Certification(Optional) Section V - Sample Questions for Training Model

Python: What’s the difference between a list and a generator? When would you use a generator in ETL?How do you read/process a large file in Python without loading it fully into memory?Explain mutable vs immutable objects with one example each.What is the difference between deep copy and shallow copy? When can shallow copy cause bugs?For I/O-bound vs CPU-bound tasks, which Python concurrency approach fits best and why?

Pyspark: What does lazy evaluation mean in Spark? Why is it useful?Explain the difference between repartition() and coalesce().What is data skew in Spark joins and one way to handle it?When would you use a broadcast join and what’s the risk?What’s the difference between cache() and persist()?

Pandas: Difference between merge() and concat() in pandas?How do you handle missing values (NaN) in pandas? Mention 2 methods.Why is vectorization faster than apply()/loops in pandas?How do you read a huge CSV efficiently in pandas (2 techniques)?What’s the difference between .loc[] and .iloc[]?

Aws S3: Why is S3 commonly used in data lakes? Mention 2 key benefits.What is a good S3 partition folder format for analytics datasets and why?How do you secure data in S3? Name two controls.What problem do many small files create in Spark/Athena, and what’s the fix?What is an S3 Lifecycle policy, and when would you use it? AWS Lambda Functions Name two Lambda limits that impact data engineering workloads.What does idempotent processing mean for Lambda triggers from S3?How do you handle retries/failures in Lambda? Mention one AWS feature.When would you choose Step Functions over a single Lambda?How would you avoid throttling if 1000s of events arrive per minute?

Sql: What is a window function and one use-case in analytics?Explain the difference between INNER JOIN and LEFT JOIN.How do you remove duplicates using ROW_NUMBER() (high level)?What is the purpose of indexes, and when can they hurt performance?Difference between WHERE and HAVING? Lyabda Functions

Technical: Skills: lyabda functions,python,pyspark,aws s3