Opening For Data Engineer@Walkingtree Tech

Must Have Skills:-

Python, Pyspark, AWS-Glue, Data Lke, Delta Lake

Key Roles and Responsibilities:

-Design and build ETL/ELT pipelines using Python and PySpark
-Process large-scale structured and semi-structured data (CSV, JSON, Parquet, Avro)
-Ensure pipelines are efficient, fault-tolerant, and scalable
-Develop distributed data transformations using PySpark DataFrames and Spark SQL
-Optimize Spark jobs (partitioning, caching, broadcast joins)
-Handle batch and (optionally) streaming data using Spark Structured Streaming
-Design data lake and data warehouse schemas (star, snowflake)
-Implement partitioning and file formats for performance (Parquet, ORC)
-Maintain data quality, consistency, and integrity
-Schedule and orchestrate pipelines using tools like:
-Apache Airflow, AWS Step Functions, or AWS Glue Workflows
-Automate data ingestion from APIs, databases, and streaming sources