Aws Data Engineer

Roles and Responsibilities:

• Develop and maintain scalable data engineering pipelines using Python and PySpark.
• Develop data processing workflows using Databricks and Apache Spark.
• Ingest data from Teradata, databases, APIs, files, and cloud storage into AWS-based data platforms.
• Write and optimize SQL queries for data extraction, transformation, and analysis.
• Build and maintain data pipelines using AWS services such as S3, Glue, Lambda, and Redshift.
• Develop and manage Databricks notebooks, jobs, clusters, and workflows.
• Extract and transform data from Teradata and migrate or integrate it with cloud-based data platforms.
• Use Git for version control and collaborative development.

Must-Have Skills

• Strong proficiency in Python for data processing and automation.
• Strong hands-on experience with PySpark and Apache Spark.
• Hands-on experience with Databricks for data engineering and analytics.
• Strong knowledge of SQL, including complex queries, joins, CTEs, window functions, and optimization.
• Strong experience with AWS cloud services, particularly S3, Glue, Lambda, and Redshift.
• Good hands-on experience with Teradata and Teradata SQL.
• Strong understanding of ETL/ELT concepts and data pipeline development.
• Experience working with large-scale data processing and distributed computing.
• Knowledge of AWS data lake and data warehouse architecture.
• Experience with Databricks workflows, jobs, notebooks, and cluster management.
• Strong analytical, problem-solving, and communication skills.