Data Engineer

Requirements:

• 4+ years of overall IT experience, including hands-on experience with big data technologies.
• Strong hands-on experience in Python and PySpark.
• Python should be used extensively for application development, ETL (Extract, Transform, Load) processes, and Data Lake curation.
• Experience in building PySpark applications using Spark DataFrames in Python with development tools such as Jupyter Notebook and PyCharm (IDE).
• Proven experience in optimizing Spark jobs that process large-scale datasets and high-volume workloads.
• Hands-on experience with version control systems, particularly Git.
• Experience working with AWS Analytics services, including Amazon EMR, Amazon Athena, and AWS Glue.
• Experience with AWS Compute and Storage services, including AWS Lambda, Amazon EC2 and Amazon S3
• Other AWS services such as Amazon SNS.
• Working knowledge of Bash/Shell scripting is an added advantage.
• Experience in designing and building ETL pipelines to ingest, copy, cleanse, and transform data across multiple formats, including CSV, TSV, XML, and JSON.
• Experience working with various file formats, including fixed-width files, delimited files, and multi-record file formats.
• Good understanding of data warehouse concepts, including facts and dimensions, star schema, and snowflake schema.
• Experience with columnar storage formats, such as Parquet, Avro, and ORC.
• Strong knowledge of data compression techniques, including Snappy and Gzip.
• Good to have experience with at least one AWS database service: Amazon Aurora, Amazon RDS, Amazon Redshift, Amazon ElastiCache, or Amazon DynamoDB
• Good to have experience with AI/ML.