Data Engineer | PySpark | Databricks | Azure Data Factory

Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark, Spark, Databricks, Azure Data Factory (ADF), and AWS Glue.
• Build and optimize ETL/ELT workflows for processing large-scale structured and unstructured data.
• Develop and manage cloud-based data solutions on AWS, Azure, and GCP platforms.
• Work with Databricks, Delta Lake, ADLS, S3, BigQuery, Redshift, and Snowflake for data ingestion, transformation, and storage.
• Implement data quality, monitoring, and performance optimization best practices.
• Collaborate with data analysts, data scientists, and business stakeholders to understand requirements and deliver scalable solutions.
• Ensure data security, governance, and compliance standards are followed across cloud platforms.
• Optimize Spark jobs and SQL queries for improved performance and cost efficiency.
• Support real-time and batch data processing frameworks.
• Participate in code reviews, troubleshooting, and production support activities.
• Create technical documentation and maintain data engineering standards and best practices.