Data Engineer - Databricks

Job Summary

Job Description - Data Engineer (AWS + Databricks). Experience: 5+ Years. Location: Flexible / Hybrid.

Key Responsibilities

• Build scalable ETL/ELT pipelines using Databricks (PySpark, Spark SQL, Delta Lake).
• Design and implement Medallion Architecture (Bronze, Silver, Gold) for Lakehouse.
• Develop batch and real-time pipelines using Spark Streaming, AWS Kinesis, or Glue Streaming.
• Manage data ingestion from diverse sources - RDBMS, APIs, S3, Kafka.
• Perform data profiling, validation, and quality checks.
• Optimize Spark jobs, partitions, and cluster configurations for performance.
• Implement and manage data infrastructure using AWS S3, Glue, Athena, Lambda, Step Functions, Redshift, EMR.
• Set up CI/CD pipelines (GitHub Actions, CodePipeline, Jenkins).
• Collaborate with analysts and data scientists to deliver analytical datasets.

Technical Skills

• Databricks: PySpark, Spark SQL, Delta Lake, Unity Catalog, Job Workflows.
• AWS: S3, Glue, Athena, Kinesis, Lambda, Step Functions, EMR, Redshift, CloudWatch.
• Programming: Python (pandas), SQL (window functions, optimization).
• IaC DevOps: Terraform / CloudFormation, Git, Jenkins, GitHub Actions.
• Performance Tuning: Spark optimizations, Z-ordering, partitioning, caching.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.