Data Engineer

Hi Folks ,

We have Organised a Walkin drive for Below post at Indore office-

Super Corridor Rd, Tigaria Badshah, Indore, Madhya Pradesh 453112

Job Title: AWS Databricks Engineer

Exp - 5+ yeas

Location-Indore

We are seeking a skilled Data Engineer with deep expertise in AWS and Databricks to design, build, and optimize our next-generation data platform. In this role, you will implement scalable data pipelines, manage cloud infrastructure, and enable advanced analytics and machine learning capabilities. You will bridge the gap between raw data and actionable business insights by leveraging the power of the Databricks Lakehouse architecture on AWS.

Data Pipeline & Architecture Engineering

• Design and deploy robust, scalable data pipelines (ETL/ELT) using Databricks, Apache Spark (Python/Scala/SQL), and AWS services.
• Implement Lakehouse architectures utilizing Delta Lake to ensure data reliability, ACID transactions, and high performance.
• Optimize Spark jobs for performance, cost, and memory management within Databricks clusters.

AWS Cloud Integration

• Manage and integrate Databricks with AWS native services including Amazon S3, AWS Glue, IAM, Amazon EMR, RDS, and Redshift.
• Implement security and governance controls, ensuring secure data access via Unity Catalog, IAM roles, and encryption (KMS).

Operations & DevOps

• Automate deployments and workflows using CI/CD pipelines (e.g., GitHub Actions, GitLab, Jenkins) and Infrastructure as Code (Terraform).
• Orchestrate workflows using Databricks Workflows, Apache Airflow, or AWS Step Functions.
• Monitor platform health, cluster utilization, and cost optimization metrics to ensure efficient cloud spending.

Technical Skills

• Databricks Expertise: Strong experience with Databricks runtimes, Delta Lake, Unity Catalog, Photon engine, and Databricks SQL.
• AWS Cloud: Proficient with core AWS data and infrastructure services (S3, IAM, VPC, EC2, Lambda, Glue).
• Programming: Advanced programming skills in Python (PySpark) or Scala, along with expert-level SQL.
• Data Modeling: Solid understanding of data warehousing concepts, star/snowflake schemas, and dimensional modeling.
• DevOps/IaC: Experience with Terraform and version control (Git) for managing data infrastructure.

Experience & Qualifications

• Education: Bachelor's degree in Computer Science, Information Technology, Data Engineering, or a related field.
• Experience: 3+ years of hands-on experience building production-grade data pipelines on Databricks and AWS.
• Certifications (Preferred):
• Databricks Certified Data Engineer Professional / Associate
• AWS Certified Data Engineer – Associate or AWS Solutions Architect