Databricks, Pyspark

Job Description:

• Join a high impact data engineering team where you ll shape modern analytics platforms and help turn raw data into trusted actionable insights
• In this role you ll lead the design and delivery of scalable data pipelines on Databricks using PySpark partnering closely with analysts data scientists and platform teams to enable faster decision making across the business
• You ll bring strong engineering discipline clean code performance tuning and reliable operations while guiding best practices and mentoring teammates
• If you enjoy solving complex data challenges optimizing distributed workloads and building systems that are resilient secure and easy to evolve this is a great opportunity to drive meaningful outcomes in a collaborative growth focused environment

Key Responsibilities:

• Key Responsibilities
• Lead the development of end to end data pipelines on Databricks using PySpark for batch and incremental processing
• Design scalable data models and curated datasets to support analytics and downstream consumption
• Write and optimize advanced SQL for transformations validations and performance critical queries
• Implement robust data quality checks reconciliation logic and monitoring to ensure trusted datasets
• Tune Spark jobs for performance and cost efficiency partitioning caching file formats cluster sizing
• Establish coding standards reusable frameworks and review practices to improve maintainability
• Collaborate with stakeholders to translate requirements into technical designs and delivery plans
• Troubleshoot production issues perform root cause analysis and drive preventive improvements
• Mentor team members and provide technical guidance across design implementation and optimization
• Minimum Qualifications
• BTECH MTECH MCA or MSC in Computer Science Information Technology or a related field
• 6 8 years of overall experience in data engineering or large scale data processing roles
• Strong hands on experience with PySpark for distributed data processing and transformation logic
• Strong hands on experience with Databricks for building running and managing data workloads
• Proficiency in Advanced SQL including complex joins window functions and query optimization
• Experience building reliable pipelines with strong focus on data quality performance and stability

Technical Requirements:

• Technology Analytics Solutions SQL Server Analytics
• Technology Big Data Data Processing PySpark
• Technology Data Engineering Databricks

Additional Responsibilities:

• Preferred Qualifications
• Experience designing lakehouse style architectures and organizing curated layers for analytics readiness
• Strong experience with Spark optimization techniques and handling large scale datasets efficiently
• Ability to build reusable PySpark utilities frameworks for ingestion transformation and validation patterns
• Experience with orchestration and scheduling approaches for dependable pipeline execution and recovery
• Proven track record of technical leadership mentoring conducting reviews and driving engineering best practices

Preferred Skills:

Technology->Data Engineering->Databricks,Technology->Big Data - Data Processing->PySpark,Technology->Analytics - Solutions->SQL Server - Analytics