Sr. Data Engineer

Sr. Data Engineer Location: India, Remote

Work Experience: 6+ Years

• 2+ years of hands-on experience with Databricks in designing, developing, and maintaining scalable data engineering solutions.
• Strong expertise in Apache Spark / PySpark, SQL, and Python for large-scale data processing and transformation.
• Strong understanding of Delta Lake and Lakehouse architecture, including data modeling, optimization, and reliable data storage patterns.
• Experience with workflow orchestration, including designing and managing data pipelines, job scheduling, dependencies, and monitoring.
• Strong knowledge of data quality, validation, reconciliation, and error-handling practices to ensure data accuracy and reliability.
• Experience working with heterogeneous enterprise data sources, including structured, semi-structured, and unstructured data.
• Azure data-platform experience is strongly preferred, particularly with Azure-based data engineering and analytics services.
• EDI knowledge is not mandatory and will be considered an added advantage.

Qualifications: Bachelors degree in computer science, Information Technology, Engineering, or equivalent.

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, SQL, and Python to support enterprise data-processing requirements.
• Build and optimize Lakehouse solutions using Databricks and Delta Lake, ensuring scalability, performance, reliability, and maintainability.
• Ingest and process data from diverse enterprise sources, including structured, semi-structured, and unstructured data, while handling different formats and data complexities.
• Develop and manage workflow orchestration for data pipelines, including scheduling, dependency management, monitoring, failure handling, and operational support.
• Implement data-quality and validation frameworks to perform data profiling, reconciliation, completeness checks, accuracy validation, and anomaly detection.
• Optimize Spark/PySpark and SQL workloads by applying appropriate partitioning, caching, query optimization, Delta Lake optimization, and other performance-tuning techniques.
• Collaborate with data engineers, architects, analysts, and business stakeholders to understand data requirements and deliver secure, reliable, and scalable Azure-based data solutions.