SENIOR SOFTWARE ENGINEER - ETL

• Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and PySpark.
• Develop and optimize data transformation logic using PySpark, Spark SQL, and SQL.
• Build and maintain Delta Lake tables following Bronze, Silver, and Gold/Medallion architecture patterns.
• Implement Delta Lake capabilities such as:
  • MERGE operations
  • Schema enforcement and schema evolution
  • Time Travel
  • OPTIMIZE and VACUUM
  • Incremental data processing
• Develop and manage Databricks Workflows/Jobs for pipeline orchestration and scheduling.
• Troubleshoot pipeline failures, performance issues, and data quality problems.
• Optimize Spark workloads by understanding partitioning, joins, caching, shuffle operations, and query execution.
• Develop reusable data engineering utilities and frameworks using PySpark and basic Python.
• Write complex SQL queries for data transformation, reconciliation, validation, and analysis.
• Integrate Databricks pipelines with external systems using REST APIs.
• Handle inbound and outbound file integrations using SFTP.
• Work with common file formats such as CSV, JSON, Parquet, and Delta.
• Work with AWS services used within the data engineering ecosystem, particularly Amazon S3, and have basic awareness of services such as IAM, Secrets Manager, CloudWatch, and Lambda.
• Implement proper exception handling, logging, auditing, monitoring, and restart/recovery mechanisms.
• Participate in code reviews and follow coding and data engineering best practices.
• Collaborate with architects, business analysts, source-system teams, QA teams, and other engineering teams.
• Support production deployments and troubleshoot production data pipeline issues.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.