Lead Big Data Engineer

About the Role:

The Team: Our enterprise data management team is at the forefront of designing and developing cutting-edge data solutions that drive strategic business outcomes across the organization. We value quick learners who thrive in a dynamic technology environment and can seamlessly transition between collaborative teamwork and independent problem-solving. The team fosters a culture of continuous learning and innovation, where members tackle emerging technologies while maintaining strong analytical rigor and effective cross-functional communication.

Responsibilities and Impact:

• Design and develop scalable data processing pipelines using distributed computing frameworks to handle large-scale enterprise datasets
• Build and maintain ETL processes that extract, transform, and load data into strategic information products supporting organizational goals
• Develop data-intensive applications and services using modern programming languages and cloud-based analytics platforms
• Implement stream processing solutions using real-time data processing technologies to support business-critical operations
• Collaborate with cross-functional teams to translate complex business requirements into robust technical solutions
• Provide production support and troubleshooting for data systems, ensuring high availability and performance

What We're Looking For:

Basic Required Qualifications:

• 8+ years of hands-on experience in technology application development and production support
• 6+ years of experience developing data pipelines that extract, transform, and load data using programming languages such as Python, Scala, or Java
• Minimum 3+ years of experience developing and supporting ETL processes using cloud-based analytics platforms (such as Databricks, Snowflake, or Azure Synapse)
• Experience building data-intensive applications using modern programming technologies (including but not limited to C#, Java, Python, Scala, and SQL)
• Proficiency with cloud computing environments (such as AWS, Azure, or Google Cloud Platform)

Additional Preferred Qualifications:

• Experience with stream processing technologies (such as Apache, Apache Pulsar, or Amazon Kinesis) for real-time data processing
• Hands-on experience with distributed processing frameworks (such as Apache Spark, Dask, or Hadoop MapReduce) for large-scale data analytics
• Knowledge of data integration tools (such as Apache NiFi, Talend, or Informatica) and database replication technologies
• Experience with CI/CD pipelines, containerization technologies (such as Docker, Podman, or containerd), and orchestration platforms (such as Kubernetes, Docker Swarm, or OpenShift)