Staff Engineer, Big Data

Total experience: 5.5+ years.
- Strong experience in Data Engineering, Cloud Engineering, or Big Data Engineering.
- Must-have expertise in Google BigQuery, Python, SQL, PySpark, GCP fundamentals, and Kubernetes.
- Strong hands-on experience with BigQuery and SQL, including writing and optimizing complex queries for large-scale data processing.
- Strong programming experience in Python, with hands-on experience in developing scalable applications and APIs.
- Strong experience with PySpark/Spark and familiarity with Big Data technologies such as Hive.
- Hands-on experience with Apache Airflow or similar orchestration services for building and managing data workflows.
- Experience with GCP serverless services, particularly Cloud Functions and Cloud Run.
- Good experience with Docker and Kubernetes for containerization, deployment, orchestration, scaling, and troubleshooting.
- Experience with Python API frameworks such as FastAPI, Flask, or Django; FastAPI is preferred.
- Experience with CI/CD pipelines, preferably using GitLab CI/CD and Octopus Deploy, along with Git-based version control.
- Good understanding of Terraform and Infrastructure as Code (IaC) for provisioning and managing cloud resources.
- Good understanding of GCP networking, VPC, load balancing, IAM, API security, monitoring, logging, and alerting.
- Strong troubleshooting, analytical, problem-solving, communication, and collaboration skills.

RESPONSIBILITIES:

- Design, implement, and maintain scalable and reliable Big Data and cloud solutions using GCP, BigQuery, PySpark, Python, and Airflow.
- Develop and maintain data workflows and pipelines using Apache Airflow and other orchestration services.
- Design, develop, and optimize BigQuery and SQL queries for high-volume data processing and analytics workloads.
- Develop and deploy cloud-native and serverless applications using GCP Cloud Functions and Cloud Run.
- Develop scalable APIs using Python frameworks such as FastAPI, Flask, or Django.
- Containerize applications using Docker and deploy, manage, and troubleshoot workloads on Kubernetes.
- Implement and maintain CI/CD pipelines for automated build, testing, and deployment using GitLab CI/CD, Octopus Deploy, or similar tools.
- Implement Infrastructure as Code (IaC) using Terraform to provision and manage GCP cloud resources.
- Configure and manage IAM policies, VPC networking, load balancing, API access controls, and security mechanisms.
- Implement monitoring, logging, alerting, and observability solutions to proactively identify and resolve system issues.
- Write comprehensive unit tests and implement quality practices to ensure application reliability and maintainability.
- Troubleshoot production issues, perform root cause analysis, and implement preventive measures to improve system stability.
- Collaborate with Data Engineering, Cloud, DevOps, Infrastructure, Security, and Application teams to deliver scalable and reliable solutions.
- Continuously improve data pipelines, cloud architecture, automation, application performance, security, and operational efficiency through industry best practices. Qualifications

Bachelor s or master s degree in computer science, Information Technology, or a related field