Senior Data Engineer

Qualification

• 3-5 Years of overall IT experience as a Data Engineer with hands on expertise in building data pipeline, data transformation and metadata creation involving large complex and disconnected data sets.
• Graduate in Computer Science, Statistics, Informatics, Information Systems or another quantitative field
• Data Engineering Certification from GCP.

Job Description

• Assemble large, complex data sets that meet functional / non-functional business requirements.
• Build processes supporting data transformation, data structures, metadata, dependency and workload management.
• Hands on in manipulating, processing and extracting value from large disconnected datasets.
• Experience in building and optimizing big data data pipelines, architectures and data sets.
• Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
• Experience supporting and working with cross-functional teams in a dynamic environment.

Skills/Tools/Techniques

• Hands on proficiency with big data tools: Hadoop, Hive, GCP BigQuery, GCP DataProc, Python/Scala
• Nice to have experience in GCP cloud services: Google cloud storage, Big query, Spanner, Cloud Pub/Sub Composer
• Experience in CI/CD using Jenkins/Gitlab
• Nice to have experience with stream-processing systems: Dataflow etc.
• Experience with data pipeline and workflow management tools: Airflow, etc.
• Strong analytic skills related to working with unstructured datasets.