Senior Data Engineer

Requirements:

• Should have a total of 4-5 years of experience in Data Engineering.
• Should have hands-on experience in distributed computing.
• Should have working experience in data architecture design.
• Strong in programming languages like Python and Java.
• Should be aware of storage and compute options and when to choose what.
• Must have one cloud hands-on experience.
• Proficient in either Apache Spark or Apache Beam or Apache Flink.
• Advanced SQL knowledge is mandatory.
• Must be aware of streaming concepts like Windowing, Late arrival, triggers, etc.
• Should have exposure to GCP tools to develop end-to-end data pipelines for various scenarios (including ingesting data from traditional databases as well as integration of API based data sources).
• Should have a business mindset to understand data and how it will be used for BI and Analytics purposes.
• Should have working experience on CI/CD pipelines, Deployment methodologies, and Infrastructure as Code.
• Should have advance understanding of different file formats, e. g., AVRO, Parquet, ORC, CSV and when to choose what.
• Good to have: hands-on experience with Kubernetes.
• Good to have: Vector-based database like Qdrant.
• Should have a good understanding on Cluster Optimisation/ Pipeline Optimisation strategies.
• Experience working with GCP tools like Storage: CloudSQL, Cloud Storage, Cloud Bigtable, BigQuery, Cloud Spanner, Cloud DataStore, Vector database.
• Ingest: Pub/Sub, Cloud Functions, AppEngine, Kubernetes Engine, Kafka, Microservices.
• Schedule: Cloud Composer, Airflow.
• Processing: Cloud Dataproc, Cloud Dataflow, Apache Spark, Apache Flink.
• CI/CD: Bitbucket+Jenkins / Gitlab.
• Infrastructure as a tool: Terraform.