Data Engineer

RAK Bank is seeking a Databricks Engineer to build a scalable data ingestion and streaming platform using Confluent connectors, Databricks Auto Loader, and Delta Lake. The role involves designing Spark Structured Streaming pipelines to ingest CDC events from diverse sources like SQL Server and Oracle, ensuring deduplication, schema evolution, and exactly-once semantics. Key responsibilities include implementing monitoring via Prometheus and Grafana, scaling ingestion through config-driven frameworks like Airflow or Delta Live Tables, and collaborating with cross-functional teams to enforce data quality and security. Candidates should have 5–8 years of experience with big data frameworks, hands-on expertise in Kafka or Azure Event Hub, and proficiency in Python, Scala, or Java. A deep understanding of Lakehouse architectures, DevOps practices using CI/CD pipelines, and strong communication skills are essential. Preferred qualifications include experience with event-driven architectures, microservices integration, and exposure to additional ingestion frameworks like NiFi or machine learning pipelines on Spark.