Kafka Engineer

Job Summary

We are looking for a Kafka Engineer with 2 to 5 years of experience to design, develop, and optimize high-throughput, low-latency data platforms. The ideal candidate will have hands-on expertise in Apache Kafka, Redis, and MongoDB, with a strong understanding of real-time event streaming, distributed caching, NoSQL databases, and performance optimization. You will work closely with application developers, DevOps engineers, and cross-functional teams to build scalable and resilient data solutions.

Key Responsibilities

Apache Kafka (Event Streaming & Messaging)

• Design, implement, and maintain high-throughput Kafka topics, partitions, and consumer groups.
• Configure Kafka producers and consumers for optimal performance and reliability.
• Ensure message delivery guarantees and manage schema evolution using Confluent Schema Registry.
• Monitor and troubleshoot Kafka clusters, consumer lag, and broker performance.

Redis (Distributed Caching & In-Memory Data Store)

• Design and implement efficient caching strategies to improve application performance and reduce database load.
• Utilize Redis data structures such as Hashes, Lists, Sets, Sorted Sets, HyperLogLogs, and Geospatial indexes.
• Configure and manage Redis Clusters, Sentinel deployments, Pub/Sub messaging, and persistence mechanisms.
• Optimize cache performance and resolve issues such as cache stampedes and memory fragmentation.

MongoDB (NoSQL Database)

• Design, implement, and optimize MongoDB databases, including Replica Sets and Sharded Clusters.
• Develop efficient BSON document models while following MongoDB best practices.
• Build optimized indexes (Compound, Partial, and Text Indexes) to improve query performance.
• Develop complex aggregation pipelines and perform query optimization.

Performance Optimization & Modernization

• Profile and optimize end-to-end data flows across Kafka, Redis, and MongoDB.
• Identify and resolve system bottlenecks including slow queries, consumer lag, caching issues, and storage inefficiencies.
• Implement modern data engineering best practices for scalable and reliable distributed systems.
• Participate in database modernization and system optimization initiatives.

Required Skills

Technical Skills

• 25 years of experience in backend or data platform engineering.
• Strong hands-on experience with:
• Apache Kafka
• Redis
• MongoDB
• Good understanding of:
• Distributed Systems
• Event-Driven Architecture
• Data Streaming
• Caching Strategies
• NoSQL Database Design
• Experience with Kafka Consumer Groups, Partitions, Schema Registry, and Message Delivery semantics.
• Knowledge of MongoDB Aggregation Framework, Indexing, Replication, and Sharding.
• Familiarity with Redis Cluster, Sentinel, Pub/Sub, and advanced Redis data structures.
• Understanding of Linux environments, REST APIs, and monitoring tools.
• Experience with Git and Agile development methodologies.

Preferred Qualifications

• Exposure to enterprise-scale database modernization or migration projects.
• Knowledge of Docker, Kubernetes, CI/CD pipelines, or cloud platforms (AWS, Azure, or GCP) is an added advantage.
• Familiarity with monitoring and observability tools such as Prometheus, Grafana, or ELK Stack.