Kafka architect

Job Description:

We are seeking an Expert Kafka professional with 10+ years of deep expertise in designing and implementing enterprise-scale event streaming solutions using Confluent Kafka. The ideal candidate will possess extensive experience in large-scale Confluent Kafka infrastructure (3+ years) and designing complex Kafka integration solutions (5+ years) that power real-time data pipelines, event-driven architectures, and mission-critical business applications. This senior role requires a technical professional who can design highly available, scalable, and secure Kafka ecosystems, establish streaming best practices, and drive the adoption of event-driven patterns across the organization. The Expert Kafka professional will serve as the subject matter expert for all Kafka-related initiatives, mentoring engineering teams, optimizing platform performance, and ensuring the reliability of streaming data infrastructure that supports business-critical operations.

Roles and Responsibilities:

• Confluent Kafka Infrastructure Architecture (3+ Years): Architect and design enterprise-scale Confluent Kafka infrastructure supporting high-throughput, low-latency event streaming requirements
• Design Kafka cluster topologies including multi-region, multi-datacenter, and hybrid cloud deployments. Plan and implement Kafka broker sizing, partition strategies, replication factors, and cluster scaling patterns.
• Architect Confluent Platform components: Confluent Server, ZooKeeper/KRaft, Schema Registry, Kafka Connect, ksqlDB, Confluent Control Center. Design high-availability and disaster recovery strategies for Kafka clusters including backup, replication, and failover mechanisms.
• Implement Kafka security architectures including SSL/TLS encryption, SASL authentication, and ACL-based authorization. Optimize Kafka infrastructure for performance, throughput, and cost-efficiency across cloud and on-premises environments.
• Evaluate and select deployment options: Confluent Cloud, Confluent Platform on-premises, or self-managed Kafka on cloud infrastructure Kafka Integration Solutions Design (5+ Years): Design end-to-end event streaming solutions using Kafka for real-time data integration and event-driven architectures .
• Architect Kafka Connect solutions for integrating with databases, cloud storage, SaaS applications, and legacy systems. Design stream processing applications using Kafka Streams, ksqlDB, or Apache Flink for real-time data transformation.
• Implement event sourcing and CQRS (Command Query Responsibility Segregation) patterns using Kafka as the event store.
• Design microservices integration patterns using Kafka for asynchronous communication and event-driven choreography. Architect data pipeline solutions for ETL/ELT, data synchronization, and real-time analytics use cases.
• Design schema management strategies using Confluent Schema Registry with Avro, Protobuf, and JSON Schema.
• Implement exactly-once processing semantics and transactional messaging patterns for critical business applications Performance Optimization & Tuning: Optimize Kafka cluster performance: throughput tuning, latency reduction, and resource utilization.
• Tune Kafka producer and consumer configurations for optimal performance and reliability Design and implement partitioning strategies for balanced load distribution and parallel processing.
• Monitor and optimize Kafka consumer lag, rebalance operations, and consumer group management. Implement Kafka compaction strategies and log retention policies for storage optimization. Tune JVM settings, garbage collection, and OS-level configurations for Kafka brokers.
• Conduct performance benchmarking and capacity planning for future growth Security & Governance: Architect comprehensive Kafka security models including authentication, authorization, and encryption.
• Implement SSL/TLS encryption for data in transit and encryption at rest for sensitive data. Design SASL-based authentication mechanisms: SASL/PLAIN, SASL/SCRAM, SASL/GSSAPI (Kerberos), SASL/OAUTHBEARER.
• Implement fine-grained ACLs and role-based access control for Kafka resources. Design data governance frameworks for event streaming including data lineage, quality, and compliance Ensure compliance with regulatory requirements: GDPR, PCI-DSS, HIPAA, SOX for streaming data
• Implement audit logging and monitoring for security events and access patterns Platform Operations & Reliability: Design monitoring and observability solutions for Kafka infrastructure using Confluent Control Center, Prometheus, Grafana
• Implement alerting mechanisms for cluster health, consumer lag, and critical metrics. Create runbooks and operational procedures for Kafka cluster maintenance and incident response Design automated failover and self-healing mechanisms for Kafka infrastructure.
• Plan and execute Kafka upgrades, migrations, and platform maintenance with minimal downtime. Implement disaster recovery strategies including MirrorMaker 2.0 for cross-cluster replication. Conduct chaos engineering and resilience testing for Kafka infrastructure Integration Architecture: Design integration patterns between Kafka and enterprise systems: databases, data lakes, data warehouses.
• Architect Kafka-to-Kafka replication strategies for multi-region and multi-cloud scenarios. Design integration with cloud-native services: AWS Kinesis, Azure Event Hubs, Google Pub/Sub Implement CDC (Change Data Capture) patterns using Debezium or similar tools with Kafka. Design API integration patterns using Kafka as the messaging backbone.