Kafka Migration Engineer

JD - Kafka to AWS MSK Migration Engineer

We are looking for a skilled Kafka Migration Engineer to support a critical migration from on-premises Apache Kafka to AWS MSK. This role covers cluster discovery, topic mapping, capacity planning, platform setup, migration execution, validation, performance tuning, automation, and legacy cluster decommissioning.

The ideal candidate will have strong hands-on experience with distributed messaging systems, a structured delivery approach, and the ability to work closely with application, infrastructure, and platform teams to ensure safe and efficient migrations.

Location

• Working Arrangement: Remote
• Preferred Office Location: Hyderabad (HYD) / Bengaluru (BLR)

Key Responsibilities

• Assess on-premises Kafka clusters by reviewing topics, partitions, consumer groups, throughput, retention policies, and dependencies to determine migration readiness.
• Analyze workloads based on message volume, criticality, latency requirements, ordering guarantees, and consumer group complexity to define appropriate MSK cluster placement and configuration.
• Plan and provision AWS MSK infrastructure, including broker sizing, storage, networking, and security configurations.
• Execute UAT and production testing using MirrorMaker 2 or similar replication tools, validate outcomes, and troubleshoot issues through resolution.
• Perform MSK performance tuning and optimization to ensure scalability, stability, and production readiness.
• Partner with application, infrastructure, and platform teams on migration planning, cutover strategy, validation criteria, and rollback procedures.
• Execute production cutovers, including consumer group migration, offset synchronization, and post-release validation.
• Support decommissioning of legacy Kafka clusters after successful migration.
• Design and enhance automation frameworks and AI-assisted solutions for repeatable, large-scale Kafka-to-MSK migrations.
• Document migration procedures, risks, lessons learned, and best practices.

Required Qualifications

Education: Bachelor's or Master's degree in Computer Science, Engineering, Applied Mathematics, or a related discipline.

Experience: 4-6+ years of hands-on engineering experience with distributed messaging systems in collaborative, team-oriented environments.

Programming: Proficiency in Python, Java, or similar scripting/programming languages.

Systems: Strong Unix/Linux fundamentals with troubleshooting experience across applications and runtime environments.

Methodology: Familiarity with SDLC, CI/CD, change management, and Kubernetes-based deployments.

Technical Competencies

Candidates are not expected to be experts in every area but should have strong experience across several of the following:

• AWS Services: ECS, EKS, Lambda, CloudWatch, and related cloud services.
• Apache Kafka: Brokers, topics, partitions, consumer groups, replication, and cluster operations.
• AWS MSK: Cluster provisioning, configuration, monitoring, performance tuning, and security (IAM, mTLS, SASL/SCRAM).
• Migration Tools: MirrorMaker 2, Confluent Replicator, or similar replication solutions.
• Cloud Migration: Workload assessment, testing, cutover planning, and post-migration validation.
• Kafka Ecosystem: Schema Registry, Kafka Connect, Kafka Streams, and related tools.
• Networking & Security: VPC Peering, PrivateLink, TLS, IAM policies, and hybrid Kafka connectivity.
• Automation: Python, Java, Shell scripting, and automation frameworks for Kafka administration and migration activities.
• APIs & Integration: REST APIs and API Gateway concepts.
• CI/CD & Version Control: GitLab, Jenkins, Maven, or equivalent tools.
• Observability: CloudWatch, Prometheus, Grafana, Splunk, or similar monitoring platforms.
• Automation Excellence: Ability to build repeatable solutions and leverage AI-assisted engineering approaches to improve migration efficiency and scale.

Core Competencies

• Strong analytical, troubleshooting, and debugging abilities.
• Ownership mindset with a focus on delivering commitments.
• Clear communication and proactive stakeholder management.
• Effective collaboration across application, infrastructure, platform, and global engineering teams.
• Ability to work across time zones and resolve issues with urgency and professionalism.
• High integrity, sound judgment, and commitment to ethical decision-making.
• Intellectual curiosity, proactive risk identification, and a continuous improvement mindset