Airflow Architect

Airflow Architect — Infrastructure Automation & Orchestration

Location: Chennai

Experience: 8–10 Years

Environment: On-Premise Data Centre & DR

About the Role

We are looking for an experienced Airflow Architect to design, build and own an enterprise-grade Apache Airflow platform across on-premise Data Centre and DR environments.

The primary focus is infrastructure and operations automation, with data pipelines as a secondary workload.

You will be the technical authority for Airflow, owning architecture, HA/DR, security, performance, upgrades and the engineering team operating the platform.

Qualifications

Bachelor's or Master's degree in Computer Science, Engineering or equivalent.

Key Responsibilities

Architecture & Platform

• Design and implement highly available Airflow on-premise.
• Architect multi-scheduler, redundant web/API, Celery worker and RabbitMQ infrastructure.
• Design and test DC/DR architecture, including RTO/RPO, failover and failback.
• Establish Dev/UAT/Pre-Prod/Prod/DR environments.
• Own Terraform/Ansible-based Infrastructure as Code and CI/CD for DAG deployment.
• Integrate with LDAP/AD/SSO, Vault/CyberArk, PKI and enterprise storage/logging.

Infrastructure Automation

Build reusable Airflow workflows for:

• OS/middleware patching and rollback
• VM/server/container provisioning
• Backup and restore validation
• Certificate and credential rotation
• Health checks and auto-remediation
• Compliance and configuration-drift checks
• DR drills and evidence collection
• Access provisioning and recertification
• Cross-system runbook automation

Develop custom Operators, Hooks, Sensors and Plugins integrating with Ansible, SSH/WinRM, REST APIs, network, storage, backup and monitoring systems.

Implement safe-automation controls such as approval gates, dry-runs, blast-radius limits, maintenance windows and audit trails.

MySQL Metadata Layer

• Architect and operate MySQL 8.x for Airflow.
• Experience with InnoDB, GTID, replication, InnoDB Cluster/Group Replication and PITR.
• Tune connection pools, InnoDB, queries and Airflow metadata tables.
• Design backup, restore, failover and schema-migration strategies.

Performance & Reliability

• Own Airflow capacity planning and performance tuning.
• Optimise schedulers, workers, concurrency, pools and queue performance.
• Implement monitoring using Prometheus/StatsD, Grafana, ELK/Loki.
• Monitor scheduler health, task latency, queue depth, DB connections and replication lag.

Security & Operations

• Own Airflow, Python dependency, OS, MySQL and RabbitMQ vulnerability management.
• Handle CVE assessment, Nessus/SonarQube findings and VAPT remediation.
• Implement TLS, RBAC, Vault-based secrets, network segmentation and privileged-access controls.
• Maintain SOPs, HLD/LLD, runbooks and operational documentation.

Leadership

• Lead a team of 8 platform/support engineers.
• Conduct architecture and code reviews.
• Own production SLAs, incidents, changes and technical roadmap.
• Represent the platform in architecture and change-management forums.

Must Have Skills

• 8–10 years Infrastructure / Platform / DevOps / SRE experience.
• 5+ years production Apache Airflow experience.
• Proven experience building and operating Airflow on-premise, including HA and DR.
• Strong MySQL 8.x administration and performance tuning.
• Production experience with Celery Executor and RabbitMQ.
• Strong Python, Linux and shell scripting.
• Hands-on Ansible/AWX and Terraform.
• CI/CD experience with Jenkins, GitLab CI or equivalent.
• Experience with Grafana, Prometheus/StatsD and ELK/Loki.
• Strong understanding of enterprise security, vulnerability management and ITIL processes.
• Proven technical leadership and production SLA ownership.

Good to Have

• Control-M / Autosys / Tidal / Rundeck migration experience.
• ServiceNow / CMDB integration.
• VMware, Veeam, NetBackup or Commvault.
• Vault, Keycloak, LDAP/Kerberos.
• Apache Airflow open-source contributions.
• CKA/CKAD, RHCE, Ansible, MySQL DBA or security certifications.
• Astronomer Apache Airflow 3 certifications.
• BFSI, Telecom or Healthcare experience.