Python Software Developer

Senior Python Engineer Data & Generation Pipelines

5+ years of experience

Full-time - Hyderabad

Work Mode - Work from Office

About the role

You'll own RedTee's data pipelines end to end. You'll start by building the scraping, cleaning, and LLM-based extraction pipelines that construct our knowledge base the source of truth the whole product depends on. As the product matures, you'll expand into the asset generation pipeline that turns knowledge into video, audio, and diagram content.

This role is for someone who thinks in pipelines: retries, idempotency, quality gates, cost per unit, provenance. You've built systems that call unreliable external services thousands of times a day and stay standing. You know the difference between a script that works once and a pipeline that runs unattended for months.

Must have

5+ years of production Python. Hands-on production experience with at least 3 of the following:

Web scraping at scale Playwright, Selenium (production, not test automation), Scrapy, or equivalent. You've dealt with rate limits, anti-bot measures, proxies, and JS-rendered content.

Pipeline orchestration Celery, Airflow, Prefect, Dagster, or custom async orchestrators with retry, backoff, and dead-letter handling.

LLM API integration OpenAI, Anthropic, or similar, with retry, fallback, structured output parsing, and cost tracking. LangChain, LlamaIndex, or hand-rolled.

Text extraction and cleaning HTML, PDF, and messy real-world documents. Tools like unstructured.io, trafilatura, pdfplumber, BeautifulSoup.

Vector databases and embedding pipelines pgvector, Qdrant, Weaviate, Pinecone, or similar.

Not a fit for this role:

Pure CRUD backend developers (Django-only, DRF-only)

QA engineers whose Selenium experience is test automation

Data scientists without production engineering experience

Anyone whose LLM experience stops at API demos or tutorials

What you'll build

First 6 months knowledge base pipelines:

Scrapers for course content sources with retry, dedup, and freshness detection

Text cleaning and chunking pipelines (HTML/PDF structured content with provenance)

Multi-agent LLM extraction and corroboration pipeline (Scout, Extractor, Corroborator, Librarian, Auditor)

Hash-versioning, provenance tracking, and incremental update system for the knowledge base

Vector store and retrieval layer feeding downstream course generation

Later expand into the asset generation pipeline:

Orchestrator that takes a structured spec and returns tagged, quality-checked assets

Integration of code-based renderers (Manim, Mermaid, D3, Motion Canvas), the Remotion video compositor, and paid APIs for TTS, avatars, and video

Automated evaluation layer that scores every generated asset before it enters the library

Cost monitoring and optimization across all paid APIs

Good to have

FFmpeg, headless browser rendering (Puppeteer, Playwright), or video compositing experience

Node/JavaScript familiarity our video compositor Remotion is React/Node-based

Cost optimization or API spend management on high-volume generative systems

Experience with vision-language models (VLMs) for automated content evaluation

Familiarity with model aggregators (fal.ai, Replicate) or self-hosting open-source models

A GitHub or public work sample showing pipeline or LLM orchestration work

Why join

Own the data spine the entire product depends on. The knowledge base you build determines the quality of every course RedTee ever produces.

Greenfield system design take working prototypes and turn them into a production pipeline, then expand your scope into generation.

Work at the intersection of scraping, LLM orchestration, and cost engineering, where the hard problem is reliability and unit economics, not novelty.

Direct influence on the technical direction of the platform. Your calls shape what we can and can't do.

How to apply Please include with your application:

Resume

GitHub / portfolio link if available

A short paragraph (510 lines) describing the most complex pipeline you've built in production what it did, where it broke, and what you learned