DELHI · FULLTIME
Lead Engineer – Data Platform & Acquisition (LLM Corpora)
EQ
Eternity Quests LLP
Delhi · onsite · Posted 39d ago
Sourced from
Not disclosed4–12 yrsfulltimeDelhi
About a minute. You'll make an account on the way through.
agent · working
- Read this posting0.2s
- Found the form on Foundit · 9 questions to answer1.0s
- Decided the answers for all of them3.2s
- Tailor your résumé to this role
- Waiting for you
Section · 01
About this role
- Location: Delhi NCR
- Core Mandate: Pipeline engineering, dedup/quality/decontamination, licensing negotiations, vendor management.
- Responsibilities:
- Architect high-throughput data pipelines capable of processing petabytes of text and multimodal data.
- Implement rigorous data deduplication, quality filtering, and benchmark decontamination protocols.
- Lead licensing negotiations for proprietary datasets and manage external vendors for large-scale digitization and crowdsourcing tasks.
Education
B.E./B.Tech/M.Tech/M.S./Ph.D. in CS, EE, Maths or related
Sourced from foundit
Section · 02
Skills
data qualitydata licensingApache SparkApache Kafkadata acquisitionapache airflowWeb CrawlingDistributed Data ProcessingRayData EngineeringPythonData DeduplicationVendor ManagementETLData PipelinePetabyte scale