Lead AI Engineer - Speech & Clinical AI

You will own the model layer for speech and diagnostic AI systems, taking open-weight ASR and language models and adapting them until they work reliably on real clinical audio and clinical reasoning tasks. This is a deeply hands-on engineering role focused on data, training runs, and evaluation. You will also guide decisions on whether a problem calls for fine-tuning, retrieval, better prompting, or a different architecture entirely.

Responsibilities:

• Fine-tune and adapt ASR models for domain vocabulary, accented and multilingual speech, noisy audio, and multi-speaker consultations.
• Fine-tune LLMs for diagnostic and clinical reasoning, symptom extraction, differential generation, structured summarization, and clinical coding.
• Run and debug training jobs end-to-end, and diagnose runs that plateau, diverge, or catastrophically forget.
• Decide when fine-tuning is the right approach versus prompting, retrieval, or changing the base model.
• Own training data strategy, including curation, labeling, synthetic generation, augmentation, deduplication, and leakage checks.
• Build evaluation frameworks that predict production behavior, including WER/CER sliced by accent and speaker, entity-level accuracy on clinical terms, and hallucination and omission rates.
• Run clinician-in-the-loop review processes and convert qualitative feedback into measurable objectives.
• Optimize models through distillation, quantization, streaming, latency reduction, batching, and cost-per-hour-of-audio improvements.
• Partner with platform and MLOps teams on serving, monitoring, drift detection, and retraining cadence.
• Set applied AI direction for the team, run design reviews, and make build-vs-buy decisions.
• Mentor 3-5 engineers on evaluation discipline and experiment hygiene.
• Communicate model capabilities and limitations clearly to product, clinical, and leadership stakeholders.

Requirements:

• 6+ years in software or ML engineering, with 3+ years training or adapting deep learning models in production.
• Strong Python skills for production-grade, tested, reviewable code.
• Working knowledge of Bash or shell scripting and SQL.
• Deep hands-on experience with PyTorch.
• Experience with Hugging Face Transformers, PEFT, TRL, Datasets, Accelerate, and Tokenizers.
• Proficiency with NumPy, SciPy, pandas, and scikit-learn.
• Expertise in full fine-tuning and parameter-efficient fine-tuning methods such as LoRA, QLoRA, adapters, and prefix tuning.
• Experience with supervised fine-tuning, instruction tuning, and preference optimization methods such as DPO, ORPO, KTO, and RLHF/PPO.
• Experience with continued pre-training or domain-adaptive pre-training.
• Strong understanding of learning-rate scheduling, warm-up, gradient accumulation, gradient clipping, mixed precision, and gradient checkpointing.
• Ability to mitigate catastrophic forgetting, overfitting, and distribution shift.
• Experience with curriculum design and data mixture or sampling ratio tuning.
• Experience with distributed training using DeepSpeed and/or PyTorch FSDP.
• Experience with multi-GPU and multi-node training, NCCL basics, sharding strategies, and OOM debugging.
• Strong ASR knowledge across Whisper, wav2vec2, HuBERT, Conformer, and RNN-T/transducer models.
• Understanding of CTC, attention encoder-decoder, and transducer architecture trade-offs.
• Experience with domain adaptation techniques such as vocabulary biasing, contextual biasing, and shallow or external LM fusion.
• Experience with forced alignment, pronunciation, and lexicon handling for clinical and drug terms.
• Knowledge of audio preprocessing, including log-mel, MFCC, VAD, resampling, and normalization.
• Experience with augmentation methods such as SpecAugment, noise, reverberation, and speed or pitch perturbation.
• Experience with speaker diarization and speaker separation using Pyannote or equivalent tools.
• Experience evaluating WER/CER by accent, speaker, cohort, and clinical term.
• Experience with streaming or low-latency ASR inference.
• Experience with toolkits such as NeMo, ESPnet, SpeechBrain, Kaldi, torchaudio, and librosa.
• Experience fine-tuning open-weight LLMs such as Llama, Qwen, Mistral, or Gemma for specialized domains.
• Experience with clinical NER and structured extraction from unstructured notes.
• Knowledge of terminology normalization across SNOMED CT, ICD-10, LOINC, RxNorm, and UMLS.
• Ability to handle failure modes such as hallucination, omission, sycophancy, and long-context degradation.
• Experience with RAG and agentic pipelines, embedding models, and vector stores.
• Strong prompt engineering and structured or constrained output generation skills.
• Experience with large-scale dataset curation, cleaning, and deduplication.
• Experience defining labeling strategy, annotation guidelines, and inter-annotator agreement.
• Experience with synthetic data generation and quality filtering.
• Experience with train/validation/test splitting and leakage detection.
• Experience building audio and text data pipelines at scale using Spark, Ray, Dask, or equivalent.
• Experience designing task-specific evaluation suites and regression harnesses.
• Knowledge of speech metrics such as WER, CER, entity-level accuracy, and keyword recall.
• Experience with generation metrics, including rubric-based grading, LLM-as-judge, factuality, and omission scoring.
• Experience with human or clinician review protocols and annotation tooling.
• Understanding of statistical significance, confidence intervals, and per-cohort error analysis.
• Experience with bias and fairness testing across accent, gender, age, and language.
• Experience with quantization methods such as INT8, INT4, GPTQ, AWQ, and bitsandbytes.
• Experience with knowledge distillation.
• Experience with export and runtime tools such as ONNX, TensorRT, and CTranslate2.
• Experience with serving stacks such as vLLM, Triton Inference Server, TorchServe, and TGI.
• Understanding of KV caching, continuous batching, latency profiling, and throughput profiling.
• Experience with cloud GPU infrastructure on AWS, GCP, or Azure, and with SLURM or Kubernetes-based scheduling.
• Experience with Docker and containerized training and serving.
• Experience with experiment tracking using Weights & Biases or MLflow.
• Experience with model and dataset versioning, artifact registries, CI/CD for ML, automated retraining, monitoring, and drift detection.
• Strong Git and code review discipline.
• Cost awareness and GPU budget management.
• Experience mentoring 3-5 engineers, running design and code reviews, setting technical standards, and translating clinical and product requirements into model objectives.
• Ability to explain model behavior, uncertainty, and limitations to non-technical stakeholders.
• Strong written communication skills for design docs, evaluation reports, and post-mortems.
• Pragmatic prioritization and comfort with ambiguity.

Nice to Have:

• C++ or Rust for inference optimization.
• Experience with JAX or TensorFlow.
• Experience with checkpoint selection, early stopping, and model merging methods such as SLERP and TIES.
• Experience with Flash Attention, xFormers, or Liger Kernel.
• Knowledge of clinical document standards such as HL7 FHIR and SOAP notes.
• Experience in healthcare, medtech, or another regulated domain.
• Knowledge of PHI/PII handling, de-identification, HIPAA, GDPR, or DPDP.
• Experience with clinical validation and SaMD, CE marking, or FDA pathways.
• Experience with model documentation, audit trails, and reproducibility for regulated release.
• Publications, open-source contributions, or benchmark results in speech or clinical NLP.
• Experience with multilingual or code-switched speech modeling.