Foundation Model Research Engineer

We are seeking a Foundation Model Research Engineer to push the boundaries of frontier AI architectures and large-scale pre-training. In this role, you will be at the cutting edge of deep learning research, inventing, implementing, and scaling novel neural network architectures beyond standard Transformer paradigms. Working at the intersection of fundamental ML research and high-performance engineering, you will design and train frontier-class models using JAX across massive accelerator topologies.

Responsibilities:

• Architectural Innovation: Research, prototype, and validate novel model architectures, attention mechanisms, dynamic routing systems, and alternative sequence-modeling frameworks (e. g., linear attention, state-space models, and hybrid architectures).
• Large-Scale Pre-Training: Lead end-to-end pre-training runs for multi-billion-parameter foundation models using JAX/MaxText, optimizing cluster-wide Model FLOPs Utilization (MFU) across multi-pod infrastructure.
• Algorithm and Kernel Engineering: Implement high-performance custom primitives, loss functions, and hardware-aware operations to maximize hardware efficiency during large-scale pre-training.
• Empirical Experimentation: Conduct systematic ablation studies, scaling law analysis, and compute-optimal trade-off evaluations to guide next-generation model design.
• Cross-Functional Collaboration: Translate research breakthroughs into robust, production-grade training frameworks to power downstream alignment, instruction-following, and post-training pipelines.

Requirements:

• Education and Expertise: Ph. D. in Computer Science, Machine Learning, or Applied Mathematics, or an equivalent track record of demonstrated impactful research in foundation models.
• Core Technical Stack: Deep hands-on expertise in JAX (or Flax/Equinox/MaxText) for scaling distributed tensor-parallel, pipeline-parallel, and data-parallel training jobs.
• Research Track Record: Active research presence, with publications focused on transformer modifications, attention variants, or pre-training dynamics.
• Deep Transformer and LLM Mechanics: Deep understanding of the internals of modern LLMs (e. g., RoPE/positional embeddings, KV-cache management, normalization schemes, MoE routing, context extension techniques).
• Distributed Systems Math: Strong understanding of compute efficiency metrics and numerical stability (BF16/FP8 training).

Good to Have / Bonus Points:

• Experience training models exceeding 100B parameters across multi-pod or multi-rack accelerator clusters (e. g., TPU v5e/v6e Trillium, H100 networks).
• Familiarity with memory-efficient attention patterns, FlashAttention implementations, or custom XLA compilation optimizations.
• Background in non-transformer sequence models (e. g., Mamba, RWKV, SSMs, or recurrent architectures) and hybrid architecture design.
• Demonstrated history of contributing to major open-source research frameworks in the JAX ecosystem.