Marin
"I am not afraid of storms, for I am learning how to sail my ship."
– Louisa May Alcott
Marin is a research program, software platform, and community for the research and development of foundation models.
Marin's concern is training large language models. This includes data curation, transformation, filtering, tokenization, pretraining, posttraining, and evaluation. Beyond the artifacts, software, and infrastructure, behind these models, Marin is committed to openly sharing all of the process knowledge required to build these models.
Marin's core value is open development. We document our processes, experiments, and decisions as they happen. Every step, from raw data to the final model, is recorded. Failed experiments are part of that record.
Marin has also been used for building audio-text models, DNA, and protein models. We encourage this work through the use of Marin as a library, in marin/experiments.
Current work
Frontier mixture-of-experts
Our current focus is pretraining, from scratch, and posttraining a large (5e24 model-FLOPs, 500 billion+ total parameters) mixture-of-experts model to succeed on tasks of importance to scientists and researchers.
Scaling suite
Delphi is Marin's open scaling suite scaling a LLM recipe from 3e18 to 1e23 FLOPs, inspired by Pythia. It has three parts: a scaling recipe that maps compute budgets to model configurations, a scaling suite trained from that recipe on the Google TPU Research Cloud, and a scaling law that uses the smaller Delphi models to predict the larger ones.
We released:
- Checkpoints for every run, available on Hugging Face at marin-community/delphi
- Training mixture pipelines that deterministically reproduce the mix from the public Nemotron-CC, StarCoderData, and ProofPile 2 in the Marin repo
- Recipe code as a forkable
CompletedAdamHParamsclass in the Marin repo - Development methodology as the
add_scaling_heuristicagent skill in the Marin repo - Plot-ready data for the Delphi figures, with one config per figure and a
wandb_urlon every row, at marin-community/delphi-blog-data
Progress was tracked in GitHub issue #1337.
Other learnings
Some additional consolidated learnings can be found on the Open Athena blog. A selection, below:
- Cluster Scheduling with Iris · scheduling jobs across heterogeneous clusters
- Improving our LLM Pretraining Efficiency · squeezing more throughput from pretraining
- Scaling Laws That Extrapolate 300× Past the Fit (Delphi) · predicting big models from small
- Mixture of Experts Quantile Balancing: Validated at 32B-A5B (1e22 FLOPs) Scale · keeping MoE experts load balanced
Other models
Previously, we used Marin to train an 8B parameter model that outperformed Llama 3.1 8B on our base-model benchmark suite. You can see the training script or read the retrospective. We also trained Marin 32B.
Learning more & using Marin
The documentation for Marin is available on ReadTheDocs or in the docs/ folder.
To get started with Marin:
- Install Marin.
- Train a tiny language model using Marin.
- See how to run a much larger DCLM 1B/1x experiment using Marin.
- See a summary of the experiments we've run.
- Join the Marin Discord to chat with the community.
Example
Marin experiments are defined as a set of steps that can depend on each other and are executed in a topological order, like a Makefile.
As a brief example of how you can use Marin, here is a complete script for training a tiny model on TinyStories. You can check out the full script for more details.
from fray.cluster import ResourceConfig
from levanter.optim import AdamConfig
from marin.execution.lazy import lower
from marin.execution.step_runner import StepRunner
from marin.experiment.data import tokenized
from marin.experiment.train import train_lm
from experiments.llama import llama_nano
from experiments.marin_tokenizer import marin_tokenizer
# 1. Tokenize the dataset as a lazy handle — nothing downloads yet.
tinystories_tokenized = tokenized(
name="tokenized/tinystories",
source="roneneldan/TinyStories",
tokenizer=marin_tokenizer,
sample_count=1000, # cap at 1 000 samples per shard to keep the tutorial fast
)
# 2. Train the model — depends on the tokenized dataset above.
nano_tinystories_model = train_lm(
name="checkpoints/marin-nano-tinystories",
version="v1",
model=llama_nano,
optimizer=AdamConfig(learning_rate=6e-4, weight_decay=0.1),
# Steps can depend on other steps: nano_tinystories_model depends on tinystories_tokenized
datasets={tinystories_tokenized: 1.0},
batch_size=4,
seq_len=2048,
num_train_steps=100,
z_loss_weight=None,
evals=None, # no point evaluating such a tiny model
resources=ResourceConfig.with_cpu(),
)
if __name__ == "__main__":
StepRunner().run([lower(nano_tinystories_model)])Here, we create two steps, one for tokenizing the dataset and one for training the model. The training step depends on the tokenized dataset step, so it will be executed after the tokenization step is completed.
With slight modifications, you can extend this to train a larger model on a larger dataset, a mixture of datasets, even scaling to very large GPU or TPU pods (or multislice TPUs!).
For Contributors
- See
CONTRIBUTING.mdfor project workflow. - See
.agents/skills/(also.claude/skills/) for loadable agent skills. For example,.agents/skills/add-dataset/has a step-by-step guide to adding new datasets.
Core Contributors
Marin's core collaborators come from Stanford CRFM and Open Athena.
Supporters
Marin's research is made possible by the generous support of our partners.
 |
The Jen-Hsun and Lori Huang Foundation |
| for TRC accelerators | for GPU clusters |
 |
 |
| for supporting development | for supporting development |
