Marin 535B-A23B MoE

18T tokens · launched Aug 20 updated just now · next in 19m · auto-generated via LLM, using mumwelt and data from GitHub, Discord, and WandB
WandB report Run description Status log
13.7% · step 53,421 of 390,251
Aug 20 2.46T of 18T tokens Nov 28

Training remains stable for about eighteen hours since the job restarted itself at 7:39 p.m. ET on September 1. A faulty InfiniBand port on one host kept dropping its link until the job lost that node. InfiniBand is the high-speed network the GPUs use to exchange gradients. The cluster's job scheduler resumed from the last checkpoint without human help. That is the second hardware fault it has cleared on its own, after the lost node on August 27. No fault has been reported since.

One architectural question stays open. ClassicLarry, who posts as Larry on Discord, opened #8818 because the weights behind the attention gate keep growing in norm, far beyond anything seen on the scaling ladder — the smaller runs trained to predict how this one would behave. The gate scales each attention head's output through a squashing function, so as the norm grows the gate saturates closed and stops responding to training. In the first layer it is already shut for almost every token in the vocabulary; deeper layers are not. Even so, tests that zero out attention in a saved checkpoint show the first ten of the model's 48 layers still matter, because most of their work runs through the feed-forward path rather than attention. ClassicLarry plans no intervention while the loss stays below the curve the ladder predicted, which it is.

The router, which picks the few experts that handle each token, grows the same way, so he proposed light weight decay on the gate and router — a steady pull of the weights back toward zero that fades out by the end of the run — to keep both trainable late on. Pull request #8833 would make that decay the default in the shared training recipe. It is still open, so the September 1 restart resumed without it. An overnight checkpoint test showed the model leans hard on the gate, so at 2:27 a.m. ET on September 2 he cut the proposed decay from 0.05 to 0.02, weak enough that the model can adapt as the norm falls. At 12:36 p.m. ET he opened #8854 to pin a permanent checkpoint just past the point where the decay would switch on, so the decayed run has a fixed comparison point against the undecayed one. Kaiyue-Wen asked what the decay costs in predictability, because the ladder runs that set this run's target loss trained without it; Larry replied that the ladder cannot settle it, since this run's gate is not tracking the ladder in the first place. Percy Liang argued that a statistic which does not hold across model sizes deserves its own investigation, separate from the run.

Sep 2
ClassicLarry #8854 open
Pins one extra permanent checkpoint at step 55,000, just past the point where the proposed gate and router weight decay would switch on. That gives a fixed comparison point between the decayed run and the undecayed one. The regular every-6,000-step schedule is unchanged.
checkpointing
ClassicLarry #8818
Outside precedent for the proposal: Microsoft's MAI-Base-1 report describes weight decay set per parameter group, weaker on attention and embedding weights than on the rest of the model.
optimizer
Larry #hero-run-2026
Lists ways a future run could handle the depth-dependent output scaling without the attention gate, including a normalized gate that centers all 48 head gates at one. Cautions against treating it as a fault at all while the loss and gradients look healthy.
training-dynamics
Larry #hero-run-2026
Plain-language summary of the attention-gate story for the channel: the model chooses to make its early layers contribute little to the output, and has co-opted the attention gate to do it by aligning all 48 head gates against the embeddings. Every layer is still learning and the first ten still matter most.
training-dynamics
Percy Liang #hero-run-2026
Argues a training statistic that does not hold across model scales deserves its own investigation, separate from the run. Asks how early the gate growth was detectable, whether the ladder suites show it, and whether the tracker should forecast more statistics than the loss and warn when one drifts off forecast.
training-dynamics
Larry #hero-run-2026
Answers the predictability objection: on the d1024 rung the decay barely moved the final loss, but the hero's attention gate is not following ladder dynamics, so the rungs cannot say whether decay pulls it closer to them or further away. A weak setting is the safer choice; he expects the gate norm to halve over roughly ten thousand steps.
optimizer
Kaiyue-Wen #hero-run-2026
First reply to the gate/router weight-decay proposal from someone other than its author: the change could help stability, but it may weaken how well the scaling ladder predicts the run's final performance.
optimizer
ClassicLarry #8833
Lowers the proposed weight decay from 0.05 to 0.02 after the checkpoint test showed how much the model depends on the attention gate. The pull request now anneals that 0.02 linearly to zero across the run, and the value can be raised later if needed.
optimizer
ClassicLarry #8818
Reads the scaling test as a reason not to collapse the gate norm quickly: the model relies heavily on the gate, and the loss curve and evaluations are both steady, so there is no cause to rush.
training-dynamics
ClassicLarry #8818
Scaled the attention-gate logits down on a step-48k checkpoint before the sigmoid. A 20 percent cut is nearly free, 40 percent degrades the model badly, and 60 percent or more collapses it, so the learned per-head gating is load-bearing.
ablation
Larry #hero-run-2026
Added a zoomed view to the public W&B report, so the recent stretch of the loss curve is readable without the whole run compressing it.
ClassicLarry #8833
Risk note on the gate/router weight decay: it could pull the gate down faster than the model adapts, which would show in the loss a day or two out; the fallback is to revert and retry weaker. A hundred-step trial on a step-48k checkpoint already shows both norms falling.
optimizer
ClassicLarry #8818
Final d1024 weight-decay ablation: decaying the router alongside the attention gate is consistently but slightly better than the baseline, and decaying the gate alone is indistinguishable from it. All differences sit inside evaluation noise, matching the d768 rung.
ablation
loom-oa-dev #8506
An InfiniBand port fault on host s55sxs64 made the node's link flap, and the training job lost it after step 50195. Iris restarted the job from the last checkpoint; the replacement attempt has clean hardware telemetry and needs no further action.
incident
Sep 1
hero-12d8b6f0-dee637
The training job logged its last step at 23:39 UTC and stopped sending heartbeats, so W&B moved the run to the crashed state. A replacement attempt picked the run back up within the hour and it is reporting again.
incident
ClassicLarry #8833 open
Turns on weight decay for the attention gate and MoE router by default in the hero recipe, so the next resume picks it up. The optimizer state is unchanged, so a checkpoint trained without decay resumes cleanly.
optimizer
Larry #hero-run-2026
Proposes weight decay of 0.05 on the attention gate and router weights, scaled down to zero as the run progresses, and asks the channel for objections.
ClassicLarry #8818
Reads the ablations as all within noise, but still argues for constraining the norm, because the hero's attention-gate scale has already grown much further than on any ladder rung.
ClassicLarry #8818
Weight-decay ablations on the d768 and d1024 ladder rungs all land inside evaluation noise; a linear decay to zero is marginally the best of them.
ablation
Helw150 #8824
Nemotron 3 Super added to the comparison. The hero's final forecast is lower on 14 of the 16 Paloma subsets.
percyliang #8824
Cautions that the compared open-weight models may have trained on the evaluation sets, so the reported margins are a lower bound on how far ahead the run is.
evals
ClassicLarry #8818
The MoE router norm follows the same growth path as the attention gate, so light weight decay on both looks low risk and would keep them learnable later in the run.
ClassicLarry #8827
New issue collects prompt completions from three saved checkpoints, as a plain-language sanity check on what the model has learned so far.
ClassicLarry #8818
Measured the first-layer attention gate for every vocabulary token on a saved checkpoint. Only 546 of 128,256 tokens open any head, and those are mostly reserved tokens and undecodable byte fragments.
training-dynamics
ClassicLarry #8818
Summary of the gate investigation: output scale does not track a layer's impact, early attention is still used, and no immediate intervention is planned. Lists the conditions that would change that.
ClassicLarry #8818
Attention-only knockout on the saved checkpoint: zeroing attention in the first ten layers hurts the evaluation loss about twice as much as zeroing it in the last ten.
ablation
Helw150 #8824
New issue compares the hero's forecast quality with Nemotron 3 and Qwen3.5 base models in bits per byte, which is comparable across tokenizers where perplexity is not.
evals
Aug 31
ClassicLarry #8818
Layer-skip ablation: dropping any ten layers from the forward pass multiplies the evaluation loss, so the early layers are not redundant despite their small output norm.
ablation
Helw150 #8435
Published the clustering and classifier weights behind the Harrier data mixture on Hugging Face, so outside groups can reproduce how the mix was selected.
data
ClassicLarry #8818
The loss projection is still ahead of the scaling-ladder schedule. Per-submodule plots show the first twenty layers moving the residual stream very little.
ClassicLarry #8818
New issue: the attention gate's weight norm has grown far beyond the scaling-ladder trajectory, concentrated in the first five layers.
training-dynamics
Aug 28
mcwitt #8754
Context-parallel branches let the hero model shape train at 262K sequence length on one rack, restored from an early hero checkpoint. Router capacity is the open problem before a long-context cooldown.
long-context
Aug 27
loom-oa-dev #8506
Node s6htxs64 lost connectivity and the 176-task gang failed. Iris retried it automatically and training resumed from the last checkpoint. No software fault preceded the failure.
incident
Aug 24
rjpower #8506
Loader-cache ablation finished: a 1 GB per-process decode cache costs almost no throughput, so the cache can stay small and give host memory back to checkpointing.
ablation
ravwojdyla #8506
A training task ran out of memory during checkpoint serialization and took the undersized coordinator with it. PR #8617 raises the coordinator to 4 GB and gives it a retry budget so one task failure cannot stop the job.
incident
rjpower #8506
Traced the leak: the checkpoint manager never clears completed TensorStore commit futures, so every save retains a staged shard per writer rank.
Aug 23
rjpower #8506
The coordinator ran out of memory while saving a checkpoint. The job was relaunched under a new coordinator instance carrying the memory fixes.
incident
Aug 22
rjpower #8506
Restarted on main with jemalloc after malloc_trim failed to stop the checkpoint-time host OOMs. Merged #8584 to abort stalled TensorStore reads that were blocking restores.
Aug 21
rjpower #8506
Restarted with bounded TensorStore restore parallelism, a 16 GB checkpoint offload cache, and a malloc_trim call after each save, to stop host OOMs recurring after checkpoints.
muchanem #8506
An uncorrectable NVLink error on node s9vrxs64 took down three tasks in the same rack.
incident
Aug 20
rjpower #8506
Host memory exhaustion recurs about every three hours. Tuning the XLA memory fraction and slop factor did not help.
incident
ravwojdyla-agent #8480 merged
Bounds the host memory a hero resume holds, after every resume attempt was killed by the container runtime once the checkpoint had loaded.
Aug 19
ClassicLarry #8435
Full model and training spec: 535B parameters with 23B active, 48 layers, expert parallelism inside each NVL72 rack, and a two-phase Harrier data mixture over 18T tokens.
spec