Marin 535B-A23B MoE

18T tokens · launched Aug 20 updated · auto-generated via LLM, using mumwelt and data from GitHub, Discord, and WandB
WandB report Run description Status log
26.4% · step 103,037 of 390,251
Aug 20 4.75T of 18T tokens Nov 27

The hero run — the 535-billion-parameter mixture-of-experts model this page tracks — is taking steps at its normal pace again after a stall this morning. It stopped at 9:38 a.m. (ET) today with every GPU busy and no data moving between them: the silent hang the team tracks as #8870, which has now stopped the run three times since it moved to the NCCL 2.30.7 build of the library that carries data between GPUs on September 9. The team has not found the cause.

A watchdog that kills the job when steps stop did so with no one stepping in. A replacement process attached at 9:55 a.m. (ET), restored the checkpoint saved at 9:33 a.m. (ET), and passed the step it hung on by 10:05 a.m. (ET) — twenty-seven minutes from the stall to the first new step, and the few steps after that checkpoint were redone, so no training progress was lost. No one has written the stall up in the status log yet.

The climb in gradient norm that Larry Dial flagged on the night of September 12 (ET) — the size of the push each update makes on the weights, which would warn of instability if it ran away — has stayed under this morning's peak since, including the readings logged after the restart, so it still reads as one spike rather than a new level.

Sep 14
hero-ragged_a2a-nccl2307-ep-step81k
[The silent hang tracked as #8870](https://github.com/marin-community/marin/issues/8870) stopped the hero for the third time on the NCCL 2.30.7 build, ending thirty-six hours and forty-four minutes of unbroken stepping. Step 103,004 finished at 13:38:26Z on a normal 16.8-second loop time; from 13:39:11Z every sampled GPU read 100% busy with memory traffic and NVLink traffic at zero and allocated memory frozen at 197.1 GB, and it held that way through 13:53:26Z with a live heartbeat, so W&B went on reporting the run as running. A new process attached to the same run id at 13:55:35Z, dropped memory to the pre-load level of 150.7 GB, restored the 13:33:08Z hourly save near step 102,986, and returned to the healthy training profile at 13:59:36Z; it replayed the steps logged past that save in about six minutes and logged step 103,005 at 14:05:40Z at a 16.78-second loop time and a train loss of 1.21491, so 27 min 14 s of wall clock was lost and no training work was. Nothing in [the status log](https://github.com/marin-community/marin/issues/8506) covers the stall yet. W&B's sampled history, which feeds this page's progress bar and chart, is current at step 102,939 and trails the run.
herohangrestart
[The gradient-norm rise Larry Dial flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) closed its block below the record. Steps 102,750-102,999 finished at a median of 0.572667 on the output projection — eleventh of the run's eighty-five closed 250-step blocks, against the 0.607133 that steps 102,250-102,499 set this morning — and the block before it closed at 0.553703, so the record still reads as one block rather than a new level. Train loss closed in line: the 500-step block 102,500-102,999 finished at 1.215161 across 240 readings, inside the 1.2111-1.2164 band the last ten closed blocks held. The hero reached step 103,003 at 13:38:09Z on an 18.4-second median loop time over its last 200 steps, and its next reading on held-out text is due at step 104,999, near 7:50 p.m. (ET). W&B's sampled history, which feeds this page's progress bar and chart, is current at step 102,939, 65 steps behind the run.
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry Dial flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has firmed up in the upper part of the run's range without approaching the record. The open block, steps 102,750-102,999, now reads a median of 0.572667 on the output projection at 21 of its 25 readings — eighth of the run's eighty-four closed 250-step blocks at the same reading count, after standing nineteenth on 0.562207 at 14 readings twenty minutes ago. The twelve closed blocks that read within 0.010 of it at 21 readings all finished between 0.549488 and 0.579827, none of them near the 0.607133 record that steps 102,250-102,499 set this morning, so the record still reads as one block rather than a new level. Train loss is unmoved with its block all but complete: the 500-step block 102,500-102,999 reads 1.215068 across 225 readings, inside the 1.2110-1.2164 band the last ten closed blocks held. The hero reached step 102,959 at 13:24:56Z on an 18.5-second median loop time over its last 200 steps, its next hourly save is due near 9:32 a.m. (ET), and its next reading on held-out text is due at step 104,999, near 7:53 p.m. (ET).
herograd-norm
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry Dial flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has come back to mid-range rather than staying low. The open block, steps 102,750-102,999, reads a median of 0.562207 on the output projection at 14 of its 25 readings — nineteenth of the run's eighty-four closed 250-step blocks at the same reading count, after opening at 0.535092 on its first 8 — and the thirteen closed blocks that read within 0.010 of it at 14 readings all finished between 0.534523 and 0.577529, none of them near the 0.607133 record that steps 102,250-102,499 set this morning. Train loss is unmoved at 1.214790 across the 190 readings so far in the 500-step block 102,500-102,999, inside the 1.2111-1.2164 band the last ten closed blocks held, and the hero passed step 102,886 at 13:02:37Z on an 18.5-second median loop time with its next reading on held-out text due at step 104,999, near 7:55 p.m. (ET).
herograd-norm
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry Dial flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) did not hold its record. Steps 102,500-102,749 closed at a median of 0.553703 on the output projection — twenty-first of the run's eighty-four closed 250-step blocks, against the 0.607133 that steps 102,250-102,499 set one block earlier — and the next block has opened lower still, at 0.535092 on 8 of its 25 readings, so the record reads as a single block rather than a new level. Train loss is unmoved: the 500-step block 102,500-102,999 reads 1.215058 across 159 readings, inside the 1.2111-1.2163 band the last ten closed blocks held. The hero reached step 102,827 at 12:44:03Z at an 18.0-second median loop time over its last 200 steps and cleared its 8:32 a.m. (ET) hourly save in a single 22.5-second step; its next reading on held-out text is due at step 104,999, near 7:35 p.m. (ET).
herograd-norm
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry Dial flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has settled mid-range with its block three quarters measured. Steps 102,500-102,749 read a median of 0.553703 on the output projection at 19 of their 25 readings — twenty-second of the run's eighty-three closed 250-step blocks measured at the same reading count, against the 0.607133 that steps 102,250-102,499 closed at to set the record. All thirteen closed blocks that read within 0.010 of it at 19 readings went on to close between 0.538465 and 0.573771, none of them near the record, so the record still reads as one block rather than a new level. Train loss stays in line: the 500-step block 102,500-102,999 now reads 1.214514 across 94 readings, inside the 1.2111-1.2163 band the last ten closed blocks held. The hero reached step 102,688 at 12:01:54Z at an 18.5-second median loop time over its last 200 steps, its longest gap 19.8 seconds, and its next reading on held-out text is due at step 104,999, near 7:56 p.m. (ET). W&B's sampled history, which feeds this page's progress bar and chart, is current at step 102,654, 34 steps behind the run.
herograd-norm
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry Dial flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has held mid-range rather than climbing back toward the record. The open 250-step block, steps 102,500-102,749, reads a median of 0.540058 on the output projection at 14 of its 25 readings — twenty-ninth of the run's eighty-three closed blocks measured at the same reading count, against the 0.607133 that steps 102,250-102,499 closed at to set the record. The eleven closed blocks that read within 0.010 of it at 14 readings went on to close between 0.487376 and 0.572443, none of them near the record. Train loss has come back into line: the 500-step block 102,500-102,999 now reads 1.21459 across 68 readings, inside the 1.2111-1.2163 band the last ten closed blocks held, after standing just above it at 1.21738 on 34 readings twenty minutes ago. The hero reached step 102,633 at 11:45:21Z at an 18.5-second median loop time over its last 200 steps, and cleared its 7:31 a.m. (ET) save in a single 27.1-second step, its slowest since 4:31 a.m. (ET). Its next reading on held-out text is due at step 104,999, near 7:55 p.m. (ET).
herograd-norm
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry Dial flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has dropped well back from the record it set an hour ago. The open 250-step block, steps 102,500-102,749, holds a median of 0.540058 on the output projection at 8 of its 25 readings — twenty-fifth of the run's eighty-three closed blocks measured at the same reading count, against the 0.607133 that steps 102,250-102,499 closed at. The eight closed blocks that read within 0.010 of it at 8 readings went on to close between 0.487376 and 0.570655, none of them near the record, so the record still reads as a spike rather than a new level. Train loss has not followed either way: the 500-step block 102,500-102,999 reads 1.21738 across 34 readings, just above the 1.2111-1.2163 band the last ten closed blocks held, but on too few readings to carry weight. The hero reached step 102,573 at 11:26:28Z at an 18.4-second median loop time over its last 200 steps, its longest gap 23.8 seconds, and its next reading on held-out text is due at step 104,999, near 7:50 p.m. (ET).
herograd-norm
AUG-LIN0-2L-DENSE-MUONH-X600-HIGH-2P2-d512-600x-lr2.2
The 2.2 rung of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) finished at 11:25:50Z and lost, so the sweep's planned range stands. Its final reading on held-out text was 4.007689 against [1.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) 4.002190, a deficit of 0.0055, and it also finished behind [1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-019-d512-600x-lr1.4) at 4.003200 — third of the five rates tested at this length, ahead of only 1.0 and 0.7. [The six-layer wave's own probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7), the only earlier rung to test a rate above its grid at this length, lost by 0.0058, so the precedent held almost exactly. All twenty-one runs in the two-layer dense Muon tree have now finished and 600x is the sweep's longest length, so this closes the tree. It also caps a pattern: 1.7 lost to 1.4 at every shorter length and won at this one by 0.0010, and 2.2 losing says the best rate stops climbing with length rather than continuing past 1.7.
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry Dial flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has set a record for the run. Steps 102,250-102,499 closed at a median of 0.607133 on the output projection, first of the run's eighty-three closed 250-step blocks and above the 0.602308 that steps 101,500-101,749 had held since Sunday. The block stood second at 18 of its 25 readings, on 0.598294, and its last seven pushed it past the record rather than back — the outcome the three closed blocks that had stood that high at 18 readings all pointed to, since each finished in the run's top three. Five of the run's seven highest-closing blocks now fall inside its last seven, against four an hour ago. Train loss has still not followed: the 500-step block 102,000-102,499 reads 1.21263 across 227 readings, inside the 1.2111-1.2164 band the last ten closed blocks held. The hero reached step 102,494 at 11:01:47Z at an 18.1-second median loop time over its last 200 steps, and its next reading on held-out text is due at step 104,999, near 7:40 p.m. (ET). W&B's sampled history, which feeds this page's progress bar and chart, is current at step 102,464.
herograd-norm
AUG-LIN0-2L-DENSE-MUONH-X600-HIGH-2P2-d512-600x-lr2.2
The 2.2 rung of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) posted its step-19,000 reading on held-out text and pulled level with the precedent that decides it: 4.0489 against [1.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) 4.0349, a deficit of 0.0141 after 0.0182 at step 18,000 — its fourth narrowing in five checks. [The six-layer wave's own probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7), the only earlier rung to test a rate above the sweep's range, was 0.0132 behind [its baseline](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) at step 19,000, so the 2.2 rung now trails the probe's shape by 0.0009, against 0.0047 at step 18,000 and 0.0161 at step 16,000. Matching the probe is still a loss: it finished 0.0058 behind and lost at every length it ran. The rung stood at 19,975 of its 21,150 steps at 11:03:12Z, stepping at about 0.98 seconds a step, for a finish near 7:22 a.m. (ET) and two readings left, at steps 20,000 and 21,000.
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry Dial flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has firmed up near the top of the run. Steps 102,250-102,499 hold a median of 0.598294 on the output projection at 18 of their 25 readings — second of the run's eighty-two closed blocks measured at the same reading count, behind only the 0.602951 of the record block 101,500-101,749. Only three closed blocks read within 0.010 of it at 18 readings, and they are the run's three highest closes: 0.591748, 0.597565 and 0.602308. Every block that stood this high at this point therefore finished in the run's top three, so this one is likely to as well — which would put four of the run's seven highest-closing blocks inside its last seven — a firmer read than the unpinned close reported twenty minutes ago at 12 readings. Train loss has still not followed: the 500-step block 102,000-102,499 reads 1.21240 across 198 readings, inside the 1.2111-1.2163 band the last ten closed blocks held. The hero reached step 102,428 at 10:42:12Z at an 18.3-second median loop time over its last 200 steps, and cleared its 6:03 a.m. (ET) hourly save in a single 21.0-second step. Its next reading on held-out text is due at step 104,999, near 7:50 p.m. (ET). W&B's sampled history, which feeds this page's progress bar and chart, has also caught up: it now reads step 102,383 against the run's 102,428, against the ~170-step lag reported an hour ago.
herograd-norm
AUG-LIN0-2L-DENSE-MUONH-X600-HIGH-2P2-d512-600x-lr2.2
The 2.2 rung of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) posted its step-18,000 reading on held-out text at its narrowest gap yet — 4.0676 against [1.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) 4.0494, a deficit of 0.0182 after 0.0215 at step 17,000 — but lost ground against the precedent that decides it. [The six-layer wave's own probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7), the only earlier rung to test a rate above the sweep's range, was 0.0135 behind [its baseline](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) at step 18,000, so the 2.2 rung is now 0.0047 further back against the probe's shape, against 0.0025 at step 17,000. That probe finished 0.0058 behind and lost at every length it ran. The rung stood at 18,888 of its 21,150 steps at 10:43:41Z, stepping at about 1.12 seconds a step and still short of its roughly 1.0 pre-restart pace, for a finish near 7:25 a.m. (ET) this morning.
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry Dial flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has opened its next block near the record rather than falling back. Steps 102,250-102,499 hold a median of 0.598445 on the output projection at 12 of their 25 readings — fourth of the run's eighty-two closed blocks measured at the same reading count, behind only the 0.608250 of 102,000-102,249, the 0.606334 of 101,500-101,749 and the 0.605051 of 100,750-100,999, which are the three highest-closing blocks the run has. All three fall inside the last seven blocks. Only four closed blocks read within 0.010 of it at 12 readings and they closed anywhere between 0.564901 and 0.602308, so the close is not yet pinned. Train loss has still not followed: the 500-step block 102,000-102,499 reads 1.21212 across 169 readings, inside the 1.2111-1.2163 band the last ten closed blocks held. The hero reached step 102,370 at 10:24:45Z at an 18.5-second median loop time, with no two-step window slower than 21.0 seconds a step since the 23.8-second step at 5:32 a.m. (ET). Its next reading on held-out text is due at step 104,999, near 7:55 p.m. (ET). Note that W&B's sampled history, which feeds this page's progress bar and chart, has returned byte-identical files for three pipeline runs and is pinned at step 102,201 — it now trails the run by about 170 steps.
herograd-norm
AUG-LIN0-2L-DENSE-MUONH-X600-HIGH-2P2-d512-600x-lr2.2
The 2.2 rung of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) posted its step-17,000 reading on held-out text and cut its gap to the narrowest of the test: 4.1085 against [1.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) 4.0871, a deficit of 0.0215 after the 0.0339 at step 16,000 that had ended three checks of narrowing. That closes most of the distance to the precedent. [The six-layer wave's own probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7), the only earlier rung to test a rate above the sweep's range, was 0.0190 behind [its baseline](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) at step 17,000 — the 2.2 rung now trails by about as much at the same point, rather than the twice-as-far it stood an hour ago. That probe went on to finish 0.0058 behind and lost at every length it ran. The rung has not recovered its pre-restart pace: it is stepping at about 1.29 seconds a step over its last 400 steps against roughly 1.0 before the break, and stood at 17,905 of its 21,150 steps at 10:24Z, a finish near 7:40 a.m. (ET).
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry Dial flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) closed its newest block below the record. Steps 102,000-102,249 finished at a median of 0.591748 on the output projection, third of the run's eighty-two closed blocks, behind the 0.602308 of the record block 101,500-101,749 and the 0.597565 of 100,750-100,999. It read 0.594565 at 18 of its 25 readings, so the last seven pulled it down rather than up, and the run's top three blocks now all fall inside the last six. Train loss has still not followed: the 500-step block 102,000-102,499 reads 1.21103 across 109 readings, the lowest of the last ten. The hero reached step 102,234 at 09:42:23Z at an 18.4-second median loop time; one step took 23.8 seconds at 5:32 a.m. (ET), the first over 19.3 seconds since 08:41Z and off the hourly-save cadence. Its next reading on held-out text is due at step 104,999, near 7:50 p.m. (ET).
herograd-norm
AUG-LIN0-2L-DENSE-MUONH-X600-HIGH-2P2-d512-600x-lr2.2
The 2.2 rung of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) posted its step-16,000 reading on held-out text at 09:33:01Z, its first since the restart, and gave back ground for the first time since step 12,000: 4.1371 against [1.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) 4.1031, a deficit of 0.0339 after 0.0300 at step 15,000 and three straight checks of narrowing. [The six-layer wave's own probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7), the only earlier rung to test a rate above the sweep's range, was 0.0178 behind [its baseline](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) at the same step — half the 2.2 rung's deficit — and went on to narrow to 0.0058 and still lose at every length it ran. The rung passed step 16,372 at 09:44:18Z at about 0.97 seconds a step, a finish near 7:00 a.m. (ET).
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry Dial flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has climbed back to near its record, so the dip reported an hour ago was one block and not a turn. The open 250-step block, steps 102,000-102,249, holds a median of 0.594565 on the output projection at 18 of its 25 readings — second of the run's 81 closed blocks measured at the same reading count, behind only the 0.602951 of the record block 101,500-101,749. Only three closed blocks read within 0.015 of it at 18 readings, and they are the run's three highest; they went on to close between 0.575310 and 0.602308, so this block will land near the top of the run whichever way it goes. Train loss has still not followed: the 500-step block 102,000-102,499 reads 1.21029 across 86 readings, the lowest of the last ten. The hero reached step 102,173 at 09:23:34Z at an 18.2-second median loop time, with no step over 19.3 seconds since 08:41Z — the 5:01 a.m. (ET) hourly-save window cost less than the 20.7 and 22.9 seconds the two saves before it did. Its next reading on held-out text is due at step 104,999, near 7:40 p.m. (ET).
herograd-norm
AUG-LIN0-2L-DENSE-MUONH-X600-HIGH-2P2-d512-600x-lr2.2
The 2.2 rung of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) restarted and lost about thirty-four minutes. Its last step before the break was 15,647 at 08:43:19Z; a new process attached to the same run id near 08:45:27Z — W&B's runtime counter reset from 18,646 seconds to 91 and the process's resident memory fell from 18.3 GB to 11.7 GB — and the first step back, 15,653, landed at 09:17:47Z, so the restore took about thirty-two minutes. The rung is stepping at about one second a step again and stood at 15,828 of its 21,150 steps at 09:24:51Z, a finish near 7:00 a.m. (ET). Its reading on held-out text is still the step-15,000 one, 4.1652 against [1.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) 4.1352, with the next due at step 16,000.
lr-sweeprestart
hero-ragged_a2a-nccl2307-ep-step81k
The first reading on held-out text since the gradient-norm record came in unchanged. The hero logged 2.221249 on Paloma at step 101,999 at 08:30:52Z, against 2.221206 at step 98,999 — a move of 0.00004, the smallest between any two readings this run has taken, and the check [Larry Dial named as what he wanted to see before treating the rise as a problem](https://discord.com/channels/1354881461060243556/1548520475397718068/1548529097792032830). The 250-step gradient-norm block that followed the record has also closed: steps 101,750-101,999 finished at 08:27:04Z with a median of 0.556654 on the output projection, seventeenth of the run's eighty-one closed blocks, against the 0.602308 that steps 101,500-101,749 set as the run high, which now stands alone. Train loss closed the same span at 1.21206 across 245 readings, the lowest of the last eight 500-step blocks. The hero reached step 102,035 at 08:42:29Z and is stepping about every 18.6 seconds; the one long step of the morning, 31.7 seconds at 08:31:17Z, was the evaluation itself.
herograd-normeval
AUG-LIN0-2L-DENSE-MUONH-X600-HIGH-2P2-d512-600x-lr2.2
The 2.2 rung of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) posted its step-15,000 reading on held-out text and narrowed again, to the smallest gap of the test: 4.1652 against [1.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) 4.1352, a deficit of 0.0300 after 0.0372 at step 14,000 and 0.0436 at 13,000. [The six-layer wave's own probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7), the only earlier rung to test a rate above the sweep's range, was 0.0211 behind [its baseline](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) at step 15,000 — still narrower than the 2.2 rung is now — and it finished 0.0058 behind and lost at every length it ran. At 08:43Z the rung is 15,647 of its 21,149 steps in at about 1.03 seconds a step, a finish near 6:17 a.m. (ET).
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry Dial flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) is holding mid-range as its 250-step block closes. The block 101,750-101,999 now has 23 of its 25 readings and a median of 0.556654 on the output projection — seventeenth of the run's eighty closed blocks measured at the same reading count, against the 0.597616 the record block 101,500-101,749 held there. All thirteen closed blocks that read within 0.010 of it at 23 readings went on to close between 0.537254 and 0.577529, none above the 0.602308 the record block finally set, so the record still reads as a spike rather than a new level. Train loss has not moved: the 500-step block 101,500-101,999 reads 1.21243 across 231 readings, inside the 1.2111-1.2164 band the last seven closed blocks held. The hero reached step 101,977 at 08:22:57Z at a 16.89-second median loop time, with no step over 20 seconds since the 20.7-second hourly save at 08:01:06Z. Its next reading on held-out text is due at step 101,999 near 4:30 a.m. (ET), the first since the peak. The sampled-history lag noted an hour ago has cleared — W&B now reports step 101,966 against the run's 101,977, so this page's progress bar and chart are current again.
herograd-norm
The 2.2 rung of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) posted its step-14,000 reading on held-out text and narrowed again, to the smallest gap of the test: 4.1937 against [1.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) 4.1566, a deficit of 0.0372 after 0.0436 at step 13,000 and 0.0495 at 12,000. The comparison that matters is [the six-layer wave's own probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7), the only earlier rung to test a rate above the sweep's range: at step 14,000 it was 0.0316 behind [its baseline](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) — narrower than the 2.2 rung is now — and it still finished 0.0058 behind and lost at every length it ran. So the 2.2 rung is tracing a losing shape from further back. At 08:22:49Z the rung is 14,497 of its 21,150 steps in, at 1.084 seconds a step, a finish near 6:23 a.m. (ET).
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry Dial flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has fallen well back from the record it set an hour ago. The open 250-step block, steps 101,750-101,999, holds a median of 0.556654 on the output projection at seventeen of its twenty-five readings — fifteenth of the run's eighty closed blocks measured at the same reading count, against the 0.597616 the record block 101,500-101,749 held there and the 0.587258 of 100,750-100,999. All twelve closed blocks that read within 0.010 of it at seventeen readings went on to close between 0.538465 and 0.577332, none of them above the 0.597565 that steps 100,750-100,999 set on Sunday evening, so the record block reads as a spike rather than a new level. Train loss has not moved: the 500-step block 101,500-101,999 reads 1.21259 across 203 readings, inside the 1.2111-1.2164 band the last seven closed blocks held. The hero cleared its thirty-first hourly save since Saturday's restart at 08:01:06Z, a single step of 20.7 seconds against an 18.2-second median, and reached step 101,920 at 08:05:07Z with no step slower than the 23.0 seconds of the 3:01 a.m. (ET) save since step 101,500. Its next reading on held-out text is due at step 101,999 near 4:30 a.m. (ET), the first since the peak. Note that W&B's sampled history, which feeds this page's progress bar and chart, has been pinned at step 101,734 for three reads while the run itself is at 101,920 — the counter trails the run by about 190 steps.
herograd-norm
The 2.2 rung of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) posted its step-13,000 reading on held-out text and cut its gap to the narrowest of the test: 4.2116 against [1.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) 4.1680, a deficit of 0.0436 after 0.0495 at step 12,000 and 0.0487 at 11,000. [The six-layer wave's own probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7), the closest precedent, narrowed at the same step in the same way — 0.0279 behind [its baseline](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4), from 0.0421 at step 12,000 — and still lost at every length it ran, so the 2.2 rung is tracing that shape from further back. At 08:06Z the rung is 13,572 of its 21,150 steps in, at about 1.08 seconds a step, a finish near 6:20 a.m. (ET).
lr-sweep
gradnorm-101316-pooled-20260914
A third reading of the gradient norm on the frozen step-101,316 checkpoint clears the ragged all-to-all backend of causing [the rise Larry Dial flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108). `gradnorm-101316-pooled-20260914` finished at 07:12:07Z on the pooled all-to-all path and returned 0.578236 on the output projection, against 0.584267 and 0.584210 from [the two ragged readings](https://wandb.ai/marin-community/marin_moe/runs/gradnorm-101316-ragged-v2-20260914) — a 1.0% spread, about a fifth of the 0.03 the run's block medians have climbed. `grad/norm/total` agrees to 0.5%; the two backends do diverge by several times on the `sconv_*` tensors, but those sit at 1e-4 and are dominated by rounding. The hero's own open 250-step block, steps 101,750-101,999, has meanwhile come off the record: at ten of its twenty-five readings it holds a median of 0.566696 — twelfth of the eighty closed blocks measured at ten readings, against 0.606334 for the record block 101,500-101,749 and 0.605051 for 100,750-100,999. The twelve closed blocks that read within 0.010 of it at ten readings all closed between 0.5385 and 0.5798, below the 0.602308 record. Train loss has not moved: the 500-step block 101,500-101,999 reads 1.21336 across 168 readings, inside the 1.2112-1.2164 band the last seven closed blocks held. The hero reached step 101,848 at 07:43:46Z at a median of 18.46 seconds a step, with no gap over 26.4 seconds in four hours, and [its next reading on held-out text](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) is due at step 101,999 near 4:30 a.m. (ET).
herograd-norm
The 2.2 rung of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) posted its step-12,000 reading on held-out text and its run of narrowing gaps stopped: 4.2521 against [1.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) 4.2026, a deficit of 0.0495 after 0.0487 at step 11,000, 0.0575 at 10,000 and 0.0576 at 9,000. [The six-layer wave's own probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7), the closest precedent, stood 0.0421 behind [its baseline](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) at step 12,000 and still lost at every length it ran, so the 2.2 rung is still further back than the precedent was. At 07:42Z the rung is 12,226 of its 21,150 steps in, at about 1.13 seconds a step, a finish near 6:30 a.m. (ET).
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry Dial flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has closed a stretch above every one the run has finished. The 250-step block covering steps 101,500-101,749 closed at 07:13:45Z with a median of 0.602308 on the output projection across its twenty-five readings — first of the run's eighty closed blocks, above the 0.597565 that steps 100,750-100,999 had held as the run high. Its high mark, 0.7782 at step 101,560, stays below the 0.8908 the run logged in the 100,250-100,499 block. The next block, steps 101,750-101,999, has opened at a median of 0.586560 on its first three readings. Train loss has not followed: the 500-step block covering steps 101,500-101,999 reads 1.21388 across the 138 readings it has, inside the 1.2111-1.2163 band the last seven closed blocks held. The hero reached step 101,781 at 07:23:38Z at a median of 18.4 seconds a step, with no gap over 23 seconds since 05:47Z, and [its reading on held-out text](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) is still the one from step 98,999, with the next due at step 101,999 near 4:30 a.m. (ET).
herograd-norm
The 2.2 rung of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) posted its step-11,000 reading on held-out text and cut its gap to the narrowest of the test: 4.2645 against [1.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) 4.2158, a deficit of 0.0487 after 0.0575 at step 10,000 and 0.0576 at step 9,000. [The six-layer wave's own probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7), the closest precedent, stood 0.0372 behind [its baseline](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) at the same step and still lost at every length it ran, so the 2.2 rung is still closing from further back than the precedent did. At 07:23Z the rung is 11,213 of its 21,150 steps in, but has slowed to about 1.25 seconds a step from 0.97, pushing its finish to near 6:50 a.m. (ET).
lr-sweep
The 2.2 rung of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) posted its step-9,000 and step-10,000 readings on held-out text and settled at its narrowest gap yet: 4.3350 and 4.2951 against [1.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) 4.2774 and 4.2376, deficits of 0.0576 and 0.0575 after the 0.0689 at step 8,000. [The six-layer wave's own probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7), the closest precedent, stood 0.0421 and 0.0437 behind [its baseline](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) at those same steps and still lost at every length it ran, so the 2.2 rung is still closing from further back than the precedent did. At 07:03Z the rung is 10,073 of its 21,150 steps in at about 0.97 seconds a step, a finish near 6:03 a.m. (ET).
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry Dial flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has taken the open block past every precedent the run has. The 250-step block covering steps 101,500-101,749 reads a median of 0.604364 on the output projection at twenty-two of its twenty-five readings — first of all seventy-nine blocks the run has closed, measured at the same reading count, and above the 0.597565 that steps 100,750-100,999 set as the closed-block high. The five closed blocks that read nearest to it at twenty-two readings went on to close between 0.5699 and 0.5976, so this one has no precedent to be read against. Its high mark, 0.7782 at step 101,560, still sits below the 0.8908 the run has already logged, and the train loss is flat: the 500-step block covering steps 101,500-101,999 reads 1.21541 across the steps it has, inside the 1.2111-1.2163 band the last eight closed blocks held. The hero cleared its hourly save at 07:01:32Z — the thirtieth since Saturday's restart — and reached step 101,710 straight after, still about eighteen seconds a step. The step and loss shown on this page come from a sampled W&B feed that trails the run by about fifty steps.
herograd-norm
The 2.2 rung of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) posted its step-8,000 reading on held-out text and narrowed its gap again: 4.3587 against [1.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) 4.2898, a deficit of 0.0689 after the 0.0741 at step 7,000 that was its widest. [The six-layer wave's own probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7), the closest precedent, stood 0.0511 behind [its baseline](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) at the same step and still lost at every length it ran, so the 2.2 rung is closing from further back than the precedent did. At 06:43Z the rung is 8,999 of its 21,150 steps in at about 1.03 seconds a step, a finish near 6:12 a.m. (ET).
lr-sweep
[The gradient-norm rise Larry Dial flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) did not stay level with the run high. The 250-step block covering steps 101,500-101,749 reads a median of 0.606334 on the output projection at sixteen of its twenty-five readings, above the 0.597565 that steps 100,750-100,999 set as the run high; the same block read 0.597616 at eleven readings, so the figure is still moving and the block is not closed. Its two high marks, 0.7782 at step 101,560 and 0.7567 at 101,570, stay below the 0.8908 the run has already logged, and training loss is flat across the same span: a median of 1.21582 against 1.21504 and 1.21513 for the two preceding 500-step blocks. The hero reached step 101,650 at 06:42:37Z, still about seventeen seconds a step, and cleared its hourly save at 06:00:43Z. The step and loss shown on this page come from a sampled W&B feed that has not moved past step 101,562 since 06:21Z, so the progress bar trails the run by about ninety steps.
herograd-norm
[The repeat of the gradient-norm measurement finished](https://wandb.ai/marin-community/marin_moe/runs/gradnorm-101316-ragged-v2-20260914) at 06:38:21Z. It restored the same `step-101316` weights onto the same sixty-four GPUs, took the same six steps over held-out text and wrote no checkpoint, and returned [the first measurement's](https://wandb.ai/marin-community/marin_moe/runs/gradnorm-101316-ragged-20260914) readings almost exactly: 0.5659, 0.6008, 0.6040, 0.6110 and 0.5842 on the output projection at steps 101,317-101,321, against 0.5662, 0.6008, 0.6040, 0.6106 and 0.5843 the first time. Both runs sit inside the band [the hero itself](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) logged around that step, so the raised norm reproduces on a fresh process and is not an artefact of the training job's own logging. Nothing in [the Discord thread](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) or on GitHub says what the measurement is for.
herograd-norm
[Matt Wittmann has started measuring the hero's gradient norm off the training job](https://wandb.ai/marin-community/marin_moe/runs/gradnorm-101316-ragged-20260914). At 05:21Z he restored the run's own `step-101316` weights onto sixteen four-GPU workers — sixty-four of the hero's seven hundred and four GPUs, the size of one of its eleven data-parallel replicas — with the model and optimizer settings unchanged, took six steps over held-out text logging every step, and wrote no checkpoint. Its output-projection readings ran 0.5662 to 0.6106 across steps 101,317-101,321, inside the band [the hero itself](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) logged around that step (0.5776 at step 101,320), so the raised norm shows up again on one eleventh of the batch and on text the run has not trained on. The probe finished at 05:39:49Z and [an identical repeat](https://wandb.ai/marin-community/marin_moe/runs/gradnorm-101316-ragged-v2-20260914) started at 06:20:03Z, still compiling at 06:28Z. Nothing in [the Discord thread](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) or on GitHub says what the measurement is for.
herograd-norm
The 2.2 rung of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) posted its step-7,000 reading on held-out text and gave back both of the checks it had closed: 4.3940 against [1.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) 4.3199, a deficit of 0.0741. That is the widest of its seven readings, after 0.0536 at step 6,000 and 0.0655 at step 5,000. [The six-layer wave's own probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7), the closest precedent, stood at 0.0530 against [its baseline](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) at the same step and still lost at every length it ran. At 06:28Z the rung is 8,105 of its 21,150 steps in at about 1.08 seconds a step, a finish near 6:23 a.m. (ET).
lr-sweep
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) opened its next block above the run high and has eased back level with it. The 250-step block covering steps 101,500-101,749 stands at a median of 0.597616 on the output projection at eleven of its twenty-five readings, against the 0.597565 that steps 100,750-100,999 set as the run high; it read 0.6151 when nine readings were in. Two single readings inside it, 0.7782 at step 101,560 and 0.7567 at 101,570, are the block's high marks and sit below the 0.8908 the run has already seen. The hero reached step 101,601 at 06:28Z, still taking a step about every seventeen seconds.
herograd-norm
The 2.2 rung of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) posted its step-6,000 reading on held-out text and narrowed its gap for a second straight check: 4.4135 against [1.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) 4.3598, a deficit of 0.0536 after 0.0655 at step 5,000 and 0.0689 at step 4,000. [The six-layer wave's own probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7), the closest precedent, stood at 0.0494 against [its baseline](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) at the same step and still lost at every length it ran, so the 2.2 rung is closing from further back than the precedent did. At 06:04Z the rung is 6,787 of its 21,150 steps in at about 1.03 seconds a step, a finish near 6:10 a.m. (ET).
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
The 250-step block covering steps 101,250-101,499 closed at a median of 0.5753 on the output projection — inside the 0.5649-0.5798 range this page projected for it, and below the 0.5976 that steps 100,750-100,999 set as the run high. It ranks fifth of the run's seventy-nine closed blocks, so [the gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) closed this stretch as a high block and not a new high. The hourly save near 2:00 a.m. (ET) also cleared: steps slowed through 06:00-06:01Z and then resumed, reaching step 101,524 at 06:05:41Z. The 500-step train-loss block covering steps 101,000-101,499 closed at 1.2150, inside the 1.2111-1.2182 band the previous twelve closed blocks held, and [the run's reading on held-out text](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) is still the one from step 98,999, with the next due at step 101,999 near 4:30 a.m. (ET).
herotraining-dynamicsgradient-norm
The 2.2 rung of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) posted its step-5,000 reading on held-out text at 1:32 a.m. (ET) and closed part of its gap for the first time: 4.4529 against [1.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) 4.3874 at the same step, a deficit of 0.0655 after 0.0689 at step 4,000. The four checks before this one all widened. The rung is 5,575 of its 21,150 steps in at about 1.03 seconds a step, a finish near 6:10 a.m. (ET).
lr-sweep
The 2.2 rung of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) posted its step-4,000 reading on held-out text and fell further behind the rate it has to beat: 4.4901 against [1.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) 4.4212, a deficit of 0.0689 that has widened at all four checks the two share. [The six-layer wave's own probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7), the closest precedent, had already started closing on [its baseline](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) by this step — 0.0531, down from 0.0653 at step 3,000 — and still lost at every length it ran. At 1:23 a.m. (ET) the rung is 4,525 of its 21,150 steps in at about 1.09 seconds a step, a finish near 6:25 a.m. (ET).
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has eased back under the run high inside the open block. The 250-step block covering steps 101,250-101,499 is fourteen of its twenty-five samples in at a median of 0.5765 on the output projection, down from the 0.6058 it read at nine samples and below the 0.5976 that steps 100,750-100,999 closed at. It is third of the run's seventy-eight closed blocks at this sample count, and the four closed blocks that read nearest to it here went on to close between 0.5649 and 0.5798 — a high block, not a new high, on the evidence so far. Neither the training loss nor [the run's reading on held-out text](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) has turned: the 500-step block covering steps 101,000-101,499 reads 1.2150 across the steps it has, inside the 1.2112-1.2182 band the last twelve closed blocks held, and the held-out reading is unchanged at step 98,999 with the next due at step 101,999 near 4:30 a.m. (ET).
herotraining-dynamicsgradient-norm
The rate above the tested range in [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) has doubled its pace again and is reading behind the rate it has to beat. At 1:08 a.m. (ET) [the 2.2 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-X600-HIGH-2P2-d512-600x-lr2.2) is 3,680 of its 21,150 steps in and taking about 1.03 seconds a step, against 2.47 over its opening 500 steps — a finish near 6:05 a.m. (ET), three hours earlier than the 9:20 a.m. (ET) this page gave forty minutes ago. Its held-out readings trail [1.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) at all three steps the two share, and the gap widens: 5.3609 against 5.3441 at step 1,000, 4.6705 against 4.6179 at 2,000, and 4.5475 against 4.4798 at 3,000. [The six-layer wave's own probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7), which lost at every length it ran, trailed [its baseline](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) by a similar 0.0240, 0.0570 and 0.0653 at those same steps and still closed to within 0.0058 by the finish, so an early deficit at this size does not settle the length.
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has opened its next block above the run high. The 250-step block covering steps 101,250-101,499 is nine of its twenty-five samples in at a median of 0.6058 on the output projection, against the 0.5976 that steps 100,750-100,999 closed at on Sunday evening. Only two of the run's seventy-eight closed blocks read this high at nine samples: one closed at 0.5976, the run high itself, and the other at 0.5591, ninth from the top — so a high early reading does not fix where this one lands. Its samples swing from 0.3713 to 0.7001, sixteen are still out, and it closes near 1:55 a.m. (ET). Neither the training loss nor [the run's reading on held-out text](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) has turned: the 500-step block covering steps 101,000-101,499 reads 1.2152 across the steps it has so far, inside the 1.2112-1.2182 band the last twelve closed blocks held, and the held-out reading is unchanged at step 98,999 with the next due at step 101,999 near 4:30 a.m. (ET).
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) cleared its twenty-eighth save since the restart and stepped straight through it. The write shows as about eighteen seconds of extra time spread across the five steps spanning 1:00 a.m. (ET), against a step time of about 17.5 seconds — the same small, contained shape as the twenty-seven before it. Step times were back near 18 seconds by step 101,322 at 1:02 a.m. (ET). [The September 12 silent hang](https://github.com/marin-community/marin/issues/8870) struck one step after a save, so each clean save is a checked-off risk; the twenty-ninth falls near 2:00 a.m. (ET).
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) did not extend to a second new high. The 250-step block covering steps 101,000-101,249 closed at 12:40 a.m. (ET) at a median of 0.5775 on the output projection, below the 0.5976 that steps 100,750-100,999 set eighty minutes earlier and below the 0.5798 that had stood as the run high before that. It is the third-highest of the run's seventy-eight closed blocks, and the first block since Sunday evening to follow a new high without matching it. Neither the training loss nor [the run's reading on held-out text](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) has turned: the 500-step block covering steps 101,000-101,499 reads 1.2154 across the steps it has so far, inside the 1.2112-1.2182 band the last twelve closed blocks held, and the held-out reading is unchanged at step 98,999 with the next due at step 101,999 near 4:31 a.m. (ET). The block after it opened at 0.6189 and closes near 1:58 a.m. (ET).
herotraining-dynamicsgradient-norm
The 2.2 rung of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) has settled to a steady pace, and it now reports about five hours earlier than its opening steps implied. At 12:24 a.m. (ET) it is 1,409 steps into 21,150 and taking about 1.6 seconds a step, against the 2.0-second average its start-up cost still carries — a finish near 9:20 a.m. (ET), inside the eleven hours [the six-layer wave's own probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7) took over the same step count rather than well outside it. The bar is unchanged: it must beat [1.7's 4.0022 on held-out text](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) to push the two-layer grid up.
lr-sweep
The two-layer wave of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) has drawn a rung above its tested range. [A 2.2 rate opened at 11:32 p.m. (ET)](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-X600-HIGH-2P2-d512-600x-lr2.2) at the fifth and longest training length, the one length [the 1.7 rate at the top of the grid won](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) — and it won it by 0.0010. It logged its first step at 11:37 p.m. (ET) and is about 650 of its 21,150 steps in; at its opening pace it closes near 2:45 p.m. (ET), slower than the eleven hours [the six-layer wave's own probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7) took over the same step count. It has to beat 1.7's 4.0022 on held-out text to push the grid up: the six-layer probe missed [its own mark](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) by 0.0058 at this length and lost at the two shorter lengths it ran, which left that grid standing.
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has opened its next block high, but under the run high the block before it just set. The 250-step block covering steps 101,000-101,249 is thirteen of its twenty-five samples in at a median of 0.5775 on the output projection, against the 0.5976 that steps 100,750-100,999 closed at; its samples swing from 0.4057 to 0.6394. Only three of the run's seventy-seven closed blocks read this high at thirteen samples, and they closed at 0.5738, 0.5649 and 0.5976 — high, middling and a new high — so a high early reading does not fix where this one lands. Twelve samples are still out and it closes near 12:38 a.m. (ET). Neither the training loss nor [the run's reading on held-out text](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) has turned: the 500-step block covering steps 101,000-101,499 reads 1.2152 at 72 samples, inside the 1.2113-1.2181 band the last twelve closed blocks held, and the held-out reading is unchanged at step 98,999 with the next due at step 101,999 near 4:30 a.m. (ET).
herotraining-dynamicsgradient-norm
hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) cleared its twenty-seventh save since the restart and stepped straight through it. The write shows as about thirteen seconds of extra time across the two steps spanning 12:00 a.m. (ET), against a step time of about 18.4 seconds — the same small, contained shape as the twenty-six before it, and cheaper than most of them. Step times were back near 18.4 seconds by step 101,122 at 12:01 a.m. (ET). [The September 12 silent hang](https://github.com/marin-community/marin/issues/8870) struck one step after a save, so each clean save is a checked-off risk; the twenty-eighth falls near 1:00 a.m. (ET).
herocheckpoint
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) closed the block now in question at a new run high. The 250-step block covering steps 100,750-100,999 closed at 11:21 p.m. (ET) at a median of 0.5976 on the output projection — the highest of the run's seventy-seven closed blocks, above the 0.5798 set over steps 99,750-99,999 on Sunday evening, and the twenty-fourth new high this job has set since it began logging at 0.3596 on Wednesday evening. The 0.0178 it adds is the largest step up in the run high since steps 95,750-95,999 closed at 0.5699 late Saturday. Its samples swing from 0.3467 to 0.7990; the block after it has one sample, at 0.4907, and closes near 12:38 a.m. (ET). Neither the training loss nor [the run's reading on held-out text](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) has turned: the 500-step block covering steps 100,500-100,999 reads 1.2151 at 268 samples, inside the 1.2112-1.2182 band the last twelve closed blocks held, and the held-out reading is unchanged at step 98,999 with the next due at step 101,999 near 4:34 a.m. (ET).
herotraining-dynamicsgradient-norm
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has eased inside the block now open, but the block still stands above the run's high. The 250-step block covering steps 100,750-100,999 is eighteen of its twenty-five samples in at a median of 0.5924 on the output projection — down from the 0.6051 it read at twelve samples, and above the 0.5798 run high set over steps 99,750-99,999. It is the highest eighteen-sample reading of the run's seventy-six closed blocks and the open one: the next, 0.5746 over steps 99,750-99,999, went on to close at that run high, and no closed block has read above 0.5798 at eighteen samples. Its samples swing from 0.3467 to 0.7990. Seven samples are still out and the last falls near 11:22 p.m. (ET). Neither the training loss nor [the run's reading on held-out text](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) has turned: the 500-step block covering steps 100,500-100,999 reads 1.2153 at 230 samples, inside the 1.2112-1.2182 band the last twelve closed blocks held, and the held-out reading is unchanged at step 98,999 with the next due at step 101,999 near 4:32 a.m. (ET).
herotraining-dynamicsgradient-norm
hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) cleared its twenty-sixth save since the restart and stepped straight through it. The write shows as about eight seconds of extra time on the single step spanning 10:59 p.m. (ET), plus about twelve seconds spread over the three steps after it, against a step time of about 18.5 seconds — the same small, contained shape as the twenty-five before it. Step times were back at 18.5 seconds by step 100,929 at 11:02 p.m. (ET). [The September 12 silent hang](https://github.com/marin-community/marin/issues/8870) struck one step after a save, so each clean save is a checked-off risk; the twenty-seventh falls near 11:59 p.m. (ET).
herocheckpoint
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) closed a third block running without a new high, and the block after it has opened at the highest half-filled reading the run has produced. The 250-step block covering steps 100,500-100,749 closed at 10:04 p.m. (ET) at a median of 0.5649 on the output projection — ninth of the run's seventy-six closed blocks, under the 0.5798 run high set over steps 99,750-99,999, and close to the 0.5620 this page projected for it at twenty-four samples. The block covering steps 100,750-100,999 is twelve of its twenty-five samples in at 0.6051, above that run high: no closed block in the run has read 0.60 at twelve samples, and the nearest, 0.5884, belongs to the block that just closed at 0.5649. Its samples swing from 0.3467 to 0.7990, and the three highest eleven-sample readings the run has given — 0.6047, 0.6044 and 0.6013 — closed at 0.5591, 0.5649 and 0.5738, so a high half-block has not once carried through to a high close. Thirteen samples are still out and the block closes near 11:23 p.m. (ET). Neither the training loss nor [the run's reading on held-out text](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) has turned: the 500-step block covering steps 100,500-100,999 reads 1.2159 at 191 samples, inside the 1.2113-1.2182 band the last twelve closed blocks held, and the held-out reading is unchanged at step 98,999 with the next due at step 101,999 near 4:28 a.m. (ET). [The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) has stepped without a break for more than twenty-five hours, and its twenty-sixth save since the restart falls near 10:59 p.m. (ET).
herotraining-dynamicsgradient-norm
hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) cleared its twenty-fifth save since the restart and stepped straight through it. The write shows as about five seconds of extra time on the single step spanning 9:59 p.m. (ET), against a step time of about 18.4 seconds — the same small, contained shape as the twenty-four before it and the cheapest of them so far, and again on the hourly slot this page named an hour ahead. [The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) is on course to close a third block running without a new high: the 250-step block covering steps 100,500-100,749 is twenty-four of its twenty-five samples in at a median of 0.5620 on the output projection — eleventh of the run's seventy-five closed blocks, under the 0.5798 run high set over steps 99,750-99,999, and down from the 0.5884 it read at twelve samples and the 0.5716 at eighteen. It closes near 10:07 p.m. (ET). The training loss has not turned either: the 500-step block covering steps 100,500-100,999 reads 1.2153 at 126 samples, inside the 1.2112-1.2182 band the last twelve closed blocks held, and [the run's reading on held-out text](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) is unchanged at step 98,999 with the next due at step 101,999, about six hours out. The twenty-sixth save falls near 10:59 p.m. (ET).
herocheckpointtraining-dynamicsgradient-norm
The two-layer wave of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) is complete, all twenty of its rungs finished, and the highest of its four rates took the fifth and last training length. [The 1.4 rate closed its 21,150 steps at 10:03 p.m. (ET) at 4.0032 on held-out text](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-019-d512-600x-lr1.4), 0.0010 behind [the 1.7 rate's 4.0022](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7), after leading at step 20,000 and reading 4.0038 at step 21,000. It gained 0.0137 over its closing 1,149 steps against the 0.0147 a win needed — close to the 0.0141 that the three earlier finishers' wind-down trend projected, and the shortfall this page called. Across the wave 1.4 wins the four shorter lengths, at 4.5915, 4.3191, 4.1208 and 4.0490, and 1.7 the longest, and 1.4's margin over 1.7 shrank at every step up in length — 0.0099, 0.0026, 0.0013, 0.0015 — before turning over to 1.7's 0.0010, so the best rate drifts toward the top of the grid as the training length grows. A winner at the top of the tested grid is the condition that opens a rung above the range. [The six-layer wave closed at the top of its own grid too](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4), one rate lower, and [the probe it drew above that range](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7) lost at all three lengths it ran, which left the grid standing. No two-layer probe has appeared as of 10:06 p.m. (ET).
lr-sweep
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has eased back inside the block now open. The 250-step block covering steps 100,500-100,749 is eighteen of its twenty-five samples in at a median of 0.5716 on the output projection — under the 0.5798 run high set over steps 99,750-99,999, and down from the 0.5884 it read at twelve samples. Held there it would close sixth of the run's seventy-five closed blocks: high, but not a new high. Seven samples are still out and the block closes near 10:07 p.m. (ET). The training loss has not turned either: the 500-step block covering steps 100,500-100,999 reads 1.2145 at 97 samples, inside the 1.2112-1.2182 band the last twelve blocks held, and [the run's reading on held-out text](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) is unchanged at step 98,999. The hero has stepped without a break for twenty-four and a half hours, and its twenty-fifth save since the restart falls near 9:59 p.m. (ET).
The two-layer wave of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) is down to one unfinished rate at its fifth and last training length, and the length is still too close to call. [The 1.0 rung closed its 21,150 steps at 9:36 p.m. (ET) at 4.0140 on held-out text](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-018-d512-600x-lr1), second of the three finishers behind [1.7's 4.0022](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) and ahead of [0.7's 4.0261](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-017-d512-600x-lr0.7) — the place this page projected for it, and it found 0.0107 over its closing 1,149 steps against the 0.0225 a win needed. [The 1.4 rate that won the four shorter lengths](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-019-d512-600x-lr1.4) is the only one still stepping, and at step 20,000 — the newest step all four reached — it leads at 4.0169, ahead of 1.7's 4.0189, 1.0's 4.0247 and 0.7's 4.0326. It must gain 0.0147 over its own closing 1,149 steps to take the length. Each finisher gained more in the wind-down to zero learning rate the higher its rate — 0.0065 at 0.7, 0.0107 at 1.0, 0.0167 at 1.7 — and reading 1.4 off that trend gives about 0.0141, a hair under what it needs. It closes near 10:02 p.m. (ET).
The two-layer wave of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) has its stalled rung back and a new leader at the newest matched step. [The 1.0 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-018-d512-600x-lr1) resumed at 9:09 p.m. (ET) after 58 minutes without a logged step at 19,879, took its step-20,000 reading at 4.0247 on held-out text, and has about 600 of its 21,150 steps left, closing near 9:36 p.m. (ET). It sits second at that step, behind [1.7's 4.0189](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) and ahead of [0.7's 4.0326](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-017-d512-600x-lr0.7), and needs 0.0225 over its last 1,149 steps to take the length from [1.7's finishing 4.0022](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) — more than the 0.0167 that 1.7 found over the same closing stretch and the 0.0065 that 0.7 found, so second is the likelier place. At step 18,000, the newest step all four rungs have reached, [the 1.4 rate that won the four shorter lengths](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-019-d512-600x-lr1.4) now leads at 4.0431, ahead of 1.0's 4.0448, 1.7's 4.0494 and 0.7's 4.0502, after reading third of the four at step 17,000. It has about 2,150 steps left and closes near 10:05 p.m. (ET), and needs 0.0409 from step 18,000 to win the length — between the 0.0472 that 1.7 gained from there and the 0.0241 that 0.7 gained, so the length still cannot be called.
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has turned back up inside the block now open. The 250-step block covering steps 100,500-100,749 is twelve of its twenty-five samples in at a median of 0.5884 on the output projection — above the 0.5798 run high set over steps 99,750-99,999, and well above the 0.5470 that the block before it closed at. A half-filled block is a weak reading: the block over steps 99,000-99,249 sat at 0.6047 at eleven samples and closed at 0.5591, and the distance between a block's halfway median and its close has run to 0.05 in both directions. This one closes near 10:06 p.m. (ET). Neither the training loss nor [the run's last reading on held-out text](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) has turned: the 500-step block covering steps 100,000-100,499 closed at 1.2163, inside the band the last six blocks held, and the Paloma reading is unchanged at step 98,999.
herotraining-dynamicsgradient-norm
hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) passed a full day of unbroken stepping at 9:13 p.m. (ET), twenty-four hours to the minute after the in-place restart that cleared [the September 12 silent hang](https://github.com/marin-community/marin/issues/8870). It has logged 4,642 steps since, from 95,973 to 100,614, and the longest gap in them is the 178 seconds its first save cost — the day covers twenty-four hourly saves, all of which it stepped straight through. This is not the run's longest clear stretch: it went about sixty-five hours between the September 10 and September 12 hangs. The twenty-fifth save falls near 9:59 p.m. (ET).
heromilestonecheckpoint
The two-layer wave of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) has a second finisher at its fifth and last training length, and the order at matched steps has inverted again. [The 0.7 rate closed its 21,150 steps at 8:52 p.m. (ET) at 4.0261 on held-out text](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-017-d512-600x-lr0.7), 0.0239 behind [the 1.7 rate that closed at 8:34 p.m. (ET) at 4.0022](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) — and yet 0.7 led 1.7 at every step the two can be compared at, by 0.0179 at step 17,000. The whole gap turned over in the schedule's wind-down to zero learning rate, where 1.7 gained 0.0849 from step 17,000 against 0.7's 0.0431. [The 1.4 rate that won the four shorter lengths](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-019-d512-600x-lr1.4) reads 4.0732 at step 17,000, so it needs 0.0710 from there to take this one — more than 0.7 found in its wind-down, less than 1.7 found in its own. It has about 3,200 steps left and closes near 10:00 p.m. (ET), and the length cannot be called before then. [The 1.0 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-018-d512-600x-lr1) has logged no step since 8:11 p.m. (ET), a stall approaching an hour with a live heartbeat and 1,270 steps left, about twenty minutes of work. If 1.7 holds the length it wins at the top of the tested grid, the case that earned [the six-layer wave a probe above its range](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7).
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has closed a second block running without a new high. The 250-step block covering steps 100,250-100,499 closed at 8:47 p.m. (ET) at a median of 0.5470 on the output projection — seventeenth of the run's seventy-five closed blocks, its lowest close since 5:31 a.m. (ET), and well under the 0.5798 run high set over steps 99,750-99,999. The block after it is five samples in and swinging between 0.47 and 0.68, too early to read. Neither the training loss nor [the run's last reading on held-out text](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) has turned. The hero also cleared its twenty-fourth save since the restart and stepped straight through it: about sixteen seconds of extra time over steps 100,531-100,534, between 8:59 and 9:01 p.m. (ET), against a step time of about 18.1 seconds. It reaches twenty-four hours of unbroken stepping at 9:13 p.m. (ET), and the twenty-fifth save falls near 9:59 p.m. (ET).
herotraining-dynamicsgradient-normcheckpoint
The two-layer wave of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) has its first finisher at the fifth and last training length, and it is the highest of the four rates. [The 1.7 rung closed its 21,150 steps at 8:35 p.m. (ET) at 4.0022 on held-out text](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7) after reading last of the four at every matched step up to 16,000 — 4.1031 there, against [0.7's 4.0814](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-017-d512-600x-lr0.7), [1.0's 4.0851](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-018-d512-600x-lr1) and [1.4's 4.0933](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-019-d512-600x-lr1.4). It made all of that back in the schedule's wind-down to zero learning rate, gaining 0.1009 from step 16,000 to the finish where 0.7 has gained 0.0551 so far. [The 1.4 rate that won the four shorter lengths](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-019-d512-600x-lr1.4) is 0.0098 ahead of 1.7 at step 16,000 and so must gain 0.0911 from there to take this one; it is about 4,300 steps back and closes near 10:02 p.m. (ET), so the length cannot be called before then. Neither rate in between is stepping: [1.0 has logged nothing since 8:11 p.m. (ET)](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-018-d512-600x-lr1), thirty-three minutes with 1,270 steps left, and [0.7 nothing since 8:32 p.m. (ET)](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-017-d512-600x-lr0.7) with 74 steps left — a pause many times the eighty seconds those steps take. Both keep a live heartbeat, the same shape as the eleven stalls this wave has taken since it launched at 1:55 p.m. (ET).
lr-sweep
The two-layer wave of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) has reversed the order of its fifth and last training length inside the wind-down to zero learning rate. [The 1.7 rate now leads at 4.0189 on held-out text at step 20,000](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7), ahead of [0.7 at 4.0326](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-017-d512-600x-lr0.7), after sitting last of the four at the matched step-15,000 check with 4.1352 against 0.7's 4.0964. Both rungs are inside the last four hundred of their 21,150 steps and close near 8:33 p.m. (ET). [The 1.0 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-018-d512-600x-lr1) has logged no step since 8:11 p.m. (ET), a sixteen-minute stall with a live heartbeat and its first of the length, and [the 1.4 rate that won the four shorter lengths](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-019-d512-600x-lr1.4) trails by about 4,900 steps at 4.1160 and closes near 9:56 p.m. (ET), so the length cannot be called until it finishes.
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has eased further. The 250-step block covering steps 100,250-100,499 is seventeen of its twenty-five samples in at a median of 0.5470 on the output projection — below the 0.5724 of the block before it, below the 0.5798 run high set over steps 99,750-99,999, and the lowest reading of any block since 5:31 a.m. (ET). This would be the second block running to end without a new high. The training loss has not turned: the 500-step block covering steps 100,000-100,499 is forty-two of its fifty samples in at 1.2185, level with the 1.2182 the 95,500-95,999 block closed at, and [the run's last reading on held-out text](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) is unchanged at step 98,999.
herotraining-dynamicsgradient-norm
hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) cleared its twenty-third save since the restart and stepped straight through it. The write shows as about twenty-five seconds of extra time over steps 100,328-100,337, between 7:58 and 8:01 p.m. (ET), against a step time of about 18.4 seconds — the same small, contained shape as the twenty-two before it, and again on the hourly slot this page named an hour ahead. The twenty-fourth save falls near 8:58 p.m. (ET).
herocheckpoint
Sep 13
[The probe above the learning-rate sweep's tested range](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7) finished at 7:35 p.m. (ET) and placed third of the five rates at its length. It closed at 3.7100 on Paloma, behind [the 1.4 rung that won the length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) at 3.7042 and [the 1.1 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-005-d512-600x-lr1.1) at 3.7076, and ahead of [0.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7) at 3.7258 and [0.45](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-018-d512-600x-lr0.45) at 3.7628. Third is where it placed at [the shortest of the six-layer wave's lengths](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X30-HIGH-1P7-d512-30x-lr1.7) and at [the middle one](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X150-HIGH-1P7-d512-150x-lr1.7) too, each time behind 1.4 and 1.1 and ahead of the two lowest rates. The probe was launched because 1.4 sits at the top of the tested grid and a winner at the edge can mean the grid is too narrow; losing at all three lengths answers that, so the six-layer wave's best rate stays inside the grid and the range does not need widening.
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) did not set a second run high. The 250-step block covering steps 100,000-100,249 closed at 7:34 p.m. (ET) at a median of 0.5724 on the output projection — fifth-highest of the run's seventy-four closed blocks, below both the 0.5798 the block before it set as the run high and the 0.5773 that had held the high since this morning. Its last three samples came in well above the 0.5611 it read at twenty-two samples, so the block rose late and still fell short of a new high. The training loss did not follow: the 500-step block covering steps 99,500-99,999 closed at 1.2157, inside the band the blocks before it held, and [the run's last reading on held-out text](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) is unchanged at step 98,999.
herotraining-dynamicsgradient-norm
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has eased back from the high it set this evening. The 250-step block covering steps 100,000-100,249 is twenty-two of its twenty-five samples in at a median of 0.5611 on the output projection — below the 0.5798 of the block that closed at 6:14 p.m. (ET) and set the run's high, and below the 0.5645 of the block before that. Three samples are still out and the block closes near 7:34 p.m. (ET). Neither [the training loss nor the run's reading on held-out text](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) has turned.
herotraining-dynamicsgradient-norm
The two-layer wave of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) is past the halfway mark of its fifth and last training length, and the rate that won the first four does not lead it. At step 12,000 — the newest step all four rungs have reached — [the 0.7 rate reads 4.1338 on held-out text](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-017-d512-600x-lr0.7), [1.0 reads 4.1488](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-018-d512-600x-lr1), [the 1.4 rate that won the shorter four reads 4.1760](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-019-d512-600x-lr1.4) and [1.7 reads 4.2026](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-020-d512-600x-lr1.7). The order can still move: this length decides inside a wind-down to zero learning rate, and the six-layer probe gained two places inside its own wind-down this evening.
lr-sweep
[The probe above the learning-rate sweep's tested range](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7) took its step-20,000 reading on held-out text at 7:02 p.m. (ET) and left last place. It reads 3.7320 there, ahead of [the 0.7 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7) at 3.7366 and [the 0.45 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-018-d512-600x-lr0.45) at 3.7688, which puts it third of the five rates at this length, and its shortfall against [the 1.4 rung that won the length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) narrowed to 0.0079 from 0.0131 at step 19,000. It has stepped without interruption since its 6:27 p.m. (ET) recovery and closes its 21,150 steps near 7:33 p.m. (ET), so its finishing place lands within the hour.
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) cleared its twenty-second save since the restart and stepped straight through it. The write shows as about twelve seconds of extra time over steps 100,130-100,135, between 6:58 and 7:00 p.m. (ET), against a step time of about 18.4 seconds — the same small, contained shape as the twenty-one before it. The twenty-third save falls near 7:58 p.m. (ET).
herocheckpoint
[The probe above the learning-rate sweep's tested range](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7) lost forty-two minutes to a stall — the first stall outside the two-layer wave that has been losing time all afternoon. It logged no step between 5:45 and 6:27 p.m. (ET), then resumed within six steps of where it stopped, so it lost machine time rather than work. Its step-19,000 reading on held-out text landed at 6:34 p.m. (ET) at 3.7556, ahead of [the 0.45 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-018-d512-600x-lr0.45) and behind the other three, and 0.0131 short of [the 1.4 rung that won this length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) at the same step against 0.0135 at step 18,000 — the shortfall has stopped closing, but this length was decided inside its final wind-down to zero learning rate, so the probe's finishing place is still open. At 1.57 s a step it closes its 21,150 steps near 7:31 p.m. (ET), forty minutes later than the stall-free projection.
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) set a new high for the run after all. The 250-step block covering steps 99,750-99,999 closed at a median of 0.5798 on the output projection, above the 0.5773 of the 97,750-97,999 block and the highest of this run's seventy-three closed blocks — its last four samples came in high and lifted it past the 0.5694 it read three-quarters of the way through, so the two blocks that had ended without a new high did not become three. The training loss did not follow: the 500-step block covering steps 99,500-99,999 closed at 1.2158, inside the 1.2115-1.2184 span of the thirteen blocks before it. The run logged step 100,000 at 6:18 p.m. (ET), and the norm sits at about two-fifths of the 1.5 [Larry called the edge of known territory](https://discord.com/channels/1354881461060243556/1548520475397718068/1548525385933783112).
herotraining-dynamicsgradient-norm
The two-layer wave of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) closed its fourth training length at 6:17 p.m. (ET), and the rate that has won every length so far won again: [1.4 finished at 4.0490 on Paloma](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-015-d512-300x-lr1.4), 0.0015 clear of [1.7 at 4.0505](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-016-d512-300x-lr1.7), with [1.0 at 4.0632](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-014-d512-300x-lr1) and [0.7 at 4.0787](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-013-d512-300x-lr0.7) behind. 1.4 trailed 1.7 by 0.0543 at the matched step 9,000 check and made all of it back in the schedule's wind-down to zero learning rate, the same reversal that decided the three lengths before this one. The win also keeps the best rate inside the tested grid rather than at its top edge, so no probe above 1.7 is due here; the fifth and last length is running, with three rungs closing near 8:26 p.m. (ET) and [the 1.4 rung that stalled this afternoon](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-019-d512-600x-lr1.4) near 10:15 p.m. (ET).
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has eased again. The 250-step block covering steps 99,750-99,999 is twenty-one of its twenty-five samples in at a median of 0.5694 on the output projection, against the 0.5746 it read at ten samples half an hour earlier — below both the 0.5773 run high, set over steps 97,750-97,999 this morning, and the 0.5738 second-highest close, so this would be the third block running to end without a new high. Neither loss has turned: the 500-step training-loss block covering steps 99,500-99,999 is forty-six of its fifty samples in at 1.2156, the highest of the last six blocks but level with the 1.2156 the 97,000-97,499 block closed at, and the norm still sits at about a third of the 1.5 an earlier healthy run reached. Both blocks close near 6:17 p.m. (ET), when the run also passes step 100,000. [Jeff H's question about filing an issue to track the rise](https://discord.com/channels/1354881461060243556/1548520475397718068/1548583749795577897) is unanswered fifteen hours on.
herotraining-dynamicsgradient-norm
hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) cleared its twenty-first save since the restart and stepped straight through it. The write shows as about thirteen seconds of extra time over steps 99,940-99,943, between 5:59 and 6:00 p.m. (ET), against a step that otherwise takes about seventeen seconds; the run logged step 99,950 at 6:02 p.m. (ET), twenty hours and forty-eight minutes after the 9:13 p.m. (ET) restart that cleared [the silent hang](https://github.com/marin-community/marin/issues/8870). The save landed on the slot this page named an hour ahead, the seventh in a row to do so. The twenty-second is due near 6:58 p.m. (ET), and the next reading on held-out text is not due until step 101,999, about ten hours out.
herocheckpoint
[The slowest rung of the two-layer wave](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-019-d512-600x-lr1.4) stalled for a second time today, the eleventh stall in [the learning-rate sweep's](https://github.com/marin-community/marin/issues/7856) newest wave. It logged no step between 5:29 and 5:52 p.m. (ET), twenty-three minutes, and resumed at the next step with nothing replayed. It now trails [its three siblings at the wave's fifth and last training length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-017-d512-600x-lr0.7) by about 4,900 steps: those three close near 8:45 p.m. (ET) and this one near 9:55 p.m. (ET), so the length cannot be called until then.
lr-sweep
The two-layer wave of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) is three of four in at its fourth training length, and the rate that has won every length so far is the one still out. [The 1.0 rate finished at 5:47 p.m. (ET)](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-014-d512-300x-lr1) with 4.0632 on Paloma, second of the three in, behind [1.7 at 4.0505](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-016-d512-300x-lr1.7) and ahead of [0.7 at 4.0787](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-013-d512-300x-lr0.7). [The 1.4 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-015-d512-300x-lr1.4) is due near 6:15 p.m. (ET) and led the matched step 9,000 check at 4.1047, 0.0095 clear of 1.7 — but the schedule's wind-down to zero learning rate decides the order, and the gain it delivers grows with the rate: from step 9,000 to the finish 0.7 gained 0.0332, 1.0 gained 0.0420 and 1.7 gained 0.0637, against the 0.0543 1.4 needs to take the length. 1.4 won the three lengths before this one, by 0.0099, then 0.0026, then 0.0013 over 1.7, so a win for 1.7 here would be its first and would put the best rate at the very top of the tested grid.
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has turned back up. The 250-step block covering steps 99,750-99,999 is ten of its twenty-five samples in at a median of 0.5746 on the output projection, above both the 0.5645 the block before it closed at and the 0.5738 second-highest close, and just under the 0.5773 run high set over steps 97,750-97,999 this morning — so a new high is back in play when this block closes near 6:15 p.m. (ET), after two blocks that eased. Neither loss has turned: the 500-step training-loss block covering steps 99,500-99,999 is thirty-five of its fifty samples in at 1.2137, inside the 1.2106-1.2158 band of the four blocks before it, and the norm still sits at about a third of the 1.5 an earlier healthy run reached. [Jeff H's question about filing an issue to track the rise](https://discord.com/channels/1354881461060243556/1548520475397718068/1548583749795577897) is unanswered fifteen hours on.
herotraining-dynamicsgradient-norm
The two-layer wave of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) is half in at its fourth training length. [The 1.7 rate finished first](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-016-d512-300x-lr1.7) at 5:25 p.m. (ET) with 4.0505 on Paloma, ahead of [0.7 at 4.0787](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-013-d512-300x-lr0.7), which closed fifteen seconds later — a gap of 0.0282. 1.7 read last of the four at the matched step 7,000 check with 4.2389 against 0.7's 4.1862, so the wind-down to zero learning rate again carried it from last to first. The order is not settled: [the 1.0 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-014-d512-300x-lr1) and [the 1.4 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-015-d512-300x-lr1.4) are still running, due near 5:46 and 6:16 p.m. (ET), and 1.4 has won all three lengths the wave has finished. No run in the wave has stopped since [the 1.4 rung at the longest length picked up again at 4:55 p.m. (ET)](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-019-d512-600x-lr1.4), the tenth and last of the afternoon's stalls; fourteen of the twenty rungs are now finished and the six still going hold a current heartbeat.
lr-sweep
[The probe above the learning-rate sweep's tested range](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7) read 3.7748 on Paloma at step 18,000 at 5:24 p.m. (ET), still last of the five, and the run it trails at that step is no longer [the 0.7 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7). [The 1.1 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-005-d512-600x-lr1.1) went ahead there, 3.7535 to 3.7572, and held the lead through step 20,000; [the 1.4 rung that won the length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) did not lead at any step before 21,000 and took the win by 0.0035 inside the last 150 steps, so this length was decided inside its final wind-down and the probe's finishing place cannot be read from its present shortfall. The probe is 0.0213 behind the step 18,000 leader and 0.0176 behind 0.7, against 0.0329 behind 0.7 at step 17,000, and at 1.67 s a step it closes its 21,150 steps near 6:52 p.m. (ET).
lr-sweep
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) cleared its twentieth save since the restart and stepped straight through it. The write shows as about twenty seconds of extra time over steps 99,742-99,746, between 4:58 and 5:00 p.m. (ET), against a step that otherwise takes about eighteen seconds; the run logged step 99,749 at 5:01 p.m. (ET), nineteen hours and forty-seven minutes after the 9:13 p.m. (ET) restart that cleared [the silent hang](https://github.com/marin-community/marin/issues/8870). The twenty-first save is due near 5:58 p.m. (ET), and the next reading on held-out text is not due until step 101,999, about eleven hours out.
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) eased for a second stretch running. The 250-step block covering steps 99,500-99,749 closed at 4:58 p.m. (ET) at a median of 0.5645 on the output projection — below both the 0.5773 run high, set over steps 97,750-97,999 this morning, and the 0.5738 the block before it closed at, so the rise has now failed twice in a row to set a new high. Neither loss has turned, and the level stays at about a third of the 1.5 an earlier healthy run reached. [Jeff H's question about filing an issue to track the rise](https://discord.com/channels/1354881461060243556/1548520475397718068/1548583749795577897) is unanswered fourteen hours on.
[The probe above the learning-rate sweep's tested range](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7) took two more readings on held-out text — steps 16,000 and 17,000 at 4:28 and 4:56 p.m. (ET) — and still reads last of the five, but its shortfall keeps narrowing. Against [the 0.7 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7), the best at both steps, the gap ran 0.0455 then 0.0329, after 0.0601 at step 15,000; against [the 1.4 rung that won the length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) it held near 0.019, and 1.4 was itself 0.0277 behind 0.7 at step 17,000 and did not lead until step 19,000. The probe is back to twenty-eight minutes per thousand steps, which closes its 21,150 steps near 6:52 p.m. (ET).
[The learning-rate sweep rung that stopped at 3:54 p.m. (ET)](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-019-d512-600x-lr1.4) is stepping again. It logged step 5,513 at 4:55:54 p.m. (ET) — the step straight after its last, so it replayed nothing and lost sixty-one minutes of machine time rather than work; this page had projected a replay of about 1,100 steps from a half-hourly save, and there was none. That is the tenth such stall in the two-layer wave since it launched at 1:55 p.m. (ET), across eight of its twenty runs, each lasting 15 to 61 minutes and each resuming at the next step. The rung now trails [its three siblings at the same length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-017-d512-600x-lr0.7) by about 3,400 steps, and the wave's fourth training length should be called near 6:10 p.m. (ET), when its own slowest rung finishes.
[The learning-rate sweep rung that stopped at 3:54 p.m. (ET)](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-019-d512-600x-lr1.4) is computing again rather than still loading. Its accelerators reached their stepping memory footprint at 4:21 p.m. (ET) and have held it at 96-100% duty since — the state the run was already stepping in within a minute of the same point at its 2:01 p.m. (ET) launch — but W&B refuses steps at or below the last one logged, so the count stays pinned at 5,512 while the run repeats the work since its last half-hourly save, about 1,100 steps or twenty-three minutes at its launch pace. That puts the counter moving again near 4:46 p.m. (ET); [its three siblings at the same length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-017-d512-600x-lr0.7) are past step 8,200.
lr-sweep
The two-layer wave's third training length is called. [The 1.0 rate finished last of the four at 4:43 p.m. (ET)](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-010-d512-150x-lr1) and placed third on Paloma at 4.1388, behind [1.4 at 4.1208](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-011-d512-150x-lr1.4) and [1.7 at 4.1221](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-012-d512-150x-lr1.7) and ahead of [0.7 at 4.1722](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-009-d512-150x-lr0.7) — 1.0 had led all four at the step 4,000 check, so the wind-down to zero learning rate moved it from first to third, the same reordering that carried 1.7 from last to within 0.0013 of the win. 1.4 has now taken all three lengths the wave has finished, with 1.7 second at each and the gap narrowing from 0.0099 to 0.0026 to 0.0013, which is the pattern the six-layer sweep's above-range probe was launched to test. The 1.0 rung reached its result despite stopping at 3:54 p.m. (ET) after step 4,233 and resuming at 4:23 p.m. (ET) with the next step, losing twenty-eight minutes and no work.
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has eased. The 250-step block covering steps 99,500-99,749 is thirteen of its twenty-five samples in at a median of 0.5574 on the output projection, against the 0.5894 it read at six samples twenty minutes earlier. That puts it below both the 0.5773 run high, set over steps 97,750-97,999 this morning, and the 0.5738 the block before it closed at, so no new high is in prospect when this block closes near 4:58 p.m. (ET). Neither loss has turned, and the level is about a third of the 1.5 an earlier healthy run reached. [Jeff H's question about filing an issue](https://discord.com/channels/1354881461060243556/1548520475397718068/1548583749795577897) is unanswered fourteen hours on.
herotraining-dynamicsgradient-norm
[The learning-rate sweep run that stopped at 3:54 p.m. (ET)](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-019-d512-600x-lr1.4) is back, under the same run id. Its accelerator telemetry resumed at 4:17 p.m. (ET), twenty-three minutes after the last step, and device memory has climbed 0 to 3.6 GB to 11.5 GB on the same schedule its 1:56 p.m. (ET) launch followed; at launch the full 53 GB training footprint arrived thirty-six minutes in, which puts the first step near 4:54 p.m. (ET). W&B now reads the run as running rather than crashed, but it has logged no step past 5,512 and refuses steps at or below the last one logged, so the step count stays pinned until any replay passes it. [Its three siblings at the same length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-017-d512-600x-lr0.7) are past step 7,150.
lr-sweep
[The 1.7 rate finished the two-layer wave's third training length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-012-d512-150x-lr1.7) at 4:15 p.m. (ET) with 4.1221 on Paloma — within 0.0013 of [the 1.4 rate's 4.1208](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-011-d512-150x-lr1.4), and well ahead of [0.7 at 4.1722](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-009-d512-150x-lr0.7). 1.7 read last of the four at the step 4,000 check with 4.2658, so the wind-down to zero learning rate carried it from last to a tie for first. [The 1.0 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-010-d512-150x-lr1) is still running and led all four at step 4,000 with 4.2328, so the length is not called.
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) cleared its nineteenth save since the restart and stepped straight through it. The write shows as about twenty-two seconds of extra time spread over steps 99,540-99,551, between 3:57 and 4:01 p.m. (ET), against a step that otherwise takes about eighteen seconds; step time was back to normal by 4:01 p.m. (ET) and the run logged step 99,570 at 4:06 p.m. (ET), eighteen hours and fifty-three minutes after the restart at 9:13 p.m. (ET) that cleared [the silent hang](https://github.com/marin-community/marin/issues/8870). The twentieth save is due near 4:58 p.m. (ET), and the next reading on held-out text is not due until step 101,999, about twelve hours out.
herocheckpointmilestone
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) did not set a run high. The 250-step block covering steps 99,250-99,499 closed at a median of 0.5738 on the output projection at 3:41 p.m. (ET) — the second-highest any block of the run has closed at, but below the 0.5773 set over steps 97,750-97,999 this morning and below the 0.5876 it was tracking at fourteen samples. The block that follows is six of its twenty-five samples in at 0.5894, too few to read. Neither loss has turned: the 500-step training-loss block covering steps 99,000-99,499 closed at 1.2120, inside the 1.2112-1.2157 band of the three blocks before it, and the norm still sits at about a third of the 1.5 an earlier healthy run reached. [Jeff H's question about filing an issue to track the rise](https://discord.com/channels/1354881461060243556/1548520475397718068/1548583749795577897) is unanswered thirteen hours on, and [the bar Larry set for acting](https://discord.com/channels/1354881461060243556/1548520475397718068/1548529097792032830) — the norm running to 5 and the loss curve then diverging — is far above the present level.
herotraining-dynamicsgradient-norm
[The probe above the learning-rate sweep's tested range](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7) took its fifteenth reading on held-out text at step 15,000 at 4:00 p.m. (ET) and still reads last of the five, but its shortfall keeps narrowing. It scored 3.8741 on Paloma against 3.8140 for [the 0.7 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7), the best at that step — a gap of 0.0601, after 0.0801 and 0.0752 at the two readings before. Against [the 1.4 rung that won the length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) the gap is 0.0211, the narrowest of the run, though 1.4 was itself 0.0390 behind 0.7 at this step and did not lead until step 19,000. The probe has slowed to 1.84 s a step from 1.66, which pushes its close from 6:49 p.m. to about 7:08 p.m. (ET).
lr-sweep
[A run in the learning-rate sweep's new two-layer wave](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-019-d512-600x-lr1.4) stopped at 3:54 p.m. (ET) after step 5,512 of 21,150, about a quarter of the way through the wave's longest training length. Its heartbeat stopped one second after its last step and W&B marks it crashed; [its three siblings at the same length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-017-d512-600x-lr0.7) were past step 6,100 and still stepping at 4:04 p.m. (ET), and nothing has been launched in its place. The lost rung carries the 1.4 rate, which won this length in [the six-layer version of the sweep](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) and only went ahead there in the final wind-down to zero, so the two-layer result at this length cannot be called without it.
lr-sweep
The two-layer wave of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) is half in at its third training length. [The 1.4 rate finished first](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-011-d512-150x-lr1.4) at 3:51 p.m. (ET) with 4.1208 on Paloma, ahead of [0.7 at 4.1722](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-009-d512-150x-lr0.7) — a gap of 0.0514, far wider than the 0.0026 that separated the top two at the length before. The order is not settled: [the 1.0 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-010-d512-150x-lr1) and [the 1.7 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-012-d512-150x-lr1.7) are still running, and 1.0 led all four at the step 4,000 check with 4.2328 against 1.4's 4.2448.
lr-sweep
The two-layer wave of [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) finished its second-shortest training length at 3:22 p.m. (ET), with all four rates in. [The 1.4 rate won again](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-007-d512-60x-lr1.4) on Paloma at 4.3191, ahead of [1.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-008-d512-60x-lr1.7) at 4.3217, then 1.0 and 0.7 — the same order as the shortest length, but the gap between the top two narrowed from 0.0099 to 0.0026. The next lengths to report are the 150x rungs, about three-quarters of the way through.
lr-sweep
[The learning-rate sweep](https://github.com/marin-community/marin/issues/7856) launched twenty more runs at 1:55 p.m. (ET), its first new wave since the sweep moved to six-layer models. The wave holds the width and the optimiser fixed and drops the model to two layers, over five training lengths and four rates — 0.7, 1.0, 1.4 and 1.7. That grid carries 1.7 as an ordinary rung at every length, where the six-layer sweep tests it only through [a single probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7). [The shortest length has already finished](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-2L-DENSE-MUONH-003-d512-30x-lr1.4): 1.4 won it on Paloma at 4.5915, with 1.7 second at 4.6014 and the two lower rates well behind, so at two layers the best rate falls inside the grid rather than above it.
lr-sweep
[The probe above the learning-rate sweep's tested range](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7) took three more readings on held-out text — steps 11,000, 12,000 and 13,000 at 2:08, 2:36 and 3:04 p.m. (ET) — and reads last of the five at each. Its shortfall against [the 0.7 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7), still the best at every one of these steps, ran 0.0964, then 0.1006, then 0.0801, the narrowest of the run so far. Against [the 1.4 rung that won the length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) the gap closed to 0.0279, but the 1.4 rung was itself 0.0522 behind 0.7 at step 13,000 and did not lead until step 19,000, so the order still settles only at the end. The probe holds 1.66 s a step, which closes its 21,150 steps near 6:49 p.m. (ET).
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) wrote training state at about 2:58 p.m. (ET) and stepped straight through it — its eighteenth save since the restart, the eighteenth to clear, and the sixth in a row to land on the slot this page named an hour ahead. The write cost about sixteen seconds, in line with the eight before it, and the run logged step 99,378 at 3:07 p.m. (ET), seventeen hours and fifty-four minutes after the restart at 9:13 p.m. (ET) that cleared [the silent hang](https://github.com/marin-community/marin/issues/8870). The nineteenth save is due near 3:58 p.m. (ET); the next reading on held-out text is not due until step 101,999, about thirteen hours out.
herocheckpointmilestone
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has not settled. The 250-step block covering steps 99,000-99,249 closed at a median of 0.5591 on the output projection — below both the 0.5773 run high, set over steps 97,750-97,999, and the 0.5729 second-highest block, so the run high this page called likely to fall at 2:28 p.m. (ET) held. The block that followed, steps 99,250-99,499, is fourteen of its twenty-five samples in at a median of 0.5876, which is above that run high again. Neither loss has turned, and the level is about a third of the 1.5 an earlier healthy run reached; [the bar Larry set for acting](https://discord.com/channels/1354881461060243556/1548520475397718068/1548529097792032830) is higher again — the norm running to 5 and the loss curve then diverging. [Jeff H's question about filing an issue to track the rise](https://discord.com/channels/1354881461060243556/1548520475397718068/1548583749795577897) is unanswered twelve hours on.
herotraining-dynamicsgradient-norm
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has turned back down. The 250-step block covering steps 99,000-99,249 is seventeen of its twenty-five samples in at a median of 0.5591 on the output projection, against the 0.6047 it read twenty minutes earlier at eleven samples — the seven samples logged since came in at a median of 0.5431. That puts the block back below both the 0.5773 run high, set over steps 97,750-97,999, and the 0.5729 second-highest block, so the run high no longer looks likely to fall when this block closes near 2:28 p.m. (ET). The block still runs above the 0.5495 the previous block closed at.
herotraining-dynamicsgradient-norm
hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) wrote training state at about 1:58 p.m. (ET) and stepped straight through it — its seventeenth save since the restart, the seventeenth to clear, and the fifth in a row to land on the slot this page named an hour ahead. The write cost about thirty seconds against the 18.3 s median step, and the run logged step 99,160 at 2:00 p.m. (ET), sixteen hours and forty-seven minutes after the restart at 9:13 p.m. (ET) that cleared [the silent hang](https://github.com/marin-community/marin/issues/8870). The eighteenth save is due near 2:58 p.m. (ET).
herocheckpointmilestone
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has turned back up. The 250-step block covering steps 99,000-99,249 is eleven of its twenty-five samples in at a median of 0.6047 on the output projection — above the 0.5773 that is the highest any block of the run has closed at, and a reversal of the 0.5495 the block before it closed at, which was the lowest in eight hours. A block this far in typically lands within 0.008 of its closing value, so this one is on course to set a run high when it closes near 2:28 p.m. (ET), though single samples in it swing between 0.42 and 0.80. The level stays far below the 1.5 Larry named as the edge of known territory, and neither loss has turned: [the hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) logged step 99,100 at 1:42 p.m. (ET), still stepping every 18.5 s, its training loss flat at 1.218, and its held-out reading unchanged since 1:12 p.m. (ET).
herotraining-dynamicsgradient-norm
[The probe above the learning-rate sweep's tested range](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7) took its tenth reading on held-out text at step 10,000 at 1:40 p.m. (ET) and still reads last of the five. It scored 4.0198 on Paloma against 3.9006 for [the 0.7 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7), the best at that step — a shortfall of 0.1192, down from 0.1357 at the ninth check and the narrowest of the run so far. Reading last here settles nothing: [the 1.4 rung that won the length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) was itself 0.0756 behind 0.7 at step 10,000, trailed it from step 2,000 through step 18,000, and only went ahead at step 19,000 as the wind-down to zero took effect. The probe is further behind at this point than that winner was, and it closes near 6:52 p.m. (ET).
lr-sweep
[The probe above the learning-rate sweep's tested range](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7) took its ninth reading on held-out text at step 9,000 and still reads last of the five, but its shortfall narrowed more than at any check before. It scored 4.0610 on Paloma against 4.0189 for [the 1.4 rung that won the length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) — a gap of 0.042, down from 0.051 one reading earlier, after seven readings in which the gap had barely moved. [The 0.7 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7) still leads at that step with 3.9253, so the narrowing does not yet change the order. The probe reached step 9,393 of 21,150 at 1:23 p.m. (ET) and holds 1.68 s a step, which closes it near 6:52 p.m. (ET).
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) eased further. The 250-step block covering steps 98,750-98,999 closed at a median of 0.5495 on the output projection — the lowest of any closed block since steps 97,250-97,499, about eight hours earlier, and the fourth of the last five to close below the 0.5773 run high, which now stands 1,250 steps back over steps 97,750-97,999. [The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) logged step 99,032 at 1:22 p.m. (ET), sixteen hours and nine minutes of unbroken stepping since the restart that cleared [the silent hang](https://github.com/marin-community/marin/issues/8870), and holds 18.5 s a step; its seventeenth hourly save since the restart is due near 1:58 p.m. (ET). [Jeff H's question about filing an issue to track the rise](https://discord.com/channels/1354881461060243556/1548520475397718068/1548583749795577897) is still unanswered eleven hours on.
herotraining-dynamicsgradient-norm
hero-ragged_a2a-nccl2307-ep-step81k
[The hero's](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) held-out reading at step 98,999 landed at 1:12 p.m. (ET) and came in at 2.2212 on Paloma, 0.0022 above the 2.2191 it read three thousand steps earlier — its first step back since step 92,999. This is the reading the page has pointed at all morning, because a turn in the training loss or the held-out loss is the condition [Larry](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) set for rolling the run back to an earlier checkpoint. A step back this small does not meet it: five of the last twelve readings have risen on the one before, one of them by 0.0030, the figure is still the second-best of the run, and it sits 0.0082 below where it stood nine thousand steps back. The training loss has not turned either — the 500-step block covering steps 98,500-98,999 closed at 1.2138, inside the 1.2113-1.2157 band of the three blocks before it.
heroevaltraining-dynamicsmilestone
[The probe above the learning-rate sweep's tested range](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7) took its eighth reading on held-out text at step 8,000 and still reads last of the five. It scored 4.0844 on Paloma, against 3.9436 for [the 0.7 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7), the best at that step, and 4.0333 for [the 1.4 rung that won the budget](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4). Its shortfall against the 1.4 rung has barely moved — 0.051 nats, against 0.053 one reading earlier — so eight checks in, nothing points to a best rate above the range the sweep covered. The probe reached step 8,720 of 21,150 at 1:04 p.m. (ET) and holds 1.68 s a step, which closes it near 6:50 p.m. (ET).
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) wrote training state at 12:58 p.m. (ET) and stepped straight through it — its sixteenth save since the restart, the sixteenth to clear, and the fourth in a row to land on the slot this page named an hour ahead. It logged step 98,964 at 1:00 p.m. (ET), fifteen hours and forty-seven minutes of unbroken stepping since the restart that cleared [the silent hang](https://github.com/marin-community/marin/issues/8870), and is still averaging 18.3 s a step. [The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) stays eased: the 250-step block covering steps 98,750-98,999 is 22 of its 25 samples in at a median of 0.565 on the output projection, below both the 0.573 the previous block closed at and the 0.577 run high, which still stands four blocks back over steps 97,750-97,999. The next reading on held-out text falls at step 98,999, about 1:10 p.m. (ET).
herocheckpointmilestone
hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) logged step 98,903 at 16:41Z and is still stepping every 18.6 s, fifteen hours and twenty-seven minutes after the restart that cleared [the silent hang](https://github.com/marin-community/marin/issues/8870). [The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) stays eased: the 250-step block covering steps 98,750-98,999 is 16 of its 25 samples in at a median of 0.548 on the output projection, below both the 0.573 the previous block closed at and the 0.577 run high set a thousand steps earlier. Neither loss has turned, the condition Larry set for rolling back to an earlier checkpoint — the 500-step training-loss median reads 1.212, level with the block before it, and model FLOPs utilisation holds at 23.3%. [Jeff H's question about filing an issue to track the rise](https://discord.com/channels/1354881461060243556/1548520475397718068/1548583749795577897) is still unanswered. The chart and progress bar came back byte-identical from the last two fetches and sit at step 98,766, so the page trails the live counter by 137 steps; that is W&B's sampled history endpoint freezing, not the run stopping. The sixteenth hourly save since the restart is due near 12:57 p.m. (ET) and the next held-out Paloma reading, at step 99,000, near 1:11 p.m. (ET).
herotraining-dynamicsgradient-norm
[The probe above the learning-rate sweep's tested range](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7) reached step 7,999 at 16:44Z, with its eighth reading on held-out text due at the next step. Its pace has been steady at 1.67 s a step over the last four thousand steps, which closes the 21,150-step run near 6:50 p.m. (ET) — about twenty-five minutes earlier than this page estimated an hour ago, when it read the pace as 1.78 s. At step 8,000 the two rungs it is measured against read 3.9436 on Paloma for [the 0.7 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7) and 4.0333 for [the 1.4 rung that won the full length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4), so the probe needs to beat 4.0333 there to move off last place.
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has eased: the 250-step block covering steps 98,750-98,999 is 10 of its 25 samples in at a median of 0.538 on the output projection, below both the 0.573 the previous block closed at and the 0.577 run high set a thousand steps earlier. Neither loss has turned, the condition Larry set for rolling back to an earlier checkpoint — training loss read 1.210 at step 98,842, logged 16:22:45Z, and model FLOPs utilisation holds at 23.3% — and [Jeff H's question about filing an issue to track the rise](https://discord.com/channels/1354881461060243556/1548520475397718068/1548583749795577897) is still unanswered. The sixteenth hourly save since the restart is due near 12:57 p.m. (ET) and the next held-out Paloma reading, at step 99,000, falls about 1:08 p.m. (ET); the progress bar and chart trail the live step counter by about 75 steps, because W&B's sampled history lags the run's tail.
herotraining-dynamicsgradient-norm
[The probe above the learning-rate sweep's tested range](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7) took its seventh reading on held-out text at step 7,000 and still reads last of the five. It scored 4.1156 on Paloma, against 3.9678 for [the 0.7 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7), the best of the four closed rungs there, and 4.0626 for [the 1.4 rung that won the full length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) — a gap that widened from 0.049 at step 6,000 to 0.053 here, undoing the narrowing the sixth reading showed. That still settles nothing: at step 7,000 the four closed rungs rank 0.7, 0.45, 1.1, 1.4, and they finish in nearly the reverse of that order, because each rate decays to zero and a higher one reads worse until its decay bites. At 1.78 s a step the probe now closes near 7:15 p.m. (ET), with its eighth reading near 12:45 p.m. (ET).
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has stopped short of a new high. The 250-step block covering steps 98,500-98,749 closed at a median of 0.5729 on the output projection — the layer that dominates the model's total gradient — the second-highest block of the run, but below the 0.5773 set over steps 97,750-97,999. So the rise has not made a new peak in the three thousand steps since, and it stays far under the 1.5 Larry named as the edge of known territory. Neither loss has turned, the condition he set for rolling back to an earlier checkpoint: training loss over steps 98,000-98,499 closed at 1.2113, the lowest 500-step median since the restart, and the block now filling reads 1.2130. Model FLOPs utilisation has held at 23.3%. The next held-out Paloma reading falls at step 99,000, about 1:10 p.m. (ET).
herotraining-dynamicsgradient-norm
hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) wrote training state at 11:57:55 a.m. (ET) and stepped straight through it — its fifteenth save since the restart, the fifteenth to clear, and the third in a row to land on the slot this page named an hour ahead. That is fourteen hours and forty-four minutes of unbroken stepping since the restart that cleared [the silent hang](https://github.com/marin-community/marin/issues/8870) at 8:54 p.m. (ET). The write took 20.6 s against a 16.9 s median, and a second 24.6 s step a minute later closed the cluster. The sixteenth save is due near 12:57 p.m. (ET).
herocheckpointmilestone
[The probe above the learning-rate sweep's tested range](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7) took its sixth reading on held-out text at 11:49 a.m. (ET) and still reads last of the five. It scored 4.1617 on Paloma at step 6,000, against 4.0008 for [the 0.7 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7), the best of the four closed rungs at that step, and 4.1123 for [the 1.4 rung that won the full length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) — a gap that narrowed from 0.058 at step 5,000 to 0.049 here, reversing the widening the fifth reading showed. That settles nothing: the four closed rungs finished in nearly the reverse of the order they hold this early, because each rate decays to zero and a higher one reads worse until its decay bites. At 1.79 s a step the probe now closes near 7:20 p.m. (ET), about half an hour later than the earlier estimate; its seventh reading falls near 12:17 p.m. (ET).
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has resumed. The 250-step block covering steps 98,500-98,749 is 21 of its 25 samples in at a median of 0.572 on the output projection — the layer that dominates the model's total gradient — against 0.555 and 0.564 for the two closed blocks that showed the levelling. That already makes it the second-highest block of the run, behind only the 0.577 set over steps 97,750-97,999, and it is still far below the 1.5 Larry named as the edge of known territory. Neither loss has turned, the condition Larry set for rolling back to an earlier checkpoint: training loss over steps 98,000-98,499 closed at the lowest 500-step median since the restart. The hero has stepped every ~18.5 s through the block, with the next held-out Paloma reading at step 99,000, about 1:10 p.m. (ET).
herotraining-dynamicsgradient-norm
[The probe above the learning-rate sweep's tested range](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7) has its fifth reading on held-out text and still reads last of the five. It scored 4.1901 on Paloma at step 5,000, against 4.0336 for [the 0.7 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7), the best of the four closed rungs at that step, and 4.1326 for [the 1.4 rung that won the full length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) — a gap that widened from 0.053 at step 4,000 to 0.058 here, reversing the narrowing the previous reading showed. The four closed rungs finished in nearly the reverse of the order they hold this early, because each rate decays to zero and a higher one reads worse until its decay bites, so the probe settles nothing before it closes near 6:50 p.m. (ET). Its sixth reading falls near 11:49 a.m. (ET).
lr-sweep
hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) wrote training state at 10:57 a.m. (ET) and stepped straight through it — its fourteenth save since the restart, the fourteenth to clear, and the second in a row to land on the slot this page named an hour ahead. That is thirteen hours and fifty minutes of unbroken stepping since [the silent hang](https://github.com/marin-community/marin/issues/8870) at 8:54 p.m. (ET). The step took 21.5 s against a 16.9 s median, the smallest write of the morning, and model FLOPs utilisation has not moved off 23.3% in eight hours. The fifteenth save is due near 11:57 a.m. (ET). [The gradient norm on the model's output layer](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has ticked back up: the block now filling, steps 98,500-98,749, reads a median of 0.572 over its first nine samples against 0.555 for the block before it, so it is above the two closed blocks that showed the levelling but still under the run's high. The next held-out Paloma reading falls at step 99,000, about 1:00 p.m. (ET).
herocheckpointmilestone
[The probe above the learning-rate sweep's tested range](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7) has its fourth reading on held-out text and still reads last of the five. It scored 4.2427 on Paloma at step 4,000, against 4.0778 for [the 0.7 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7), the best of the four closed rungs at that step, and 4.1896 for [the 1.4 rung that won the length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4). Its gap to 1.4 narrowed from 0.065 at step 3,000 to 0.053 here. That still settles nothing: the four closed rungs ranked 0.7, 1.1, 0.45, 1.4 at step 4,000 and finished in almost the reverse order, because each rate decays to zero and a higher one reads worse until its decay bites. The probe is 4,500 steps into 21,149 at 1.7 seconds a step and should close near 6:50 p.m. (ET).
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has levelled. The 250-step block covering steps 98,250-98,499 closed at a median of 0.555 on the output projection — the layer that dominates the model's total gradient — the second block in a row to close below the high set over steps 97,750-97,999, and the lower of the two. The level is still high for the run: its six highest blocks all fall inside the last three thousand steps. Neither the training loss nor the held-out loss has turned, which is the condition Larry set for rolling back to an earlier checkpoint.
herotraining-dynamicsgradient-norm
[The probe above the learning-rate sweep's tested range](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7) has its first three readings on held-out text, and it reads last of the five at every one. It scored 4.5180, 4.3580 and 4.2789 on Paloma at steps 1,000, 2,000 and 3,000, against 4.1337 for [the 0.7 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7) and 4.2136 for [the 1.4 rung that won the budget](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4). That order does not hold: at step 3,000 the four closed rungs rank 0.7, 1.1, 0.45, 1.4, and at their last step they rank 1.4, 1.1, 0.7, 0.45 — the rate that led there finished third and the rate that trailed there finished first, because every rate decays to zero and a higher one reads worse until its decay bites. The probe is 3,000 steps into 21,149 at about 1.7 seconds a step and should close near 6:50 p.m. (ET).
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) has now stepped for twelve hours and forty-five minutes without a break, since the restart at 9:13 p.m. (ET) that cleared [the silent hang](https://github.com/marin-community/marin/issues/8870). It wrote training state between 9:57 and 9:59 a.m. (ET) and stepped straight through — the thirteenth save since the restart, the thirteenth to clear, and the first to land on the slot this page named in advance. Saves have kept the same hourly minute all night, so the fourteenth is due near 10:57 a.m. (ET). [The gradient norm on the model's output layer](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) is still inside the band it has held since the restart: the block now filling and the one before it both read below the run's high, set two blocks back, and far below the 1.5 Larry named as the edge of known territory. The next held-out Paloma reading falls at step 99,000, about 1:10 p.m. (ET).
herocheckpointmilestone
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has turned down. The 250-step window covering steps 98,000-98,249 closed at a median of 0.580, below the 0.594 peak set two windows earlier over steps 97,750-97,999, so the run's highest reading is no longer its newest; every window since [the restart](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) sits inside 0.53-0.59, far below the 1.5 Larry named as the edge of known territory, and neither the training loss nor the Paloma held-out loss has turned.
herotraining-dynamicsgradient-norm
Helw150 h100-mix25-20260912-d1536
[The mixture-swap ladder's largest model](https://wandb.ai/marin-community/marin_moe/runs/h100-mix25-20260912-d1536) passed step 30,757 at 8:52 a.m. (ET) — the step where [August's run at this size](https://wandb.ai/marin-community/marin_moe/runs/h100-ladder-d1536-ep8x8-bs1024-791tpp-20260825-rno2a-interactive) crashed — so it now trains with no control to read it against. Its last matched evaluation, at step 27,227, put the new mixture 0.14% ahead on Paloma held-out text. That is a third of the way through, and at [the size below](https://wandb.ai/marin-community/marin_moe/runs/h100-mix25-20260912-d1024), where both runs reached the end, the gap stayed within 0.24% through seven tenths of the run and only opened over the final quarter, to 0.70% — after the point where [the treatment switches to the mixture's second set of weights](https://github.com/marin-community/marin/issues/9126). The stretch that produced the gain there cannot be read at this size, whatever Tuesday's finish shows. All three narrower models finished ahead of their August controls, by 0.56%, 0.61% and 0.70%.
ladderdata-mixturemilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) wrote training state between 8:57 and 8:58 a.m. (ET) and stepped straight through it — its twelfth save since the restart, and the twelfth to clear without a stall. That is eleven hours and forty-five minutes of unbroken stepping since [the silent hang](https://github.com/marin-community/marin/issues/8870) at 8:54 p.m. (ET). [The gradient-norm block that read 0.564 part-filled an hour ago](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has closed at 0.580 over steps 98,000-98,180, so the fall back reported then did not hold: it is the second-highest block of the run, behind 0.594. The eleven blocks of 250 steps since step 95,500 all sit between 0.550 and 0.594, so the figure is holding a band rather than climbing, and neither loss curve has turned with it. The next held-out Paloma reading falls at step 99,000, about 1:15 p.m. (ET).
herocheckpointtraining-dynamicsgradient-norm
Helw150 h100-mix25-20260912-d1536
[The mixture-swap ladder's largest width](https://wandb.ai/marin-community/marin_moe/runs/h100-mix25-20260912-d1536) will not finish before [tomorrow's data-mixture decision](https://github.com/marin-community/marin/issues/9126). It is about a third of the way through its 90,767 steps and gaining roughly 1,080 steps an hour, which puts its close near 4:30 p.m. (ET) on Tuesday. So the decision rests on the three narrower widths already in, the last of which [closed overnight](https://wandb.ai/marin-community/marin_moe/runs/h100-mix25-20260912-d1024); [August's run at this width](https://wandb.ai/marin-community/marin_moe/runs/h100-ladder-d1536-ep8x8-bs1024-791tpp-20260825-rno2a-interactive) stopped at step 30,757, and the new one passes that mark this hour, beyond which there is no baseline to read it against.
ladderdata-mixture
hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has not become a trend. Its median per 250 steps reads 0.594 over steps 97,750-98,000 — the highest window of the run, 0.010 above the previous high — but the partial window after it falls back to 0.564, and the eleven windows since step 95,250 have all sat between 0.551 and 0.594. That is a band, not a climb, and it stays far below the 1.5 Larry cited from an earlier healthy run as the edge of known territory. Neither loss curve has turned with it: the held-out Paloma reading has improved at each of the last two evals, and the next falls at step 99,000, about 1:05 p.m. (ET).
herotraining-dynamicsgradient-norm
WhenWen AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7
[A probe above the tested range](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X600-HIGH-1P7-d512-600x-lr1.7) started at 8:42 a.m. (ET), eleven minutes after rate 1.4 took the longest run length — the sweep opens one automatically whenever the top rate of its planned range wins a length. This is the third such extension and the first at the longest length; the two before it, [at the shortest](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X30-HIGH-1P7-d512-30x-lr1.7) and [at 150x](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X150-HIGH-1P7-d512-150x-lr1.7), both came back worse than the rate they were testing. The four finished runs at this length took about ten hours each, so it should close near 6:40 p.m. (ET).
ladderlr-sweep
WhenWen AUG-LIN0-6L-DENSE-MUONH-020 / -005
[Rate 1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) took [the learning-rate sweep's](https://github.com/marin-community/marin/issues/7856) longest run length at 8:31 a.m. (ET), finishing at 3.7041 on Paloma held-out text against [rate 1.1's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-005-d512-600x-lr1.1) 3.7076 — a 0.0035 win from 0.012 behind at step 17,000, gained over its final decay to zero, the same come-from-behind it produced [at the length below](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-017-d512-300x-lr1.4). That closes the twenty cells the sweep set out to run. The two top rates end near-level overall: 1.4 wins the shortest, the middle and the longest lengths, 1.1 wins the other two, and at 300x the gap was 0.0003. [Rate 0.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7) finished third here and rate 0.45 last.
ladderlr-sweepmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) wrote training state between 7:56 and 7:58 a.m. (ET) and stepped straight through it — its eleventh save since the restart, and the eleventh to clear without a stall. That is ten hours and forty-five minutes of unbroken stepping since [the silent hang](https://github.com/marin-community/marin/issues/8870) at 8:54 p.m. (ET). Steps have come every 17.8 s since the write against an 18.5 s median over the hour before it, and the twelfth save is due about 8:57 a.m. (ET).
herocheckpointmilestone
[The gradient-norm rise Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has not resumed. Its median per 250 steps reads 0.584 over steps 97,500-97,750 — the highest window of the run, but level with the 0.581 it read at step 95,750, two thousand steps and about ten hours earlier, so the figure has been moving inside a 0.55-0.58 band rather than climbing. It stays far below the 1.5 Larry cited from an earlier healthy run as the edge of known territory, and neither loss curve has turned with it: the next held-out Paloma reading falls at step 99,000, about 1:20 p.m. (ET), and a turn there is the condition he set for acting.
[Rate 0.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7) finished [the learning-rate sweep's](https://github.com/marin-community/marin/issues/7856) longest run length at 7:23 a.m. (ET) at 3.7258 on Paloma held-out text, 0.018 behind [rate 1.1's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-005-d512-600x-lr1.1) 3.7076, so the late gain it needed over its final decay did not arrive: it took 0.011 off its step-20,000 reading where it needed 0.029. Three of the four rates have now closed here and rate 1.1 leads all three; [rate 1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) alone is still running, about 2,300 steps out and due to close near 8:31 a.m. (ET). It reads 3.7613 at step 18,000 against rate 1.1's 3.7535 at the same step, and at [the length below](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-017-d512-300x-lr1.4) it stood 0.015 behind rate 1.1 with a seventh of its steps left and finished within 0.0003 of it, so this length is not settled.
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) wrote training state between 6:56 and 6:58 a.m. (ET) and stepped straight through it — its tenth save since the restart, and the tenth to clear without a stall. That is nine hours and forty-five minutes of unbroken stepping since [the silent hang](https://github.com/marin-community/marin/issues/8870) at 8:54 p.m. (ET), so the hourly save window, where the September 10 hang struck, is no longer the open question it was overnight.
herocheckpointmilestone
WhenWen AUG-LIN0-6L-DENSE-MUONH-019 / -005
[Rate 0.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7) is losing ground at [the learning-rate sweep's](https://github.com/marin-community/marin/issues/7856) longest run length: 3.7366 on Paloma held-out text at step 20,000 at 6:49 a.m. (ET), 0.014 behind [rate 1.1's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-005-d512-600x-lr1.1) 3.7225 at the same step, against 0.005 behind one thousand steps earlier. About 600 steps remain, so it finishes near 7:20 a.m. (ET) and 1.1's win at this length now looks settled.
ladderlr-sweep
WhenWen AUG-LIN0-6L-DENSE-MUONH-020 / -005
[Rate 1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) has cut its gap to the leader by a third at [the learning-rate sweep's](https://github.com/marin-community/marin/issues/7856) longest run length: 3.7964 on Paloma held-out text at step 17,000 at 6:34 a.m. (ET), 0.012 behind [rate 1.1's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-005-d512-600x-lr1.1) 3.7848 at that step, after trailing by 0.018 at step 16,000. It has about 3,700 of its 21,150 steps left — the stretch over which rate 1.1 gained 0.077 as its learning rate decayed to zero — and it came back from 0.075 behind to finish level [at the length below](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-017-d512-300x-lr1.4), so 0.012 settles nothing; it should close about 8:28 a.m. (ET). [Rate 0.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7) is about 1,250 steps from finishing, due about 7:20 a.m. (ET), and must still gain 0.038 from its step-19,000 reading to pass rate 1.1's finished 3.7076.
ladderlr-sweep
WhenWen AUG-LIN0-6L-DENSE-MUONH-019 / -005
The lead at [the learning-rate sweep's](https://github.com/marin-community/marin/issues/7856) longest run length has changed hands. [Rate 0.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7) read 3.7454 on Paloma held-out text at step 19,000 at 6:21 a.m. (ET), 0.005 behind [rate 1.1's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-005-d512-600x-lr1.1) 3.7404 at the same step, after leading by 0.019 at step 15,000 — rate 1.1's decay to zero bit first and took the lead at step 18,000. Rate 0.7 has 2,149 steps of its own decay left and must gain 0.038 to pass rate 1.1's finished 3.7076, against the 0.033 rate 1.1 gained over that same final span; it should close about 7:22 a.m. (ET). [Rate 0.45's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-018-d512-600x-lr0.45) finished 3.7628 is already beaten, so it is out of the length, and [rate 1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) reads 3.8244 at step 16,000 — 0.018 behind rate 1.1 there, closing about 8:30 a.m. (ET), and it closed a gap three times that size over its final decay at the length below.
ladderlr-sweepmilestone
WhenWen AUG-LIN0-6L-DENSE-MUONH-018-d512-600x-lr0.45
[Rate 0.45](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-018-d512-600x-lr0.45) closed [the learning-rate sweep's](https://github.com/marin-community/marin/issues/7856) 21,150-step run length — its longest — at 6:01 a.m. (ET) at 3.7628 on Paloma held-out text, 0.055 behind [rate 1.1's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-005-d512-600x-lr1.1) 3.7076. That is the second of the four rates to finish here, and both so far put rate 1.1 in front, as it was at [the fourth length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-017-d512-300x-lr1.4) overnight. The length is not settled: [rate 0.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7) is about 18,300 steps in and already reads 3.7572 with its decay still to run, and [rate 1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) is about 15,900 steps in at 3.8530.
ladderlr-sweepmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) wrote training state between 5:56 and 5:58 a.m. (ET) and stepped straight through it — its ninth save since the restart, and the ninth to clear without a stall. That is eight hours and forty-five minutes of unbroken stepping since [the silent hang](https://github.com/marin-community/marin/issues/8870) was cleared. The save window is where this run has failed before — the September 10 hang stopped it one step after an hourly save committed — so a clean save is the test that matters, not the stepping between them. Steps have come every 18.5 s since the write against an 18.1 s median over the hour before it.
herocheckpointmilestone
WhenWen AUG-LIN0-6L-DENSE-MUONH-X150-HIGH-1P7-d512-150x-lr1.7
[Rate 1.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X150-HIGH-1P7-d512-150x-lr1.7) closed [the learning-rate sweep's](https://github.com/marin-community/marin/issues/7856) 5,288-step run length at 5:39 a.m. (ET) at 3.8805 on Paloma held-out text, behind both [rate 1.4's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-014-d512-150x-lr1.4) 3.8615 and [rate 1.1's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-013-d512-150x-lr1.1) 3.8684, so the probe above the planned range came back worse and rate 1.4 keeps the length. That closes the second of the two boundary extensions this wave opened — [the first, at the shortest length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X30-HIGH-1P7-d512-30x-lr1.7), also lost — so the four rates the sweep set out to try stand at every length it has settled. It trailed from its first reading and never recovered over its decay, unlike the come-from-behind finishes seen at the longer lengths. [Rate 0.45 at the longest length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-018-d512-600x-lr0.45) is the next to finish, due about 6:00 a.m. (ET).
ladderlr-sweepmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) wrote training state between 4:55 and 4:59 a.m. (ET) and stepped straight through it — its eighth save since the restart, and the eighth to clear without a stall. That is seven hours and forty-five minutes of unbroken stepping since [the silent hang](https://github.com/marin-community/marin/issues/8870) was cleared. The save window is where this run has failed before — the September 10 hang stopped it one step after an hourly save committed — so a clean save is the test that matters, not the stepping between them. Steps have come every 18.6 s since the write against an 18.2 s median over the hour before it.
herocheckpointmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The gradient-norm climb Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) has stopped. Its median per five hundred steps rose from 0.530 at step 94,000 to 0.572 at 95,500 and has held there since — 0.570, 0.562 and 0.564 at steps 96,000, 96,500 and 97,000 — so it has not risen in the nine and a half hours since 6:29 p.m. (ET). The output projection, the layer that turns the model's last hidden state into word scores, still carries nearly all of the total, 0.553 of 0.564, well short of the 1.5 Larry cited from an earlier healthy run as the edge of known territory. Neither loss curve has turned with it: the held-out Paloma reading stands at 2.2191 from step 96,000 and the next one falls at step 99,000, about 1:15 p.m. (ET).
herotraining-dynamicsgradient-norm
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) wrote training state between 3:55 and 3:58 a.m. (ET) and stepped straight through it — its seventh save since the restart, and the seventh to clear without a stall. That is six hours and forty-five minutes of unbroken stepping since [the silent hang](https://github.com/marin-community/marin/issues/8870) was cleared, and the eighth save is due about 4:55 a.m. (ET). Steps have come every 17.9 s since the save against an 18.4 s median over the hour before it, so the write cost the run nothing beyond its own three minutes.
herocheckpointmilestone
WhenWen AUG-LIN0-6L-DENSE-MUONH-X150-HIGH-1P7
[The learning-rate sweep](https://github.com/marin-community/marin/issues/7856) opened the second boundary extension of its six-layer wave at 2:15 a.m. (ET): [rate 1.7 at the 5,288-step run length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X150-HIGH-1P7-d512-150x-lr1.7), added because [rate 1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-014-d512-150x-lr1.4) won that length at the top of the four rates the sweep tries, so one rate faster goes out to find whether the best rate lies outside the set. Its first reading trails — 4.6305 on Paloma held-out text at step 1,000, against [1.4's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-014-d512-150x-lr1.4) 4.5552 and [1.1's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-013-d512-150x-lr1.1) 4.4872 at the same step — but every rate decays to zero and a faster one reads worse until its decay bites, so the close about 5:14 a.m. (ET) is what settles it. [The same probe at the shortest length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X30-HIGH-1P7-d512-30x-lr1.7) came back worse and left the original four rates standing.
ladderlr-sweepmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The gradient norm Larry flagged](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) is still climbing, and the climb is flattening. Its median per thousand steps reads 0.383 at step 81,000, 0.450 at 84,000, 0.503 at 89,000 and 0.564 at 96,000 — half as much again over sixteen thousand steps, but 0.061 over the last seven thousand against 0.067 over the first three thousand. Most of the total is the output projection, the layer that turns the model's last hidden state into word scores and the one Larry named as the driver, at 0.438 of the 0.547 total — well short of the 1.5 he cited from an earlier healthy run as the edge of known territory. Neither loss curve has turned with it: the held-out Paloma reading fell from 2.2302 at step 93,000 to 2.2191 at 96,000, and a turn is the condition he set for acting.
herotraining-dynamicsgradient-norm
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
Correction to the entry below: the sampled feed that fills the chart and the progress bar on this page has caught up with [the hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k). It returned step 97,061 at 3:20 a.m. (ET) against the run's own 97,070 at 3:22 a.m. (ET), so the 115-step lag reported at 2:57 a.m. (ET) is closed and the page now trails the run by under ten steps. The hero itself has stepped every 18.3 s for six hours and nine minutes since [the 9:13 p.m. (ET) restart](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) cleared [the silent hang](https://github.com/marin-community/marin/issues/8870), and its seventh hourly save of training state is due about 3:55 a.m. (ET).
heromilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) wrote training state between 2:56 and 2:57 a.m. (ET) and stepped straight through it — its sixth save since the restart, and the sixth to clear without a stall. That is five hours and forty-four minutes of unbroken stepping since [the silent hang](https://github.com/marin-community/marin/issues/8870) was cleared, and the seventh save is due about 3:56 a.m. (ET). The chart and the progress bar on this page trail the run by about 115 steps: the sampled feed that fills them has returned the same frozen tail since 2:40 a.m. (ET), while the run itself logged step 97,001 at 3:01 a.m. (ET) and is stepping every 18 s.
herocheckpointmilestone
Jeff H #hero-run-2026
Jeff H asked at 2:38 a.m. (ET) whether [the hero's rising gradient norm](https://discord.com/channels/1354881461060243556/1548520475397718068/1548520790742274108) should be filed as an issue to track, four hours after Larry opened the thread on it and called the rise interesting rather than dangerous. Nobody has answered in the three quarters of an hour since, so the climb is watched in chat only — no issue covers it and [the status log](https://github.com/marin-community/marin/issues/8506) does not mention it.
herotraining-dynamicsgradient-norm
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) wrote training state between 1:55 and 1:58 a.m. (ET) and stepped straight through it — its fifth save since the restart, and the fifth to clear without a stall. That is four hours and forty-four minutes of unbroken stepping since [the silent hang](https://github.com/marin-community/marin/issues/8870) stopped it at 8:54 p.m. (ET), and it is past the point in the cycle where the September 10 hang struck. The sixth save is due about 2:55 a.m. (ET).
herocheckpointmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) wrote training state between 12:55 and 12:57 a.m. (ET) and stepped straight through it — its fourth save since the restart, and the fourth to clear without a stall. That is three hours and forty-four minutes of unbroken stepping since [the silent hang](https://github.com/marin-community/marin/issues/8870) was cleared. The pace gained at the restart has faded: steps took 17.2 s over the first hour back and now take 18.6 s, level with the 18.5 s median the run held before the hang, so the gain was the warm restart, not a lasting change. [The status log](https://github.com/marin-community/marin/issues/8506) still carries nothing on the stop or the restart.
herocheckpointmilestone
WhenWen AUG-LIN0-6L-DENSE-MUONH-005
[Rate 1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-005-d512-600x-lr1.1) is the first run to finish [the learning-rate sweep's](https://github.com/marin-community/marin/issues/7856) fifth and longest run length, closing its 21,150 steps at 12:56 a.m. (ET) at 3.7076 on Paloma held-out text — ahead of the 3.7709 the same rate posted at the length below, as a run four times as long should be. The other three rates are a quarter to half way through and cannot be compared until they decay to zero, but at matched steps [rate 0.45](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-018-d512-600x-lr0.45) sits 0.023 ahead of 1.1's own reading and [rate 0.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7) 0.047 ahead, with [rate 1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) behind. Read that lead with care: at the length below, [rate 1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-017-d512-300x-lr1.4) closed a gap of 0.075 over its final decay. Rate 0.45 is due about 5:46 a.m. (ET), rate 0.7 about 7:02 a.m. (ET) and rate 1.4 about 8:15 a.m. (ET).
ladderlr-sweepmilestone
WhenWen AUG-LIN0-6L-DENSE-MUONH-017 / -004
[Rate 1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-017-d512-300x-lr1.4) finished [the learning-rate sweep's](https://github.com/marin-community/marin/issues/7856) fourth run length at 12:28 a.m. (ET) at 3.7712 on Paloma held-out text, three ten-thousandths behind [rate 1.1's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-004-d512-300x-lr1.1) 3.7709 — the closest of the four lengths settled so far, and too close to separate the two rates. No probe above the grid follows, because 1.4 sits above 1.1 and lost, so the search stays inside the rates already run. The two have traded the lead at every length — 1.4 at 1,058 and 5,288 steps, 1.1 at 2,115 and 10,575 — and only the 21,150-step length is left, where [rate 1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-005-d512-600x-lr1.1) is a few hundred steps from finishing.
ladderlr-sweepmilestone
WhenWen AUG-LIN0-6L-DENSE-MUONH-017
[Rate 1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-017-d512-300x-lr1.4), the last of the four rates outstanding at [the learning-rate sweep's](https://github.com/marin-community/marin/issues/7856) fourth run length, has closed almost all of its gap to the leader: 3.7889 on Paloma held-out text at step 10,000 against [rate 1.1's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-004-d512-300x-lr1.1) 3.7856 at the same step, so 0.003 behind, after trailing by 0.075 at step 3,000. It has about 120 of its 10,575 steps left and should close about 12:27 a.m. (ET); rate 1.1 gained 0.015 over its own last 574 steps, so either rate can take the length, and rate 1.4 won the length below this one.
ladderlr-sweepmilestone
WhenWen AUG-LIN0-6L-DENSE-MUONH-019 / -005
[Rate 0.7 at the sweep's longest run length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7) has passed the steps its restart had to repeat — 6,207 against the 5,797 its counter sat at — so the last of tonight's four rounds of stops is closed out and all four rates at that length are stepping with current heartbeats. [Rate 1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-005-d512-600x-lr1.1) is about 1,100 of 21,150 steps from being the first to finish the length, due about 12:52 a.m. (ET); [the sweep's issue](https://github.com/marin-community/marin/issues/7856) still records no cause for any of tonight's stops.
ladderlr-sweeprestart
[The 4.0-billion-parameter rung](https://wandb.ai/marin-community/marin_moe/runs/h100-mix25-20260912-d1024) of [the mixture-swap scaling ladder](https://github.com/marin-community/marin/issues/9126) finished at 12:20 a.m. (ET) at step 30,551 — the same step [August's run at that width](https://wandb.ai/marin-community/marin_moe/runs/h100-ladder-d1024-ep8x4-bs1024-791tpp-20260825-rno2a-interactive) reached, so the two are matched token for token. Read without the router's capacity limit, the measure this page's own eval curve uses, the new mixture leads: 2.7616 against 2.7793. Read with the limit on, it trails, 3.1780 against 3.1442. That repeats the split at the two smaller widths, where the new mixture led by 0.017 on the first measure at both — so its lead is flat across width, not growing — while its deficit on the second narrows here, 0.034 against about 0.050. This is the last width with a finished August run to read against: [August's 11.5-billion rung](https://wandb.ai/marin-community/marin_moe/runs/h100-ladder-d1536-ep8x8-bs1024-791tpp-20260825-rno2a-interactive) crashed at step 30,757 of 90,767, and [the new one](https://wandb.ai/marin-community/marin_moe/runs/h100-mix25-20260912-d1536) is about a third of the way through that budget.
WhenWen moe-lgr-9110-larry-d768-muonh
[The wider run in the corrected-routing test](https://wandb.ai/marin-community/dial_moe/runs/moe-lgr-9110-larry-d768-muonh) finished its full 16,874 updates at 11:58 p.m. (ET) at 3.2197 on Paloma held-out text, against [its control's](https://wandb.ai/marin-community/dial_moe/runs/moe_may_compute_opt_mla_hd192_vd256_qc_latentmoe_half_norm_gatednorm_xsa_d768) 3.2149 at the same step, so 0.005 behind. It trailed the control on sixteen of the seventeen paired readings and led on one, by 0.009, so [the correction](https://github.com/marin-community/marin/issues/9110) closes this width with no gain shown.
dialroutingmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) wrote training state between 11:54 and 11:57 p.m. (ET) and stepped straight through it — its third save since the restart, and the third to clear without a stall. That is two hours and forty-four minutes of unbroken stepping since [the silent hang](https://github.com/marin-community/marin/issues/8870) was cleared; [the status log](https://github.com/marin-community/marin/issues/8506) still carries nothing on the stop or the restart.
herocheckpointmilestone
WhenWen AUG-LIN0-6L-DENSE-MUONH-019
Correction: [rate 0.7 at the learning-rate sweep's longest run length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7) did not stay down. A new process attached at 11:47 p.m. (ET), seven minutes after the 11:40 p.m. (ET) stop, and its chips were back at full duty by 11:58 p.m. (ET). Its step counter stays pinned at 5,797 of 21,150 until the replay passes that mark, so read the flat count as a restore, not a second stall; [the sweep's issue](https://github.com/marin-community/marin/issues/7856) still records no cause for tonight's four stops.
ladderlr-sweeprestart
WhenWen AUG-LIN0-6L-DENSE-MUONH-019
[Rate 0.7 at the learning-rate sweep's longest run length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7) stopped again at 11:40 p.m. (ET), forty-nine minutes after it came back from the last stop and at 5,797 of its 21,150 steps. Its heartbeat and its chip readings both end within the same twelve seconds, so the process exited rather than stalled, and nothing has started in its place. This is the fourth round of stops [the sweep](https://github.com/marin-community/marin/issues/7856) has taken tonight; its issue still records no cause for any of them, and its other four live runs are stepping normally.
ladderlr-sweepcrash
WhenWen moe-lgr-9110-larry-d768-muonh
[The wider run in the corrected-routing test](https://wandb.ai/marin-community/dial_moe/runs/moe-lgr-9110-larry-d768-muonh) took its fifteenth and sixteenth paired readings at 11:16 and 11:38 p.m. (ET) — 3.2730 and 3.2390 on Paloma held-out text, against [its control's](https://wandb.ai/marin-community/dial_moe/runs/moe_may_compute_opt_mla_hd192_vd256_qc_latentmoe_half_norm_gatednorm_xsa_d768) 3.2661 and 3.2327 at the matching points, so 0.007 and 0.006 behind. It is about 480 of the 16,874 updates the control ran from the finish and should close about 11:58 p.m. (ET). Fifteen of the sixteen readings sit behind the control and all sixteen fall within 0.02 of it either way, so [the correction](https://github.com/marin-community/marin/issues/9110) closes this width with no gain shown.
experimentmoeroutingmilestone
WhenWen moe-lgr-9110-larry-d768-muonh
Correction: [the wider run in the corrected-routing test](https://wandb.ai/marin-community/dial_moe/runs/moe-lgr-9110-larry-d768-muonh) did not end at 14,252. A new process attached to the same run at 10:52 p.m. (ET) and logged step 14,253 at 10:59 p.m. (ET), straight after its last, so it repeated nothing and lost three hours and twenty-five minutes of wall clock only. It has about 2,400 of the 16,874 updates [its control](https://wandb.ai/marin-community/dial_moe/runs/moe_may_compute_opt_mla_hd192_vd256_qc_latentmoe_half_norm_gatednorm_xsa_d768) ran left and should close about 11:50 p.m. (ET), so [the correction](https://github.com/marin-community/marin/issues/9110) gets its last three paired readings after all; through fourteen it trails the control on thirteen, by 0.003 to 0.018 on Paloma held-out text, where lower is better.
dialroutingrestart
WhenWen AUG-LIN0-6L-DENSE-MUONH-016
[Rate 0.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-016-d512-300x-lr0.7) finished [the learning-rate sweep's](https://github.com/marin-community/marin/issues/7856) fourth run length at 10:59 p.m. (ET) at 3.7899 on Paloma held-out text, 0.019 behind [rate 1.1's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-004-d512-300x-lr1.1) 3.7709 and ahead of [rate 0.45's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-015-d512-300x-lr0.45) 3.8364, so 1.1 leads the 10,575-step length with three of the four rates in. Only [rate 1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-017-d512-300x-lr1.4) is outstanding, due about 12:40 a.m. (ET); it won the length below this one, so the pick is not settled.
ladderlr-sweepmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) wrote training state between 10:54 and 10:56 p.m. (ET) and stepped straight through it — its second save since the restart, and the first on the hourly slot the 9:53 p.m. (ET) held-out evaluation reset. That leaves an hour and forty minutes of unbroken stepping since [the silent hang](https://github.com/marin-community/marin/issues/8870) was cleared, with no second stall; [the status log](https://github.com/marin-community/marin/issues/8506) still carries nothing on the stop or the restart.
herocheckpointrestart
WhenWen AUG-LIN0-6L-DENSE-MUONH-005 / -019
Two more runs at [the learning-rate sweep's](https://github.com/marin-community/marin/issues/7856) longest length stopped and returned tonight — [rate 1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-005-d512-600x-lr1.1) went quiet at 10:37 p.m. (ET) and logged its next step at 10:50, and [rate 0.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7) went quiet at 9:59 p.m. (ET) and returned at 10:51. Each resumed at the step straight after its last, so the sweep lost no steps in this third round of stops tonight, and its issue still records no cause for any of them.
ladderlr-sweeprestart
WhenWen AUG-LIN0-6L-DENSE-MUONH-015 / -020
[Rate 0.45](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-015-d512-300x-lr0.45) closed [the learning-rate sweep's](https://github.com/marin-community/marin/issues/7856) fourth run length at 10:30 p.m. (ET) at 3.8364 on Paloma held-out text, behind [rate 1.1's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-004-d512-300x-lr1.1) 3.7709, so 1.1 still leads that 10,575-step length with two of its four rates in; [rate 0.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-016-d512-300x-lr0.7) sits 0.014 behind 1.1 at step 10,000 with its final decay still to run, so the length is not settled. [Rate 1.4 opened at the longest length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-020-d512-600x-lr1.4) a minute later, the fourth and last of the four rates to start there, so every run this tree will make is now open.
ladderlr-sweepmilestone
Larry #hero-run-2026
Larry opened a thread at 10:28 p.m. (ET) on the hero's total gradient norm, which has climbed since the September 9 switch to the near-dropless expert-parallel backend — W&B puts it near 0.57 over the last thousand steps against 0.39 over the first thousand after the switch, and nearly all of that magnitude comes from the output layer, whose gradient scale is set by the learnable gain on the final normalisation. He read the rise as the model predicting more confidently now that the router drops almost no tokens rather than as a fault, and named the trigger for acting: a gradient norm past 5 together with a diverging loss curve, which would mean restarting from an earlier checkpoint and either raising the logit penalty that keeps output scores bounded or adding weight decay to the final normalisation.
herotraining-dynamicsgradient-norm
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) finished repeating the steps it had lost and logged step 95,973 at 9:46 p.m. (ET), within a minute of the estimate in the entry below, so the counter is moving again and the run is now past step 95,972, where [the silent hang](https://github.com/marin-community/marin/issues/8870) stopped it. It then cleared a held-out evaluation at step 95,999 between 9:53 and 9:55 p.m. (ET) — the lowest of the five readings this run has logged on that text, and the first to fall after three that drifted upward — and the step-time pattern of a checkpoint write followed from 9:56 to 10:00 p.m. (ET). Steps now come every 17.2 s against the 18.5 s median before the hang, with no second stall in the forty-eight minutes since the healthy profile returned; [the status log](https://github.com/marin-community/marin/issues/8506) still carries nothing on either the stop or the restart.
herorestarteval
mcwitt #8870
[The silent-hang issue](https://github.com/marin-community/marin/issues/8870) — open since September 3 with no named cause — got a revised diagnosis at 9:26 p.m. (ET), thirteen minutes after the hero restarted from tonight's stall. The comment runs one line and links only to an internal incident note this page cannot read, so whether the revision covers tonight's stop is not stated; [the status log](https://github.com/marin-community/marin/issues/8506) still carries nothing on either the stop or the restart.
herohang
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) restarted in place at 9:13 p.m. (ET), nineteen minutes after [the silent hang](https://github.com/marin-community/marin/issues/8870) stopped it, and the healthy pattern of GPU memory and link traffic returned at 9:17 p.m. (ET) and has held since. It resumed from the 8:25 p.m. (ET) save of state, so it repeats about ninety-five steps and the step count stays frozen at 95,972 until near 9:47 p.m. (ET); no status-log entry covers either the stop or the restart.
herorestarthang
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) logged its last step, 95,972, at 8:54 p.m. (ET) and has taken none since; from 8:55 p.m. (ET) every GPU W&B samples reads 100% busy with memory traffic and link traffic both at zero and allocated memory frozen at 197.9 GB — the signature of [the silent hang](https://github.com/marin-community/marin/issues/8870), open since September 3 with no named cause. The job still sends a live heartbeat, so W&B reports the run healthy and no restart has fired thirteen minutes in; [the September 10 hang](https://github.com/marin-community/marin/issues/8506#issuecomment-5614639876) ran about fifteen minutes before the progress watchdog killed the gang and cost twenty-one minutes in all. The last hourly write of state closed at 8:25 p.m. (ET) near step 95,877, so a restart should repeat about ninety-five steps, near half an hour, and the stall ends sixty-five hours of unbroken stepping at an 18.5-second median.
herocrashhang
WhenWen AUG-LIN0-6L-DENSE-MUONH-018
Correction: [rate 0.45 at the learning-rate sweep's longest run length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-018-d512-600x-lr0.45) repeated nothing. The entry below forecast a forty-minute replay of 1,380 steps; the run instead logged step 1,381 at 8:48 p.m. (ET), straight after its last step before the stop, so it was restored from a save at that same step and lost only wall-clock time — thirty-four minutes between the two steps. It has stepped since and now stands level with [rate 0.7 at the same length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7), which came back the same way, so [the sweep](https://github.com/marin-community/marin/issues/7856) lost no steps in last night's cluster of stops.
ladderlr-sweeprestart
WhenWen AUG-LIN0-6L-DENSE-MUONH-018
[Rate 0.45 at the learning-rate sweep's longest run length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-018-d512-600x-lr0.45) started again at 8:30 p.m. (ET), sixteen minutes after the stop W&B had marked a crash, and its chips have carried a normal training load since — so the five runs that stopped together at about 7:50 p.m. (ET) are all back, and the cause of any of those stops is still recorded nowhere on [the sweep's issue](https://github.com/marin-community/marin/issues/7856). Its step counter will stay put until it repeats the 1,380 steps it lost, about forty minutes at its earlier pace, because W&B refuses a step number it already holds.
ladderlr-sweeprestart
WhenWen AUG-LIN0-6L-DENSE-MUONH-015 … -019
Five of the six six-layer runs [the learning-rate sweep](https://github.com/marin-community/marin/issues/7856) had live stopped logging within sixty-seven seconds of each other, from 7:49 to 7:50 p.m. (ET), and four came back in place between 8:09 and 8:19 p.m. (ET), each at the step after its last, so no steps were repeated or lost — among them [rate 0.7 at the longest run length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7), reported crashed in the entry below, which is stepping again. [Rate 0.45 at that length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-018-d512-600x-lr0.45) came back at 8:13 p.m. (ET), logged forty-two steps, stopped a minute later, and W&B has marked it crashed with nothing in its place; no comment on the sweep's issue covers the stop, and its cause is not recorded anywhere.
ladderlr-sweepcrash
Sep 12
WhenWen AUG-LIN0-6L-DENSE-MUONH-019
[The learning-rate sweep's newest run](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7) crashed at 7:50 p.m. (ET), sixteen minutes after it started, at 277 of the 21,150 steps of [the sweep's](https://github.com/marin-community/marin/issues/7856) longest run length. Nothing has relaunched in its place, and it is the second run the sweep has lost today — [the probe that would settle its ranking of four optimisers](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-4-d512-600x-lr4.3) has sat crashed since 1:49 p.m. (ET). The longest length keeps two live runs, [rate 1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-005-d512-600x-lr1.1) and [rate 0.45](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-018-d512-600x-lr0.45); the chips are TPU, not the hero's.
ladderlr-sweepcrash
WhenWen AUG-LIN0-6L-DENSE-MUONH-004 / -019
[Rate 1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-004-d512-300x-lr1.1) closed first at [the learning-rate sweep's](https://github.com/marin-community/marin/issues/7856) fourth run length, 10,575 steps, at 3.7709 on Paloma held-out text; the other three rates at that length stand between a tenth and just past half of the way through. [Rate 0.7 opened at the longest length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-019-d512-600x-lr0.7) at 7:34 p.m. (ET), the third of the four rates to start there.
ladderlr-sweep
WhenWen moe-lgr-9110-larry-d768-muonh
[The wider run in the corrected-routing test](https://wandb.ai/marin-community/dial_moe/runs/moe-lgr-9110-larry-d768-muonh) stopped at 14,252 of the 16,874 updates its control ran, about an hour short of the finish, and W&B has marked it crashed; no replacement has started. Its fourteenth and last paired reading, at 13,999 updates, was 3.3119 on Paloma held-out text against [its control's](https://wandb.ai/marin-community/dial_moe/runs/moe_may_compute_opt_mla_hd192_vd256_qc_latentmoe_half_norm_gatednorm_xsa_d768) 3.3059 — 0.006 behind, and all fourteen sat within 0.02 of the control either way, so [the correction](https://github.com/marin-community/marin/issues/9110) closes this width with no gain shown.
experimentmoerouting
WhenWen AUG-LIN0-6L-DENSE-MUONH-014
[Rate 1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-014-d512-150x-lr1.4) closed [the learning-rate sweep's](https://github.com/marin-community/marin/issues/7856) third run length at 3.8615 on Paloma held-out text, ahead of [rate 1.1's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-013-d512-150x-lr1.1) 3.8684, so the fastest of the four rates wins the 5,288-step length after 1.1 won the length below it. The pick sits at the top of the range the sweep tries, which is where its automation opens an extra run at a faster rate; at [the shortest length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X30-HIGH-1P7-d512-30x-lr1.7) that extra run came back worse and the original four rates stood.
ladderlr-sweepmilestone
WhenWen moe-lgr-9110-larry-d768-muonh
[The wider run in the corrected-routing test](https://wandb.ai/marin-community/dial_moe/runs/moe-lgr-9110-larry-d768-muonh) took its thirteenth paired reading at 7:07 p.m. (ET): 3.3519 on Paloma held-out text at 13,000 updates against [its control's](https://wandb.ai/marin-community/dial_moe/runs/moe_may_compute_opt_mla_hd192_vd256_qc_latentmoe_half_norm_gatednorm_xsa_d768) 3.3452 at the same point — 0.007 behind, against 0.003 at the reading before. It has covered 13,000 of the 16,874 updates the control ran and should close about 8:35 p.m. (ET). Thirteen readings in, [the correction](https://github.com/marin-community/marin/issues/9110) has shown no gain at this width.
experimentmoerouting
WhenWen AUG-LIN0-6L-DENSE-MUONH-017 / -018
[The learning-rate sweep](https://github.com/marin-community/marin/issues/7856) opened two more six-layer runs just after 7:05 p.m. (ET), reaching into its two longest lengths. [Rate 1.4 at 10,575 steps](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-017-d512-300x-lr1.4) fills the last of the four rates tried at that length, and [rate 0.45 at 21,150 steps](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-018-d512-600x-lr0.45) is the second rate to start at the longest, where [rate 1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-005-d512-600x-lr1.1) has been running since 2:16 p.m. (ET).
ladderlr-sweep
WhenWen AUG-LIN0-6L-DENSE-MUONH-012 / -013
Two more of [the learning-rate sweep's](https://github.com/marin-community/marin/issues/7856) 5,288-step six-layer runs closed at 7:04 p.m. (ET). [Rate 1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-013-d512-150x-lr1.1) scored 3.8684 on Paloma held-out text and [rate 0.45](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-012-d512-150x-lr0.45) 3.9407, against [rate 0.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-003-d512-150x-lr0.7) 3.8810 earlier in the evening. Rate 1.1 leads this third length, the same pick as the length below it, with [rate 1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-014-d512-150x-lr1.4) about 300 steps from closing the set.
ladderlr-sweepmilestone
WhenWen moe-lgr-9110-larry-d768-muonh
[The wider run in the corrected-routing test](https://wandb.ai/marin-community/dial_moe/runs/moe-lgr-9110-larry-d768-muonh) took its twelfth paired reading at 6:45 p.m. (ET): 3.3895 on Paloma held-out text at 12,000 updates against [its control's](https://wandb.ai/marin-community/dial_moe/runs/moe_may_compute_opt_mla_hd192_vd256_qc_latentmoe_half_norm_gatednorm_xsa_d768) 3.3862 at the same point — 0.003 behind, inside the 0.01 band for the second reading running. It has covered 12,900 of the 16,874 updates the control ran and should close about 8:35 p.m. (ET); twelve readings in, [the correction](https://github.com/marin-community/marin/issues/9110) has shown no gain at this width.
experimentmoerouting
WhenWen moe-lgr-9110-larry-d768-muonh
[The wider run in the corrected-routing test](https://wandb.ai/marin-community/dial_moe/runs/moe-lgr-9110-larry-d768-muonh) took its eleventh paired reading at 6:23 p.m. (ET): 3.4406 on Paloma held-out text at 11,000 updates against [its control's](https://wandb.ai/marin-community/dial_moe/runs/moe_may_compute_opt_mla_hd192_vd256_qc_latentmoe_half_norm_gatednorm_xsa_d768) 3.4304 at the same point — 0.010 behind, against 0.007 at the reading before. It has covered 11,000 of the 16,874 updates the control ran and should close about 8:35 p.m. (ET). Eleven readings in, [the correction](https://github.com/marin-community/marin/issues/9110) has still shown no gain at this width.
experimentmoerouting
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) reached step 95,408 at 6:01 p.m. (ET) at about eighteen seconds a step, and passes sixty-two hours of unbroken stepping at 6:11 p.m. (ET). Its 5:24 p.m. (ET) hourly save of state cleared in about ninety seconds, the same shape as the saves before it, and the next falls about 6:24 p.m. (ET). The next reading on held-out text comes at step 95,999, about three hours away.
herocheckpointmilestone
WhenWen moe-lgr-9110-larry-d768-muonh
[The wider run in the corrected-routing test](https://wandb.ai/marin-community/dial_moe/runs/moe-lgr-9110-larry-d768-muonh) took its tenth paired reading at 6:00 p.m. (ET): 3.4797 at 9,999 updates against [its control's](https://wandb.ai/marin-community/dial_moe/runs/moe_may_compute_opt_mla_hd192_vd256_qc_latentmoe_half_norm_gatednorm_xsa_d768) 3.4724 at 10,000 — 0.007 behind, back inside the 0.01 band after the 0.018 gap of the reading before. It has now passed 10,000 of the 16,874 updates the control ran and should close about 8:35 p.m. (ET). Across all ten readings it has stayed within 0.02 of the control either way, so [the correction](https://github.com/marin-community/marin/issues/9110) has still shown no gain at this width.
experimentmoerouting
WhenWen moe-lgr-9110-larry-d768-muonh
[The wider run in the corrected-routing test](https://wandb.ai/marin-community/dial_moe/runs/moe-lgr-9110-larry-d768-muonh) took its eighth paired reading at 5:38 p.m. (ET): 3.5251 at 8,999 updates against [its control's](https://wandb.ai/marin-community/dial_moe/runs/moe_may_compute_opt_mla_hd192_vd256_qc_latentmoe_half_norm_gatednorm_xsa_d768) 3.5073 at 9,000 — 0.018 behind. That is as wide a gap as the run's first reading showed and the first above 0.01 since, ending six readings in a row within 0.01 of the control either way. The run has covered 9,179 of the 16,874 updates the control ran, so [the correction](https://github.com/marin-community/marin/issues/9110) has still shown no gain at this width, and the gap has stopped closing.
experimentmoerouting
WhenWen AUG-LIN0-6L-DENSE-MUONH-016
[The learning-rate sweep](https://github.com/marin-community/marin/issues/7856) opened [rate 0.7 at 10,575 steps](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-016-d512-300x-lr0.7) at 5:37 p.m. (ET), giving its fourth run length a third rate beside [0.45](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-015-d512-300x-lr0.45) and [1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-004-d512-300x-lr1.1). Every shorter length has been decided from a set of four rates, so this brings the fourth length within one rate of the same grid. The run should close about 11:25 p.m. (ET). The chips are TPU, not the hero's.
ladderlr-sweep
WhenWen AUG-LIN0-6L-DENSE-MUONH-011
Both of [the learning-rate sweep's](https://github.com/marin-community/marin/issues/7856) two shortest six-layer lengths now have an answer, and the answers differ. [The 1.7 probe at 1,058 steps](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X30-HIGH-1P7-d512-30x-lr1.7) closed at 5:27 p.m. (ET) at 4.2722 on Paloma held-out text, behind both [1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-008-d512-30x-lr1.4) at 4.2589 and [1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-001-d512-30x-lr1.1) at 4.2653, so the faster rate the sweep added was unnecessary and that length keeps 1.4 — the fastest rate its set contains. Eight minutes later [rate 1.4 at 2,115 steps](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-011-d512-60x-lr1.4) closed at 4.0535, behind [1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-002-d512-60x-lr1.1) at 4.0442, so the longer length picks a slower rate and needs no probe. The best rate therefore falls as the runs get longer, which is the first thing the six-layer wave can say about rate against length. The chips are TPU, not the hero's.
ladderlr-sweepmilestone
WhenWen AUG-LIN0-6L-DENSE-MUONH-X30-HIGH-1P7
[The learning-rate sweep's first probe at a rate faster than the fixed set it tries at each run length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X30-HIGH-1P7-d512-30x-lr1.7) took its first reading at 5:25 p.m. (ET): 4.2990 on Paloma held-out text at 1,000 of its 1,058 updates, behind both [rate 1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-008-d512-30x-lr1.4) at 4.2823 and [rate 1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-001-d512-30x-lr1.1) at 4.2814 at the same point. On that reading the best rate at this length sits inside the set already tested and the extension is unnecessary, but 1.1 and 1.4 swapped places between this point and their own final updates, so the probe's close at about 5:30 p.m. (ET) is what settles it. The chips are TPU, not the hero's.
ladderlr-sweep
WhenWen moe-lgr-9110-larry-d768-muonh
[The wider run in the corrected-routing test](https://wandb.ai/marin-community/dial_moe/runs/moe-lgr-9110-larry-d768-muonh) added two paired readings, the newest at 5:17 p.m. (ET): 3.6117 at 7,000 updates against [its control's](https://wandb.ai/marin-community/dial_moe/runs/moe_may_compute_opt_mla_hd192_vd256_qc_latentmoe_half_norm_gatednorm_xsa_d768) 3.6020, then 3.5649 at 8,000 against 3.5614 — 0.010 and 0.003 behind. Six readings in a row now fall within 0.01 of the control either way, and the run has passed half the 16,874 updates the control ran, so [the correction](https://github.com/marin-community/marin/issues/9110) still shows no effect at this width.
experimentmoerouting
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) cleared its hourly save of state near 4:24 p.m. (ET) like the ones before it, and reached step 95,220 at 5:03 p.m. (ET) at about eighteen seconds a step. The save slot has drifted a minute an hour over the past day, from :22 on September 11 to :24 now, and the next save falls about 5:24 p.m. (ET). Sixty-one hours of unbroken stepping falls at 5:11 p.m. (ET), and the next reading on held-out text at step 95,999, about four hours away.
herocheckpointmilestone
WhenWen AUG-LIN0-6L-DENSE-MUONH-003
Three more of the learning-rate sweep's six-layer runs closed. At 2,115 steps [rate 0.45](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-009-d512-60x-lr0.45) scored 4.1508 on Paloma held-out text and [0.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-010-d512-60x-lr0.7) scored 4.0630, both behind [1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-002-d512-60x-lr1.1) at 4.0442, so that grid still improves with every faster rate and its answer waits on [1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-011-d512-60x-lr1.4), due about 5:40 p.m. (ET). At 5,288 steps [rate 0.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-003-d512-150x-lr0.7) finished at 5:00 p.m. (ET) at 3.8810, against 4.9142 for [the one-layer run at the same rate and step budget](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-015-d512-150x-lr0.7) — the largest six-layer gain the wave has shown. The chips are TPU, not the hero's.
ladderlr-sweepmilestone
WhenWen AUG-LIN0-6L-DENSE-MUONH-014
[The learning-rate sweep](https://github.com/marin-community/marin/issues/7856) opened three more six-layer runs. [Rate 1.7 at 1,058 steps](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-X30-HIGH-1P7-d512-30x-lr1.7) went out at 4:50 p.m. (ET) as the wave's first automatic boundary extension: each length is tested on a fixed set of rates, 1.4 was the fastest rate at this length and it won, so the sweep adds one rate faster to find out whether the best rate lies outside the set. At one layer the same 1.7 probe failed to beat 1.1 here. [Rate 1.4 at 5,288 steps](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-014-d512-150x-lr1.4) and [rate 0.45 at 10,575 steps](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-015-d512-300x-lr0.45) followed at 4:59 p.m. (ET), giving the third length a fourth rate and the fourth length a second one. The probe should close about 5:30 p.m. (ET), the 5,288-step run about 8 p.m. (ET) and the 10,575-step run about 10:30 p.m. (ET).
ladderlr-sweep
WhenWen moe-lgr-9110-larry-d512-muonh
[The narrower run in the corrected-routing test](https://wandb.ai/marin-community/dial_moe/runs/moe-lgr-9110-larry-d512-muonh) finished all 10,979 of its updates at 3:40 p.m. (ET) and ended level with [its matched control](https://wandb.ai/marin-community/dial_moe/runs/moe_may_compute_opt_mla_hd192_vd256_qc_latentmoe_half_norm_gatednorm_xsa_d512) on Paloma held-out text, 3.5694 against 3.5700. [The wider run](https://wandb.ai/marin-community/dial_moe/runs/moe-lgr-9110-larry-d768-muonh) has passed 6,600 updates, against the 16,874 [its own control](https://wandb.ai/marin-community/dial_moe/runs/moe_may_compute_opt_mla_hd192_vd256_qc_latentmoe_half_norm_gatednorm_xsa_d768) ran, and has closed the gap its early readings showed: 0.018 behind at about 2,000 updates, then 0.007 behind, 0.003 behind, 0.009 ahead and 0.003 behind at the four readings since. So far [the correction](https://github.com/marin-community/marin/issues/9110) neither helps nor hurts at either width.
experimentmoeroutingmilestone
WhenWen AUG-LIN0-6L-DENSE-MUONH-011
[The learning-rate sweep](https://github.com/marin-community/marin/issues/7856) opened three more six-layer runs at 4:29 p.m. (ET), two minutes after its 1,058-step grid closed. [Rate 1.4 at 2,115 steps](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-011-d512-60x-lr1.4) brings that length to the same four rates the 1,058-step grid carried — 0.45, 0.7, 1.1 and 1.4 — so the two lengths can now be read rate for rate; [0.45](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-012-d512-150x-lr0.45) and [1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-013-d512-150x-lr1.1) at 5,288 steps widen the third length, which until now carried only [0.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-003-d512-150x-lr0.7). The 2,115-step run should close about 6:05 p.m. (ET) and the two 5,288-step runs about 8:25 p.m. (ET). The chips are TPU, not the hero's.
ladderlr-sweep
WhenWen AUG-LIN0-6L-DENSE-MUONH-008
The learning-rate sweep's shortest six-layer grid closed at 4:27 p.m. (ET). On Paloma held-out text [rate 1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-008-d512-30x-lr1.4) scored best at 4.2589, ahead of [1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-001-d512-30x-lr1.1) at 4.2653, [0.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-007-d512-30x-lr0.7) at 4.2896 and [0.45](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-006-d512-30x-lr0.45) at 4.4133. The winner sits at the top of the grid and above the 1.1 that [the one-layer stand-in at the same length picked](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X30-HIGH-1-d512-30x-lr1.1), so the sweep's usual answer is a probe at a faster rate and the six-layer choice at this length is not settled. The chips are TPU, not the hero's.
ladderlr-sweepmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) passed sixty hours of unbroken stepping at 4:11 p.m. (ET), reaching step 95,104 at 4:28 p.m. (ET) at seventeen to eighteen seconds a step. Its 3:23 p.m. (ET) hourly save of state passed like the ones before it, holding the :23 slot it has used every hour since September 11, and the next falls at 4:23 p.m. (ET). Its next reading on held-out text falls at step 95,999, about four and a half hours away.
herocheckpointmilestone
WhenWen AUG-LIN0-6L-DENSE-MUONH-006
[The learning-rate sweep](https://github.com/marin-community/marin/issues/7856) opened five more six-layer runs at 3:50 p.m. (ET), all at the two shortest run lengths: rates [0.45](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-006-d512-30x-lr0.45), [0.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-007-d512-30x-lr0.7) and [1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-008-d512-30x-lr1.4) at 1,058 steps, and [0.45](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-009-d512-60x-lr0.45) and [0.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-010-d512-60x-lr0.7) at 2,115. Until now the wave carried one rate per length, copied from the one-layer stand-ins, so it could only say whether six layers beat one; a grid of rates at each of these two lengths lets it say whether the rate the stand-ins picked is still the best one at depth. The two 2,115-step runs should close about 5:01 p.m. (ET).
ladderlr-sweep
WhenWen AUG-LIN0-6L-DENSE-MUONH-002
[The second of the learning-rate sweep's six-layer runs](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-002-d512-60x-lr1.1) finished all 2,115 of its steps at 3:31 p.m. (ET), on the estimate published half an hour earlier. It scored 4.0442 on Paloma held-out text against 5.0622 for [the one-layer run at the same rate and the same step budget](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X60-HIGH-1-d512-60x-lr1.1), about the same size of gain the shortest pair showed.
ladderlr-sweepmilestone
WhenWen moe-lgr-9110-larry-d512-muonh
[The test of a corrected way of routing tokens to experts](https://github.com/marin-community/marin/issues/9110) has its first matched evidence. [The narrower replacement](https://wandb.ai/marin-community/dial_moe/runs/moe-lgr-9110-larry-d512-muonh) has covered about 9,000 of its 10,980 updates and now reads level with [its matched control](https://wandb.ai/marin-community/dial_moe/runs/moe_may_compute_opt_mla_hd192_vd256_qc_latentmoe_half_norm_gatednorm_xsa_d512) on Paloma held-out text — 3.6712 against 3.6730 at the latest paired reading, and every reading since about 3,000 updates within 0.016 either way — where the stale baselines the test used before put it 2.6 behind. [The wider replacement](https://wandb.ai/marin-community/dial_moe/runs/moe-lgr-9110-larry-d768-muonh) has two readings so far: 0.004 behind [its control](https://wandb.ai/marin-community/dial_moe/runs/moe_may_compute_opt_mla_hd192_vd256_qc_latentmoe_half_norm_gatednorm_xsa_d768) at about 1,000 updates and 0.018 behind at about 2,000.
experimentmoeroutingmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) passed fifty-eight hours of unbroken stepping at 2:11 p.m. (ET) and fifty-nine at 3:11 p.m. (ET), reaching step 94,895 at 3:22 p.m. (ET) at seventeen to nineteen seconds a step. Its 2:22 p.m. (ET) hourly save of state passed like the ones before it, and the next falls at 3:22 p.m. (ET), holding the :22 slot it has used every hour since 2:19 p.m. (ET) on September 11. Its next reading on held-out text falls at step 95,999, about five and a half hours away.
herocheckpointmilestone
Helw150 9126
[A new issue](https://github.com/marin-community/marin/issues/9126) asks the team to judge the four-width mixture-swap ladder against [the original hero scaling ladder](https://github.com/marin-community/marin/issues/8435) before a data swap planned for the hero on Monday, September 14 — the first firm date for changing what the run trains on. Its own reading so far: before the switch the candidate mixture is slightly ahead at the two middle widths, the lead narrows by half-way, and RedPajama, Penn Treebank, Wikipedia, Wikitext and programming-language text all get worse. It warns that the two ladders differ in hardware, expert layout and evaluation setup, so the comparison is not matched.
ladderdata-mixhero
WhenWen AUG-LIN0-6L-DENSE-MUONH-001
[The shortest of the learning-rate sweep's five six-layer runs](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-001-d512-30x-lr1.1) finished all 1,058 of its steps at 3:02 p.m. (ET), thirty-four minutes after its first step. It scored 4.2653 on Paloma held-out text where [the one-layer run at the same rate and the same step budget](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X30-HIGH-1-d512-30x-lr1.1) scored 5.2601, so six layers pay off at the rate the one-layer tree picked for this length. Only one rate was launched per length, so the wave says whether the picks hold at depth only once the four longer runs land: [2,115 steps](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-002-d512-60x-lr1.1) about 3:33 p.m. (ET), [5,288 steps](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-003-d512-150x-lr0.7) about 5:32 p.m. (ET), [10,575 steps](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-004-d512-300x-lr1.1) about 7:54 p.m. (ET) and [21,150 steps](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-005-d512-600x-lr1.1) about 1:26 a.m. (ET) Sunday. The chips are TPU, not the hero's.
ladderlr-sweepmilestone
WhenWen AUG-LIN0-6L-DENSE-MUONH
[The learning-rate sweep](https://github.com/marin-community/marin/issues/7856) left its one-layer stand-in models at 2:16 p.m. (ET) and opened five six-layer ones, one per run length, each at the rate the one-layer Muon tree had just picked for that length: 1.1 at [1,058](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-001-d512-30x-lr1.1), [2,115](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-002-d512-60x-lr1.1), [10,575](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-004-d512-300x-lr1.1) and [21,150 steps](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-005-d512-600x-lr1.1), and 0.70 at [5,288](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-6L-DENSE-MUONH-003-d512-150x-lr0.7), the one length where the grid's own top rate held. Width, batch size and step budgets match the one-layer trees exactly, so depth is the only change, and the last of those picks landed four hours earlier at [10:33 a.m. (ET)](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X600-HIGH-1P4-d512-600x-lr1.4). The chips are TPU, not the hero's.
ladderlr-sweepmilestone
WhenWen moe-lgr-9110-larry-d512-muonh
Replacements for the two stopped runs started at 2:06 and 2:12 p.m. (ET) on a later commit — 23898fb87 against 86e5826e4 — with every other logged setting identical and `muonh` added to both names. [The narrower replacement](https://wandb.ai/marin-community/dial_moe/runs/moe-lgr-9110-larry-d512-muonh) scored 4.5822 on Paloma held-out text at step 999, against 7.1434 for [the run it replaces](https://wandb.ai/marin-community/dial_moe/runs/moe-lgr-9110-larry-d512) at the same step and 4.5498 for the control [#9110 reads it against](https://github.com/marin-community/marin/issues/9110#issuecomment-5647677405), so the large early gap belongs to something that commit changed, not to the routing change under test. No comment covers the relaunch; the newest #9110 entry in the corpus is the 1:59 p.m. (ET) one, and the mirror lags.
experimentmoeroutingrestart
WhenWen AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-4-d512-600x-lr4.3
[The plain-gradient rate-4.3 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-4-d512-600x-lr4.3) died four minutes after it resumed. It logged step 11,590 of 21,149 at 1:49 p.m. (ET), its accelerator telemetry stopped in the same minute, and W&B now marks the run crashed — where its two earlier stops today kept the process reporting in and reloaded under this same run id, so this is a harder failure than either. Nothing has attached to the run in the fourteen minutes since, and the probe decides whether the plain-gradient family settles on rate 2.7 at the longest run length, so that pick now waits on a relaunch.
ladderlr-sweepcrash
WhenWen moe-lgr-9110-larry-d512
[The test of a corrected way of routing tokens to experts](https://github.com/marin-community/marin/issues/9110) restarted at 1:40 p.m. (ET) against matched controls — Larry's own d512 and d768 runs, on the same schedules and batch sizes — because the stored baselines it had used before were trained under a different architecture, recipe and chip generation and could not attribute the gap. Both new runs died about twenty minutes in: [the narrower](https://wandb.ai/marin-community/dial_moe/runs/moe-lgr-9110-larry-d512) at step 2,330 of 10,980 and [the wider](https://wandb.ai/marin-community/dial_moe/runs/moe-lgr-9110-larry-d768) at 796 of 16,875, with W&B marking both crashed. [The issue's first reading](https://github.com/marin-community/marin/issues/9110#issuecomment-5647677405), posted at 1:59 p.m. (ET), put the narrower run 2.6 behind its control on held-out text at step 999 and described both runs as still training. The chips are TPU, not the hero's.
experimentmoeroutingcrash
WhenWen AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-4-d512-600x-lr4.3
[The plain-gradient rate-4.3 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-4-d512-600x-lr4.3) is stepping again. It logged step 11,436 at 1:45 p.m. (ET) — the step straight after the last one before the stop, so no steps were replayed — and the stop cost fifty-three minutes, four fewer than its first stop today; the recovery put the model on the accelerators at 1:33 p.m. (ET) and reached its first step twelve minutes later, where the first stop took seventeen. It is past half of its 21,149 steps at about one second a step and should close near 4:25 p.m. (ET) if it does not stop a third time; no cause is recorded for either stop.
ladderlr-sweeprestart
WhenWen moe-lgr-9110-d768-rmsgated
[The wider of the two runs testing a corrected way of routing tokens to experts](https://wandb.ai/marin-community/marin_moe/runs/moe-lgr-9110-d768-rmsgated) finished all 19,378 of its steps at 12:37 p.m. (ET), completing the two-width report [#9110](https://github.com/marin-community/marin/issues/9110) was waiting on. It scored 3.5582 on Paloma held-out text against [the stored baseline of the same width](https://wandb.ai/marin-community/marin_moe/runs/moe_may_compute_opt_d768_ep1) at 3.2273, about the same gap [the narrower run showed](https://wandb.ai/marin-community/marin_moe/runs/moe-lgr-9110-d512-rmsgated). #9110 had already refused the first gate on the narrower width, so the wider models it could have unlocked will not be built, and the issue repeats that the stored baselines were trained under a different architecture, recipe and chip generation than these runs, so neither width isolates the gating change. The chips are TPU, not the hero's.
experimentmoeroutingmilestone
WhenWen AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-4-d512-600x-lr4.3
[The plain-gradient rate-4.3 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-4-d512-600x-lr4.3) is re-loading on the same profile as its first stop, and a little ahead of it. Its replacement process cleared the accelerators at 1:00 p.m. (ET), eight minutes after the last step at 12:52 p.m. (ET), and has compiled one program at a time since: the queue of compiled programs on its first accelerator has grown from zero to eight and the mean time it spends running one from nothing to about 105,800 microseconds. The first stop's recovery reached the same two marks six minutes later in its own run, then put the model on the accelerators two minutes after that and logged its first step seventeen minutes later. On that alignment this one loads near 1:27 p.m. (ET) and steps again near 1:44 p.m. (ET), inside the fifty-seven minutes the first stop cost. It still holds step 11,435 of 21,149, and no cause is recorded for either stop.
ladderlr-sweep
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) passed fifty-seven hours of unbroken stepping at 1:11 p.m. (ET) and entered its hourly save of state at 1:22 p.m. (ET), holding the :22 slot it has used every hour since 2:19 p.m. (ET) on September 11. The save is leaving the mark it always leaves — steps widened from eighteen seconds to about twenty-two, and the fraction of the GPUs' compute in use fell from 23% to 16% — and the run reached step 94,508 at 1:24 p.m. (ET) while still inside it. Its next reading on held-out text falls at step 95,999, about seven and a half hours away.
herocheckpointmilestone
WhenWen AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-4-d512-600x-lr4.3
[The plain-gradient rate-4.3 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-4-d512-600x-lr4.3) stopped for the second time today. It logged step 11,435 of 21,149 at 12:52 p.m. (ET) and none since; its accelerators fell idle three minutes later, and from 12:55 p.m. (ET) its memory and compile times have been climbing back from nothing — the same restart profile as [its first stop](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-4-d512-600x-lr4.3), which began at 9:53 a.m. (ET), took fifty-seven minutes, and resumed at the step it had left rather than replaying any. On that precedent it should be stepping again near 1:45 p.m. (ET). Its last reading on held-out text before the stop, 5.2655 at step 11,000, still sits behind [rate 2.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-3-d512-600x-lr2.7) 5.1516 at the same step, and that tree's finished rates reorder themselves after this point, so the gap still forecasts nothing.
ladderlr-sweep
WhenWen AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-4-d512-600x-lr4.3
The plain-gradient probe now reads behind at matched steps, and its own family says that forecasts nothing. [The rate-4.3 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-4-d512-600x-lr4.3) scored 5.3146 on Paloma held-out text at step 10,000, against [rate 2.7's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-3-d512-600x-lr2.7) 5.2121 at the same step. But the four finished rates of this tree reorder themselves between that step and the close: at step 10,000 they ranked [1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-1-d512-600x-lr1.1) 5.1139, [1.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-2-d512-600x-lr1.7) 5.1322, [0.70](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-025-d512-600x-lr0.7) 5.1345 and 2.7 last at 5.2121, and at step 21,149 they rank 2.7 first at 4.8915, 1.7 at 4.9009, 1.1 at 4.9303 and 0.70 last at 4.9845 — the worst at half budget finished best, and the best at half budget finished third. Every rate decays to zero, so a higher one reads worse until its decay bites. The probe passed half of its 21,149 steps at 12:30 p.m. (ET) and has stepped at about one second a step with no pause since the one that ended at 10:50 a.m. (ET), so it should close near 3:30 p.m. (ET).
ladderlr-sweep
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) passed fifty-six hours of unbroken stepping at 12:11 p.m. (ET) and cleared its hourly save of state at 12:23-12:24 p.m. (ET), holding the :22 slot it has used every hour since 2:19 p.m. (ET) on September 11. It reached step 94,312 at 12:24 p.m. (ET) at nineteen to twenty seconds a step, and its next reading on held-out text falls at step 95,999, about nine hours away.
herocheckpointmilestone
held h100-mix25-20260912-d1024/d1536
The new four-width scaling ladder's two unfinished rungs are much further out than the first two: [the 4.0-billion-parameter rung](https://wandb.ai/marin-community/marin_moe/runs/h100-mix25-20260912-d1024) is two-fifths through its 30,552 steps and should close around midnight (ET), and [the 11.5-billion one](https://wandb.ai/marin-community/marin_moe/runs/h100-mix25-20260912-d1536) a tenth through its 90,767 and not before September 15. The widest comparison will also be one-sided — [August's run at that width](https://wandb.ai/marin-community/marin_moe/runs/h100-ladder-d1536-ep8x8-bs1024-791tpp-20260825-rno2a-interactive) crashed at step 30,757 of the same 90,767-step budget, so only [the 4.0-billion width](https://wandb.ai/marin-community/marin_moe/runs/h100-ladder-d1024-ep8x4-bs1024-791tpp-20260825-rno2a-interactive) still has a finished August run to be read against.
ladderdata-mix
held h100-mix25-20260912-d768
Correction: the new four-width scaling ladder is not behind the August one. Read the way [the hero chart on this page](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) reads its own eval curve — with the router's per-expert capacity limit switched off, so no token is dropped — both finished widths beat their August counterparts: [the 0.6-billion-parameter rung](https://wandb.ai/marin-community/marin_moe/runs/h100-mix25-20260912-d512) scored 3.3063 against [August's 3.3234](https://wandb.ai/marin-community/marin_moe/runs/h100-ladder-d512-ep8-bs1024-791tpp-20260824-rno2a), and [the 1.6-billion one](https://wandb.ai/marin-community/marin_moe/runs/h100-mix25-20260912-d768) 3.0012 against [August's 3.0181](https://wandb.ai/marin-community/marin_moe/runs/h100-ladder-d768-2xep8-bs1024-791tpp-20260824). The two entries that reported these rungs quoted the other reading, taken with the limit on, where the same two runs trail by about 0.05 — the new models lose about 0.066 more to dropping at both widths. The no-drop reading is also the quieter of the two: [one mixture's four random seeds](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed3-from10pct-20260829) span 0.0031 on it against 0.0126 with the limit on, so the 0.017 lead is about five times its seed noise. The earlier caveat stands: three weeks of code changes and an extra mixture-stage switch separate the two ladders, so neither reading isolates the mixture the new ladder exists to test.
ladderdata-mixcorrection
held h100-mix25-20260912-d768
[The second rung of the four-width scaling ladder](https://wandb.ai/marin-community/marin_moe/runs/h100-mix25-20260912-d768) finished all 11,420 of its steps at 11:37 a.m. (ET) and scored 3.5218 on Paloma held-out text, behind [the August run of the same width, step budget and batch size](https://wandb.ai/marin-community/marin_moe/runs/h100-ladder-d768-2xep8-bs1024-791tpp-20260824) at 3.4727. [The first rung showed the same gap](https://wandb.ai/marin-community/marin_moe/runs/h100-mix25-20260912-d512) — 3.8072 against [August's 3.7565](https://wandb.ai/marin-community/marin_moe/runs/h100-ladder-d512-ep8-bs1024-791tpp-20260824-rno2a) — and 0.05 is about four times the widest spread that repeats of one mixture under different random seeds produce, so the two widths agree. The new ladder switches data mixture twice, at steps 2,880 and 9,120, where the August one switched only at 9,120, and three weeks of code changes separate the two ladders as well, so the gap is not the mixture weights alone. [The four-billion-parameter rung](https://wandb.ai/marin-community/marin_moe/runs/h100-mix25-20260912-d1024) is a third of the way through its 30,552 steps and has August's own full-budget 3.1442 to beat.
ladderdata-mixmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) passed fifty-five hours of unbroken stepping at 11:12 a.m. (ET) and cleared its hourly save of state at 11:22-11:24 a.m. (ET), holding the :22 slot it has used every hour since 2:19 p.m. (ET) on September 11. The save left the mark it always leaves — steps widened from nineteen seconds to about twenty-two, and the fraction of the GPUs' compute in use fell from 23% to 19% for four steps — and both had returned to normal by step 94,116, so this save was as ordinary as the 8:22 and 10:22 a.m. (ET) ones. The lag the last report flagged has closed: the step count and chart above now trail the run by about fifteen steps, or five minutes.
herocheckpointmilestone
WhenWen AUG-LIN0-1L-DENSE-SGDMH-X600-HIGH-1-d512-600x-lr1.1
[The Nesterov-momentum family's rate-1.1 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-X600-HIGH-1-d512-600x-lr1.1) finished at 11:05 a.m. (ET) at 4.8026 on Paloma held-out text, ahead of [its rate-1.4 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-X600-HIGH-1P4-d512-600x-lr1.4) at 4.8091 and of [its grid's own best rate of 0.70](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-025-d512-600x-lr0.7) at 4.8228. The climb turns back at 1.4, so this family settles on rate 1.1 at the longest run length — one step above the range the grid searched, and the same pick it made at [the 10,575-step budget](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-X300-HIGH-1-d512-300x-lr1.1). No further momentum probe is open at this length.
ladderlr-sweepmilestone
WhenWen AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-4-d512-600x-lr4.3
[The plain-gradient family's rate-4.3 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-4-d512-600x-lr4.3) is stepping again, fifty-seven minutes after it stopped at 9:53 a.m. (ET). A replacement process started at 10:01 a.m. (ET) and logged its first step at 10:50 a.m. (ET), the step straight after the last one before the stall, so the run's step count did not go backwards; it has since passed a fifth of its 21,149 steps at about one second a step.
ladderlr-sweep
WhenWen AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-4-d512-600x-lr4.3
[The plain-gradient family's rate-4.3 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-4-d512-600x-lr4.3) has logged no step since 9:53 a.m. (ET), when it reached step 3,783 of 21,149 at about one and a half seconds a step, while its process has kept reporting in as late as 10:44 a.m. (ET). A pause of this length is not new in this fleet — [the Muon rate-1.4 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X600-HIGH-1P4-d512-600x-lr1.4) stopped for thirty-nine minutes at 6:50 a.m. (ET) and still finished every step — so this is a stall to watch rather than a failure. The probe decides whether the plain-gradient family settles on rate 2.7 at the longest run length, as it did at [the 5,288-step](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X150-HIGH-4-d512-150x-lr4.3) and [10,575-step](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X300-HIGH-4-d512-300x-lr4.3) budgets.
ladderlr-sweep
WhenWen AUG-LIN0-1L-DENSE-MUONH-X600-HIGH-1P4-d512-600x-lr1.4
[Muon's second probe above the 21,149-step grid](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X600-HIGH-1P4-d512-600x-lr1.4) finished at 10:33 a.m. (ET) at 4.7809 on Paloma held-out text, behind [the rate-1.1 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X600-HIGH-1-d512-600x-lr1.1) at 4.7719 and only seven ten-thousandths in front of [the grid's own best rate of 0.70](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-025-d512-600x-lr0.7) at 4.7817. The climb has turned back, so Muon settles on rate 1.1 at the longest run length — one step outside the range the grid searched, and the same pick it made at [the 10,575-step budget](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X300-HIGH-1-d512-300x-lr1.1). No further Muon probe is open at this length.
ladderlr-sweepmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) passed fifty-four hours of unbroken stepping at 10:12 a.m. (ET) and cleared its hourly save of state at 10:22-10:23 a.m. (ET), holding the :22 slot it has used every hour since 2:19 p.m. (ET) on September 11. It reached step 93,917 at 10:23 a.m. (ET), stepping every seventeen to nineteen seconds between saves. The page trails the run again: the step count and chart above sit about sixty-seven steps, or twenty minutes, behind, because the sampled feed W&B serves this page lags the run itself, which is stepping normally.
herocheckpointmilestone
WhenWen AUG-LIN0-1L-DENSE-SGDMH-X600-HIGH-1P4-d512-600x-lr1.4
[The Nesterov-momentum family's rate-1.4 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-X600-HIGH-1P4-d512-600x-lr1.4) finished at 10:24 a.m. (ET) at 4.8091 on Paloma held-out text, ahead of [its grid's own best rate of 0.70](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-025-d512-600x-lr0.7) at 4.8228, so this family's best rate also lies above the range the grid searched. [Its rate-1.1 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-X600-HIGH-1-d512-600x-lr1.1) sits about a thousand steps from the end and reads 4.8106 unfinished, so it decides whether the peak falls at 1.1, as it did at [the 10,575-step budget](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-X300-HIGH-1-d512-300x-lr1.1), or at 1.4.
ladderlr-sweepmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) reached step 93,849 at 10:02 a.m. (ET), stepping every seventeen to nineteen seconds. That makes fifty-three hours and fifty minutes of unbroken stepping since the September 10 restart. The caveat in the last entry has reversed: the lag between this page and the run has closed rather than grown, and the step count and chart above now trail the run by about eighteen steps, or five minutes, not sixty. The last hourly save of state cleared at 9:22-9:23 a.m. (ET); the next is due near 10:22 a.m. (ET), so none fell in this window.
herocheckpoint
WhenWen AUG-LIN0-1L-DENSE-MUONH-X600-HIGH-1-d512-600x-lr1.1
[Muon's first probe above the 21,149-step grid](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X600-HIGH-1-d512-600x-lr1.1) finished all its steps at 9:50 a.m. (ET) at 4.7719 on Paloma held-out text, ahead of [the grid's own best rate of 0.70](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-025-d512-600x-lr0.7) at 4.7817. Muon's best rate at the longest run length therefore lies outside the range the grid searched, as it does at [the 10,575-step budget](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X300-HIGH-1-d512-300x-lr1.1). [A second probe at rate 1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X600-HIGH-1P4-d512-600x-lr1.4) is nine-tenths through and decides whether the climb goes further.
ladderlr-sweepmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) cleared its hourly save of state at 9:22-9:23 a.m. (ET), holding the :22 slot it has used every hour since 2:19 p.m. (ET) on September 11, and stepped every eighteen to nineteen seconds for the twenty minutes after it. That makes fifty-three and a half hours of unbroken stepping since the September 10 restart, and every hourly save in that stretch has passed. A caveat about this page, revised from the last report: the step count and chart above trail the run again, by about sixty steps or twenty minutes, rather than the sixteen steps reported at 9:05 a.m. (ET). The lag is in the sampled feed W&B serves this page, not in the run itself, which reached step 93,785 at 9:42 a.m. (ET).
herocheckpoint
WhenWen AUG-LRC-1L-SGDH-X600-HIGH-2-d512-600x-lr1.7
The mixture-of-experts tree on a flat rate closed out its probes above the 21,149-step grid. [Rate 2.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LRC-1L-SGDH-X600-HIGH-3-d512-600x-lr2.7) finished at 9:36 a.m. (ET) at 4.4993 on Paloma held-out text and [rate 1.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LRC-1L-SGDH-X600-HIGH-2-d512-600x-lr1.7) at 9:43 a.m. (ET) at 4.3194, both behind [its 0.70 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LRC-1L-SGDH-025-d512-600x-lr0.7) at 4.2557 and [its rate-1.1 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LRC-1L-SGDH-X600-HIGH-1-d512-600x-lr1.1) at 4.2599. That tree settles on 0.70, the highest rate its grid offered, with 1.1 four thousandths behind — a flat stretch at the edge of the search rather than a curve still falling past it, which is what [the dense plain-gradient tree](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-3-d512-600x-lr2.7) shows at the same run length.
ladderlr-sweepmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
Correction to the caveat in the last report: this page no longer trails the run by fifty minutes. W&B began returning fresh sampled points again at 8:40 a.m. (ET), so the chart and the step count above are now about sixteen steps, or five minutes, behind the run. The run has stepped without a break for nearly fifty-three hours at eighteen and a half seconds a step, and its last save of state ran from 8:22 to 8:23 a.m. (ET); the next is due near 9:22 a.m. (ET).
herocheckpointmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run passed fifty-two hours of unbroken stepping at 8:14 a.m. (ET). It cleared its hourly save of state at 7:22-7:23 a.m. (ET) and entered the next one at 8:22 a.m. (ET), holding the :22 slot it has used every hour through the night. Steps come every eighteen to twenty seconds between saves and stretch to twenty-four across one. One caveat about this page: the chart and the step count above trail the run itself, because W&B has returned the same last sampled point since 7:40 a.m. (ET) — about 170 steps, or fifty minutes, behind where the run is now.
herocheckpointmilestone
WhenWen AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-4-d512-600x-lr4.3
[A fourth probe above the 21,149-step grid](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-4-d512-600x-lr4.3) opened at 8:25 a.m. (ET) at rate 4.3, twenty-three minutes after [the rate-2.7 probe closed in front at 4.8909 on Paloma held-out text](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-3-d512-600x-lr2.7). Rate 4.3 lost at [the 5,288-step](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X150-HIGH-4-d512-150x-lr4.3) and [10,575-step](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X300-HIGH-4-d512-300x-lr4.3) budgets, so this run decides whether the longest budget settles on 2.7 as well.
ladderlr-sweep
WhenWen AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-3-d512-600x-lr2.7
[The third probe above the 21,149-step grid](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-3-d512-600x-lr2.7) finished at 8:02 a.m. (ET) at 4.8909 on Paloma held-out text, ahead of [the second probe at rate 1.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-2-d512-600x-lr1.7) at 4.9002 and of [the grid's own best rate of 0.70](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-025-d512-600x-lr0.7) at 4.9845. The three steps up the rate have gained 0.0544, then 0.0299, then 0.0093, so the climb is flattening but has not turned back — at [the 10,575-step budget](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X300-HIGH-4-d512-300x-lr4.3) the next rate, 4.3, was the first outright loss. Rate 2.7 now leads at three run lengths in a row, and no rate-4.3 probe has opened at this one.
ladderlr-sweepmilestone
WhenWen AUG-LIN0-1L-DENSE-SGDH-X300-HIGH-4-d512-300x-lr4.3
[The fourth probe above the 10,575-step grid](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X300-HIGH-4-d512-300x-lr4.3) finished at 7:41 a.m. (ET) at 5.0069 on Paloma held-out text, behind [the third probe at rate 2.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X300-HIGH-3-d512-300x-lr2.7) at 4.9773, so the family without momentum settles on 2.7 at this run length too. That matches [the 5,288-step budget, which turned back at the same rate at 6:20 a.m. (ET)](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X150-HIGH-4-d512-150x-lr4.3), and leaves [only its 21,149-step probe at rate 2.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-3-d512-600x-lr2.7) still running.
ladderlr-sweepmilestone
held h100-mix25-20260912-d512
[The first rung of the new four-width scaling ladder](https://wandb.ai/marin-community/marin_moe/runs/h100-mix25-20260912-d512) finished all 3,930 of its steps at 7:23 a.m. (ET) and scored 3.8072 on Paloma held-out text — behind every one of [the eight mixtures the fifth data-mixture wave compared](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-dd6c1760c4691b33-seed0-from10pct-20260829), which span 3.7959 to 3.8025, and inside the 3.8000-to-3.8126 range that [the chosen mixture's own four random seeds](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed3-from10pct-20260829) already cover. Two settings separate it from those rungs: it trains from step 0 rather than from a stored checkpoint at a tenth of the run, and it moves to the second mixture stage at step 1,008 rather than 384, so the gap is not a clean seed repeat. [The other three widths](https://wandb.ai/marin-community/marin_moe/runs/h100-mix25-20260912-d1536) are still running on much longer budgets.
ladderdata-mixmilestone
WhenWen #9110 — first gate fails at d512 open
#9110 has recorded its verdict on the corrected way of routing tokens to experts: the narrower run returned 0.14 on the compute-saving measure the test uses, where 1.0 is break-even, so it fails the first gate and the two wider models it could have unlocked will not be built. The wider running model, moe-lgr-9110-d768-rmsgated, is near 6,000 of its 19,378 steps and will finish only to complete the two-width report. #9110 repeats that both comparisons are against baselines trained under a different recipe on different chips, so neither isolates the gating change. The chips are TPU, not the hero's.
experimentmoeroutingcorrection
WhenWen moe-lgr-9110-d512-rmsgated
[The narrower of the two runs testing a corrected way of routing tokens to experts](https://wandb.ai/marin-community/marin_moe/runs/moe-lgr-9110-d512-rmsgated) finished all 13,642 of its steps at 6:45 a.m. (ET), the first complete result [#9110](https://github.com/marin-community/marin/issues/9110) has produced after two scheduler preemptions. It scored 3.8425 on Paloma held-out text over 1.79 billion tokens, behind [the stored baseline of the same width](https://wandb.ai/marin-community/marin_moe/runs/moe_may_compute_opt_d512_ep1) at 3.5422 over 1.44 billion — worse on the larger token budget, and its training loss trails by the same 0.30. #9110 states that the stored baselines were trained under a different recipe on v4 chips against this run's v5p, which limits what the gap can be attributed to, and it will not decide the first gate until both widths are in: [the wider run](https://wandb.ai/marin-community/marin_moe/runs/moe-lgr-9110-d768-rmsgated) is near 5,100 of its 19,378 steps. The chips are TPU, not the hero's.
experimentmoeroutingmilestone
WhenWen #7856 LR sweep — 600x above-grid probes
The first results from rates above the grid at the longest run length are in, and the two plain-gradient trees answer differently. [The dense tree's rate-1.7 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-2-d512-600x-lr1.7) finished its 21,149 steps at 6:27 a.m. (ET) scoring 4.9002 on Paloma held-out text and [its rate-1.1 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-1-d512-600x-lr1.1) at 6:36 a.m. (ET) scoring 4.9301, both ahead of [the grid's own best rate of 0.70](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-025-d512-600x-lr0.7) at 4.9845 — the curve is still falling past the edge of the search, and [a rate-2.7 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-3-d512-600x-lr2.7) is three-quarters through. [The mixture-of-experts tree on a flat rate](https://wandb.ai/marin-community/marin_moe/runs/AUG-LRC-1L-SGDH-X600-HIGH-1-d512-600x-lr1.1) came back level instead: its rate-1.1 probe finished at 6:33 a.m. (ET) at 4.2599, four thousandths behind [its own 0.70 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LRC-1L-SGDH-025-d512-600x-lr0.7) at 4.2557, and its 1.7 and 2.7 probes trail by 0.11 and 0.29 at their last shared step, so that tree's best rate sits at the top of the range already tested.
ladderlr-sweepmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run passed fifty hours of unbroken stepping at 6:14 a.m. (ET) and cleared its hourly save of state at 6:22-6:23 a.m. (ET), in the same slot it has used every hour since 2:19 p.m. (ET) on September 11. The 5:41 a.m. (ET) scoring on held-out text did not push that save an hour later, as the last report expected it would, so the save cycle runs on its own clock rather than restarting after each scoring. Steps come every eighteen to nineteen seconds on either side of the save, and stretch to twenty-three across it.
herocheckpointmilestone
WhenWen AUG-LIN0-1L-DENSE-SGDH-X150-HIGH-4-d512-150x-lr4.3
[The fourth probe above the 5,288-step grid](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X150-HIGH-4-d512-150x-lr4.3) finished at 6:20 a.m. (ET) at 5.1417 on Paloma held-out text, 0.0394 behind [the third probe at rate 2.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X150-HIGH-3-d512-150x-lr2.7) at 5.1023. The family without momentum had gained 0.0561 and then 0.0023 on the two steps before it, so rate 4.3 is the first outright loss and the best rate at this budget is now pinned at 2.7. [The 2,115-step budget turned back the same way one rate lower](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X60-HIGH-3-d512-60x-lr2.7), which is what a search that is finding real optima rather than running out of grid should look like. [Rate 4.3 is still running at the 10,575-step budget](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X300-HIGH-4-d512-300x-lr4.3), and no fifth probe opened at the shorter one.
ladderlr-sweepmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
[The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) scored itself on held-out text at 5:41 a.m. (ET), one step that took about a minute against the usual eighteen seconds. At 2.2302 the score sits above the 2.2294 of three thousand steps earlier, which sat above 2.2285 — the first time in the run that two readings in a row have moved the wrong way. The two rises come to 0.0017 together, less than the single 0.0030 rise at step 83,999 that the run then recovered, and the score drifted a net 0.0016 the wrong way across the twelve thousand steps from 59,999 to 71,999 before resuming its fall, so the series has held flat this long before. The scoring also restarts the hourly save timer, as [the 2:19 p.m. (ET) scoring on September 11](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) did, so the next save is due near 6:41 a.m. (ET) rather than in the :21 slot it has held every hour since.
heroevalstraining-dynamics
WhenWen AUG-LIN0-1L-DENSE-SGDMH-X300-HIGH-2-d512-300x-lr1.7
[The Nesterov-momentum family's second probe above its 10,575-step grid](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-X300-HIGH-2-d512-300x-lr1.7) finished at 5:31 a.m. (ET) at 4.8805 on Paloma held-out text, behind [its own first probe at the lower rate](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-X300-HIGH-1-d512-300x-lr1.1) at 4.8658. Both families that use momentum have now turned back at this run length — [Muon reversed the same way](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X300-HIGH-2-d512-300x-lr1.7) twenty-eight minutes earlier — and neither has opened a third probe.
ladderlr-sweep
WhenWen moe-lgr-9110-d512-rmsgated / d768-rmsgated
[Both runs of the corrected way of routing tokens to experts](https://wandb.ai/marin-community/marin_moe/runs/moe-lgr-9110-d512-rmsgated) are training again, twelve hours after they stopped: the wider model logged its first new step at 5:18 a.m. (ET) and the narrower at 5:23 a.m. (ET), each picking up from the checkpoint saved just before the stop, so the narrower repeated about 570 steps and the wider about ten. [#9110](https://github.com/marin-community/marin/issues/9110) gives the cause the crashed state hid: a scheduler took their machines, and a second attempt was queued automatically, so neither run was lost. The narrower has since passed 8,000 of its 13,642 steps and scored 4.0415 on held-out text at step 7,999, while [the wider](https://wandb.ai/marin-community/marin_moe/runs/moe-lgr-9110-d768-rmsgated) is near 2,700 of 19,378. The chips are TPU, not the hero's.
experimentmoeroutingrestart
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run passed forty-nine hours of unbroken stepping at 4:14 a.m. (ET) and cleared its hourly save of state at 5:21-5:22 a.m. (ET), on the cycle it has held since 2:19 p.m. (ET). Four saves have come and gone since the last report of this cycle, each spread over one to three steps, and steps come every eighteen to nineteen seconds on either side of them.
herocheckpointmilestone
larrydial dynact-latent-in-double-d512-fa4fix
[The stalled attention-latent probe](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-in-double-d512-fa4fix) resumed at 4:26 a.m. (ET), ninety-five minutes after its last step, and closed all 6,000 steps at 5:04 a.m. (ET). A second break of twelve minutes at 4:44 a.m. (ET) was followed by a lasting rise in its training loss, and it finishes at 3.9312 on Paloma held-out text against the 3.5908 it had already reached 1,200 steps earlier, so the wave's largest arm returns no usable score.
ladderablationcrash
WhenWen AUG-LIN0-1L-DENSE-MUONH-X300-HIGH-2-d512-300x-lr1.7
[Muon's second probe above its 10,575-step grid](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X300-HIGH-2-d512-300x-lr1.7) finished at 5:03 a.m. (ET) at 4.8390 on Paloma held-out text, behind [its own first probe at the lower rate](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X300-HIGH-1-d512-300x-lr1.1) at 4.8219. Muon's climb above that grid therefore stops one step up, and no third probe opened at that run length.
ladderlr-sweep
WhenWen AUG-LIN0-1L-DENSE-SGDMH-X600-HIGH-*
[Nesterov momentum opened its first probe above the 21,149-step grid](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-X600-HIGH-1-d512-600x-lr1.1) at 4:28 a.m. (ET) and [a second at rate 1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-X600-HIGH-1P4-d512-600x-lr1.4) at 4:42 a.m. (ET), matching the pair Muon opened at 4:03 a.m. (ET). All three families that decay the rate to zero now have probes running above their longest grid.
ladderlr-sweepmilestone
WhenWen AUG-LIN0-1L-DENSE-SGDH-X150/X300-HIGH-4
The family without momentum opened a fourth probe above its grid at rate 4.3, at [the 5,288-step budget](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X150-HIGH-4-d512-150x-lr4.3) at 4:34 a.m. (ET) and [the 10,575-step budget](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X300-HIGH-4-d512-300x-lr4.3) at 4:42 a.m. (ET). [Its third probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X150-HIGH-3-d512-150x-lr2.7) had gained 0.0023 on Paloma at the shorter budget against about 0.056 for each of the two steps before it, so the search keeps climbing on a margin that has nearly closed.
ladderlr-sweep
held h100-mix25-20260912-d512/d768/d1024/d1536
[A four-width scaling ladder](https://wandb.ai/marin-community/marin_moe/runs/h100-mix25-20260912-d512) started at 4:16 a.m. (ET), training 0.6, 1.6, 4.0 and 11.5-billion-parameter models on one, two, four and eight H100 nodes — [the shapes added to the H100 ladder in August](https://github.com/marin-community/marin/pull/8680), which exist to [fit a power law across widths for the hero](https://github.com/marin-community/marin/issues/8435#issuecomment-5344533796). Every rung is tagged for data mixture `996f489106c7b922`, the one [the fitted mixture model](https://github.com/marin-community/marin/pull/8633) picks rather than the one that scored best; that mixture's [four repeats under different random seeds](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed3-from10pct-20260829) span 0.0125 on Paloma, about twice the 0.0066 separating the wave's eight distinct mixtures. No issue, pull request, or Discord message announces the cohort.
ladderdata-mixscaling-ladder
larrydial dynact-latent-in-double-d512-fa4fix
[The stalled attention-latent probe](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-in-double-d512-fa4fix) has logged no step for seventy-eight minutes, since 3,744 at 2:51 a.m. (ET), through three replacement processes at 2:53, 3:16 and 3:39 a.m. (ET). Its GPUs read and write memory throughout, 12 to 51 per cent of the time, so each attempt is doing work rather than sitting idle — the opposite of the hero's silent hang, and the shape of a restart replaying steps W&B will not record because they sit at or below 3,744. The current attempt has held twenty-eight minutes, longer than either of the two before it.
ladderablationcrash
WhenWen h100-d512-mixprior wave 5
The data-mixture wave is complete, and it ranks nothing. [The eighth and last distinct mixture](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-dd6c1760c4691b33-seed0-from10pct-20260829) finished its 3,929 steps at 4:05 a.m. (ET) at 3.798155 on Paloma held-out text, inside the range the other seven set; the eight run from 3.795922 to 3.802502, a spread of 0.0066. [A fourth seed of one mixture](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed3-from10pct-20260829) closed at 4:00 a.m. (ET) at 3.800688, and that one mixture's four seeds span 0.0125 — still nearly twice the spread across the eight different mixtures.
ladderdata-mixmilestone
WhenWen AUG-LIN0-1L-DENSE-MUONH-X600-HIGH-*
The Muon family opened [two probes above its grid at the longest run length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X600-HIGH-1-d512-600x-lr1.1) at 4:03 a.m. (ET), after that grid picked the highest rate on offer. Both were stepping within six minutes. [The second sits at rate 1.4](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X600-HIGH-1P4-d512-600x-lr1.4), between the 1.1 and 1.7 steps the shorter run lengths used, so the search is being filled in as well as widened; Nesterov momentum is now the only decay-to-zero family with no above-grid probe at this length.
ladderlr-sweep
larrydial dynact-latent-in-double-d512-fa4fix
[The stalled attention-latent probe](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-in-double-d512-fa4fix) took a second replacement process at 3:39 a.m. (ET) and has still logged no step since 3,744 at 2:51 a.m. (ET), fifty-four minutes earlier. Its three siblings have all stopped — [two with a full score](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-out-double-d512-fa4fix) and [one after 231 steps](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-out-half-d512-fa4fix) — so it is the only arm of the rebuilt wave still owed a result.
ladderablationcrash
WhenWen h100-d512-mixprior wave 5
[A seventh distinct data mixture](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-ba9dd4219b35ff93-seed0-from10pct-20260829) finished its 3,929 steps at 3:36 a.m. (ET), scoring 3.798866 on Paloma held-out text — again inside the range the earlier six covered, so the seven still span 0.0066 against the 0.0125 that [one mixture rerun under three random seeds](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed1-from10pct-20260829) spans on its own. Correction: [the rung reported at 3:30 a.m. (ET) as having recorded nothing](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-dd6c1760c4691b33-seed0-from10pct-20260829) had resumed seconds before that read and is now at step 3,528, the wave's last unfinished mixture.
ladderdata-mix
WhenWen h100-d512-mixprior wave 5
Two stalled data-mixture rungs are stepping again and a third has stopped. [The rung frozen at step 3,817 since 2:45 a.m. (ET)](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-ba9dd4219b35ff93-seed0-from10pct-20260829) resumed at 3:30 a.m. (ET), forty-one minutes after its replacement process started, and [the fourth-seed repeat](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed3-from10pct-20260829) has climbed to step 3,330 since its own replacement at 3:01 a.m. (ET); [a third rung](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-dd6c1760c4691b33-seed0-from10pct-20260829) logged its last step, 3,283, at 3:12 a.m. (ET), was replaced at 3:14 a.m. (ET), and has recorded nothing since.
ladderdata-mixcrash
WhenWen AUG-LRC-1L-SGDH-X600-HIGH-2-d512-600x-lr1.7
Correction: the learning-rate probe reported crashed at 2:41 a.m. (ET) is running again, not lost. [The rate-1.7 probe at the 21,149-step run length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LRC-1L-SGDH-X600-HIGH-2-d512-600x-lr1.7) took a replacement process at 3:07 a.m. (ET) and has resumed from the step 4,360 it had reached, passing 4,450 at 3:30 a.m. (ET).
ladderlr-sweep
larrydial dynact-latent-* d512 fa4fix
Two of the rebuilt attention-latent probes finished their full 6,000 steps, and the three scores rank by model size rather than by shape. [The arm that doubles what leaves the attention latent](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-out-double-d512-fa4fix) closed at 3:28 a.m. (ET) at 3.4690 on Paloma held-out text and [the arm that halves what enters it](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-in-half-d512-fa4fix) at 3:22 a.m. (ET) at 3.5336, on either side of [the unchanged baseline](https://wandb.ai/marin-community/marin_moe/runs/dynact-baseline-fsdp-d512) at 3.4972 — the same order as their parameter counts of 749M, 598M and 446M.
ladderablation
WhenWen #7856 LR sweep — 600x grids
The last of the three longest learning-rate grids closed at 3:22 a.m. (ET), and every family that decays the rate to zero settled on 0.70, the highest rate the grid offers — [Muon](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-025-d512-600x-lr0.7) at 4.7817 on Paloma, [Nesterov momentum](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-025-d512-600x-lr0.7) at 4.8228 and [plain gradient descent](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-025-d512-600x-lr0.7) at 4.9845, with [the Muon 0.45 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-024-d512-600x-lr0.45) reporting last at 4.8006. Only [the plain-gradient tree](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-1-d512-600x-lr1.1) has probes above that grid running so far, so two boundary extensions are still owed.
ladderlr-sweep
WhenWen h100-d512-mixprior-5f6e920ed52cd6d0-seed0-from10pct-20260829
[A sixth distinct data mixture](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-5f6e920ed52cd6d0-seed0-from10pct-20260829) finished its 3,929 steps at 3:21 a.m. (ET), scoring 3.798062 on Paloma held-out text — inside the range the first five already covered. The six mixtures now span 0.0066, still narrower than the 0.0125 that [a single mixture rerun under three random seeds](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed1-from10pct-20260829) spans on its own, so the wave still tells no mixture apart from a repeat of itself.
ladderdata-mix
larrydial dynact-latent-in-double / out-half d512 fa4fix
A third rebuilt attention-latent probe stopped: [the arm that doubles what enters the latent](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-in-double-d512-fa4fix) logged its last step, 3,744, at 2:51 a.m. (ET), took a replacement process at 3:16 a.m. (ET), and has recorded nothing since. W&B has also marked [the fourth arm](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-out-half-d512-fa4fix) finished at step 231, so it ended at 2:36 a.m. (ET) rather than stalling, and stopped short of its first evaluation at step 600.
ladderablationcrash
WhenWen h100-d512-mixprior-ba9dd4219b35ff93-seed0-from10pct-20260829
The data-mixture stalls came back within a minute of being declared over. [The rung reported stepping again at 2:45 a.m. (ET)](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-ba9dd4219b35ff93-seed0-from10pct-20260829) logged its last step, 3,817, twenty-four seconds after that read and has recorded nothing in the seventeen minutes since. [The fourth-seed repeat](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed3-from10pct-20260829) stopped at step 3,036 at 2:59 a.m. (ET), 893 steps short of the full run, and W&B briefly marked it complete before its process reported in again. Both keep a current heartbeat, which is the same shape as the stalls earlier tonight: the work continues and the step counter does not move.
ladderdata-mixcrash
larrydial dynact-latent-in-half-d512-fa4fix
The rebuilt image has cleared the fault for three of the four attention-shape probes. [The arm that halves what enters the attention latent](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-in-half-d512-fa4fix) is at step 4,083, [the arm that doubles what leaves it](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-out-double-d512-fa4fix) at 3,727 and [the arm that doubles what enters it](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-in-double-d512-fa4fix) at 3,744 — each past the step at which its own copy in the withdrawn wave died, and past that wave's best showing of 2,990. [The fourth arm](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-out-half-d512-fa4fix) has logged nothing since step 231 at 2:36 a.m. (ET), so the single failure stands on its own.
ladderarchitecture
WhenWen h100-d512-mixprior-df90e8b7f0951a11-seed0-from10pct-20260829
A sixth data-mixture rung finished, and a mixture repeated under three random seeds now spreads wider than the different mixtures do. [The rung](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-df90e8b7f0951a11-seed0-from10pct-20260829) logged its last step, 3,929, at 2:48 a.m. (ET) and scored 3.7990 on held-out text, fourth of the five distinct mixtures home so far; those five run from 3.7959 to 3.8025, a spread of 0.0066. Against that, [one mixture run three separate times under different seeds](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed1-from10pct-20260829) scored 3.8000, 3.8025 and 3.8126 — a spread of 0.0125, nearly twice the spread across five different mixtures. The wave therefore separates no mixture from a re-roll of the same one, and an ordered list of its finishers would report only which seed each drew.
ladderdata-mix
larrydial dynact-latent-out-half-d512-fa4fix
One of the four rebuilt attention-shape probes has already stopped again. [The arm that halves what leaves the attention latent](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-out-half-d512-fa4fix) went through three replacement processes between 2:18 and 2:30 a.m. (ET) — its GPU memory fell to the pre-load level and climbed back each time — reached step 231 and exited at 2:36 a.m. (ET), and has logged nothing in the ten minutes since. Its three siblings from the same 2:14 a.m. (ET) rebuild are at steps 2,195, 2,011 and 2,998 and stepping, so this is one arm failing, not the cohort. It is also [the arm that stopped first in the withdrawn wave](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-out-half-d512), which the rebuilt image has not fixed.
ladderarchitecturecrash
kaiyuew #7856 LR sweep
At the longest run length all three decay-to-zero families now point above their grid. [The Muon family's 0.70 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-025-d512-600x-lr0.7) finished its 21,149 steps at 2:26 a.m. (ET) scoring 4.7817 on held-out text, the best of the four it has home, and [the 0.45 rung still running](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-024-d512-600x-lr0.45) trails it by 0.010 at their last shared step — the same top-of-grid shape [the plain-gradient family](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-025-d512-600x-lr0.7) closed with at 4.9845. [The Nesterov family's 0.70 rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-025-d512-600x-lr0.7) leads [its own 0.45](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-024-d512-600x-lr0.45) by 0.022 with 1,400 steps left and is heading the same way. One run length down, [a 1.7 probe opened for the Nesterov family at 2:31 a.m. (ET)](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-X300-HIGH-2-d512-300x-lr1.7), the next step up after [its 1.1 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-X300-HIGH-1-d512-300x-lr1.1) beat the grid at 2:01 a.m. (ET).
ladderlrdensemilestone
kaiyuew h100-d512-mixprior wave 5
The wave-5 stalls have cleared. [The rung that had logged nothing for forty-nine minutes](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-dd6c1760c4691b33-seed0-from10pct-20260829) passed its stuck step 2,439 and reached 2,678 by 2:44 a.m. (ET), and at that moment all five unfinished rungs were recording steps — the first time since 8:40 p.m. (ET) that none was silent. Seconds later [one of the five stopped at step 3,817](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-ba9dd4219b35ff93-seed0-from10pct-20260829), 112 short of the 3,929 its siblings finish at, and W&B marked it `finished`; rungs in this cohort have returned from that state before, so whether it resumes is open.
ladderdata-mixrestart
WhenWen AUG-LRC-1L-SGDH-X600-HIGH-2-d512-600x-lr1.7
A learning-rate probe crashed a fifth of the way through its run. [The rate-1.7 probe at the 21,149-step run length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LRC-1L-SGDH-X600-HIGH-2-d512-600x-lr1.7) logged its last step, 4,360, at 2:41 a.m. (ET); W&B marked it crashed, its heartbeat has not returned in the twenty-three minutes since, and no replacement has opened. Its two siblings at that run length are still going — [rate 1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LRC-1L-SGDH-X600-HIGH-1-d512-600x-lr1.1) at step 11,669 and [rate 2.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LRC-1L-SGDH-X600-HIGH-3-d512-600x-lr2.7) at 3,608 — so the tree keeps its search above the grid, minus one rate. This tree is the mixture-of-experts model on a flat rate, not the dense models the other sweep entries follow.
ladderlr-sweepcrash
WhenWen AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-3-d512-600x-lr2.7
The plain-gradient family's climb above its grid has stopped at the 5,288-step run length. [Its probe at rate 2.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X150-HIGH-3-d512-150x-lr2.7) finished at 2:17 a.m. (ET) scoring 5.1023 on held-out text, ahead of [the 1.7 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X150-HIGH-2-d512-150x-lr1.7) at 5.1046 by two thousandths — the two steps before it were worth 0.058 and 0.056, so this is the top of the curve rather than another gain, and [the grid's own best rate of 0.70](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-015-d512-150x-lr0.7) now trails the optimum by 0.116. No fourth probe has opened at that run length. One minute later the same family [opened a 2.7 probe at the 21,149-step run length](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-3-d512-600x-lr2.7), where [its 1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-1-d512-600x-lr1.1) and [1.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-2-d512-600x-lr1.7) probes are both a quarter of the way through.
ladderlrdensemilestone
held h100-d512-mixprior wave 5
A fifth data-mixture rung finished its 3,929 steps, and the wave now has a measure of its own noise. [The rung that had been frozen since 12:54 a.m. (ET)](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-b45c8fcea03f37d8-seed0-from10pct-20260829) closed at 2:17 a.m. (ET) scoring 3.7974 on held-out text. The five weightings home so far span 3.7959 to 3.8025, a spread of 0.0066; [one weighting that was repeated under a second random seed](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-5c32d535fde96e01-seed1-from10pct-20260829) scored 3.8006 against [its own first run](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-5c32d535fde96e01-seed0-from10pct-20260829)'s 3.7969, a gap of 0.0038 from the seed alone. Five different mixtures therefore separate by less than twice what one mixture separates from itself, so nothing here has yet beaten seed noise. A second seed repeat is still running. Of the five rungs still going, [one has logged nothing since step 2,439 at 1:35 a.m. (ET)](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-dd6c1760c4691b33-seed0-from10pct-20260829) despite a replacement process at 1:36 a.m. (ET) — forty-nine minutes silent, the longest in the wave — and the other four are stepping.
ladderdata-mixmilestone
larrydial dynact-latent-* d512 fa4fix
The remaining three attention-shape probes came back, on the same rebuilt image as the first. [The arm that halves what enters the attention latent](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-in-half-d512-fa4fix), [the one that doubles what leaves it](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-out-double-d512-fa4fix) and [the one that halves what leaves it](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-out-half-d512-fa4fix) all restarted from step 0 between 2:13 and 2:14 a.m. (ET), each carrying the identical package change as [the arm that went first at 2:00 a.m. (ET)](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-in-double-d512-fa4fix): the fused attention kernel `flash-attn-4` moves from build b16 to b28 and the communication library steps back from 2.31.2 to 2.30.7. A whole cohort rebuilt together confirms this was an operator withdrawal, not four separate faults. Two corrections to earlier entries. This wave does not test expert design: all four arms keep the standard expert block and change the width of the attention latent — the narrow buffer each layer's attention reads from and writes to — while [the first wave](https://wandb.ai/marin-community/marin_moe/runs/dynact-dual-updown-d512) did the reverse. And the four are not matched on size: they carry 446M, 522M, 749M and 900M parameters against [the baseline](https://wandb.ai/marin-community/marin_moe/runs/dynact-baseline-fsdp-d512)'s 598M, because the latent width is itself what sets their parameter count. A win here would not separate a better shape from more parameters.
ladderarchitecturerestartcorrection
kaiyuew h100-d512-mixprior wave 5
Two of the four stalled data-mixture rungs are stepping again, and two are not. [The rung frozen since 12:54 a.m. (ET)](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-b45c8fcea03f37d8-seed0-from10pct-20260829) took a replacement process at 1:26 a.m. (ET) and reached step 3,713 by 2:04 a.m. (ET), and [a second](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-5f6e920ed52cd6d0-seed0-from10pct-20260829) is at 2,350. [A third](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-ba9dd4219b35ff93-seed0-from10pct-20260829), which had been stepping normally, took a replacement of its own at 1:51 a.m. (ET) and is at 2,953. The two that have not recovered are [one silent since step 3,335 at 1:23 a.m. (ET)](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-df90e8b7f0951a11-seed0-from10pct-20260829), which drew a fresh replacement at 1:58 a.m. (ET), and [one silent since step 2,439 at 1:35 a.m. (ET)](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-dd6c1760c4691b33-seed0-from10pct-20260829). Four of the ten rungs have finished their 3,929 steps.
ladderdata-mixrestart
kaiyuew #7856 LR sweep
The flat-rate arm's longest set is closed, and its winner sits inside the grid. [The last of the five rates, 0.45](https://wandb.ai/marin-community/marin_moe/runs/AUG-LRC-1L-DENSE-SGDMH-024-d512-600x-lr0.45), finished its 21,149 steps at 2:01 a.m. (ET) scoring 4.9514 on held-out text, behind [0.2](https://wandb.ai/marin-community/marin_moe/runs/AUG-LRC-1L-DENSE-SGDMH-022-d512-600x-lr0.2) at 4.9276 and [0.32](https://wandb.ai/marin-community/marin_moe/runs/AUG-LRC-1L-DENSE-SGDMH-023-d512-600x-lr0.32) at 4.9394. The five scores form one clean curve with its low point one rate in from each edge, so this arm needs no probe above its range.
ladderlr-sweep
kaiyuew #7856 LR sweep
The climb above the grid is no longer one family's story — all three that decay the rate to zero now do it. At the 10,575-step run length [the Nesterov-momentum family's probe at rate 1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-X300-HIGH-1-d512-300x-lr1.1) finished at 2:01 a.m. (ET) scoring 4.8658 on held-out text against [its grid's best rate of 0.70](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-020-d512-300x-lr0.7) at 4.8797, and [the Muon family's probe at the same rate](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X300-HIGH-1-d512-300x-lr1.1) had already done the same at 1:45 a.m. (ET), 4.8219 against [0.70](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-020-d512-300x-lr0.7)'s 4.8430. Both gains are smaller than [the plain-gradient family's 0.053 two hours earlier](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X300-HIGH-1-d512-300x-lr1.1), and [the Muon family's next probe at 1.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X300-HIGH-2-d512-300x-lr1.7) has opened without logging a step.
ladderlr-sweep
larrydial dynact expert-design test
All four second-wave expert-design probes stopped within twenty-nine seconds of each other, and one has relaunched on a newer attention kernel. [The two that were still healthy](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-out-double-d512) logged their last steps at 1:58 a.m. (ET) and none of the four has reported since — a whole cohort going quiet at once is the signature of an operator withdrawal, not a fault. At 2:00 a.m. (ET) [the arm that doubles what enters the attention latent restarted from step 0 under a new name](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-in-double-d512-fa4fix) with the same model configuration; the only substantive change in its package list is the fused attention kernel `flash-attn-4`, which moves from build b16 to b28, alongside a step back in the NCCL communication library from 2.31.2 to 2.30.7.
laddermoerestart
kaiyuew h100-d512-mixprior wave 5
Two of the stalled data-mixture rungs are recording steps again. [The fourth-seed repeat](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed3-from10pct-20260829), frozen at step 1,345 since 11:04 p.m. (ET), passed that step at about 1:38 a.m. (ET) and reached 1,536 by 1:47 a.m. (ET) — its 12:49 a.m. (ET) replacement process spent roughly fifty minutes redoing work W&B declines to record twice. [A second rung](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-ba9dd4219b35ff93-seed0-from10pct-20260829) is stepping again at 2,830. Four others are frozen with a replacement attached and nothing logged since: [one](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-b45c8fcea03f37d8-seed0-from10pct-20260829) from 12:54 a.m. (ET), [one](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-df90e8b7f0951a11-seed0-from10pct-20260829) from 1:23 a.m. (ET), [one](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-5f6e920ed52cd6d0-seed0-from10pct-20260829) from 1:24 a.m. (ET) and [one](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-dd6c1760c4691b33-seed0-from10pct-20260829) from 1:35 a.m. (ET).
ladderdata-mixrestart
kaiyuew h100-d512-mixprior-2d02e594203d6b7c-seed0-from10pct-20260829
Correction, and a fourth rung home. [A fourth data-mixture rung](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-2d02e594203d6b7c-seed0-from10pct-20260829) logged its last step, 3,929, at 1:45 a.m. (ET) and scored 3.7981 on Paloma, the held-out text benchmark this ladder selects on. The three scores given in the 1:14 a.m. (ET) entry — 3.3032, 3.3045 and 3.3062 — are not Paloma scores; the three rungs read [3.7959](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-8d5f0aed232ce1b7-seed0-from10pct-20260829), [3.8006](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-5c32d535fde96e01-seed1-from10pct-20260829) and [3.8025](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-ee3799698c71ea8f-seed0-from10pct-20260829), which leaves their order unchanged. So the four finished weightings sit within seven thousandths of each other, not three — still too narrow to separate the mixtures from run-to-run noise.
ladderdata-mixmilestonecorrection
larrydial dynact-latent-* d512
Two of the four new expert-design probes stopped within twenty minutes of starting. [The one that halves what leaves the attention latent](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-out-half-d512) logged nothing after step 1,247 at 1:38 a.m. (ET) and took a replacement process at 1:45 a.m. (ET); [the one that doubles what enters it](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-in-double-d512) stopped at step 609 at 1:31 a.m. (ET) and has logged nothing under its 1:39 a.m. (ET) replacement. W&B briefly marked the first `finished`, which here means one process exited, not that the 6,000-step run is done. [The other two](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-in-half-d512) are past step 1,950 and still stepping.
ladderarchitecturecrash
WhenWen AUG-LIN0-1L-DENSE-SGDH-X300-HIGH-1-d512-300x-lr1.1
The one sweep family whose best rate keeps rising gained another step. [A probe at rate 1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X300-HIGH-1-d512-300x-lr1.1), opened above the five-rate grid at the 10,575-step run length, finished at 1:39 a.m. (ET) scoring 5.0256 on held-out text against [the grid's best rate of 0.70](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-020-d512-300x-lr0.7) at 5.0786. The 0.053 gain matches the two this family won at the 5,288-step length, where 0.70, 1.1 and 1.7 scored 5.2182, 5.1607 and 5.1046 in turn. The search here stays open: [1.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X300-HIGH-2-d512-300x-lr1.7) and [2.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X300-HIGH-3-d512-300x-lr2.7) opened at 12:44 a.m. (ET) and are near step 3,000.
ladderlrdensemilestone
held h100-d512-mixprior wave 5
Five of the ten data-mixture rungs are now recording no steps. [One that was still stepping minutes earlier](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-df90e8b7f0951a11-seed0-from10pct-20260829) stopped at 1:23 a.m. (ET) and [another](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-5f6e920ed52cd6d0-seed0-from10pct-20260829) at 1:24 a.m. (ET); replacement processes attached to both and to [a third](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-b45c8fcea03f37d8-seed0-from10pct-20260829) between 1:25 and 1:26 a.m. (ET) — three inside one minute, which points to a single shared trigger rather than three separate faults. They join [the fourth-seed repeat](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed3-from10pct-20260829), silent since 11:04 p.m. (ET), and [one more](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-ba9dd4219b35ff93-seed0-from10pct-20260829), silent since 12:40 a.m. (ET). None is wedged: the three new processes read no GPU traffic, a few per cent, and full traffic respectively, which is the spread of a checkpoint reload caught at three different moments.
ladderdata-mixcrash
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run passed forty-five hours of unbroken stepping at 1:14 a.m. (ET) and cleared its hourly save of state at 1:22 a.m. (ET), on the cycle it has held since 2:19 p.m. (ET). The save spilled into one step and added about four seconds — the lightest of the night, against ten to twenty seconds spread over three or four steps on the earlier ones. Steps come every nineteen seconds on either side of it.
herocheckpointmilestone
WhenWen AUG-LRC-1L-DENSE-SGDMH-021 … 025
The first run length to settle without needing a wider range. Four of the five rates in the longest set of the arm that holds the learning rate flat, instead of decaying it to zero, finished their 21,149 steps at 1:19 a.m. (ET), and [0.2](https://wandb.ai/marin-community/marin_moe/runs/AUG-LRC-1L-DENSE-SGDMH-022-d512-600x-lr0.2) won on held-out text at 4.9276 against [0.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LRC-1L-DENSE-SGDMH-021-d512-600x-lr0.1)'s 4.9757 and [0.32](https://wandb.ai/marin-community/marin_moe/runs/AUG-LRC-1L-DENSE-SGDMH-023-d512-600x-lr0.32)'s 4.9394. The winner sits one rate inside the grid rather than on its edge, so no `-HIGH` probe follows — unlike the decaying arm, which has spawned extension runs all night. [The fifth rate, 0.45](https://wandb.ai/marin-community/marin_moe/runs/AUG-LRC-1L-DENSE-SGDMH-024-d512-600x-lr0.45), is still stepping and is bracketed by two rates that both scored worse.
ladderlrdensemilestone
larrydial dynact-latent-* d512
A second wave of four expert-design probes started at 1:19 a.m. (ET), and it supplies the size-matched control the first wave lacked. All four keep the standard expert and resize the attention latent instead — [halving](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-in-half-d512) or [doubling](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-in-double-d512) what enters it, [halving](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-out-half-d512) or [doubling](https://wandb.ai/marin-community/marin_moe/runs/dynact-latent-out-double-d512) what leaves it — which lands two of them at 900.4M and 749.4M parameters, within a million of [the dual up-down arm](https://wandb.ai/marin-community/marin_moe/runs/dynact-dual-updown-d512) and [the coupled-pair arm](https://wandb.ai/marin-community/marin_moe/runs/dynact-coupled-d512-v3) they can now be read against. [The 598M baseline](https://wandb.ai/marin-community/marin_moe/runs/dynact-baseline-fsdp-d512) scored 3.4972 on held-out text; all four are under 400 of their 6,000 steps, too early to rank.
ladderarchitecture
held h100-d512-mixprior-ee3799698c71ea8f-seed0-from10pct-20260829
A third data-mixture rung ran its full budget. [It](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-ee3799698c71ea8f-seed0-from10pct-20260829) logged step 3,929 of 3,930 at 1:14 a.m. (ET) and scored 3.3045 on held-out text, just behind [the first to finish](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-8d5f0aed232ce1b7-seed0-from10pct-20260829) at 3.3032, while [the seed-1 draw](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-5c32d535fde96e01-seed1-from10pct-20260829) closed at 1:09 a.m. (ET) on 3.3062. Three different weightings inside three thousandths of a nat is a narrow spread — too narrow, so far, to separate the mixtures from run-to-run noise.
ladderdata-mixmilestone
kaiyuew h100-d512-mixprior-8d5f0aed232ce1b7-seed0-from10pct-20260829
The fifth wave's first data-mixture rung reached the end of its run. [One of the eight fresh weightings](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-8d5f0aed232ce1b7-seed0-from10pct-20260829) logged its final step, 3,929 of 3,930, at 1:08 a.m. (ET) and scored 3.7959 on held-out text, having recorded every step since it picked up at 393 at 10:16 p.m. (ET) with no gap longer than two minutes; [the second-seed repeat](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-5c32d535fde96e01-seed1-from10pct-20260829) is one step behind it. Three of the ten are recording no steps — [the fourth-seed repeat](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed3-from10pct-20260829) since 11:04 p.m. (ET), [one rung](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-ba9dd4219b35ff93-seed0-from10pct-20260829) since 12:40 a.m. (ET) and [another](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-b45c8fcea03f37d8-seed0-from10pct-20260829) since 12:54 a.m. (ET) — each under a process started after its own last recorded step.
ladderdata-mixmilestone
kaiyuew h100-d512-mixprior-5f6e920ed52cd6d0-seed0-from10pct-20260829
[The fresh weighting](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-5f6e920ed52cd6d0-seed0-from10pct-20260829) started recording steps again at 12:55 a.m. (ET), one hour and forty-four minutes after it stopped. Its first new step, 1,514, follows its last recorded one directly, so nothing had to be written twice — but it came 55 minutes after the process that wrote it started at 12:00 a.m. (ET), which is the repeated work W&B declines to record a second time. [The fourth-seed repeat](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed3-from10pct-20260829) is still stopped at step 1,345 from 11:04 p.m. (ET), its newest process started at 12:49 a.m. (ET), and [a third rung](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-ba9dd4219b35ff93-seed0-from10pct-20260829) has recorded nothing since 12:40 a.m. (ET) under a process started three minutes later. Both move data across their GPUs on the same profile as the rungs still stepping. None of it touches the hero.
ladderdata-mixcrash
WhenWen AUG-LIN0-1L-DENSE-MUONH-X60-HIGH-2-d512-60x-lr1.7
The Muon family's second run length is settled, and its optimum is one step above the range the sweep first tried. At 2,115 steps [the 1.7 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X60-HIGH-2-d512-60x-lr1.7) finished at 12:54 a.m. (ET) scoring 5.0722 on held-out text — ahead of the range's best rate of 0.70 at 5.0802, but behind [the 1.1 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X60-HIGH-1-d512-60x-lr1.1)'s 5.0622 — so 1.1 stands and the search at that run length ends. Both of this family's closed run lengths settle at 1.1, while the family without momentum kept climbing with length, 1.1 then 1.7.
ladderlrdensemilestone
kaiyuew h100-d512-mixprior-996f489106c7b922-seed3-from10pct-20260829
Correction: [the fourth-seed repeat](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed3-from10pct-20260829) is restarting in a loop, not sitting still. Its W&B process-start time moved from 12:02 to 12:49 a.m. (ET) between two reads eight minutes apart, and each reading is later than the run's own last step at 11:04 p.m. (ET) — a time only a replacement process writes — so the earlier entry denying any restart was wrong. [The fresh weighting](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-5f6e920ed52cd6d0-seed0-from10pct-20260829) reads the same way from 12:00 a.m. (ET), and neither has recorded a step in over ninety minutes.
ladderdata-mixcrash
kaiyuew h100-d512-mixprior-dd6c1760c4691b33-seed0-from10pct-20260829
The stalls have reached two more of the wave's ten rungs, and one of them cleared. [One rung](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-dd6c1760c4691b33-seed0-from10pct-20260829) resumed at 12:46 a.m. (ET) after eleven minutes without a step — its third gap tonight, after one of sixty-six minutes and one of nine — while [a fourth](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-ba9dd4219b35ff93-seed0-from10pct-20260829) has logged nothing since 12:40 a.m. (ET). None of it touches the hero.
ladderdata-mixrestart
WhenWen AUG-LIN0-1L-DENSE-SGDH-X600-HIGH-1-d512-600x-lr1.1
The sweep stopped waiting for one probe to close before opening the next. Six higher-rate probes opened between 12:43 and 12:45 a.m. (ET) across the two families without momentum — among them a second and a third step above the range at the 10,575-step run length while [the first step up there](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X300-HIGH-1-d512-300x-lr1.1) is still running, and the first two steps up at 21,149 steps, the longest run length the sweep tests.
ladderlrdensemilestone
WhenWen AUG-LIN0-1L-DENSE-SGDH-X150-HIGH-3-d512-150x-lr2.7
The third probe the 5,288-step run length was due opened at 12:37 a.m. (ET) at rate 2.7, nineteen minutes after [the 1.7 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X150-HIGH-2-d512-150x-lr1.7) closed ahead of the grid. At the 2,115-step run length the same third step came back worse, and that is what closed the search there.
ladderlrdense
WhenWen AUG-LIN0-1L-DENSE-SGDMH-X150-HIGH-1-d512-150x-lr1.1
The other two families probed the same 5,288-step run length and both came back behind their grids, so each settles at the grid's top rate of 0.70. [The momentum family's probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-X150-HIGH-1-d512-150x-lr1.1) finished at 12:33 a.m. (ET) scoring 4.9795 on held-out text against the grid's 4.9785, and [the Muon family's](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X150-HIGH-1-d512-150x-lr1.1) at 12:24 a.m. (ET) scored 4.9195 against 4.9142 — it had trailed at step 2,000 and never caught up, unlike its shorter sibling, which did.
ladderlrdensemilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run passed forty-four hours of unbroken stepping at 12:14 a.m. (ET) and cleared its hourly save of state at 12:21 a.m. (ET), on the cycle it has held since 2:19 p.m. (ET). Two steps in that window took 22 to 23 seconds against the run's 18.5-second median, matching the cheapest of the saves before it. W&B still shows the run training, at step 91,969 at 12:23 a.m. (ET).
herocheckpointmilestone
WhenWen h100-d512-mixprior wave 5
Correction: the two silent data-mixture rungs have not been restarted. [The fourth-seed repeat](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed3-from10pct-20260829) has logged nothing since step 1,345 at 11:04 p.m. (ET) and [the fresh weighting](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-5f6e920ed52cd6d0-seed0-from10pct-20260829) nothing since 1,513 at 11:11 p.m. (ET) — eighty and seventy-three minutes. Allocated memory has held between 65.3 and 68.5 GB on all eight GPUs of each, never dropping to the pre-load level a new process would show, so the four replacements the last read reported did not happen. Both are working — their GPUs move data on the same profile as the eight rungs still stepping — but neither is recording steps, and the reason is unknown. None of it touches the hero.
ladderdata-mixh100
WhenWen AUG-LIN0-1L-DENSE-SGDH-X150-HIGH-2-d512-150x-lr1.7
The family without momentum went two steps above one of its grids and won again. At 5,288 steps the 1.7 probe finished at 12:18 a.m. (ET) scoring 5.1046 on held-out text, ahead of [the 1.1 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X150-HIGH-1-d512-150x-lr1.1)'s 5.1607 and of the grid's best rate of 0.70 at 5.2182 — each step up worth about 0.056, with no sign of flattening. Its two shorter budgets each closed when a probe came back worse, so this one is due a third probe at 2.7; none had opened eight minutes after the result.
ladderlrdensemilestone
WhenWen AUG-LIN0-1L-DENSE-MUONH-X30/X60-HIGH-2
The Muon family's shortest budget is settled, and the next one up stepped higher without waiting for it. At 1,058 steps [the 1.7 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X30-HIGH-2-d512-30x-lr1.7) finished at 11:51 p.m. (ET) scoring 5.2653 on held-out text — worse than [1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X30-HIGH-1-d512-30x-lr1.1)'s 5.2601 but still ahead of the grid's best rate of 0.70 at 5.2786 — so 1.1 stands as the best rate there and the search at that budget ends. Four minutes before that result landed, a 1.7 probe opened at the 2,115-step budget at 11:47 p.m. (ET), twenty-five minutes after [the 1.1 probe there](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X60-HIGH-1-d512-60x-lr1.1) closed at 5.0622 against its grid's 5.0802. So each budget escalates on its own result, not on the shorter budget's outcome.
ladderlrdensemilestone
WhenWen h100-d512-mixprior wave 5
Two of the fifth wave's ten data-mixture rungs have recorded no progress for an hour. [The fourth-seed repeat](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed3-from10pct-20260829) logged its last step, 1,345 of 3,930, at 11:04 p.m. (ET), and [one of the fresh weightings](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-5f6e920ed52cd6d0-seed0-from10pct-20260829) its last, 1,513, at 11:11 p.m. (ET). Each has had its process replaced four times since — memory falling to the pre-load level and climbing back at 11:07, 11:14, 11:40 p.m. and 12:02 a.m. (ET) for the first, and 11:13, 11:22, 11:51 p.m. and 12:00 a.m. (ET) for the second — with the healthy training profile in between, so both are repeating work W&B will not record until they pass their last logged step. The other eight rungs of the wave have run without a replacement and are stepping normally. None of it touches the hero.
ladderdata-mixh100
The run passed forty-three hours of unbroken stepping at 11:14 p.m. (ET) and cleared its hourly save of state at 11:20–11:23 p.m. (ET), on the cycle it has held since 2:19 p.m. (ET). Four steps in that window took 22 to 24 seconds against the run's 18.5-second median, about nineteen seconds of extra wall clock and the same cost as every save before it. W&B still shows the run training, at step 91,840 at 11:44 p.m. (ET), and the single 21.4 percent reading of compute utilisation inside the save window sits against a 23.3 percent median on either side.
The Muon family's two longest higher-rate probes are behind the grids they extend at the first point where both have been scored. At step 2,000 the 5,288-step probe at rate 1.1 reads 5.2642 on held-out text against 5.1886 for the grid's best rate of 0.70, and [the 10,575-step probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X300-HIGH-1-d512-300x-lr1.1) reads 5.2625 against 5.2106. That order need not hold: [the 2,115-step probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X60-HIGH-1-d512-60x-lr1.1) also trailed at its step-1,000 eval, 5.4026 against 5.3438, and finished ahead. [The new 1,058-step probe at rate 1.7](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X30-HIGH-2-d512-30x-lr1.7) has not reached its first eval, and the 2,115-step budget has drawn no probe above 1.1 in the twenty-one minutes since its own closed.
WhenWen AUG-LIN0-1L-DENSE-MUONH-X30/X60-HIGH
The Muon family's first two probes above its own grids both won, and the sweep answered with a second step up. At 1,058 steps [rate 1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X30-HIGH-1-d512-30x-lr1.1) scored 5.2601 on held-out text against the grid's best of 5.2786 at rate 0.70, and at 2,115 steps [1.1](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-MUONH-X60-HIGH-1-d512-60x-lr1.1) scored 5.0622 against 5.0802 — both by about 0.018. Nineteen minutes after the shorter one closed at 11:04 p.m. (ET), a 1.7 probe opened at that budget at 11:23 p.m. (ET), the first time this family has gone two steps past one of its grids. The 2,115-step probe closed at 11:22 p.m. (ET) and has drawn no follow-up yet.
ladderlrdensemilestone
WhenWen AUG-LIN0-1L-DENSE-SGDH-X60-HIGH-3-d512-60x-lr2.7
In the family without momentum the 2,115-step budget is settled. Its third probe, at rate 2.7, finished at 11:22 p.m. (ET) scoring 5.4091 on held-out text, worse than [the 1.7 probe](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X60-HIGH-2-d512-60x-lr1.7)'s 5.3037, so 1.7 stands as the best rate there and the search at that budget ends. Both of this family's short budgets are now closed — 1.1 at 1,058 steps, 1.7 at 2,115 — so the best rate rises with run length. Its 5,288- and 10,575-step probes are still running.
ladderlrdensemilestone
WhenWen AUG-LIN0-1L-DENSE-SGDMH-X150/X300-HIGH-1
The tree the last pass skipped drew its probes sixteen minutes later. The 5,288-step rung above rate 0.70 opened at 10:56 p.m. (ET) and [the 10,575-step rung](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDMH-X300-HIGH-1-d512-300x-lr1.1) at 11:02 p.m. (ET), both at rate 1.1 — the first two probes above a grid this tree has ever drawn, and they close the gap the 10:30 p.m. (ET) pass left. Their grids had closed at 5:18 and 8:30 p.m. (ET), so the two waits were five hours thirty-eight minutes and two hours thirty-two minutes, inside the range the other trees have shown. The tree's 21,150-step grid is still running, so it has no further grid to extend yet.
ladderlrdensemilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run passed forty-two hours of unbroken stepping at 10:14 p.m. (ET) and cleared its hourly save of state at 10:20–10:21 p.m. (ET), on the cycle it has held since 2:19 p.m. (ET). The save cost two slow steps, the same few minutes every save before it has cost, and speed and loss read the same on either side. W&B still shows the run training, at step 91,653 at 10:45 p.m. (ET).
herocheckpointmilestone
WhenWen AUG-*-HIGH-* boundary extensions
The higher-rate tests the sweep owed all opened at once. After one hour and forty-seven minutes with no new run of any kind, eight opened between 10:30 and 10:41 p.m. (ET) across three of the sweep's four trees — among them the 10,575-step rung above 0.70 that the Muon tree had owed since 8:32 p.m. (ET) and the one [the tree without momentum](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X300-HIGH-1-d512-300x-lr1.1) had owed since 7:49 p.m. (ET). The Muon tree drew four in the pass, one at each budget it has finished and every one at rate 1.1, though its 1,058-step and 2,115-step grids closed at 3:10 and 3:56 p.m. (ET). So the automation releases these in sweeps rather than as each grid closes, and a grid can wait seven hours for its probe.
ladderlrdensemilestone
WhenWen AUG-LIN0-1L-DENSE-SGDMH-020-d512-300x-lr0.7
The tree that adds momentum was the one tree the pass skipped. Its 5,288-step and 10,575-step grids both scored best at the top rate 0.70 — 4.9785 and 4.8797 on held-out text, ahead of the rate below by 0.046 and 0.056 — which is the condition that opens a rung above the grid in every other tree. Its two shortest grids picked rates inside the grid, 0.32 and 0.45, so momentum moves the best rate down at short budgets and out to the edge at long ones. No probe above 0.70 has ever opened on this tree.
ladderlrdense
WhenWen AUG-LIN0-1L-DENSE-SGDH-X60-HIGH-3-d512-60x-lr2.7
In the tree without momentum the best rate climbs with run length, and the search has closed at one budget. At 1,058 steps [1.1 beat the grid's top rate](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X30-HIGH-1-d512-30x-lr1.1), 5.5797 against 5.6330 on held-out text, and [1.7 came back worse](https://wandb.ai/marin-community/marin_moe/runs/AUG-LIN0-1L-DENSE-SGDH-X30-HIGH-2-d512-30x-lr1.7) at 5.7033, so that budget is bracketed and drew no third probe. At 2,115 steps 1.7 is still winning, 5.3037 against 5.3529, so a fourth rate of 2.7 opened at 10:35 p.m. (ET); the 5,288-step budget moved up to 1.7 and the 10,575-step budget opened its first probe at 1.1, both at 10:41 p.m. (ET).
ladderlrdense
WhenWen AUG-LRC-1L-SGDH-X600-HIGH-1-d512-600x-lr1.1
The sweep opened its first probe above a grid at the longest run it tests, 21,150 steps. On the tree that holds the rate constant and uses experts, that grid closed at its top rate 0.70 at 12:40 p.m. (ET) scoring 4.2557 on held-out text, and the 1.1 probe opened at 10:30 p.m. (ET), nine hours and fifty minutes later. The same tree's two earlier probes of 1.1 both came back worse than 0.70 — 4.5454 against 4.4685 at 5,288 steps and 4.3604 against 4.3341 at 10,575 — so this one is likelier to confirm the grid than to move it. No budget beyond this one exists, so it is the last probe the tree can draw.
ladderlrmoemilestone
larrydial dynact-dual-updown-d512
The expert-design test has its answer, and the answer is size. The arm set to `DUAL_UP_DOWN` finished its 6,000 steps at 10:12 p.m. (ET) at 3.4981 on held-out text, 0.0009 behind [the baseline](https://wandb.ai/marin-community/marin_moe/runs/dynact-baseline-fsdp-d512)'s 3.4972 while carrying 899.6 million parameters against the baseline's 597.6 million. [The arm that doubles expert width](https://wandb.ai/marin-community/marin_moe/runs/dynact-double-neurons-d512) is the only one ahead, at 3.4742, and it carries 1.05 billion. So the two arms that finished split on parameter count, not on design.
laddermoeh100milestone
larrydial dynact-coupled-d512-v3
The smallest arm of that test is still stopped at step 2,369 of 6,000, with no replacement twenty-seven minutes after it went quiet at 9:59 p.m. (ET) — its sixth stop across three run names. It holds 748.6 million parameters, the closest of the three to the baseline, and its last score on held-out text is from step 1,799, so the cheapest design in the test has no result at the length the others report.
laddermoeh100
WhenWen h100-d512-mixprior wave 5
A fifth wave of data-mixture tests opened at 10:14–10:15 p.m. (ET) with ten rungs, the smallest wave the series has run after forty, fifteen and two of twenty. Eight carry weightings the ladder has not tried; the other two repeat earlier weightings at a fresh random seed — a fourth seed for [the mixture whose three seeds spread 0.013](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed3-from10pct-20260829), the widest spread measured so far, and a second for [one of the fourth wave's better weightings](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-5c32d535fde96e01-seed1-from10pct-20260829) at 3.7969. Every one runs 3,930 steps on a single eight-GPU H100 node, as before.
ladderdata-mixh100
WhenWen h100-d512-mixprior wave 4
All fifteen rungs of the fourth wave finished their 3,929 steps by 1:12 p.m. (ET), scoring 3.7924 to 3.8126 on held-out text. The twelve fresh weightings span only 3.7924 to 3.8017, narrower than the 0.013 that [one weighting's own three seeds](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed2-from10pct-20260829) cover, so the wave again separates no weighting from another. Two weightings tie for its best score, at 3.7924.
ladderdata-mixh100milestone
larrydial dynact-coupled-d512-v3
The arm that pairs experts stopped a sixth time, at step 2,369 of 6,000 at 9:59 p.m. (ET). It ran further than any earlier attempt — the best before was 1,439 — and its GPUs read the healthy training profile in the last sample before the stop, with none of the idle-but-pegged pattern that marks a hang. W&B marks the run `finished` and its heartbeat has not moved since 9:59 p.m. (ET); no replacement has opened in the ten minutes since.
laddermoeh100
larrydial dynact-dual-updown-d512
The arm set to `DUAL_UP_DOWN` is minutes from finishing and sits level with the baseline rather than ahead of it. At the nine steps the two share it reads within 0.004 of [the baseline](https://wandb.ai/marin-community/marin_moe/runs/dynact-baseline-fsdp-d512) on held-out text, the gap changing sign from one reading to the next — 3.5503 against 3.5493 at step 5,399. Earlier reads on this page said the two arms under test keep the baseline's size; the run configs say otherwise — the baseline holds 597.6 million parameters, this arm 899.6 million and [the paired-expert arm](https://wandb.ai/marin-community/marin_moe/runs/dynact-coupled-d512-v3) 748.6 million, against 1.05 billion for [the arm that doubles expert width](https://wandb.ai/marin-community/marin_moe/runs/dynact-double-neurons-d512). No arm in the test is size-matched to the baseline.
laddermoeh100milestone
larrydial dynact-double-neurons-d512
The first arm of the expert-design test finished. The arm that simply doubles the width of every expert ran all 6,000 steps without a break and stopped at 9:30 p.m. (ET), scoring 3.474 on held-out text against [the baseline](https://wandb.ai/marin-community/marin_moe/runs/dynact-baseline-fsdp-d512)'s 3.497 — ahead at all ten readings the two share, by 0.010 at step 599 widening to 0.023 at the end. It also carries 1.05 billion parameters against the baseline's 0.60 billion, so it measures what extra width buys rather than settling the comparison the test was built for. None of it touches the hero.
laddermoeh100milestone
larrydial dynact-dual-updown-d512
The arm set to `DUAL_UP_DOWN` is past the step where it stopped. It restored an early checkpoint and trained for forty minutes without logging a step — W&B refuses a step at or below the last one it recorded, so the replay left no trace — then logged step 3,289 at 9:39 p.m. (ET), a minute from what the last read predicted, and has reached 3,873 since; the stop did not repeat at 3,288. At the matched step 3,599 it reads 3.730 on held-out text against [the baseline](https://wandb.ai/marin-community/marin_moe/runs/dynact-baseline-fsdp-d512)'s 3.731, so the 50 percent more parameters it carries have bought nothing measurable.
laddermoeh100
larrydial dynact-coupled-d512-v3
The arm that pairs experts stopped a fifth time, at step 745 of 6,000 at 9:25 p.m. (ET). A new process attached to the same name at 9:26 p.m. (ET), fell to the pre-load memory level, restored an earlier checkpoint and replayed for ten minutes before logging step 746 at 9:39 p.m. (ET); its GPUs read the healthy profile throughout the replay, so this was a restart and not a hang. It has run to step 1,242 since, past the 1,073 where [the run it replaces](https://wandb.ai/marin-community/marin_moe/runs/dynact-coupled-d512-v2) first stopped but short of the 1,439 where that run ended. It still has one reading on held-out text, at step 599, so it cannot be read against the other three.
laddermoeh100
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run passed forty-one hours of unbroken stepping at 9:14 p.m. (ET) and cleared its hourly save of state at 9:20–9:22 p.m. (ET), in the slot the last read predicted. Its share of peak compute fell from 23.2 to 17.6 percent across four steps and was back at 23.2 percent by 9:22 p.m. (ET), with loss unchanged on either side. The seven saves before it, back to 2:19 p.m. (ET), each dipped to between 17.5 and 19.2 percent, so the save is costing no more as the run grows. W&B still shows the run training, at step 91,392 at 9:26 p.m. (ET).
herocheckpointmilestone
larrydial dynact-coupled-d512-v3
The fourth attempt at the paired-expert arm is now training. It logged step 0 at 9:03 p.m. (ET) — a start from the beginning, not a resume — and reached step 745 of 6,000 by 9:26 p.m. (ET), retracing [the run it replaces](https://wandb.ai/marin-community/marin_moe/runs/dynact-coupled-d512-v2) to within 0.006 in training loss at every step the two share. W&B now marks that run crashed, where [the first attempt](https://wandb.ai/marin-community/marin_moe/runs/dynact-coupled-d512) was marked finished. The two stops fell at steps 156 and 1,439, so they do not land at a fixed point in training; at its present rate this attempt passes 1,439 at about 9:43 p.m. (ET). None of it touches the hero.
laddermoeh100
larrydial dynact-dual-updown-d512
The arm set to `DUAL_UP_DOWN` has logged no step past 3,288 of 6,000 in the thirty minutes since a new process attached to its name at 8:56 p.m. (ET). Its GPUs are moving real data in bursts rather than sitting idle, so it is repeating work it had already logged — W&B refuses a step at or below the last one it recorded. If it too restarted from the beginning, its established rate puts it back past 3,288 at about 9:40 p.m. (ET), and only then will it show whether the stop repeats. [The third arm](https://wandb.ai/marin-community/marin_moe/runs/dynact-double-neurons-d512) has run unbroken to step 5,659 of 6,000.
laddermoeh100
WhenWen AUG-*-d512-600x
The sweep has opened no run of any kind since 8:43 p.m. (ET), forty minutes of silence against the 82-to-118-second promotions it ran earlier in the evening. Three of its 10,575-step grids closed at their top rate — at 7:49, 8:18 and 8:32 p.m. (ET) — and none has yet drawn the rung above 0.70 that [the automatic boundary extension](https://github.com/marin-community/marin/issues/7856#issuecomment-5638088947) opens in that case, now an hour and a half after the first of the three. All twenty rungs at the sweep's largest budget, 21,149 steps, are running and no 10,575-step rung is left open, so those extensions are the only work the sweep still owes.
ladderlrdense
larrydial dynact-coupled-d512-v3
The arm that pairs experts was reopened under a new name at 9:00 p.m. (ET), after a fourth stop. [The run it replaces](https://wandb.ai/marin-community/marin_moe/runs/dynact-coupled-d512-v2) came back at 8:53 p.m. (ET) and resumed at step 1,074 instead of repeating the 1,073 it had already logged — so its counter never froze, as the last read expected — then trained 365 steps and stopped at 8:58 p.m. (ET). The new run carries the same `COUPLED_PAIR_UP` setting, model shape and tags; at 9:02 p.m. (ET) it had logged no step and its GPUs still sat at the pre-load memory level, so it is loading rather than training. This arm still has no score on held-out text past step 1,199 of 6,000, so it cannot be read against the other three.
laddermoeh100
larrydial dynact-dual-updown-d512
A second arm of the same test stopped as well, so the trouble is not confined to the one setting under test, as the earlier read had it. The arm set to `DUAL_UP_DOWN` logged its last step, 3,288 of 6,000, at 8:54 p.m. (ET), and a new process attached to the same name at 8:56 p.m. (ET); by 9:02 p.m. (ET) its GPUs were moving real data while the counter still read 3,288, because W&B drops the steps it is repeating. [The third arm](https://wandb.ai/marin-community/marin_moe/runs/dynact-double-neurons-d512) has run unbroken to step 3,727, and at the matched step 2,999 it reads 3.782 on held-out text against 3.792 for [the finished baseline](https://wandb.ai/marin-community/marin_moe/runs/dynact-baseline-fsdp-d512) and 3.794 for the restarted arm — a spread far too small to separate the designs. None of the four touches the hero.
laddermoeh100
WhenWen AUG-LIN0-1L-DENSE-MUONH-016 … 020
On the schedule whose rate decays to zero, the Muon optimiser closed its 10,575-step grid at 8:32 p.m. (ET) and, like the two optimisers before it, scored best at the top of the grid: 4.843 on held-out text at 0.70, then 4.855, 4.888, 4.960 and 5.172 down to 0.10. Three grids have now closed at that same edge — at 7:49, 8:18 and 8:32 p.m. (ET) — and no rung above 0.70 had opened at that budget in any of the three by 8:48 p.m. (ET), an hour after the first closed and twice as long as the three earlier [boundary extensions](https://github.com/marin-community/marin/issues/7856#issuecomment-5638088947) took. The automation does reach this budget: it opened [a 1.1 rung there on the mixture-of-experts tree](https://wandb.ai/marin-community/marin_moe/runs/AUG-LRC-1L-SGDH-X300-HIGH-1-d512-300x-lr1.1) at 6:47 a.m. (ET) today.
ladderlrdensemilestone
WhenWen AUG-*-d512-600x
All four dense trees now have their full set of five rates open at the sweep's largest budget, 21,150 steps. The last two fifth rungs opened at 8:33 and 8:43 p.m. (ET), eleven and fifteen minutes after the matching 10,575-step rung below them closed — a slower handover than the 82-to-118-second promotions earlier in the evening. That budget is the last the sweep tests, so these twenty runs are its final rung set and no larger one will follow.
ladderlrdensemilestone
larrydial dynact-coupled-d512-v2
The arm that pairs experts stopped for the third time tonight, after [the first attempt](https://wandb.ai/marin-community/marin_moe/runs/dynact-coupled-d512) ended at step 156. It logged its last step, 1,073 of 6,000, at 8:28 p.m. (ET); a process attached in its place at 8:32 p.m. and ran about three minutes, and a second attached at 8:37 p.m. and has been moving real data through the GPUs since 8:39 p.m. (ET) rather than spinning idle. Its step counter stays at 1,073 while it repeats the work, because W&B refuses a step at or below the last one logged. Its two sibling arms have run without a break, so the fault follows the one setting under test.
laddermoeh100
larrydial dynact-baseline-fsdp-d512
The baseline of that test finished its 6,000 steps at 8:32 p.m. (ET), scoring 3.497 on held-out text. Its two arms still running are about 40 percent of the way in and sit on top of it: at the matched step 2,399 they read 3.855 and 3.863 against the baseline's 3.867, a spread far too small to separate the three designs. None of them touches the hero.
laddermoeh100
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run passed forty hours of unbroken stepping at 8:14 p.m. (ET) and cleared its next hourly save of state at 8:20–8:21 p.m. (ET), again spread over three steps. Speed and loss read the same on either side of the save, and W&B still shows the run training.
herocheckpointmilestone
larrydial dynact-coupled-d512-v2 / -double-neurons / -dual-updown
The test of the hero's model shape on one eight-GPU H100 node reopened at 8:10 p.m. (ET) with three arms in place of one, three minutes after [the first arm](https://wandb.ai/marin-community/marin_moe/runs/dynact-coupled-d512) stopped at step 156 of 6,000. The one setting under test changed form: what was a true/false flag is now a named choice, `expert_variant`, and the three runs read `COUPLED_PAIR_UP`, `STANDARD` with each expert twice as wide, and `DUAL_UP_DOWN`. At step 599, the first point all four score held-out text, [the baseline](https://wandb.ai/marin-community/marin_moe/runs/dynact-baseline-fsdp-d512) reads 4.269 and the arms 4.258, 4.272 and 4.274 — a spread far too small to separate them this early. None of it touches the hero.
laddermoeh100
WhenWen AUG-LIN0-1L-DENSE-SGDMH-016 … 020
On the schedule whose rate decays to zero, the optimiser with momentum closed its 10,575-step grid at 8:18 p.m. (ET) and, like the one without momentum before it, scored best at the top of the grid: 4.880 on held-out text at 0.70, then 4.936, 4.977 and 5.234 down to 0.10, with 0.20 still short of the budget at 5.055. The Muon tree is walking the same path, its 0.45 rung closed at 8:10 p.m. (ET) and its 0.70 rung still open and ahead of it. Four more rungs at the largest budget opened between 8:10 and 8:20 p.m. (ET), but no rate above 0.70 had opened at the 10,575-step budget in either tree by 8:24 p.m. (ET) — thirty-five minutes after the first of the two closed, longer than the roughly half hour the three earlier [boundary extensions](https://github.com/marin-community/marin/issues/7856#issuecomment-5638088947) took.
ladderlrdensemilestone
larrydial dynact-coupled-d512
The test started at 7:14 p.m. (ET) now has its second arm. A run opened at 7:56 p.m. (ET) on another eight-GPU H100 node with the same model shape, batch and 6,000-step budget as [the baseline](https://wandb.ai/marin-community/marin_moe/runs/dynact-baseline-fsdp-d512); the two configurations differ in exactly one model setting, `coupled_pair_up`, and the new run carries an extra `dynamic-activation` tag. It restarted once at 8:04 p.m. (ET), then stopped at step 156 of 6,000 at 8:07 p.m. (ET), which W&B marks `finished` rather than crashed; no replacement had appeared by 8:09 p.m. (ET). The two training losses sit within 0.04 of each other at step 150, far too early to separate them. Neither run touches the hero.
laddermoeh100
WhenWen AUG-*-d512-600x
The sweep promotes one rate at a time, not one grid at a time. Across the four trees now reaching the largest budget, 21,149 steps, every rung there opened 82 to 118 seconds after the rung at the same rate closed the 10,575-step budget below — including the fifth rung the earlier read had recorded as missing, which opened at 7:46 p.m. (ET), two minutes after its own predecessor. By 8:05 p.m. (ET) two trees had all five rungs open at that budget and two had three and one, matching how many of each tree's 10,575-step rungs have closed.
ladderlrdensemilestone
WhenWen AUG-LIN0-1L-DENSE-SGDH-016 … 020
On the schedule whose rate decays to zero, the optimiser without momentum closed its 10,575-step grid at 7:49 p.m. (ET), and every rate scored worse than the one above it: 5.079 on held-out text at 0.70, then 5.148, 5.201, 5.286 and 5.469 down to 0.10. So the best setting is again the top of the five rates the grid tests — the fourth budget in this tree to close at its own edge. No rung above 0.70 had opened there by 8:05 p.m. (ET), which is early: the three earlier extensions appeared about half an hour after their grids closed.
ladderlrdensemilestone
Sep 11
WhenWen AUG-LRC-1L-DENSE-SGDMH-016 … 024
The optimiser that adds momentum closed its constant-rate grid at the 10,575-step budget between 7:35 and 7:44 p.m. (ET), and its best setting sits inside the grid: 0.32 scored 5.022 on held-out text against 5.025 at 0.20 and 5.104 at 0.70, the top rate the grid tests, so no rung above it is due. The plain training loss picks 0.20 instead, also inside the grid. Four of the five rungs at the sweep's largest budget, 21,149 steps, opened at 7:36–7:37 p.m. (ET); the fifth had not appeared by 7:44 p.m. (ET), when the last 10,575-step rung finished.
ladderlrdensemilestone
ravwojdyla #9120 open
A pull request opened at 6:49 p.m. (ET) rewrites the ranges on the hero's dashboard charts: speed starts at zero, training loss gets a 1.20-to-1.60 band, and both widen to hold outliers instead of dropping the highest and lowest two percent of samples, which had hidden the extremes worth seeing. It also drops the earliest scores on held-out text from the default view, because those first high readings squeeze every later change flat, and adds a control that puts them back.
monitoring
larrydial dynact-baseline-fsdp-d512
A new test of the hero's model shape started at 7:14 p.m. (ET) on one eight-GPU H100 node. It splits the work across those GPUs by data rather than by expert, routes tokens through a path that sets no per-expert capacity limit, and takes a batch of 128 sequences against the 1,024 of [the ladder rungs on the same shape](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-197c9f5ceff6b9ee-seed0-from10pct-20260829), so its speed cannot be read against theirs. It is set to stop after 6,000 steps and was at step 1,035 at 7:31 p.m. (ET). Its name marks it a baseline, and no issue naming the comparison it opens has appeared yet. It does not touch the hero.
laddermoeh100
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run passed thirty-nine hours of unbroken stepping at 7:14 p.m. (ET) and cleared its hourly save of state at 7:19–7:21 p.m. (ET), adding about fifteen seconds across the three steps it spanned — a heavier save than the single-step one an hour before, and again in the slot that one predicted. The run reached step 91,011 at 7:29 p.m. (ET), still taking a step every eighteen seconds.
herocheckpointmilestone
WhenWen AUG-LIN0-1L-DENSE-SGDH-X150-HIGH-1 / X60-HIGH-2
Two more rates opened above the grid closed, and both beat it, so the search reopens at two budgets that had looked done. At the 5,288-step budget 1.1 scored 5.161 on held-out text at 7:01 p.m. (ET) against 5.218 at 0.70, the top of the five rates the grid tests; at the 2,115-step budget the second rate opened above it, 1.7, scored 5.303 at 6:57 p.m. (ET) against 5.353 at 1.1. Both won on the plain training loss as well. Of the three budgets this tree has closed, only the 1,058-step one has settled, and [#7856](https://github.com/marin-community/marin/issues/7856#issuecomment-5638088947) had opened no further rung at 7:06 p.m. (ET).
ladderlrdensemilestone
WhenWen AUG-LRC-1L-DENSE-SGDH-X150-HIGH-1
On the schedule whose rate never decays, the rate opened above the grid came back worse and closed that search. At the 5,288-step budget 1.1 scored 5.402 on held-out text at 6:28 p.m. (ET), behind 0.70's 5.316 — so for the optimiser without momentum at a constant rate, the best setting is the top of the five the grid tests, with nothing better above it. The matching probes on the schedule whose rate decays to zero went the other way, so the two schedules disagree on where the useful range of rates ends.
ladderlrdensemilestone
mcwitt lb0911moebared3
The sixth and last one-rack test finished at 6:48 p.m. (ET), closing the comparison. The three runs named `bare` read 22.85, 22.82 and 22.82 percent model FLOPs utilisation against 22.70, 22.74 and 22.70 for the three named `current`, so the two groups do not overlap, while all six final losses agree to within 1.1e-4. The groups differ in exactly one installed package, the compiled GPU plugin wheel, so that package costs about 0.12 of a percentage point of speed and nothing in loss over thirty steps. None of the six touched the hero, and no seventh had started at 7:06 p.m. (ET).
herobenchone-rackmilestone
mcwitt #9118 open
A second pull request opened at 5:46 p.m. (ET), alongside [#9119](https://github.com/marin-community/marin/pull/9119), makes the model's short convolution work when one sequence is split across GPUs: each shard swaps a strip of its neighbour's activations before computing, so values and gradients match the unsplit result. It is stacked on #9117 and, like #9119, is part of [#8754](https://github.com/marin-community/marin/issues/8754), the long-context phase planned for the end of the run.
long-context
WhenWen AUG-LIN0-1L-DENSE-SGDH-X30-HIGH-2
The second rate opened above the grid closed worse, and the search at that budget is done. At the 1,058-step budget 1.7 scored 5.703 on held-out text at 6:38 p.m. (ET), behind both 1.1 (5.580) and 0.70 (5.633), the top of the five rates the grid tests — so for the optimiser without momentum on the schedule whose rate decays to zero, the best setting there is 1.1, with a tested rate on either side of it. The matching 1.7 probe at the 2,115-step budget was still running at 6:49 p.m. (ET) and [#7856](https://github.com/marin-community/marin/issues/7856#issuecomment-5638088947) had opened no third extension.
ladderlrdensemilestone
mcwitt lb0911moebared3
A sixth one-rack test started at 6:30 p.m. (ET), and the six are two arms rather than six repeats of one setting. The three named `bare` and the three named `current` differ in exactly one installed package — the compiled GPU plugin wheel, `0.11.1+marin.47928lbbare` against `0.11.1+marin.47928lbcurrent` — with the same host, configuration, tags and every other version identical. Across the five that have closed the `bare` arm read 22.85 and 22.82 percent model FLOPs utilisation against 22.70, 22.74 and 22.70 for `current`, so the arms do not overlap, while their final losses agree to 1.1e-4. The sixth was stepping at 6:49 p.m. (ET); none of the six touched the hero.
herobenchone-rack
WhenWen AUG-LIN0-1L-DENSE-SGDH-X30-HIGH-2 / X60-HIGH-2
The sweep stepped a second time beyond its own grid: two runs at 1.7 opened at 6:09 and 6:10 p.m. (ET) at the 1,058- and 2,115-step budgets, above the 1.1 runs that closed at 5:53 and 6:06 p.m. (ET) and beat 0.70, the top of the five rates the grid tests. This is the first grid from which [#7856](https://github.com/marin-community/marin/issues/7856#issuecomment-5638088947)'s automatic boundary extension has had to reach out twice, so for the optimiser without momentum on the schedule whose rate decays to zero the best setting is still above everything tried. Both runs were stepping at 6:23 p.m. (ET).
ladderlrdensemilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run passed thirty-eight hours of unbroken stepping at 6:14 p.m. (ET) and cleared its hourly save of state at 6:19 p.m. (ET) inside one step, for about eight seconds of added time — a lighter save than the four-step, twenty-second one an hour before. It landed in the slot the last read predicted. The run reached step 90,797 at 6:23 p.m. (ET), still taking a step every eighteen seconds.
herocheckpointmilestone
mcwitt lb0911moecurrentd3
The fifth one-rack test of the hero's current settings ran its thirty steps between 5:59 and 6:17 p.m. (ET) and finished. Across all five the final loss spans 0.00011 and model FLOPs utilisation 0.15 of a percentage point — the last two runs widened neither — so a one-rack reading repeats to far inside the roughly half a percentage point by which it sits below the hero's own utilisation. No sixth had started at 6:23 p.m. (ET), and none of the five touched the hero.
herobenchone-rackmilestone
WhenWen AUG-LIN0-1L-DENSE-SGDH-X30-HIGH-1 / X60-HIGH-1
The first two rates opened above the grid closed, at 5:53 and 6:06 p.m. (ET), and both beat it: at 1.1 the 1,058- and 2,115-step budgets scored 5.580 and 5.353 on held-out text against 5.633 and 5.430 at 0.70, the top of the five rates the grid tests, and won on the plain training loss as well. So on the schedule whose rate decays to zero the best setting for the optimiser without momentum lies outside the range the sweep searches, not at its edge. [#7856](https://github.com/marin-community/marin/issues/7856#issuecomment-5638088947) opened these probes automatically; the same probe run this morning on the mixture-of-experts model under a constant rate finished worse than 0.70, so the result does not carry across schedules.
ladderlrdensemilestone
mcwitt lb0911moecurrentd3
A fifth one-rack test of the hero's current settings started at 5:59 p.m. (ET), twelve minutes after [the fourth](https://wandb.ai/marin-community/marin_moe/runs/lb0911moecurrentd2) finished, under the same configuration and tags as the four before it. It had logged no step at 6:06 p.m. (ET). Like them it restores the first hero run's August checkpoint and stops thirty steps later, so it does not touch the hero.
herobenchone-rack
WhenWen h100-d512-mixprior — fourth wave
Two more mixtures finished at 6:02 and 6:03 p.m. (ET) at 3.8000 and 3.8010 on the held-out text, both inside the 3.7924-to-3.8017 span the wave already held. Eleven of the wave's twelve mixtures are now in and that span has not widened, so it stays narrower than the 0.013 [one mixture's three seeds](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed2-from10pct-20260829) show between themselves. The last rung is still running.
ladderdata-mix
WhenWen moe-lgr-9110-d512-rmsgated / d768-rmsgated
Both full runs of the corrected way of routing tokens to experts died within four minutes of each other, the wider model at 5:24 p.m. (ET) and the narrower at 5:28 p.m. (ET), each a long way short of its step target and each left in the crashed state. [#9110](https://github.com/marin-community/marin/issues/9110) requires a finished run and a final checkpoint before it will read a result, so the corrected design is still untested, and no replacement had been submitted as of 5:49 p.m. (ET). The chips are TPU, not the hero's.
experimentmoeroutingcrash
WhenWen AUG-LIN0-1L-DENSE-SGDH-X30-HIGH-1 / X150-HIGH-1
Two more rates above the grid, both 1.1, opened at 5:24 p.m. (ET) at the 1,058- and 5,288-step budgets, for the optimiser without momentum on the schedule whose rate falls to zero. With the 2,115-step extension from 5:20 p.m. (ET), all three budgets that tree has closed read best at 0.70, the top of the five rates it tests, on the held-out score and the training loss alike — so the sweep's automatic boundary extension [#7856](https://github.com/marin-community/marin/issues/7856#issuecomment-5638088947) has now had to probe beyond every one of them.
ladderlrdensemilestone
mcwitt lb0911moecurrentd2
A fourth one-rack test of the hero's current settings ran its thirty steps between 5:34 and 5:46 p.m. (ET), from the same August checkpoint and under the same configuration and tags as the three before it. Across all four the final loss spans 0.00011 and model FLOPs utilisation 0.15 of a percentage point, so a one-rack reading repeats to far inside the roughly half a percentage point by which it sits below the hero's own utilisation. None of the four touched the hero.
herobenchone-rackmilestone
WhenWen h100-d512-mixprior — fourth wave
A ninth mixture in the fourth wave finished at 5:32 p.m. (ET) at 3.7967 on the held-out text, inside the 3.7924-to-3.8017 span the other eight already hold. The wave's ranking therefore still rests on less spread than [one mixture's three seeds](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed2-from10pct-20260829) show between themselves. Three rungs are still running.
ladderdata-mix
mcwitt #9119 open
A pull request splits each training sequence across a group of GPUs so the hero's model shape can train on 262,144-token inputs, sixty-four times the length it trains on now. Its only measurement is [this afternoon's one-rack test of that path](https://wandb.ai/marin-community/marin_moe/runs/lc-main-ragged-262k-0911), and the description says that figure shows throughput, not training quality. It is part of [#8754](https://github.com/marin-community/marin/issues/8754), the work toward a long-context phase at the end of the run.
long-context
WhenWen AUG-LIN0-1L-DENSE-SGDH-X60-HIGH-1
A second rate above the grid, 1.1, opened at 5:20 p.m. (ET), this time on the schedule whose rate falls to zero, for the optimiser without momentum at the 2,115-step budget. That grid closed at 3:56 p.m. (ET) with its best at 0.70, the top of the five rates it tests, on the held-out score and the training loss alike — 5.430 and 4.324 — so the sweep's automatic boundary extension [#7856](https://github.com/marin-community/marin/issues/7856#issuecomment-5638088947) opened a rung beyond it. The first such extension fired at 4:53 p.m. (ET) on the constant-rate tree. Fourteen of the fifteen decay-to-zero rungs at the 10,575-step budget are now open, the newest at 5:20 p.m. (ET).
ladderlrdensemilestone
WhenWen h100-d512-mixprior — seed repeats
Two reruns of one data mixture at fresh random seeds finished at 5:13 and 5:20 p.m. (ET), scoring 3.8126 and 3.8025 on the held-out text against [the same mixture's first seed](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-996f489106c7b922-seed0-from10pct-20260829) at 3.8000 — a spread of 0.013 for one unchanged mixture. The eight different mixtures that have finished in the current wave span 0.009, from 3.7924 to 3.8017, so changing the random seed now moves the score further than changing the mixture does. A second mixture's one rerun, which finished at 4:12 p.m. (ET), had spread only 0.002, so the size of this effect is still measured on two mixtures. Four rungs of the wave are still running.
ladderdata-mixmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run passed thirty-seven hours of unbroken stepping at 5:14 p.m. (ET) and cleared an hourly save of state at 5:19–5:21 p.m. (ET), adding about twenty seconds across the four steps it spanned. The save landed in the moved slot the last read predicted, which confirms that this afternoon's scoring on held-out text restarted the hourly timer rather than a save being missed. It reached step 90,599 at 5:23 p.m. (ET), still taking a step every eighteen seconds.
herocheckpointmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run cleared an hourly save of state at 4:19–4:20 p.m. (ET), adding about ten seconds across the four steps it spanned, and has taken a step every eighteen seconds since, reaching step 90,530 at 5:01 p.m. (ET). The save slot sits at :19 past the hour rather than the :52 it held this morning: the 2:18 p.m. (ET) scoring on held-out text restarted the hourly timer, so no save was ever due at the :51 slot. The next is due near 5:19 p.m. (ET), and thirty-seven hours of unbroken stepping falls at 5:14 p.m. (ET).
herocheckpoint
WhenWen AUG-LRC-1L-DENSE-SGDH-X150-HIGH-1
A sixth rate, 1.1, opened at 4:53 p.m. (ET) above the five the constant-rate grid tests, for the optimiser without momentum at the 5,288-step budget — the per-budget boundary extension [#7856](https://github.com/marin-community/marin/issues/7856#issuecomment-5638088947) described at 1:20 p.m. (ET). It fired because that grid's best score on held-out text sits at 0.70, the top rate it tests, 5.316 against 5.331 at 0.45, while the plain training loss reads best at 0.45 — the first closed grid on which the two measures pick different rates, and the sweep decides on the held-out score. The same probe ran this morning on the mixture-of-experts version of the same one-layer model, where 1.1 finished worse than 0.70 and left the grid's top standing.
ladderlrdensemilestone
WhenWen AUG-LIN0 / AUG-LRC-1L-DENSE-011 … 020
The optimiser without momentum closed all five of its decay-to-zero rungs at the 5,288-step budget at 4:51 p.m. (ET), its best at 0.70 — the top of the grid — on the training loss and the held-out score alike, so that grid also reads best at its edge. The newest optimiser closed its constant-rate grid at the same budget at 4:46 p.m. (ET) with its best at 0.32, the middle rate, on both measures. Ten of the fifteen decay-to-zero rungs at the next budget up, 10,575 steps each, opened between 4:47 and 5:00 p.m. (ET).
ladderlrdense
mcwitt lb0911moebared2
The third one-rack test of the hero's current settings finished all thirty of its steps at 4:59 p.m. (ET), nineteen minutes after it started. All three tests run today carry the same settings and the same tags and ended within 0.0002 of each other on loss and within 0.15 of a percentage point on model FLOPs utilisation, so together they measure how much a one-rack reading of the hero's recipe moves between runs — far less than the half-point it sits below the hero. None of the three touched the hero.
herobenchone-rackmilestone
WhenWen h100-d512-mixprior — fourth wave
A seventh mixture in the fourth wave finished at 4:56 p.m. (ET) at 3.8005 on the held-out text; the seven now span 3.7924 to 3.8005, about four times the 0.002 measured so far between a mixture and its rerun at a second seed. Two rungs are behind the step counts W&B shows: one has logged no step since 2:07 p.m. (ET) and took a fresh process at 4:42 p.m. (ET), and another read finished eighteen steps short at 5:08 p.m. (ET) before a new process attached at 5:10 p.m. (ET). Both are repeating steps W&B will not record a second time, so a frozen counter here does not mean a stopped rung.
ladderdata-mix
WhenWen moe-lgr-9110-d512-rmsgated / d768-rmsgated
Both full runs of the new way of routing tokens to experts started on TPU at 4:48 p.m. (ET), at two model widths, after [#9110](https://github.com/marin-community/marin/issues/9110) corrected the design at 2:08 p.m. (ET): the step that normalises what the router reads is kept, and the learned gate is applied after it rather than in its place. The five-step trial of the corrected version was skipped and the full runs submitted directly, and the earlier interpretation's runs are withdrawn. The chips are not the hero's.
experimentmoerouting
A third one-rack test of the hero's current settings started at 4:40 p.m. (ET), forty minutes after [the second](https://wandb.ai/marin-community/marin_moe/runs/lb0911moebared1) finished, and W&B records the same configuration and the same tags as both tests before it. Like them it restores the first hero run's August checkpoint at step 42,000 and is set to stop thirty steps later, so it does not touch the hero. It had logged no step at 4:45 p.m. (ET).
Six of the fifteen rungs in the fourth wave of data-mixture tests finished their 3,929 steps between 4:08 and 4:39 p.m. (ET). One of them is the series' first finished repeat of a mixture at a second random seed: it scored 3.7995 on the held-out text against [the first seed's](https://wandb.ai/marin-community/marin_moe/runs/h100-d512-mixprior-80a4d4891e8356d1-seed0-from10pct-20260829) 3.8017, a difference of 0.002, where the six finished mixtures span 0.008 between best and worst — so the ranking among them rests on about four times the one seed-to-seed difference measured so far. Nine rungs are still running, between steps 1,327 and 3,737.
Four of the newest optimiser's five constant-rate rungs at the 5,288-step budget finished at 4:36 p.m. (ET), and its lowest final loss came at 0.32, the middle rate, at 4.189; the fifth rung, at the top rate, is about 135 steps short and already behind at 4.379. The other two optimisers closed that budget this morning with their best at 0.20 and 0.45, so all three constant-rate grids now settle inside the grid rather than at either end, but on three different rates, where at half the budget all three settled on 0.32. Four rungs at the next budget up, 10,575 steps each, opened at 4:37 and 4:38 p.m. (ET); the fifth has not opened.
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run cleared its thirty-sixth hourly save of state near 3:51 p.m. (ET), adding about nine seconds across the five steps it spanned, and passed thirty-six hours of unbroken stepping at 4:14 p.m. (ET). It reached step 90,404 at 4:23 p.m. (ET), still taking a step every eighteen seconds. The thirty-seventh save is due near 4:51 p.m. (ET).
herocheckpointmilestone
ClassicLarry #8827
Larry published the launcher and multi-prompt sampler behind the written samples taken from saved checkpoints, on the branch `grug/hero-inference-check`, so the step-6k, 24k, 42k and 78k reads can be repeated by hand.
heroeval
WhenWen AUG-LIN0 / AUG-LRC-1L-DENSE-006 … 015
All fifteen rungs at twice the smallest token budget on the schedule whose rate falls to zero are now finished, the last at 3:56 p.m. (ET). The optimiser without momentum and the one that orthogonalises its updates both read their lowest final loss at 0.70, the top of the grid — 4.324 and 3.960 — which reverses the reading of an hour ago, when the second of those still had its best at 0.45. The newest optimiser, which adds plain momentum, keeps its best at 0.45: 4.204 against 4.210 at 0.70, an interior best by 0.006. At the same budget the constant-rate schedule puts all three optimisers' best at 0.32, the middle rate of the five, so the two schedules keep selecting different ends of the grid. All fifteen rungs at the next budget up, 5,288 steps each, are now open, the last at 3:57 p.m. (ET).
ladderlrdensemilestone
mcwitt lb0911moebared1
A second one-rack test of the hero's current settings started at 3:41 p.m. (ET) and finished all thirty of its steps at 4:00 p.m. (ET). W&B records the same configuration and the same tags as [the first test](https://wandb.ai/marin-community/marin_moe/runs/lb0911moecurrentd1) forty-eight minutes earlier, and it ended on the same loss to three decimals, 1.304, at 22.8% model FLOPs utilisation against the first's 22.7% and seventeen seconds a step against the hero's eighteen. Like the first, it restored the first hero run's August checkpoint at step 42,000, stopped thirty steps later, and never touched the hero.
herobenchone-rackmilestone
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run cleared its thirty-fifth hourly save of state at 2:51 p.m. (ET), adding about three seconds to the single step it spanned, and passed thirty-five hours of unbroken stepping at 3:14 p.m. (ET). It reached step 90,213 at 3:24 p.m. (ET), still taking a step every eighteen seconds; four steps from 3:19 p.m. (ET) ran two to six seconds slow before the pace returned. The thirty-sixth save is due near 3:51 p.m. (ET).
herocheckpoint
mcwitt lb0911moecurrentd1
The one-rack test of the hero's current settings finished all thirty of its steps at 3:11 p.m. (ET), twelve minutes after its first, and W&B marks it finished. It held 22.8% model FLOPs utilisation against the hero's 23.3% and took a step every seventeen seconds against the hero's eighteen, on one rack against the hero's eleven. Its per-step losses ranged from 1.23 to 1.32, against the hero's 1.23, and it never touched the hero.
herobenchone-rackmilestone
WhenWen AUG-LIN0-1L-DENSE-SGDH / MUONH / SGDMH-006 … 015
Eleven of the fifteen rungs at twice the smallest token budget on the schedule whose rate falls to zero finished their 2,115 steps at 3:09 p.m. (ET). The optimiser without momentum read its lowest final loss at 0.70, the top of the grid and the same rate that won at the smaller budget, at 4.324; the two optimisers that add momentum read their lowest finished loss at 0.45 instead, 4.038 and 4.204, but both still have their 0.70 rung running and one of those opened only at 3:10 p.m. (ET), a tenth of the way in. Eleven rungs at the next budget up, 5,288 steps each, opened between 3:10 and 3:15 p.m. (ET); four have not opened.
ladderlrdense
WhenWen AUG-LRC-1L-DENSE-SGDMH-010 … 015
The newest optimiser's fifth constant-rate rung at twice the smallest token budget, the top rate of the five, finished at 3:08 p.m. (ET) at 4.527 — the worst of its five. With all five in, that grid's lowest final loss stays at 0.32, the middle rate, so the constant-rate schedule holds its interior best at this budget too. The fifth rung at the next budget up opened at 3:10 p.m. (ET), eleven minutes after the other four, completing that grid of five.
ladderlrdense
mcwitt lb0911moecurrentd1
A one-rack test of the hero's current settings started at 2:53 p.m. (ET) on sixty-four GPUs, against the hero's seven hundred and four. It restores the first hero run's August checkpoint at step 42,000 and is set to stop thirty steps later, so it does not touch the hero itself. It logged its first step at 2:59 p.m. (ET) and settled at 22.8% model FLOPs utilisation, seventeen seconds a step, against the hero's 23.3% at eighteen. Its one departure from the hero's settings is that it keeps the master copy of the weights in host memory rather than on the GPUs.
herobenchone-rack
WhenWen AUG-LRC-1L-DENSE-SGDMH-006 … 014
Four of the newest optimiser's five constant-rate rungs at twice the smallest token budget finished their 2,115 steps at 2:57 p.m. (ET), forty-six minutes after they opened. Its lowest final loss again came at 0.32, the middle rate of the five — 4.293, down from 4.812 at the smaller budget — so the constant-rate schedule has now put this optimiser's best in the middle of the grid at two budgets running, where the decay-to-zero schedule has so far put every optimiser's best at the top of it. The fifth rung, at the top rate, is about 115 steps short. Four rungs at the next budget up, 5,288 steps each, opened at 2:58 and 2:59 p.m. (ET); the fifth has not opened.
ladderlrdense
WhenWen AUG-LIN0-1L-DENSE-SGDH / MUONH / SGDMH-001 … 010
The last rung at the smallest token budget on the schedule whose rate falls to zero finished at 2:39 p.m. (ET). Of the rates that have finished, all three optimisers read their lowest final loss at 0.70, the top of the grid, where on the constant-rate schedule all three read theirs at 0.32, the middle of it — but the newest optimiser's 0.70 beats its 0.45 by 0.0007, and one rung at 0.32 is still sixty steps from the end and already within 0.025 of its own optimiser's 0.70, so the top of the grid is not yet a clear winner. Fourteen of the fifteen rungs at twice the budget are open, the newest at 2:44 p.m. (ET).
ladderlrdense
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run passed thirty-four hours of unbroken stepping at 2:14 p.m. (ET) and stepped past 90,000 at 2:19 p.m. (ET), still taking a step every nineteen seconds. It scored itself on the held-out text at 2:18 p.m. (ET), holding up stepping for about a minute on the one step it spanned — the first such scoring since 11:01 p.m. (ET) yesterday, which cost about the same. The score, 2.229, sits where the last four scorings put it, between 2.228 and 2.235 across nine thousand steps, against the 2.04 registered for the end of the run. The thirty-fifth hourly save of state is due near 2:52 p.m. (ET).
heroeval
WhenWen AUG-LRC-1L-DENSE-SGDMH-001 … 010
The third optimizer finished all five of its rungs at the smallest token budget of the constant-rate sweep at 2:09 p.m. (ET), twenty-nine minutes after they opened. Its best final loss, 4.812 at rate 0.32, falls between the first optimizer's 4.985 and the second's 4.602 at the same budget and the same rate, so the momentum it adds recovers about half the distance between the two. It opened four rungs at twice that budget at 2:10 and 2:11 p.m. (ET) and the fifth at 2:23 p.m. (ET).
ladderlrdense
WhenWen AUG-LIN0-1L-DENSE-SGDH / MUONH / SGDMH-001 … 010
Thirteen of the fifteen rungs on the schedule whose rate falls to zero finished their 1,058 steps at 2:21 and 2:22 p.m. (ET), twenty-seven minutes after they opened. Under the two optimizers with all five rates finished, the lowest final loss came at the highest rate tested — 4.400 and 4.791 — where the constant-rate sweep's best at the same budget sat at the middle rate of the five. A schedule that ends at zero should read lower at its final step, so the rate that wins, not the size of the gap, is the comparison to carry. The sweep opened twelve rungs at twice the budget from 2:23 p.m. (ET); one rung at the smaller budget stalled for seventeen minutes from 2:04 p.m. (ET) and is about 700 steps behind the rest.
ladderlrdense
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run cleared its thirty-fourth hourly save of state on time at 1:52 p.m. (ET), adding about five seconds to the single step it spanned. It reached step 89,949 at 2:02 p.m. (ET), still taking a step every eighteen seconds, and passes thirty-four hours of unbroken stepping at 2:14 p.m. (ET).
herocheckpoint
mcwitt lc-main-ragged-262k-0911
The 262,144-token test finished all forty of its steps at 1:46 p.m. (ET), seventy minutes after it started, and W&B marks it finished. It held 10.0% model FLOPs utilisation against the hero's 23.3% and took a step every hundred seconds against the hero's eighteen; its per-step losses ranged from 0.73 to 1.57 and averaged 1.11, against the hero's 1.22.
long-contextheromilestone
WhenWen moe-lgr-9110-d512-gated-smoke
The five-step trial of a new way of routing tokens to experts, reported here as crashed, was restarted in place at 1:48 p.m. (ET) and ran all five steps, finishing at 1:51 p.m. (ET). Its loss held near 11.80 throughout, the value an untrained model of this size starts at — the run checks only that the code under [#9110](https://github.com/marin-community/marin/issues/9110) runs end to end.
ladderrouter
WhenWen AUG-LIN-1L-DENSE-SGDH / MUONH-001 … 005
All ten rungs of the falling-rate sweep under [#7856](https://github.com/marin-community/marin/issues/7856) stopped within fourteen seconds of each other at 1:40 p.m. (ET), between steps 860 and 875 of the 1,058 each was set to run, and W&B marks them crashed. None had restarted twenty minutes later.
ladderlrdensecrash
WhenWen AUG-LIN0-1L-DENSE-SGDH / MUONH / SGDMH-001 … 005
A third learning-rate schedule opened at 1:54 p.m. (ET) under [#7856](https://github.com/marin-community/marin/issues/7856), fourteen minutes after the falling-rate sweep stopped: fourteen rungs at the smallest token budget across all three optimizers, five rates each, whose rate falls linearly to zero instead of to a twentieth of its starting value. Each runs the same 1,058 steps and all fourteen are past step 190; the fifteenth rung, the third optimizer's highest rate, has not opened.
ladderlrdense
WhenWen #7856
The ten test models that stopped at 1:40 p.m. (ET) were cancelled on purpose, not lost to a fault: their learning rate fell to a twentieth of its starting value rather than to zero, which was not the comparison the sweep is meant to make, so all three of those trees were withdrawn and relaunched under new names. The same note adds a third optimiser — gradient descent with momentum — with matched constant-rate and decay-to-zero grids of twenty-five settings each.
ladderlrcorrection
WhenWen AUG-LRC-1L-DENSE-SGDMH-001 … 005
A third optimizer joined the constant-rate sweep at 1:41 p.m. (ET) under [#7856](https://github.com/marin-community/marin/issues/7856): five rungs at the smallest token budget, without experts, set to the same 1,058 steps and the same five rates as the first optimizer's rungs. Its only settings that differ add momentum of 0.95 with Nesterov's correction. All five were still starting up and had logged no step at 1:45 p.m. (ET).
ladderlrdense
WhenWen moe-lgr-9110-d512-gated-smoke
A five-step trial of a new way of routing tokens to experts started on four TPU v5 chips at 1:36 p.m. (ET), the first run of its group and tagged to [#9110](https://github.com/marin-community/marin/issues/9110). It registered its model — 597 million parameters over six layers and 384 experts — then stopped reporting at 1:37 p.m. (ET) without logging a training step, and W&B marks it crashed. The chips are not the hero's.
ladderrouter
WhenWen #9110
A change to how the model picks which experts handle each token is written and checked in: the step that normalises the features the router reads is replaced by a learned gate, and the router and the experts both read the gated features. Four numerical tests pass, one of them reproducing the hero's present behaviour exactly when the change is switched off, and the five-step trial run was submitted.
ladderrouter
held h100-d512-mixprior × 15
The fourth wave of data-mixture tests opened at 1:11 p.m. (ET) with fifteen rungs, twelve of them mixtures the ladder has not run before. The other three repeat two earlier mixtures at a second and a third random seed — the first repeats the series has run — and every rung is set to the 3,929 steps the third wave used.
ladderdata
WhenWen AUG-LIN-1L-DENSE-SGDH / MUONH-001 … 005
A second learning-rate sweep opened at 1:15 p.m. (ET) under [#7856](https://github.com/marin-community/marin/issues/7856): ten rungs at the smallest token budget, five rates under each of the two optimizers, matching the sweep now finishing except that the rate falls linearly to a twentieth of its starting value instead of holding constant. Each rung runs 1,058 steps and all ten are past step 190.
ladderlrdense
mcwitt lc-main-ragged-262k-0911
The 262,144-token test has logged twenty-five of its forty steps, the newest at 1:21 p.m. (ET), still one every hundred seconds and holding 10.1% model FLOPs utilisation against the hero's 23.2%. Its per-step losses swing between 0.87 and 1.57 and average 1.12, close to the hero's 1.22; at this rate it stops near 1:48 p.m. (ET).
long-contexthero
WhenWen #7856
The falling-rate half of the sweep was widened to twenty-five settings per optimiser — five training lengths by five learning rates — matched across two optimisers. A first submission was withdrawn before any model started because it asked each host for two hundred gigabytes of memory against the hundred and ninety-two the cluster allows; the resubmission corrected that figure and changed nothing else.
ladderlrdense
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run cleared its thirty-third hourly save of state on time at 12:52 p.m. (ET), adding about fourteen seconds across the two steps it spanned. It reached step 89,751 at 1:02 p.m. (ET), still taking a step every nineteen seconds, and passes thirty-three hours of unbroken stepping at 1:14 p.m. (ET).
herocheckpoint
mcwitt lc-main-ragged-262k-0911
The 262,144-token test has logged twelve of the forty steps it is set to run, one every hundred seconds against the hero's nineteen, and reads 10.0% model FLOPs utilisation against the hero's 23.2%. Its loss sits near 0.91, below the hero's 1.21, as expected when each token is predicted from sixty-four times the context. At this rate it stops near 1:48 p.m. (ET).
long-contexthero
WhenWen AUG-LRC-1L-SGDH-025-d512-600x-lr0.7
The last of the ten rungs at the sweep's largest token budget finished its 21,150 steps at 12:40 p.m. (ET), the second optimizer's highest rate. Every model the sweep planned with experts has now finished, five rates under each optimizer at each of its five token budgets.
ladderlrmilestone
mcwitt lc-main-ragged-262k-0911
mcwitt started the hero's own model shape at 262,144 tokens per sequence — sixty-four times the 4,096 the hero trains on — at 12:36 p.m. (ET), on the same 64 GB200 GPUs and the same pinned step-81716 checkpoint the 4,096-token reference run used yesterday. It splits each sequence across four GPUs and its experts across sixteen, the layout [#8141](https://github.com/marin-community/marin/issues/8141) sets as the target; it is set to stop after 40 steps and took its first two at 12:42 and 12:44 p.m. (ET), about ninety-nine seconds apart against the hero's eighteen.
long-contexthero
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run passed thirty-two hours of unbroken stepping at 12:14 p.m. (ET) and reached step 89,624 at 12:23 p.m. (ET), still taking a step every eighteen seconds. Its thirty-third hourly save of state is due near 12:52 p.m. (ET).
heromilestone
WhenWen AUG-LRC-1L-DENSE-MUONH-024 / 025
The sweep opened its last two rungs at the 21,150-step budget without experts at 12:05 and 12:08 p.m. (ET), the fourth and fifth rates under the second optimizer. All ten rungs at that budget now run, five rates under each optimizer.
ladderlrdense
WhenWen AUG-LRC-1L-DENSE-MUONH-019 / 020
The last two of the five rates at the 10,575-step budget without experts finished under the second optimizer at 12:04 and 12:07 p.m. (ET). That budget is now complete under both optimizers, leaving the 21,150-step rungs without experts as the only ones still running.
ladderlrdense
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run cleared its thirty-second hourly save of state on time at 11:52 a.m. (ET), adding about twelve seconds across the three steps it spanned. It reached step 89,555 at 12:02 p.m. (ET), still taking a step every eighteen seconds, and passes thirty-two hours of unbroken stepping at 12:14 p.m. (ET).
herocheckpoint
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run passed thirty-one hours of unbroken stepping at 11:14 a.m. (ET) and reached step 89,489 at 11:42 a.m. (ET), still taking a step every nineteen seconds. Its thirty-second hourly save of state has not appeared yet; it is due near 11:52 a.m. (ET).
herocheckpoint
WhenWen AUG-LRC-1L-025, AUG-LRC-1L-SGDH-024
Two more rungs at the sweep's largest token budget finished their 21,150 steps, at 11:18 and 11:41 a.m. (ET), leaving one of the ten. The first optimizer has now run all five of its rates there and puts its best result at the second-lowest; the second has only its highest rate left, at step 18,703, and there each finished rate has beaten the one below it.
ladderlr
WhenWen AUG-LRC-1L-DENSE-SGDH-019 / 020, MUONH-016 / 017 / 018
The sweep closed its 10,575-step budget without experts under the first optimizer, the last two of its five rates finishing at 11:23 and 11:39 a.m. (ET). Under the second optimizer three rates finished there between 11:30 and 11:39 a.m. (ET); the other two have reached steps 9,312 and 9,086.
ladderlrdense
WhenWen AUG-LRC-1L-DENSE-SGDH-024 / 025, MUONH-021 / 022 / 023
The sweep opened its fourth and fifth rungs at the 21,150-step budget without experts at 11:25 and 11:41 a.m. (ET), so all five of its rates now run there under the first optimizer. It opened the first three rates at that budget under the second optimizer between 11:32 and 11:41 a.m. (ET).
ladderlrdense
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run cleared its thirty-first hourly save of state on time at 10:52 a.m. (ET), adding about eight seconds to the one step it spanned — the lightest of the day's saves. It reached step 89,360 at 11:02 a.m. (ET), still taking a step every eighteen seconds, and passes thirty-one hours of unbroken stepping at 11:14 a.m. (ET).
herocheckpoint
WhenWen AUG-LRC-1L-SGDH-X300-HIGH-1-d512-300x-lr1.1
The sweep's second trial of rate 1.1 — above the five rates it normally tests — finished its 10,575 steps at 10:49 a.m. (ET). It ended ahead of the three lowest of those five and behind the two highest, a better showing than its first trial at half the length, which had finished behind all five.
ladderlr
WhenWen AUG-LRC-1L-DENSE-SGDH-018 / 022 / 023
A third of the five rungs without experts at the 10,575-step budget finished under the first optimizer at 10:45 a.m. (ET), at the middle of its five rates; the other two have reached steps 9,451 and 8,397. The sweep also opened two more rungs at the 21,150-step budget without experts, at 10:42 and 10:47 a.m. (ET), at the second and third of those rates.
ladderlrdense
WhenWen AUG-LRC-1L-DENSE-SGDH-016 / 017
The first two of the five rungs without experts at the 10,575-step budget finished under the first optimizer, at 10:34 and 10:41 a.m. (ET), at the two lowest of its five rates. Three are still running there, between steps 6,856 and 10,333.
ladderlrdense
WhenWen AUG-LRC-1L-DENSE-SGDH-021-d512-600x-lr0.1
The sweep opened its first rung without experts at 21,150 steps at 10:36 a.m. (ET), at the lowest of its five rates. That budget is twice the longest it had run without experts and matches the longest it runs with them.
ladderlrdense
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run passed thirty hours of unbroken stepping at 10:14 a.m. (ET), its longest clear stretch since the switch to this run on September 9. It reached step 89,224 at 10:21 a.m. (ET), still taking a step every nineteen seconds, and its thirty-first hourly save of state is due near 10:52 a.m. (ET).
heromilestone
WhenWen AUG-LRC-1L-022 / 023 / 024, AUG-LRC-1L-SGDH-021 / 022
Five more rungs at the sweep's largest token budget finished their 21,150 steps between 10:03 and 10:20 a.m. (ET), leaving three of the ten. One optimizer has only its highest rate still running and puts its best result so far at a middle rate; the other has its two highest rates still running, and there each finished rate beat the one below it.
ladderlr
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run cleared its thirtieth hourly save of state on time at 9:51 a.m. (ET), adding about thirteen seconds across the three steps it spanned. It reached step 89,160 at 10:01 a.m. (ET), still taking a step every nineteen seconds, and passes thirty hours of unbroken stepping at 10:14 a.m. (ET).
herocheckpoint
WhenWen AUG-LRC-1L-SGDH-023, AUG-LRC-1L-021
The first two rungs at the sweep's largest token budget finished their 21,150 steps, at 9:36 and 9:40 a.m. (ET) — the middle of its five rates under one optimizer and the lowest rate under the other. Eight rungs are still running at that budget, between steps 13,233 and 20,211.
ladderlr
WhenWen AUG-LRC-1L-DENSE-MUONH-014 / 015
The sweep closed its middle token budget without experts under the second optimizer, the last two of its five rates finishing their 5,288 steps at 9:03 and 9:07 a.m. (ET). That budget is now complete under both optimizers.
ladderlrdense
WhenWen AUG-LRC-1L-DENSE-MUONH-019 / 020
The sweep opened its last two rungs at the largest token budget without experts at 9:04 and 9:09 a.m. (ET), at the fourth and fifth of its five rates. All five rates now run there under both optimizers, 10,575 steps each.
ladderlrdense
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run cleared its twenty-ninth hourly save of state on time at 8:51 a.m. (ET), adding about twenty seconds across the four steps it spanned. It passed twenty-eight hours of unbroken stepping at 8:14 a.m. (ET) and reached step 88,962 at 9:01 a.m. (ET), still taking a step every nineteen seconds. Its thirtieth save is due near 9:51 a.m. (ET).
herocheckpoint
held h100-d512-mixprior × 40
The third wave of data-mixture tests is complete. All forty rungs have finished their 3,929 steps, the last of them at 8:31 a.m. (ET). No fourth wave has opened.
ladderdata
WhenWen AUG-LRC-1L-DENSE-SGDH-014 / MUONH-011 / 012 / 013
The sweep closed its middle token budget without experts under the first of its two optimizers at 8:40 a.m. (ET), when the rung that had repeated work after a restart finished its 5,288 steps. Under the second optimizer three of the five rates finished there between 8:25 and 8:40 a.m. (ET); the other two have reached steps 4,383 and 4,090.
ladderlrdense
WhenWen AUG-LRC-1L-DENSE-SGDH-019 / 020, MUONH-016 / 017 / 018
The sweep filled its largest token budget without experts under the first optimizer at 8:41 a.m. (ET) with the fifth and last of its five rates, and opened the first three rates there under the second optimizer between 8:31 and 8:42 a.m. (ET). Each of these rungs runs 10,575 steps.
ladderlrdense
WhenWen AUG-LRC-1L-DENSE-SGDH-015-d512-150x-lr0.7
The fourth of the five rungs without experts at the sweep's middle token budget finished its 5,288 steps at 8:23 a.m. (ET), at the highest of the five rates. Only the rung whose process restarted at 7:22 a.m. (ET) is left, and it has reached step 4,286.
ladderlrdense
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run cleared its twenty-eighth hourly save of state on time at 7:51 a.m. (ET), adding about fourteen seconds across the two steps it spanned. It reached step 88,769 at 8:01 a.m. (ET), still taking a step every seventeen seconds, and its twenty-ninth save is due near 8:51 a.m. (ET).
herocheckpoint
held h100-d512-mixprior × 40
Thirty-nine of the forty rungs in the third wave of data-mixture tests have finished their 3,929 steps, the latest at 8:02 a.m. (ET). One rung is left and is stepping steadily, at 3,451.
ladderdata
WhenWen AUG-LRC-1L-DENSE-SGDH-014-d512-150x-lr0.45
The sweep rung whose step counter had stood still at 2,911 since 7:14 a.m. (ET) logged step 2,912 at 8:02 a.m. (ET), carrying on from where it stopped after forty-eight minutes of repeating work it had already recorded. It has since reached step 3,137 of 5,288.
ladderlrdenserestart
WhenWen AUG-LRC-1L-DENSE-SGDH-017 / 018
The sweep opened two more rungs without experts at its longest training length, at 7:44 and 7:47 a.m. (ET), at the second and third of its five rates. Three of the five are now running there under the first of the sweep's two optimizers.
ladderlrdense
WhenWen AUG-LRC-1L-DENSE-SGDH-011 / 012 / 013
Three of the five rungs without experts at the sweep's middle token budget finished their 5,288 steps under the first optimizer, at 7:34, 7:41 and 7:45 a.m. (ET). Of the two left, one has reached step 3,027 and the other has had its counter frozen since 7:14 a.m. (ET) while a replacement process repeats steps it already logged.
ladderlrdense
WhenWen AUG-LRC-1L-DENSE-SGDH-016-d512-300x-lr0.1
The sweep opened its first rung without experts at the largest token budget at 7:35 a.m. (ET), at the lowest of its five rates. It runs 10,575 steps, twice the length of the budget below it, under the first of the sweep's two optimizers.
ladderlrdense
WhenWen AUG-LRC-1L-DENSE-MUONH-010 / 015
The last of the five rates without experts at the second-smallest token budget finished its 2,115 steps at 7:28 a.m. (ET), completing that budget under the second optimizer. The sweep opened its fifth rung at the middle budget two minutes later, so all five rates are running there as well.
ladderlrdense
held h100-d512-mixprior × 40
The rung whose step counter had stood still since 5:21 a.m. (ET) in the third wave of data-mixture tests began logging again at 7:27 a.m. (ET), carrying on from the next step after two hours of repeating work it had already recorded. Both rungs still going are now stepping, at 3,072 and 3,671 of 3,929.
ladderdatarestart
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run passed twenty-seven hours of unbroken stepping at 7:14 a.m. (ET) and reached step 88,631 at 7:22 a.m. (ET), still taking a step every eighteen seconds. Its twenty-eighth hourly save of state has not appeared yet; it is due near 7:51 a.m. (ET).
herocheckpoint
WhenWen AUG-LRC-1L-DENSE-MUONH-009 / 014
Four of the five rates without experts at the sweep's second-smallest token budget have now finished their 2,115 steps under its second optimizer, between 6:47 and 7:23 a.m. (ET). A fourth rung opened at the middle budget at 7:25 a.m. (ET), leaving only the highest rate unopened there.
ladderlrdense
held h100-d512-mixprior × 40
One of the two rungs still going in the third wave of data-mixture tests began logging steps again at 7:07 a.m. (ET), half an hour after its previous process stopped at step 2,837, and carried on from the next step rather than repeating any. It has since reached step 3,199 of 3,929. The other rung has logged nothing since 5:21 a.m. (ET) while its replacement process repeats steps already recorded.
ladderdatarestart
WhenWen AUG-LRC-1L-DENSE-MUONH-010, DENSE-SGDH-014
Two of the sweep's rungs without experts had their processes replaced and are repeating steps they already logged, so their step counters stand still: one stopped at step 2,014 at 6:56 a.m. (ET) and the other at step 2,911 at 7:14 a.m. (ET). Both replacements are doing real work on the GPUs. The data-mixture wave on the same pool shows the same pattern.
ladderlrdenserestart
WhenWen AUG-LRC-1L-SGDH-X300-HIGH-1-d512-300x-lr1.1
The learning-rate sweep opened a second rung at rate 1.1 at 6:47 a.m. (ET), a minute after its first at that rate finished worse than all five lower rates. This one runs 10,575 steps, twice the length of the first, under the same optimizer.
ladderlr
WhenWen AUG-LRC-1L-DENSE-MUONH-011 / 012 / 013
The sweep opened three rungs without experts at its middle token budget under the second optimizer, between 6:49 and 7:03 a.m. (ET). They cover the three lowest of its five rates; the first optimizer reached that budget about an hour earlier.
ladderlrdense
held h100-d512-mixprior × 40
Thirty-eight of the forty rungs in the third wave of data-mixture tests have finished their 3,929 steps, two of them at 6:52 and 7:01 a.m. (ET). The last two each have a fresh process doing real work while their step counters stay frozen, so both are repeating steps they already logged; each rung saves its state only once an hour.
ladderdatarestart
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run cleared its twenty-seventh hourly save of state on time at 6:51 a.m. (ET), holding up stepping by about twenty seconds across four steps. That leaves it just short of twenty-seven hours of unbroken stepping, and the twenty-eighth save is due near 7:51 a.m. (ET).
herocheckpoint
held h100-d512-mixprior × 40
A thirty-sixth of the forty rungs in the third wave of data-mixture tests finished its 3,929 steps at 6:40 a.m. (ET). Of the four still going, one has logged no step since 5:21 a.m. (ET) and a second stopped at 6:37 a.m. (ET), both with processes that keep reporting in to W&B.
ladderdatacrash
WhenWen AUG-LRC-1L-DENSE-SGDH-009 / 015
The last of the sweep's five models without experts at the second-smallest token budget finished its 2,115 steps at 6:44 a.m. (ET), completing that set under the first of its two optimizers. A fifth rung opened at the middle budget two minutes later, so all five rates are now running there as well.
ladderlrdense
WhenWen AUG-LRC-1L-SGDH-X150-HIGH-1-d512-150x-lr1.1
The sweep's rung at rate 1.1 finished its 5,288 steps at 6:45 a.m. (ET), two hours after it opened. It ended worse than all five lower rates at the same token budget, and the lowest there is still the rung at 0.45.
ladderlr
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run passed twenty-six hours of unbroken stepping at 6:14 a.m. (ET). It cleared its twenty-sixth hourly save of state on time at 5:51 a.m. (ET), holding up stepping by about fourteen seconds across four steps, and the twenty-seventh is due near 6:51 a.m. (ET).
herocheckpoint
held h100-d512-mixprior × 40
Two of the four stalled rungs in the third wave of data-mixture tests began stepping again at 6:20 and 6:24 a.m. (ET), one after fifty-four minutes without a step and the other after three hours and eight minutes. Each carried on from the step it stopped at. Thirty-five of the forty rungs have now finished, and the two rungs still quiet have fresh processes that have yet to log a step.
ladderdatarestart
WhenWen AUG-LRC-1L-DENSE-MUONH-009 / 010, SGDH-013 / 014
The learning-rate sweep opened four more models without experts between 6:05 and 6:16 a.m. (ET). Its second optimizer now covers all five rates at the second-smallest token budget, and the first optimizer has four of the five open at the middle budget.
ladderlrdense
held h100-d512-mixprior × 40
Thirty-two of the forty rungs in the third wave of data-mixture tests have now run their full 3,929 steps, four of them since 5:50 a.m. (ET). Four of the eight still going have logged no step for between twenty minutes and two hours fifty minutes while their processes keep reporting in; the longest has stood at step 2,572 since 3:15 a.m. (ET).
ladderdatacrash
WhenWen AUG-LRC-1L-DENSE-SGDH-006 / 007 / 008
Three of the five rungs without experts at the sweep's second-smallest token budget finished their 2,115 steps between 5:56 and 6:03 a.m. (ET), under the first of its two optimizers. The remaining two there are at step 156 and step 1,992.
ladderlrdense
WhenWen AUG-LRC-1L-DENSE-SGDH-009-d512-60x-lr0.45
The sweep's rung at rate 0.45 and the second-smallest token budget, reported stopped at 5:38 a.m. (ET) without a step, began stepping at 6:01 a.m. (ET) under a later process. That is thirty-eight minutes between the rung opening and its first step.
ladderlrdenserestart
WhenWen AUG-LRC-1L-DENSE-SGDH-011 / 012
The sweep opened its first two rungs without experts at the middle of its five token budgets, 5,288 steps, at 5:58 and 6:01 a.m. (ET). The models without experts had reached only the two smallest budgets until now.
ladderlrdense
WhenWen AUG-LRC-1L-DENSE-MUONH-006 / 007 / 008
The sweep opened three rungs without experts at the second-smallest token budget under its second optimizer, between 5:50 and 5:51 a.m. (ET). They cover the three lowest of its five rates; the matching rungs under the first optimizer were finishing at the same time.
ladderlrdense
held h100-d512-mixprior × 40
Four of the eleven rungs still going in the third wave of data-mixture tests have logged no step for between twenty minutes and two and a half hours, while their processes keep reporting in to W&B. The longest has stood at step 2,572 of 3,929 since 3:15 a.m. (ET); twenty-nine of the forty rungs have now finished, the newest at 5:43 a.m. (ET).
ladderdatacrash
WhenWen AUG-LRC-1L-DENSE-MUONH-004 / 005
The learning-rate sweep opened its last two models without experts under the second optimizer at 5:40 a.m. (ET), at the smallest token budget. All five of its rates now have a dense counterpart under both optimizers it tests, and three of the five have finished.
ladderlrdense
WhenWen AUG-LRC-1L-DENSE-SGDH-009-d512-60x-lr0.45
The sweep's dense rung at rate 0.45 and the second-smallest token budget stopped at 5:38 a.m. (ET) without logging a step, fifteen minutes after it opened. The other four rates at that budget are stepping, between step 734 and step 1,588 of 2,115.
ladderlrdensecrash
held h100-d512-mixprior × 40
All five rungs of the third wave of data-mixture tests that stopped between 4:55 and 5:01 a.m. (ET) had a fresh process take over between 5:17 and 5:26 a.m. (ET). Each saves its state only once an hour, so all five repeat work. A sixth rung stopped at 5:27 a.m. (ET) at step 3,121 of 3,929.
ladderdatarestart
held h100-d512-mixprior × 40
Twenty-three of the forty rungs in the third wave of data-mixture tests have now run their full 3,929 steps, seven of them since 5:01 a.m. (ET) and the newest at 5:26 a.m. (ET). The sixteen still going range from step 2,263 to step 3,928.
ladderdata
WhenWen AUG-LRC-1L-DENSE-SGDH × 5
The learning-rate sweep opened all five of its rates at the second-smallest token budget for models without experts, the last two at 5:23 a.m. (ET). The rung at the lowest rate, which three processes had opened without logging a step, began stepping under a fourth at 5:09 a.m. (ET).
ladderlrdense
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run passed twenty-five hours of unbroken stepping at 5:14 a.m. (ET). It cleared its twenty-fifth hourly save of state on time at 4:51 a.m. (ET), holding up stepping by about sixteen seconds across four steps, and the twenty-sixth is due near 5:51 a.m. (ET).
herocheckpoint
WhenWen AUG-LRC-1L-DENSE-MUONH-002 / 003
The sweep opened two more models without experts under its second optimizer at 5:09 a.m. (ET), at the smallest token budget. Three of the five rates there now have a dense counterpart to the 128-expert rungs.
ladderlrdense
WhenWen AUG-LRC-1L-DENSE-SGDH-006-d512-60x-lr0.1
The learning-rate sweep opened its first model without experts at the second-smallest token budget at 4:51 a.m. (ET), 2,115 steps. A second process took it over at 4:58 a.m. and a third at 5:09 a.m., and it has yet to log a step.
ladderlrdensecrash
WhenWen AUG-LRC-1L-DENSE-SGDH × 5
All five of the sweep's models without experts finished their 1,058 steps at the smallest token budget, the last four between 5:08 and 5:09 a.m. (ET). The sweep now has a full set of dense results at that budget under the first of its two optimizers.
ladderlrdense
held h100-d512-mixprior × 40
Five rungs in the third wave of forty data-mixture tests stopped between 4:55 and 5:01 a.m. (ET), all of them past step 2,500 of the 3,929 each is set to run, and none had restarted by 5:12 a.m. (ET). Sixteen rungs have now finished, the sixteenth at 4:51 a.m. (ET).
ladderdatacrash
held h100-d512-mixprior × 40
Fifteen of the forty rungs in the third wave of data-mixture tests have now finished their 3,929 steps, all between 4:22 and 4:47 a.m. (ET). The rungs that a restart set back trail well behind — the furthest has 2,098.
ladderdata
WhenWen AUG-LRC-1L-023 / SGDH-021
The two rungs at the sweep's largest token budget that stopped at 4:14 a.m. (ET) and were marked crashed resumed at 4:47 and 4:50 a.m. (ET), one under each optimizer. Each carried on from the step it stopped at, so neither repeated work, and both are now near step 10,000 of 21,150.
ladderlrrestart
WhenWen AUG-LRC-1L-DENSE-MUONH-001-d512-30x-lr0.1
The learning-rate sweep finished its first model without a mixture of experts at 4:43 a.m. (ET), 1,058 steps at the smallest token budget, and opened the matching model under its second optimizer two minutes later. The dense comparison now runs under both optimizers the sweep is testing.
ladderlrdense
WhenWen AUG-LRC-1L-SGDH-X150-HIGH-1-d512-150x-lr1.1
The learning-rate sweep opened a rung at 1.1 at 4:38 a.m. (ET), a rate above the five it has tested so far, which topped out at 0.7. It runs the 5,288 steps of the sweep's middle token budget under the first of the two optimizers.
ladderlr
held h100-d512-mixprior × 40
The first rungs of the third wave of forty data-mixture tests reached the end of their run at 4:22 a.m. (ET), just under three hours after the wave began, and five had finished by 4:24 a.m. (ET). Each rung starts a tenth of the way in, from a shared checkpoint, and the slowest are still near half way.
ladderdata
WhenWen AUG-LRC-1L-DENSE-SGDH × 5
The learning-rate sweep started five new rungs at its smallest token budget between 4:12 and 4:15 a.m. (ET), one per rate. They carry no experts, unlike the 128-expert models in every rung so far, so they give the sweep a dense comparison under one of its two optimizers.
ladderlrdense
WhenWen AUG-LRC-1L-SGDH-025-d512-600x-lr0.7
The learning-rate sweep filled its largest token budget at 4:15 a.m. (ET) with the tenth and last rung there, so both optimizers now run all five rates over the full 21,150 steps. The rungs opened earliest at that budget are about a third of the way through.
ladderlr
WhenWen AUG-LRC-1L-023 / SGDH-021
Two rungs at the sweep's largest token budget stopped five seconds apart at 4:14 a.m. (ET) and were marked crashed, one under each optimizer. Both were about a third of the way through and neither has been relaunched.
ladderlrcrash
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
The run passed twenty-four hours of unbroken stepping at 4:14 a.m. (ET). It cleared its twenty-fourth hourly save of state on time at 3:50 a.m. (ET), holding up stepping by about nine seconds across two steps, and the twenty-fifth is due near 4:50 a.m. (ET).
herocheckpoint
held h100-d512-mixprior × 40
Restarts in the third wave of forty data-mixture tests have reached twenty-eight of the forty, three of them since 3:10 a.m. (ET). Those three are repeating work now — each has a live process but has logged no new step, because W&B refuses a step at or below the one it last recorded. The twelve rungs never restarted lead at about step 3,170 of the 3,930 each is set to run; the furthest behind has 1,366.
ladderdatarestart
WhenWen AUG-LRC-1L-SGDH-024-d512-600x-lr0.45
The learning-rate sweep opened a ninth rung at its largest token budget at 3:39 a.m. (ET), the fourth of its five rates under the second optimizer, so only the highest rate there is still unopened. One of the rungs already at that budget was restarted at 2:55 a.m. (ET) after twenty-two minutes without a step and now trails its five siblings by about a thousand steps.
ladderlr
held h100-d512-mixprior × 40
The quiet rungs in the third wave of forty data-mixture tests are restarts, not hangs. Twenty-five of the forty have had a fresh process take over their run since the wave began at 1:22 a.m. (ET), and thirteen were still repeating work at 3:00 a.m. (ET) — each saves its state only once an hour, so a rung that falls over early has to redo everything. The fifteen never restarted lead at about step 2,340 of the 3,930 each is set to run; the furthest behind has 853.
ladderdatarestart
hero-ragged_a2a-nccl2307-ep-step81k
The run reaches twenty-three hours of unbroken stepping at 3:14 a.m. (ET). It cleared its twenty-third hourly save of state on time at 2:50 a.m. (ET), holding up stepping by about twenty seconds across three steps, and the twenty-fourth is due near 3:50 a.m. (ET).
infracheckpointing
held h100-d512-mixprior × 40
The stalls in the third wave of forty data-mixture rungs have spread. Five rungs have logged no step for between ten and thirty minutes while their processes stay alive and report in, and nine now trail the leaders — near step 1,900 of the 3,930 each is set to run — by more than four hundred steps.
ladderdatahang
held h100-d512-mixprior × 40
The third wave of forty data-mixture rungs has begun to go quiet the way the wave before it did. Two rungs have logged no step since 1:59 and 2:00 a.m. (ET) while staying alive and reporting in, and a third was quiet for twenty-four minutes before resuming at 2:12 a.m. (ET); the leaders are near step 1,490 of the 3,930 each is set to run.
ladderdatahang
WhenWen AUG-LRC-1L-025-d512-600x-lr0.7
The learning-rate sweep opened an eighth rung at its largest token budget at 2:23 a.m. (ET), the fifth and last of its rates there, so one of the two optimizers it compares now runs every rate at that budget. The other optimizer has three of five open there and two unfinished rungs at the budget below.
ladderlr
WhenWen AUG-LRC-1L-024-d512-600x-lr0.45
The learning-rate sweep closed its second-largest token budget under one of its two optimizers: the last two of its five rates finished 10,574 steps at 2:18 and 2:20 a.m. (ET). A seventh rung opened at the largest budget at 2:20 a.m. (ET), adding a fourth rate there, while the two matching rungs under the other optimizer are still stepping.
ladderlr
hero-ragged_a2a-nccl2307-ep-step81k
The run cleared its twenty-second hourly save of state at 1:50 a.m. (ET), on time and in three steps that each took about five seconds longer than usual. The twenty-third is due near 2:50 a.m. (ET), and the run reaches twenty-two hours of unbroken stepping at 2:14 a.m. (ET).
infracheckpointing
hero-ragged_a2a-nccl2307-ep-step81k
The run passed twenty-one hours of unbroken stepping at 1:14 a.m. (ET). It cleared its twenty-first hourly save of state at 12:50 a.m. (ET), holding up stepping by about nine seconds across two steps, and the twenty-second is due near 1:50 a.m. (ET).
infracheckpointing
WhenWen AUG-LRC-1L × 6
The learning-rate sweep opened a fifth and largest token budget between 1:37 and 1:39 a.m. (ET), with six rungs at the three lowest of its five rates under both optimizers. Each is set to run 21,150 steps, twice the length of the rungs at the budget that was largest until now; four of those earlier rungs, at the two highest rates, are still stepping.
ladderlr
held h100-d512-mixprior × 40
A third wave of data-mixture rungs started at 1:22 a.m. (ET), forty at once — twice the size of each of the two batches run the day before. Every one carries a weighting that none of the earlier forty used, on the same small model shape and the same run length.
ladderdata
hero-ragged_a2a-nccl2307-ep-step81k
The run passed twenty hours of unbroken stepping at 12:14 a.m. (ET). It cleared its twentieth hourly save of state at 11:50 p.m. (ET), holding up stepping by about eleven seconds across two steps, and the twenty-first is due near 12:50 a.m. (ET). It also paused about seventy-five seconds at 11:01 p.m. (ET) to score itself on held-out text, its second such score in this run and an improvement on the first, taken fifteen hours earlier.
infracheckpointingeval
Larry #hero-run-2026
Larry answered Sheng Zha's router proposal at 12:22 a.m. (ET), calling the unit-norm router weights worth trying and the sampling idea new to him. He said the two warning signs behind the proposal do not apply here: the router loss term that was creeping up is recorded for inspection but not trained against, and the dropped tokens came from the first expert-parallel implementation, which the current one has all but removed.
routerstability
WhenWen AUG-LRC-1L × 40
A forty-model learning-rate sweep for the hero recipe began stepping on TPUs at 5:26 p.m. (ET) and has run through the night. It covers five learning rates, from a tenth to seven tenths of the rate the recipe recommends, across four token budgets and two optimizers, on one-layer 128-expert models of width 512. Thirty rungs have finished; the last ten, all on the largest budget, started between 9:33 p.m. and 12:07 a.m. (ET).
ladderlr
hero-ragged_a2a-nccl2307-ep-step81k
The run has now stepped without a break for close to nineteen hours. It cleared its eighteenth and nineteenth hourly saves of state at 9:49 and 10:51 p.m. (ET), holding up stepping by about eleven and five seconds — the two lightest saves of the stretch — and the twentieth is due near 11:51 p.m. (ET).
infracheckpointing
hero-ragged_a2a-nccl2307-ep-step81k
The run has now stepped without a break for more than sixteen and a half hours, its longest unbroken stretch on the software build put into production on September 9. It cleared its fifteenth, sixteenth and seventeenth hourly saves of state on time in the 6:48, 7:48 and 8:48 p.m. (ET) slots, each holding up stepping by about fifteen seconds or less — the lightest run of saves so far.
infracheckpointing
held h100-d512-mixprior × 20
The twentieth and last rung of the second batch finished its 3,929 steps at 8:40 p.m. (ET), so all twenty weightings now have a score at the same length. Its fourth quiet spell ran forty-five minutes and ended on its own, like the three before it, and it then ran its final 120 steps in seven minutes.
ladderdata
Sheng Zha #hero-run-2026
Sheng Zha proposed four changes to the router — the part that picks which experts see each token — to hold training steady: scale each row of the router weight to unit length, add noise to the routing or sample from it rather than taking the top experts outright, and subtract the mean from the router bias. Sheng Zha reported training stably to trillion-parameter scale from scratch with those changes and no later patching, and said the weight decay put on the router on September 4 appears to have stopped the drift in the router's own loss terms.
routerstability
held h100-d512-mixprior-8b014e07
The last unfinished rung of the second batch has logged no step since 7:48 p.m. (ET), at step 3,809 of 3,929, while its process stays alive and reports in. It is this rung's fourth long quiet spell today, after gaps of eighty-one, ninety-four and thirty-seven minutes; each earlier one ended on its own.
ladderhang
Sep 10
held h100-d512-mixprior × 20
Nineteen of the twenty data-mixture rungs have now finished all 3,929 steps, the last four between 6:54 and 7:09 p.m. (ET). That includes the rung reported here as 61 steps short: its replay from the last save passed the old mark and ran to the end, so its weighting needs no rerun.
ladderdata
held h100-d512-mixprior-f297c5b7
A fifteenth rung of the second batch finished its 3,929 steps at 6:29 p.m. (ET), the fifth to finish in the past hour. Four rungs are stepping normally again after their quiet spells, each logging a step within the last minute, which leaves one rung of the twenty outstanding.
ladder
mcwitt lc-main-ragged-4k-0910
mcwitt ran the hero's own model shape for 40 steps on 64 GB200 GPUs at the 4,096 tokens per sequence the hero trains on, restoring the pinned step-81716 checkpoint. It finished in eighteen minutes and used 24.8 percent of peak compute, which gives the same-day test of 32,768-token sequences a reference point at today's length.
seqlenep
held h100-d512-mixprior-09c11815
The rung that stopped 61 steps short was picked up by a new process at 6:03 p.m. (ET), and W&B moved it back from finished to running. Its GPU is working, but its step counter is still at the step it reached at 5:16 p.m. (ET): W&B refuses a step at or below a run's last logged one, so the replay from its last save stays invisible until it passes that mark.
ladderhang
held h100-d512-mixprior-09c11815
The rung quiet since 5:16 p.m. (ET) ended at step 3,868, 61 steps short of the 3,929 its siblings run, and W&B marks it finished rather than crashed — its last score on held-out text is from step 3,723, so its weighting has no score at the length the eleven completed rungs report. Two other quiet rungs resumed stepping at 5:56 and 6:00 p.m. (ET), leaving three of the eight rungs still running quiet, two of them for over an hour.
ladderhang
held h100-d512-mixprior × 20
The quiet spells in the second batch of twenty data-mixture rungs have spread. Six of the nine unfinished rungs have logged no step for fifteen minutes or more — two of them for over an hour, and one stopped 61 steps short of its 3,929-step finish — though all stay alive and report in; the rung quiet since 4:56 p.m. (ET) resumed after forty-five minutes and an eleventh rung finished at 5:33 p.m. (ET).
ladderhang
hero-ragged_a2a-nccl2307-ep-step81k
The run cleared its fourteenth hourly save of state in the 5:48 p.m. (ET) slot it was due, slowing four steps by about five seconds in total — the lightest hold of any of its saves, against about ten seconds at the save before it. It has now stepped without a break for close to thirteen and a half hours.
infracheckpointing
held h100-d512-mixprior-8b014e07
The rung reported as stopped at step 2,673 has not stopped: W&B still lists it as running with a heartbeat one minute old, and it has simply logged no step since 4:43 p.m. (ET). Three of the twenty rungs are quiet in that way, the others since 4:20 p.m. and 4:56 p.m. (ET), while ten have finished all 3,929 steps and seven are still stepping.
ladderhang
held h100-d512-mixprior-a6b2cadc
The rung that had logged no step since 3:00 p.m. (ET) started stepping again at about 5:08 p.m. (ET), from step 2,223, after two hours quiet. Two other rungs are still quiet, one since 4:20 p.m. (ET) and one since 4:56 p.m. (ET).
ladderhang
held h100-d512-mixprior-8b014e07
A rung of the second batch ended at step 2,673 of 3,929, twenty minutes after its last logged step at 4:43 p.m. (ET). It is the first quiet rung to stop without finishing, and its weighting will need a rerun; ten other rungs have completed all their steps.
ladderhang
hero-ragged_a2a-nccl2307-ep-step81k
The run cleared its thirteenth hourly save of state in the 4:48 p.m. (ET) slot it was due, holding up stepping for about a minute across two steps — half the hold of the save before it. It has now stepped without a break for close to thirteen hours.
infracheckpointing
held h100-d512-mixprior-7026cf51
A second rung of the second batch has logged no step since 4:20 p.m. (ET), at step 2,659 of 3,929, while its process stays alive and reports in — the same pattern as the rung stopped since 3:00 p.m. (ET). Six of the twenty rungs have now finished and twelve are still stepping.
ladderhang
mcwitt #8934
mcwitt closed the issue that had tracked this cluster's fabric faults since September 5 and moved its evidence to #9086. He kept one finding from it: the GPU link it named five days ago is the link that failed at 4:08 a.m. (ET) and took the hero down.
incidenthardware
held h100-d512-mixprior × 20
The first four of the second batch of twenty data-mixture rungs finished their 3,929 steps between 4:22 and 4:25 p.m. (ET). Fifteen are still stepping, the slowest a third of the way through.
ladderdata
held h100-d512-mixprior-a6b2cadc
One rung of the second batch has logged no step since 3:00 p.m. (ET), at step 2,223 of 3,929, while its process stays alive and reports in. Its siblings, launched in the same minute, are past step 2,800.
ladderhang
mcwitt lc-ragged-cp2-smoke-0910
mcwitt ran a ten-step test on a single GB200 node that splits a 32,768-token sequence across two GPUs — eight times the 4,096 tokens the hero trains on — while routing tokens to experts over the transport the hero uses. It trained from scratch and finished in 88 seconds with no hang.
seqlenep
mcwitt #9086
A new issue asks for the per-link error counters of the GPU-to-GPU fabric to be collected and kept. The link that failed at 4:08 a.m. (ET) had been climbing for five days, but the only record of that came from a sweep run by hand at a terminal: the tables the dashboards read hold no rows at all, in any cluster. It also puts the two rack-scale link failures of the last month at about the rate a published Llama 3 run reported, and training availability at 96 percent over the last fourteen days.
incidenthardwaremonitoring
hero-ragged_a2a-nccl2307-ep-step81k
The run cleared its twelfth hourly save of state in the 3:48 p.m. (ET) slot it was due, holding up stepping for about two minutes across five steps — the same hold as the eleventh. Stepping returned to its normal pace straight after, so the run has stepped without a break for nearly twelve hours and the thirteenth save is due near 4:48 p.m. (ET).
infracheckpointing
hero-ragged_a2a-nccl2307-ep-step81k
The run cleared its eleventh hourly save of state in the 2:48 p.m. (ET) slot it was due, holding up stepping for about two minutes across five steps — twice the hold of the ninth and tenth, and the longest since the eighth. Stepping returned to its normal pace straight after, so the run has stepped without a break for nearly eleven hours and the twelfth save is due near 3:48 p.m. (ET).
infracheckpointing
held h100-d512-mixprior × 20
Eighty minutes in, the second batch of twenty data-mixture rungs has split in two. Thirteen are about half way through their 3,930 steps and on course to finish in under three hours, against the eight the morning batch took; the other seven are between a sixth and a third of the way through, at roughly a third the pace, and all twenty still report progress.
ladderdata
mcwitt #8870
The procedure for capturing evidence from a hang while it is happening is now validated, with its steps and its limits written up. The source audit posted fifty minutes earlier could not name the operation the job blocks on, so this capture is what the next hang would have to supply.
hangdebugging
mcwitt #9082 open
Adds a projected finish date for the hero to the run dashboard, so the cost of a stop is visible as a date rather than as lost hours. The estimate divides progress by wall clock over whatever window the dashboard shows, which counts downtime and replayed steps against the run; on live data it read November 18.
monitoring
hero-ragged_a2a-nccl2307-ep-step81k
The run cleared its tenth hourly save of state in the 1:48 p.m. (ET) slot it was due, holding up stepping for about fifty seconds — the same short hold as the ninth, and the two briefest of the ten. Stepping has held its normal pace since, so the run has stepped without a break for more than nine and a half hours and the next save is due near 2:48 p.m. (ET).
infracheckpointing
mcwitt #8870
An audit of the source code for the transport that moves data between GPUs did not explain the hang. It found two ways that code can deadlock, but both are in paths the hero does not use, and it found no missing synchronisation in the path the hero does use — though a step stalled earlier in the chain could still leave the transport waiting, so the audit does not clear the software.
hangdebugging
mcwitt #9077
A new issue proposes replacing the hand-written kernels that run the hero's expert layers with a standard operation from the compiler, so the project no longer maintains its own copy of that code. A one-rack test restored from the hero's step-54000 checkpoint matched the current kernels on loss and came within about one point of them on compute use, but two of the four one-rack test runs hung with the same silent signature as #8870 while none of the current-kernel comparison runs did.
kernelshang
held h100-d512-mixprior × 20
A second batch of twenty data-mixture rungs started between 1:24 and 1:26 p.m. (ET), less than five hours after the morning's twenty finished without separating one weighting from another. All twenty weightings are new — none repeats one from the morning batch — and each rung again trains a small model for 3,930 steps on one eight-GPU node; forty minutes in, each is about a quarter of the way through, well ahead of the eight hours the morning batch took. No issue or status-log entry covers the batch.
ladderdata
hero-ragged_a2a-nccl2307-ep-step81k
The run cleared its ninth hourly save of state in the 12:47 p.m. (ET) slot it was due, holding up stepping for about fifty seconds — half the hold of the save before it. Stepping has held its normal pace since, so the run has stepped without a break for more than eight and a half hours and the next save is due near 1:48 p.m. (ET).
infracheckpointing
hero-ragged_a2a-nccl2307-ep-step81k
The run has now stepped for longer than the attempt that hung this morning: it passed that attempt's seven-hour-fifty-nine-minute life at 12:13 p.m. (ET) and has held its normal pace since. That makes this the longest unbroken stretch on the software build put into production yesterday, and the ninth hourly save of state is still due near 12:47 p.m. (ET).
infraep
hero-ragged_a2a-nccl2307-ep-step81k
The run cleared its eighth hourly save of state, holding up stepping for about two minutes from 11:47 a.m. (ET), and has now stepped without a break for nearly eight hours. That approaches the age at which this morning's first stop struck, one step after the eighth save of the attempt before it.
infracheckpointing
hero-ragged_a2a-nccl2307-ep-step81k
The run cleared its seventh hourly save of state, holding up stepping for about a minute and a quarter from 10:47 a.m. (ET), and has now stepped without a break for seven hours since the restart. The stretch that ended in this morning's first stop ran just under eight hours on the same build, so the run is close to matching its best showing on this software; what training uses of each GPU's memory has still not moved since the 7:46 a.m. (ET) held-out scoring, and the next save is due near 11:47 a.m. (ET).
infracheckpointing
hero-ragged_a2a-nccl2307-ep-step81k
The run passed six hours of unbroken stepping since the restart and cleared its sixth hourly save of state, holding up stepping for a little over a minute and a half from 9:47 a.m. (ET). That save came an hour after the one before it, so the save cadence the 7:46 a.m. (ET) held-out scoring reset is holding, and the next is due near 10:47 a.m. (ET). What training uses of each GPU's memory has still not moved since that scoring, and steps have come every eighteen seconds throughout.
infracheckpointing
hero-ragged_a2a-nccl2307-ep-step81k
The near-full memory every sampled GPU has held since the 7:46 a.m. (ET) held-out scoring is set aside, not in use. The run's own counters show what training uses unchanged at 35.6 GiB a card across the two hours since, while the most it has ever set aside rose at that step from 103.6 to 120.9 GiB against a 138.2 GiB limit — so the program kept the larger working area the scoring needed and about 17 GiB a card stays free at that high mark. Stepping has held its normal pace and two hourly saves of state have completed since.
memoryeval
hero-ragged_a2a-nccl2307-ep-step81k
The hourly save of state came at 8:46 a.m. (ET), one hour to the minute after the run finished scoring itself on the held-out sets — five step gaps of 21 to 30 seconds against a 17-second median, about two minutes of held-up stepping. So the scoring, not the 8:12 a.m. (ET) slot it appeared to miss, sets the save cadence now; the write went through with every sampled GPU still holding the near-full memory that scoring took, and stepping returned to its normal pace straight after.
infracheckpointing
held h100-d512-mixprior × 20
All twenty data-mixture rungs have now finished their full 3,930 steps, the last of them at 8:39 a.m. (ET). Its score on the Paloma held-out set, 3.320, lands mid-band: the twenty span 3.304 to 3.338, the same range the first nineteen set, so no one weighting of the mixture separated itself and the best rung sits about three hundredths below the worst. No issue or status-log entry covers the batch or its result.
ladderdata
hero-ragged_a2a-nccl2307-ep-step81k
The hourly save of state did not come at the 8:12 a.m. (ET) slot the three saves before it used, and none had appeared by 8:47 a.m. (ET); stepping through that window held its normal pace. A pause with the save signature — five step gaps of 22 to 30 seconds against a 17-second median — came instead at 7:49 a.m. (ET), four minutes after the held-out scoring, which points to the save timer restarting from that write and puts the next save near 8:49 a.m. (ET). The run is configured to save once an hour, and this morning's first stop struck one step after a save.
infracheckpointing
hero-ragged_a2a-nccl2307-ep-step81k
The run scored itself on the held-out sets for the first time since the restart, holding up stepping for about four minutes from 7:42 a.m. (ET). The average over the sixteen Paloma sets came out at 2.234603, above the 2.23162 of three thousand steps earlier and the first rise after four straight falls; seven of the sixteen sets rose and nine fell, and the four the [September 9 argument](https://discord.com/channels/1354881461060243556/1546615008706830388/1547479632394190868) turned on — gab, ptb, wikitext_103 and twitterAAE — rose together, against that day's reading that they had stopped moving as a group. The second held-out suite, uncheatable_eval, improved over the same interval. The check also left every sampled GPU holding about a tenth more memory than before, near the limit of the card, and it has not been given back in the seventy steps since, though training has read its normal profile throughout; the [August 20 fault](https://github.com/marin-community/marin/issues/8508) that killed all 704 processes was an allocation that could not find room.
evalmemory
held h100-d512-mixprior × 19
Nineteen of the twenty data-mixture rungs have now finished their full 3,930 steps, the last two at 6:47 and 7:19 a.m. (ET). Their scores on the Paloma held-out set span 3.304 to 3.338, so no one weighting of the mixture has separated itself; the twentieth rung is about three quarters of the way through.
ladderdata
hero-ragged_a2a-nccl2307-ep-step81k
The run reached three hours of unbroken stepping since the restart and cleared its third hourly save of state, holding up stepping for about two minutes from 7:11 a.m. (ET). This morning's first stop came one step after a save, so each save cycle passed is evidence that the fault has not come back.
infraepcheckpointing
hero-ragged_a2a-nccl2307-ep-step81k
The run cleared its second hourly save of state since the restart, holding up stepping for about two minutes from 6:11 a.m. (ET). Its first stop this morning came at exactly this point in the save cycle, so the run has now passed that point twice without fault.
infraepcheckpointing
held h100-d512-mixprior × 17
Seventeen of the twenty data-mixture rungs have now finished their full 3,930 steps, the last of them at 5:50 a.m. (ET). Their scores on the Paloma held-out set span 3.304 to 3.338, so no one weighting of the mixture has separated itself; three rungs are still stepping.
ladderdata
The replay finished. The run passed step 83,486 — the last step before the 4:08 a.m. (ET) NVLink fault — at 5:07 a.m. (ET), after fifty-nine minutes of re-running work that W&B does not record, and has stepped every seventeen seconds since. It then held up stepping for about two minutes from 5:12 a.m. (ET) to write its first save of state since the restart, so it has now cleared the point in the save cycle where this morning's first hang struck.
mcwitt #8506
mcwitt corrects the cause of the 4:08 a.m. (ET) stop: an uncorrectable NVLink hardware error on four tasks in rack index 9, not the silent hang. The scheduler retried at once, and the new attempt has to replay about fifty-five minutes because the failed one never reached its hourly save. That rack has held the same physical nodes in every placement this week and was already flagged for slow traffic between racks, which mcwitt calls the strongest hardware evidence yet that the #8870 hangs start in the fabric.
infraepincident
hero-ragged_a2a-nccl2307-ep-step81k
A new process took over the run at 4:11 a.m. (ET), under three minutes after the last step — far quicker than the sixteen minutes the stall detector took on the hang before. It reloaded a save of state and read the normal training profile from 4:14 a.m. (ET). Because the newest save dates from 2:55 a.m. (ET) and the failed attempt wrote none of its own, the run is repeating roughly fifty minutes of stepping, and W&B records none of it: the step count stays at 4:08 a.m. (ET)'s mark until the run passes it, at about 5:08 a.m. (ET).
infraepincident
hero-ragged_a2a-nccl2307-ep-step81k
The run stopped a second time, seventy-five minutes after the first, at step 83,486 logged at 4:08 a.m. (ET). Every sampled GPU then read fully busy with no memory or NVLink traffic, which looks like the silent hang; the job's own logs, posted at 4:12 a.m. (ET), instead show an uncorrectable NVLink hardware error that killed four tasks in one rack.
infraepincident
loom-oa-dev #8506
The incident record for the 2:55 a.m. (ET) hang says what it was not. The cluster's own diagnostic interface stayed responsive with all seventy-six communication groups alive, yet sixty-five of them held frozen, unequal operation counts across ranks; no Kubernetes disruption, GPU fault or network error came first. Which rank, kernel or network event starts it remains unknown.
infraepincident
held h100-d512-mixprior × 6
Six of the twenty data-mixture rungs started at 12:36 a.m. (ET) have finished their full 3,930 steps, the six of them between 3:34 and 3:41 a.m. (ET). Their scores on the Paloma held-out set land in a narrow band, 3.309 to 3.338, so no one weighting of the mixture stands out yet; the other fourteen are still stepping.
ladderdata
mcwitt #8506
mcwitt confirms the recovery and corrects the save it restored: `step-83305`, not the `step-83306` named minutes earlier. The new attempt logged its first step at 3:15 a.m. (ET), repeated three steps at losses identical to the ones before the hang — evidence the restore was faithful — and passed the step that had hung at 3:16 a.m. (ET). Cost: twenty-one minutes and three repeated steps.
infraepincident
mcwitt #8506
mcwitt logs the 2:55 a.m. (ET) hang as the first on the NCCL 2.30.7 build, after about 1,590 clean steps. The rack holding the lead process dropped to about half the CPU use of the other ten, and a 3:02 a.m. (ET) sample showed one member of every cross-rack collective exactly three operations behind, with no error printed. The stall detector killed the job at 3:10 a.m. (ET), 924 seconds past the step start, and the scheduler retried. The tally now stands at four hangs in four attempts on the older build against one in about 1,900 steps on this one.
infraepincident
hero-ragged_a2a-nccl2307-ep-step81k
The run is training again. A new process took over the run at 3:12 a.m. (ET), seventeen minutes after its last step; the GPUs went idle and released their memory at 3:13 a.m. (ET), reloaded, and read the normal training profile from 3:15 a.m. (ET). The first step after the restart came at 3:16 a.m. (ET) and was 83,307 — the step straight after the last one logged before the hang — so no steps were repeated and about twenty-one minutes of wall-clock time were lost. Steps have come every nineteen seconds since. No status-log entry covers the hang or the restart.
infraepincident
hero-ragged_a2a-nccl2307-ep-step81k
The run stopped taking steps. Step 83,306, logged at 2:54 a.m. (ET), is its last; from 2:57 a.m. (ET) every sampled GPU reads fully busy with no memory or NVLink traffic and allocated memory frozen — the idle-but-pegged signature of the #8870 silent hang. The job is still alive with a current heartbeat, so W&B still reports the run as running and no restart has fired. It went quiet at the point its eighth hourly save of state was due, after seven that each completed.
infraepincident
Larry #hero-run-2026
Larry reads the training loss as descending overall but swinging up and down in clusters along the way, and asks whether the held-out loss moves the same way step by step. The guess offered is that a few parameter groups, such as the output layer, circle their best value over a few hundred steps. Nobody has replied.
training-dynamics
mcwitt #8506
mcwitt records the checkpoint test and says why it mattered. This run saves its weights in a new layout, holding the full-precision copy on the GPUs; every restore until now had read a save from the older run in the previous layout and converted it while loading, so the new layout had never been read back. A one-rack diagnostic job restored the hero's own 12:54 a.m. (ET) save and trained twenty steps from it, at a mean loss of 1.247 with the router's balance measures in their normal range — a damaged save reads as a loss above 2. All sixteen tasks exited cleanly, nothing was written into the run's own files, and that save is now pinned so the hero's next save cannot delete it, leaving a point to fall back to for fourteen days.
infracheckpointsep
mcwitt restore-nccl2307-82906
A side job read the hero's newest hourly save back and trained from it. `restore-nccl2307-82906` started at 1:06 a.m. (ET), loaded the step-82,906 checkpoint the hero had written about ten minutes earlier, and logged twenty steps, 82,906 to 82,925, between 1:11 and 1:17 a.m. (ET) before stopping at its configured limit; W&B holds it finished. It ran on 16 tasks against the hero's 176 — one rack of the eleven, and a batch one eleventh the size — and wrote no checkpoints of its own, so it tests that the save can be restored and trained from, not that the hero itself survives a stop and resume. [The hero](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) kept stepping throughout. No status-log entry covers the test.
infraep
hero-ragged_a2a-nccl2307-ep-step81k
The relaunched run has now trained for just over six hours without a break and has committed six hourly saves of its state at full scale. It stood at step 82,938 at 1:04 a.m. (ET), 1,222 steps past the step-81,716 checkpoint it restored, at a median 17.5 seconds a step; the longest pause in the whole stretch was the 54 seconds taken by the first save, and every save since has held up stepping for under 40 seconds. That is the first sustained full-scale evidence for the check that [mcwitt's audit of the silent hangs](https://github.com/marin-community/marin/issues/8870#issuecomment-5609573276) left open on the newer communication-library build — that the run holds for a long stretch at full scale. The audit's second open check, a deliberate stop and resume, has not been tried.
infraep
held h100-d512-mixprior × 20
Twenty data-mixture rungs of the hero scaling ladder started on H100s at 12:36 a.m. (ET), the first such batch since September 5. Each is a six-layer, 384-expert model at width 512, one node apiece, running 3,930 steps on its own weighting of the 223 cluster-and-quality buckets the Harrier mixture is drawn from; all twenty were roughly halfway through and stepping at 2:00 a.m. (ET). They carry the `hero` and `scaling-ladder` tags and the `moe-hero-ep-scaling-ladder` group, but no issue or status-log entry covers the batch.
ladderdata
hero-ragged_a2a-nccl2307-ep-step81k
The replay is finished, it matched, and the run is now past the ground its predecessor covered. Step 81,916 — the last of the two hundred steps the relaunch set out to repeat — was logged at 7:56:34 p.m. (ET), and the run passed 81,919, the final step the [pooled-wave control](https://wandb.ai/marin-community/marin_moe/runs/hero-wd-gate-router-p02-step58k) ever reached, at 7:57:30 p.m. (ET). It stood at step 81,935 at 8:02:35 p.m. (ET), sixteen steps beyond that mark, so everything from here is training the project did not have. The run has stepped without a break for sixty-six minutes since its first step at 6:56:15 p.m. (ET) — 220 steps at a median 17.1 seconds, no gap longer than 54 seconds — and W&B holds it running with a heartbeat current to the minute. Its GPUs read the profile of training rather than of a stall: the memory subsystem is busy at 22-25% of capacity with tens of gigabytes a second moving on the links between GPUs, the opposite of the pegged-but-idle pattern the day's four hangs showed. The check the relaunch exists to make has now passed in full: over all 201 steps of the repeated window, loss against the [pre-merge trial run](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-ep-step81k) agrees to a mean difference of 0.00008 nats, in a band from −0.00046 to +0.00030. The released build [#9062](https://github.com/marin-community/marin/pull/9062) therefore reproduces the private one the trial ran, and the [data-loader fix](https://github.com/marin-community/marin/pull/9040) that landed on main in between has not changed the order the training data arrives in or what the model learns from it. Set against the control on the 26 steps the two share, loss is 0.0065 nats lower; MFU averages 21.9% with a median of 23.2% against the control's 21.5%; the share of tokens dropped for want of room at their expert averages 7.3 in 100,000 against the control's 3.4 in 100; routing entropy is unchanged at 5.949; and throughput averages 2.57 million tokens a second.
infraep
Sep 9
mcwitt #8506
mcwitt rules the redeploy a pass and hands it the run of record. The run relaunched from main passed step 81,916 — the last of the two hundred steps it repeated — at 7:57 p.m. (ET) with no hang, retry or error, and committed its first hourly checkpoint at scale at 7:54 p.m. (ET). Two comparisons close out the trial. Against the [pre-merge trial run](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-ep-step81k) on the same batches, loss differs by a mean of 0.00008 nats and never by more than 0.00046, so the released build [#9062](https://github.com/marin-community/marin/pull/9062) reproduces the private one — and the [data-loader fix](https://github.com/marin-community/marin/pull/9040) that reached main in between did not disturb the data order. Against the [pooled-wave control](https://wandb.ai/marin-community/marin_moe/runs/hero-wd-gate-router-p02-step58k) over 202 paired steps, loss is 0.0065 nats lower; MFU reads 23.11% against 21.46%, the token drop rate 7.8 in 100,000 against 3.4 in 100, and peak GPU memory 103.6 against 136.3 GiB, with load-balancing loss, router z loss and gradient norm all within a hair of the control. The lineage is now written down: the old hero's step-81716 checkpoint feeds [this run](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) on main commits, and the trial's tree is retained as an artifact only.
infraep
loom-oa-dev #8945
The cluster's own metrics service for the machine running the hero is answering queries again, and the incident record names the cause. Finelog on cw-us-east-08a had [refused queries since 6:22 p.m. (ET)](https://github.com/marin-community/marin/issues/8506#issuecomment-5609817090), which left run health to be read from W&B and fired a false alarm that the run had gone silent. The service reports ready as of 7:26 p.m. (ET). The fault was in startup: listing the objects it holds walks the whole tree, and on this cluster that outgrew the probe that decides when the service is ready, so it never came up. At least 6,122 rows of measurements from the run's lead task were dropped by the writing client while it was down, and slow retried writes keep the incident mitigated rather than fixed — work tracked on [#8945](https://github.com/marin-community/marin/pull/8945), which makes these regional relays survive a rebuild of their stored objects.
infraincident
hero-ragged_a2a-nccl2307-ep-step81k
The trial run was stopped and relaunched under a new name that records the communication library it was built against. `hero-ragged_a2a-ep-step81k` logged its last step, 82,025, at 6:49:53 p.m. (ET). Over the next twenty seconds its GPUs wound down — utilisation, memory traffic and traffic on the links between GPUs all falling toward zero — and at 6:50:10 p.m. (ET) they stopped reporting at all. That is the profile of processes exiting, and it is the opposite of the hangs earlier today, which held the GPUs at full utilisation and kept reporting throughout. W&B has since moved the run to the crashed state on its missing heartbeat. Two minutes after the shutdown, at 6:52:17 p.m. (ET), [a new run](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) started on the same machine, restored the same pinned step-81716 checkpoint, and logged step 81,716 at 6:56:15 p.m. (ET). It has stepped every 19 seconds since, reaching 81,744 at 7:05:34 p.m. (ET), and its GPUs read the profile of training: memory subsystem busy near a quarter of capacity, tens of gigabytes a second on the links between GPUs, and the matrix units active close to half the time. Its recorded software differs from the stopped attempt in one line: the layer that runs the compiled model on the GPUs is build `b040736f2c4a` rather than `b183c591142c`. Both are compiled against the newer NVIDIA communication library the job loads, which is what the new name reports. A fresh run id is the project's [standing convention](https://github.com/marin-community/marin/pull/8868) for a relaunch that repeats steps, because W&B refuses to log a step below a run's last one. The cost is that the 309 steps the stopped attempt logged — 106 of them past anything the [old run](https://wandb.ai/marin-community/marin_moe/runs/hero-wd-gate-router-p02-step58k) reached — are being run again. Nothing on the [status log](https://github.com/marin-community/marin/issues/8506) covered the change when this was written, because the mirror this page reads lags GitHub by about an hour. The [entry explaining it](https://github.com/marin-community/marin/issues/8506#issuecomment-5609817090) had in fact been posted at 6:51 p.m. (ET).
infraep
mcwitt #8506
mcwitt explains the swap this page could not account for: the hero was redeployed from the main branch. The wheel that carried the trial sat on a local branch behind a signed URL, so the run was not reproducible from main. [#9062](https://github.com/marin-community/marin/pull/9062) merged the same change as a released build, landing on main as `04fb348456`. The trial coordinator was cancelled at 6:50:30 p.m. (ET) at about step 82,020 after 300 clean steps; its tree is kept as the trial artifact and is not a lineage source. The [new run](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-nccl2307-ep-step81k) was [launched](https://github.com/marin-community/marin/issues/8506#issuecomment-5609812430) at 6:50:42 p.m. (ET) from a pristine worktree at that commit, restoring the old hero's pinned step-81716 checkpoint so the lineage stays on main commits. Its first 200 steps replay the validated window against the pooled-wave control, which also checks the released binary against the pre-merge build the trial ran. Main also took [#9040](https://github.com/marin-community/marin/pull/9040), a fix to two dataset loaders, between the two commits; the paired replay will expose it if it changes data order. A caution closes the note: the regional telemetry service for this cluster has refused queries since about 6:22 p.m. (ET) and the hub has nothing newer from it, so a `TrainingTelemetryGone` alarm fired at 6:36 p.m. (ET) for the trial run while W&B showed it stepping normally. Liveness is being read from W&B until that service recovers.
infraepincident
hero-ragged_a2a-ep-step81k
The trial window is complete and the run has moved past the ground the old run covered. Step 81,916 — the two-hundredth and last step of the window opened at the cutover — was logged at 6:16 p.m. (ET), and the run carried on through the control's final step, 81,919, at 6:17 p.m. (ET), so every step from 81,920 is new training rather than a replay. It stood at step 81,956 at 6:28 p.m. (ET), and W&B reports it running with a current heartbeat. The attempt has now stepped for 59 minutes without a break: 199 steps at a median 17.1 seconds each, with no gap longer than 62 seconds. Scored over the whole window against the [control](https://wandb.ai/marin-community/marin_moe/runs/hero-wd-gate-router-p02-step58k) on the same batches, three of the four conditions in mcwitt's [gate](https://github.com/marin-community/marin/issues/8506#issuecomment-5606002395) are met. Loss ran 0.0064 nats below the control across the 26 steps W&B retained for it in this range, in a narrow band from 0.0060 to 0.0067, against the ~0.006 the dropless routing was expected to earn — inside the gate's bf16-noise allowance. The share of tokens dropped for want of room at their expert averaged 7.7 in 100,000 and peaked at 1.4 in 10,000, against a ceiling of 1 in 1,000 and the control's 3.4 in 100. MFU averaged 23.1% with a median of 23.5%, against the control's 21.5% and a floor of about 21%. Routing entropy was unchanged at 5.949. The fourth condition — no hang, no retry, no watchdog kill — is not met, but every hang fell on the build that the sixth attempt replaced, so the window was carried by software the gate was not written for. Whether that counted as a pass was mcwitt's call; he [ruled it a pass](https://github.com/marin-community/marin/issues/8506#issuecomment-5609573154) at 6:24 p.m. (ET), a ruling this page could not yet see when this was written.
infraepthroughput
mcwitt #8506
mcwitt rules the trial a pass. The sixth attempt (`coord-72aad6e7`) completed the 200-step window 81,716–81,916 at 6:15 p.m. (ET) with no hang, retry or error, and wrote its first hourly checkpoint at scale. Paired against the pooled-wave control on the same batches over 196 joined steps: loss 0.0064 nats lower (−0.0070 to −0.0058); tokens dropped at a full expert 1 in 10,000 against the control's 3.4 in 100; MFU 23.3% against 21.5%; throughput 2.73M tokens a second against 2.52M; load-balancing loss and routing entropy unchanged; peak GPU memory 103.6 GiB against 136.3. He sets that against the same day's failures — the wheel compiled against the older communication-library headers hung four attempts out of four. The hourly save also exposed a monitoring trap: hub ingestion from this cluster lagged 7-8 minutes from 6:13 p.m. (ET), so hub reads showed no step, a stopped heartbeat and no pod telemetry while the regional store had the run past step 81,930; liveness must be read from the regional deployment. The plan he states is the one that followed within half an hour: merge [#9062](https://github.com/marin-community/marin/pull/9062) as a released build and redeploy from main under a new run id, restarting from the same step-81716 checkpoint so the lineage stays reproducible. [Trial report](https://wandb.ai/marin-community/marin_moe/reports/ep_change:-hero-ragged-a2a-swap,-200-step-trial-(step-81716)--VmlldzoxNzkwMjQyNQ==).
infraepthroughput
mcwitt #8870
mcwitt reviews the evidence on the silent hang and withdraws part of it. A table of the four eleven-rack ragged all-to-all runs that ran clean shows two of them used the older compiled headers, so his earlier statement that every clean reproduction used the newer headers was wrong: verified exposure on the newer headers is 402 completed steps across two runs, against 237 clean steps on the old ones. The earlier clean windows are also not comparable — the first two omitted the checkpointer's per-step broadcast, and the third restored it but added bounded barriers, wait instrumentation, initialisation waits and post-collective device-to-host copies, so its success cannot isolate the header change. Today's production attempt is the stronger evidence: 704 GPU processes on the released kernels, with the source difference from the failing build reduced to a [two-line build-default change](https://github.com/marin-community/xla/pull/22), and without the September 4 diagnostic barriers and copies — which argues those are not needed for the improvement. He is explicit that this is one uninstrumented success with build, allocation and timing differences, not a randomised wheel A/B or a tested restart sequence, and that a hardware fabric fault stays credible given the [September 5 NVLink error](https://github.com/marin-community/marin/issues/8870#issuecomment-5554893088) on the pooled-wave hero. Preliminary paired numbers over matched steps 81,726–81,758 show loss differing by at most 0.000110 nats, with MFU 0.48 points lower on the new wheel — observational, and not enough to establish a regression either way. Sustained training and restart validation remain open, tracked on the [incident record](https://echo.oa.dev/wiki/378).
infraepkernelsincident
mcwitt #9062 merged
[#9062](https://github.com/marin-community/marin/pull/9062) pins the GPU compiler runtime to the wheel built against NCCL 2.30.7 headers, so the compiled headers match the communication-library runtime the job already loads. It moves one build hash and leaves everything else in the lock file unchanged. The release passed ragged all-to-all correctness checks on two GB200 GPUs, including confirmation that the ragged all-to-all device kernel actually ran. Its stated evidence is the day's split at hero scale: the old wheel hung four attempts out of four with the [#8870](https://github.com/marin-community/marin/issues/8870) signature — one rack at half CPU, every cross-rack AllReduce with one member exactly three operations behind — while a pre-merge build of this change ran 225 clean steps including a 5.4 TB checkpoint. The PR also points the hero launcher at a new run id starting from the pooled-wave lineage's step-81716 handoff, giving the promoted wheel its own run tree rather than restoring the trial run's. It merged as `04fb348456`, and the hero was relaunched on it at 6:51 p.m. (ET).
infrakernelsep
hero-ragged_a2a-ep-step81k
A fifth attempt is moving data, the first sustained work since the run stopped taking steps. The fourth attempt held the idle-but-pegged state until 5:07 p.m. (ET) and was then cancelled. A new process attached to the same run name at 5:12:53 p.m. (ET), dropped its GPU memory to the level held before any model is loaded, climbed back through a checkpoint restore by 5:16 p.m. (ET), and from 5:16:40 p.m. (ET) has read the profile of training: memory subsystem busy at about a fifth to a quarter of capacity and steady traffic on the links between GPUs, unbroken for nine minutes. Each failed attempt today produced at most one such sample before falling back to full utilisation with no traffic. No training step has been logged, and W&B's step history still ends at 81,757, where the first attempt stopped. That absence is not yet a verdict: on the first attempt ten minutes passed between the checkpoint restore and the first step, and that included a cold compile this attempt does not need. mcwitt's [recurrence study](https://github.com/marin-community/marin/issues/8506#issuecomment-5608545188) caps automatic restarts at five, so this is the last attempt before the rollback the trial gate calls for.
infraep
#8506
mcwitt's account of the fourth hang corrects this page's count of the attempts and names the change that followed. Attempt 4 (`coord-f48b4a48`) restored the checkpoint, cleared its loader at 4:57 p.m. (ET), then took no step for ten minutes, with every task on rack index 2 running 3.6–4.4 CPU cores against 7.6–9.0 elsewhere — the same per-rack asymmetry that caught the two attempts before it. An NCCL RAS sample at 5:01 p.m. (ET) put every cross-rack AllReduce three operations apart, and py-spy dumps from the same minute placed every rank, stalled and healthy alike, inside the compiled training step rather than in the data loader or the compiler. The tally on main `9ccc1bd5e4` is four hangs in four attempts, on rack indices 7, 9, 1 and 2, after 43, 0, 0 and 0 completed steps. Attempt 5 (`coord-c6b2ede7`) was cancelled two minutes after submit, before it restored anything, in order to change wheels. Attempt 6 — the one now training — launched at 5:11 p.m. (ET) as `coord-72aad6e7` from an unpushed local branch: `9ccc1bd5e4` plus a single change, the PJRT wheel `0.11.1+marin.b183c591142c`, whose device kernels are compiled against NCCL 2.30.7 headers, in place of `3a8f3392723e`, compiled against 2.29.7. The run's own package set installs `nvidia-nccl-cu13==2.30.7`, so the new kernels are built against the same version of the collective library the job actually loads. Everything else, including the step-81716 handoff, is unchanged, and the wheel sits under a seven-day temporary prefix. This is the second time a fault in this backend has turned on how the PJRT device kernels were built: [#8313](https://github.com/marin-community/marin/issues/8313) traced a first-step crash at this expert-parallel width to a 32-peer bound inside a barrier kernel, and fixed it with a rebuilt wheel rather than a code change in the model.
infraepkernelsincident
hero-ragged_a2a-ep-step81k
The replacement run has hung twice more, and neither attempt reached a training step. The attempt that attached at 4:41 p.m. (ET) followed the launch profile in order — GPU memory dropped to the level held before any model is loaded, climbed through a checkpoint restore, and produced one burst of full-power compute at 4:44 p.m. (ET) — and then stopped. From that burst onward its GPUs read full utilisation with their memory subsystem idle and no traffic on the links between them, and their resident memory sat at exactly the figure the first attempt held while hung. That process was killed about four minutes later. A third attempt attached at 4:51 p.m. (ET), restored a checkpoint by 4:54 p.m. (ET), reached the same resident-memory figure at 4:56 p.m. (ET), and has not moved since: full utilisation, no memory traffic, no link traffic, and — unlike the second attempt — not one burst of real work at any point. For scale, the first attempt went from restore to full-rate training in half a minute and logged its first step two minutes after that; this one has been past that mark for seven minutes. Its heartbeat to W&B stopped at 4:53 p.m. (ET), two minutes into the attempt, although the machine monitor on the same job keeps reporting and W&B still lists the run as running. The last step the run logged was at 3:59 p.m. (ET), an hour ago. This is the signature of the open fault [#8870](https://github.com/marin-community/marin/issues/8870): a job spinning inside an exchange between machines that never completes. Nobody has claimed any of the restarts. Nothing has been posted to the [status log](https://github.com/marin-community/marin/issues/8506) since the 3:49 p.m. (ET) [first-steps note](https://github.com/marin-community/marin/issues/8506#issuecomment-5607788308), and no rollback run has appeared in the project. The [plan](https://github.com/marin-community/marin/issues/8506#issuecomment-5606002395) mcwitt wrote before the cutover called for an immediate rollback on the first hang — relaunching the old configuration and resuming [the run this one replaced](https://wandb.ai/marin-community/marin_moe/runs/hero-wd-gate-router-p02-step58k) from that tree's own checkpoint — and the go/no-go it set for about 4:45 p.m. (ET) has passed.
infraep
mcwitt #8506
mcwitt posted a recurrence study, and it corrects this page's count of the restarts. There have been more attempts than W&B's rewritten start time revealed: the attempt launched at 4:23 p.m. (ET) restored the checkpoint, cleared its data loader, and hung before logging a step; the one launched at 4:39 p.m. (ET) did the same; a fourth launched at 4:48:59 p.m. (ET). That is three hangs in three attempts, on rack indices 7, 9 and 1 — a different rack each time — one after forty-three steps and the other two before taking a step. Detection no longer waits on the watchdog: a check now runs every thirty seconds on the per-rack CPU load reported by the job scheduler, flagging any rack below 0.6 times the median, and it called two of the hangs within about two minutes of the loader clearing. Automatic restart is capped at five attempts. Every attempt uses the same commit and the same handoff checkpoint.
infraep
hero-ragged_a2a-ep-step81k
The hung process was killed and the run is trying again. After thirty-nine minutes in which [the replacement run](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-ep-step81k) held its GPUs at full utilisation while their memory and the links between them stayed idle, that process ended between 4:38 and 4:41 p.m. (ET), and a new one attached to the same run name at 4:41:09 p.m. (ET). Its opening minutes repeat the 3:37 p.m. (ET) launch in order — GPU memory dropped back to the level held before any model is loaded, then climbed through a checkpoint restore, then produced one burst of full-power compute at 4:44 p.m. (ET) — but it reached that point in three minutes rather than eleven, the executable no longer needing a cold compile. Memory is still rising toward the level the previous attempt held, so the attempt is still starting; no step has been logged, and it is too early to say whether this one will run. Nobody has claimed the restart. Nothing has been posted to the [status log](https://github.com/marin-community/marin/issues/8506) since the 3:49 p.m. (ET) [first-steps note](https://github.com/marin-community/marin/issues/8506#issuecomment-5607788308), and when the [September 3 attempt](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-ep-step54k) hung on this same routing backend it was the progress watchdog that killed and relaunched it. That precedent is not encouraging: the September 3 relaunch hung again after a single step, and the hero was rolled back to the pooled-wave configuration within the hour. The restart also decides the trial. The [gate](https://github.com/marin-community/marin/issues/8506#issuecomment-5606002395) mcwitt wrote before the cutover required no hang, no retry and no watchdog kill across the two hundred steps that were to determine whether the new routing stays, and the run has now produced at least the first two of the three, forty-two steps in. The go/no-go the gate named, about 4:45 p.m. (ET), has passed with the run restarting rather than stepping.
infraep
mcwitt #8870
The first hang is written up on [#8870](https://github.com/marin-community/marin/issues/8870), and it narrows the fault without naming it. In the seconds after the last completed step, every one of the 704 GPU processes reported a data fetch that took at least ten seconds, and most of them at least twenty. Ten minutes later, stack captures covering all 704 showed every main thread inside the training step and every prefetch thread waiting to hand work to a queue that was already full — so a data-loader wait is not what holds the stall open, whatever started it. Snapshots of the collective library found every rank present and none unresponsive, with a stable gap in operation counts on one communicator, which mcwitt notes does not prove the collective is the cause: an earlier operation on one GPU can keep a rank from reaching the next one. His hypothesis is that the slow fetch made ranks arrive unevenly and exposed a timing defect in the ragged exchange or in some other GPU operation. He marks it a hypothesis, and the test that would separate the two transports under a deliberate rank delay has not been run. The attempt was built with the upgraded [QuACK 0.6.4](https://github.com/marin-community/marin/pull/8959) kernels, so that upgrade alone does not prevent the hang.
infraepkernels
hero-ragged_a2a-ep-step81k
The replacement run has stopped taking steps, and the way it stopped matches the one fault the trial was gated against. Its last step, 81,757, was logged at 3:59 p.m. (ET). Twenty-seven minutes later nothing has followed, against a measured pace of one step every 18.3 seconds, so roughly ninety steps are missing. Nothing has failed in the ordinary sense: W&B holds the run in the running state, its heartbeat is current to within a minute, and its GPU memory stays pinned at 89.9 per cent, so the processes are alive and the model is still resident. What has stopped is the work. The GPUs report full utilisation while their memory subsystem sits idle — DRAM activity fell from about 23 per cent to under a thousandth of a per cent at 3:59:50 p.m. (ET) and has not moved since — and NVLink traffic between them has been zero over the same span. A GPU that is fully busy while moving no data is spinning inside a collective exchange that never completes. The [September 3 attempt](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-ep-step54k) on this same routing backend entered the identical state, full utilisation with no memory or link traffic, and never left it; that attempt reached about seventy steps of its window, and this one reached forty-two of two hundred. Every gated measure was passing up to the stop. One gate was that the run must not hang, so on its own terms the trial has now failed, and the [plan](https://github.com/marin-community/marin/issues/8506#issuecomment-5606002395) calls for an immediate rollback: relaunch `trigger_hero.sh` at commit `0908920c86`, which resumes the run it replaced from that tree's own newest checkpoint, saved at step 81,900 at 3:28 p.m. (ET). The root cause of [#8870](https://github.com/marin-community/marin/issues/8870) is still open, and its strongest lead — a fabric fault on the racks that land at replica indices 8 to 10 — is one the trial could not design around. Nothing has been posted to the [status log](https://github.com/marin-community/marin/issues/8506) since the 3:49 p.m. (ET) first-steps note, which set the decision point at about 4:45 p.m. (ET).
infraep
mcwitt #8506
mcwitt claimed the first hang and the restart that followed, which this page had recorded as unexplained. The trial ran forty-three steps on signature against the control before step 81,759 began, after a wait of thirty to forty seconds for the data loader's next batch, and never finished. No rank logged an error. Through the stall one rack of tasks sat at half the CPU load of the other ten, and a snapshot of the collective library showed every cross-rack communicator with one member exactly three operations behind. The watchdog could not fire, because a new job gets an hour of startup grace, so mcwitt cancelled the coordinator by hand at 4:23:25 p.m. (ET) and relaunched the same run id sixteen seconds later from the same commit and the same handoff checkpoint. He states the reason plainly: the run stays up to measure whether the hang recurs, and the rollback to the old configuration remains one command.
infraep
mcwitt #8506
mcwitt posted the first-steps report the plan promised, and it read clean on every gate. The replacement restored the pinned step-81716 checkpoint at 3:38 p.m. (ET), with the path confirmed in its own log, and logged step 81,720 at 3:48 p.m. (ET), fourteen minutes after launch. Over the first five steps matched against the [old run](https://wandb.ai/marin-community/marin_moe/runs/hero-wd-gate-router-p02-step58k) on the same batches, every measure sat on the right side of its threshold: loss 0.0062 nats below the control, which is the credit the dropless routing was expected to earn and the same offset seen in the September 2 trial; the share of tokens sent to an expert with no room left, and so dropped, at 9 in 100,000 against the control's 3.4 in 100; throughput 2.75M tokens a second against 2.52M; and routing entropy unchanged at 5.949, so expert load is as even as before. He also confirmed that the old tree's hourly save wrote step 81,900 at 3:28 p.m. (ET) without disturbing the protected step-81716 checkpoint the replacement needs for a rollback, and set the go/no-go at step 81,916, about 4:45 p.m. (ET).
infraep
mcwitt hero-ragged_a2a-ep-step81k
The replacement hero run started at 3:37 p.m. (ET), four minutes after the old one stopped. It restores the checkpoint [forced and pinned](https://github.com/marin-community/marin/issues/8506#issuecomment-5606812293) at step 81,716 — the last of the four checkpoint paths it is given points into the old run's storage at that step — and it holds the same shape of machine: eleven racks, 176 tasks of four GB200s each, the same data mixture, the same optimizer, and the same 390,251-step finish line. Two things differ, and they are the two the cutover was for. It uses the [routing backend](https://github.com/marin-community/marin/pull/9043) withdrawn on September 3, which moves tokens between experts in one ragged exchange instead of fixed waves, and it keeps its master copy of the weights in full precision on the GPU. It is also built against the [upgraded kernels](https://github.com/marin-community/marin/pull/8959): the package list W&B stores with the run names QuACK 0.6.4, where the old run's names 0.6.1. So both changes are in the image that launched, whatever the pull requests themselves say. The run has logged no training step yet, eight minutes in. That is normal rather than a warning — the [September 3 attempt](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-ep-step54k) on this same backend took about thirty-four minutes from launch to its first step. W&B holds it in the running state and its heartbeat is current. The [trial report](https://wandb.ai/marin-community/marin_moe/reports/ep_change:-hero-ragged-a2a-swap,-200-step-trial-%28step-81716%29--VmlldzoxNzkwMjQyNQ==) that will place it beside the old run over the window is live.
infraep
mcwitt #8506
The cutover is on the record, and it settles a question this page had left open. mcwitt cancelled the old run's coordinator at 3:33 p.m. (ET) and confirmed it terminal at 3:34 p.m. (ET) with step 81,919 logged, so the 200-step control window from step 81,716 to 81,916 is complete. The replacement launched a minute later from a pristine worktree at main commit `9ccc1bd5e4`, which he states carries both [#8959](https://github.com/marin-community/marin/pull/8959) and [#9043](https://github.com/marin-community/marin/pull/9043). Both changes therefore did land on the main branch before launch, correcting this page's earlier reading that neither had merged — that reading came from the pull-request record, which had not caught up. The job restores step-81716 through `--initialize-from-checkpoint` and runs on eleven NVL72 racks in `cw-us-east-08a` as `IRIS_USER=marin`. He expected a restore and a cold compile of the ragged executable before the first step, roughly thirty minutes. Rollback is defined as relaunching `trigger_hero.sh` at commit `0908920c86`, which resumes the old tree's newest checkpoint.
infraep
mcwitt hero-wd-gate-router-p02-step58k
The run this page has followed since September 4 is over. It logged its last step, 81,919, at 3:33 p.m. (ET) and its last heartbeat a half minute later; W&B moved it to the crashed state, which is also what a cancelled coordinator looks like from the outside. The timing says this was the planned stop and not a fault: the matched control window ran from the pinned checkpoint at step 81,716 to step 81,916, mcwitt [put its end](https://github.com/marin-community/marin/issues/8506#issuecomment-5606002395) near 3:35 p.m. (ET), and the run stopped three steps past it. This page's read twenty minutes earlier could not tell whether the window had closed, because W&B's sampled history trailed the live run by about 130 steps; the run summary, which does not lag, settles it. The window is therefore complete, and the old run supplies the whole control side of the comparison. The step count and loss curve shown above still come from the sampled history of this finished run and will not move again until the page is pointed at [its replacement](https://wandb.ai/marin-community/marin_moe/runs/hero-ragged_a2a-ep-step81k).
infraep
mcwitt 8959-expert-capture
A fourth job under [#8959](https://github.com/marin-community/marin/pull/8959)'s label started at 2:53 p.m. (ET) and failed on its own code before it took a step. It reloads the same step-80912 checkpoint the gradient arms used, on the same spare rack of 64 GB200s, and is written to stop after a single step. Its script takes a fingerprint of the weights at every step: two numbers out of the first expert in each of the model's 48 layers, which is a cheap way to check whether anything moved. JAX refused the read. The 384 experts of a layer are split across 64 positions on the rack, and the script asks for one expert on its own, which that split does not allow; every rank in the log ends on the same refusal. The job logged no step, ran one minute forty-six seconds, and W&B holds it in the finished state even so. It was built against the upgraded kernels. It is not one of the tests the pull request names, and nothing about it has been posted there.
infrakernels
mcwitt 8959-gradient-control
The frozen-weight gradient check is finished, and the two kernel versions produce the same arithmetic. The [control arm](https://wandb.ai/marin-community/marin_moe/runs/8959-gradient-control-20260909) stopped at 2:51 p.m. (ET) at step 80,920, the last of the nine steps it was set to take, matching the [upgraded arm](https://wandb.ai/marin-community/marin_moe/runs/8959-gradient-treatment-20260909) step for step from the same step-80912 checkpoint with every weight held fixed. Their stored settings differ only in name and storage path. Which kernel version each job carries — the thing this page could not confirm earlier today — is in fact recorded, in the package list W&B stores with every run: the upgraded arm was built against QuACK 0.6.4 and the control against 0.6.1, the version the hero itself is running. On loss the arms are identical: the same value at all nine steps, to every digit W&B stores. On gradients they are close enough to settle the question. The total gradient norm differs between the arms by two parts in ten thousand on average and by ten parts at worst, against a spread of twelve per cent across the nine steps themselves. Taken group by group — the model has 1,380 of them, so 11,979 comparisons in all — a little over half agree within a thousandth and more than nine in ten within a hundredth. The widest gap is seven per cent and only one other exceeds five; both fall on parameter groups whose norms are a hundredth or less of the total, where the arithmetic has the fewest digits to work with. Neither arm sits above the other. This is what the morning's matched pair could not show: those two started training, so their weights parted after the first step and a kernel difference could not be told from ordinary drift. Speed cannot be read from these arms, because each step writes 1,380 norms and so takes about three times the morning pair's. The [reference arm](https://wandb.ai/marin-community/marin_moe/runs/8959-gradient-reference-20260909) that started between the two is now in the crashed state, having logged no step. mcwitt has posted none of this to [#8959](https://github.com/marin-community/marin/pull/8959), which is still a draft. The hero trained through all of it on the older kernels, on its own eleven racks, and W&B holds it in the running state.
infrakernels
mcwitt 8959-gradient-control
The frozen-weight gradient check now has the other half it was missing. At 2:37 p.m. (ET) mcwitt started a control arm, eleven minutes after the [upgraded arm](https://wandb.ai/marin-community/marin_moe/runs/8959-gradient-treatment-20260909) finished. It matches that arm on everything W&B records: the same spare rack of 64 GB200s, the same production checkpoint at step 80,912, the same nine-step stop, the routing backend the run is being asked to return to, master weights held on the GPU, no checkpoints saved, and a record written at every step. The two should therefore differ only in the kernel library each was built against, which W&B does not record, so this page cannot confirm which version the control carries. It has logged no steps yet. A third job, labelled [reference](https://wandb.ai/marin-community/marin_moe/runs/8959-gradient-reference-20260909), started between the two at 2:27 p.m. (ET); it logged no steps either and stopped reporting to W&B at 2:36 p.m. (ET), a minute before the control began, although W&B still lists it as running. None of the three has been mentioned on [#8959](https://github.com/marin-community/marin/pull/8959), which is still a draft. The hero trained through all of it on its own eleven racks.
infrakernels
mcwitt #8506
The cutover has started: the checkpoint the replacement run would begin from has been written and pinned. At 2:26 p.m. (ET) mcwitt forced a save on the live run through the training-control endpoint, taking step 81,716 out of turn rather than waiting for the next scheduled permanent one. It committed two minutes later — 5.36 TB across 26,795 objects — and at 2:30 p.m. (ET) its metadata was rewritten so the hourly cleanup treats it as permanent and cannot delete it. This is the step the [plan](https://github.com/marin-community/marin/issues/8506#issuecomment-5606002395) posted an hour earlier called for, and it costs about one step of training. The old run keeps going from there: it now trains the two hundred steps that serve as the matched control, ending at step 81,916, which mcwitt put near 3:35 p.m. (ET). The launcher change is not a separate pull request after all — it went into [#9043](https://github.com/marin-community/marin/pull/9043), which now points the trigger script at this checkpoint under the run id `hero-ragged_a2a-ep-step81k`. So the cutover is blocked on two merges and nothing else: [#8959](https://github.com/marin-community/marin/pull/8959) and [#9043](https://github.com/marin-community/marin/pull/9043) both have to land on the main branch. mcwitt also linked the [report](https://wandb.ai/marin-community/marin_moe/reports/ep_change:-hero-ragged-a2a-swap,-200-step-trial-%28step-81716%29--VmlldzoxNzkwMjQyNQ==) that will show the old and new runs side by side over the window.
infracheckpointing
mcwitt 8959-gradient-treatment
The frozen-weight arm on the upgraded kernels finished on time and recorded what it was written to record. It stopped at 2:26 p.m. (ET) at step 80,920, the last of the nine steps it was set to take, fourteen minutes and twenty-three seconds after its first, and W&B holds it in the finished state. Each step carries a gradient norm for every parameter group in the model — about 1,380 of them — and a total; the totals sit between 0.385 and 0.433 across the nine steps and its loss between 1.190 and 1.282, which is the ordinary spread of a fixed model reading nine different batches, since no weight ever moved. On its own the arm settles nothing, because the point of the exercise is to run the same nine steps on the old kernels and compare the two sets of gradients. A second job, labelled [reference](https://wandb.ai/marin-community/marin_moe/runs/8959-gradient-reference-20260909), started a minute later at 2:27 p.m. (ET). Neither this result nor the morning's two has been posted to [#8959](https://github.com/marin-community/marin/pull/8959).
infrakernels
mcwitt #8959
The validation results are now on the pull request, correcting what this page said at its last read. At 2:15 p.m. (ET) mcwitt posted a [W&B report](https://wandb.ai/marin-community/marin_moe/reports/PR-8959-QuACK-0.6.4-deployment-validation:-d768-and-hero-restores--VmlldzoxNzkwMDk3OQ==) to [#8959](https://github.com/marin-community/marin/pull/8959), its first comment since September 8. The report's title names the two tests the pull request asks for — the pair of hero restores run this morning and the from-scratch soak that finished clean at 1:41 p.m. (ET) — and no others. It was posted thirty-six minutes before the frozen-weight gradient check finished at 2:51 p.m. (ET), so that check, which mcwitt ran on his own and which is the only one of the day that compares the two kernel versions on identical weights, cannot be in it. No text accompanied the link, so the report is the whole of what has been said about the results.
kernels
mcwitt #8506
Sets out a plan to stop this run today and relaunch it on the expert-routing backend that was withdrawn a week ago. At 1:26 p.m. (ET) mcwitt posted the sequence to the status log. Merge the kernel-library upgrade ([#8959](https://github.com/marin-community/marin/pull/8959)), then the backend change ([#9043](https://github.com/marin-community/marin/pull/9043)). Force a checkpoint on the live run through the training-control endpoint about seventy minutes before the cutover and mark it so the hourly cleanup cannot delete it. Open a launcher pull request pointing the trigger script at that checkpoint under a new run id. Let the current run train two hundred steps past it, so the old and new runs can be read against each other on the same batches. Then cancel the old coordinator and launch from a clean checkout. Waiting for the next scheduled permanent checkpoint was rejected: it lands about 3 a.m. (ET) tomorrow and the one after on Thursday, while forcing a checkpoint costs a single step, which is what the September 4 pause cost. Beyond the backend, the new run would carry one train-step executable per rank ([#8911](https://github.com/marin-community/marin/pull/8911)), the rebuilt routing kernel wheel ([#8976](https://github.com/marin-community/marin/pull/8976)), and two fixes to attention and the loss calculation ([#8960](https://github.com/marin-community/marin/pull/8960), [#8961](https://github.com/marin-community/marin/pull/8961)). The trial passes only if the new run's loss matches the old one within the noise of its own arithmetic once a known small offset is allowed for, dropped tokens stay below a thousandth of assignments, speed is at least the old run's, and nothing hangs, retries or trips the progress watchdog; anything short of that rolls back at once to the commit the run is on now, which resumes its own newest checkpoint. mcwitt says the runbook scripts are being dry-run read-only against the cluster, and that a W&B report and each transition — forced checkpoint, cancel, launch, first steps, go/no-go — will be posted here before launch. The reason the backend was withdrawn on September 3, two silent hangs on one rack ([#8870](https://github.com/marin-community/marin/issues/8870)), is still unexplained; the plan names a fabric fault on the racks that land at three particular positions as the strongest lead and says the trial cannot avoid them.
infraincidentthroughput
mcwitt #9043 open
Puts up the change that would return this run to the way of moving tokens between experts that it lost on September 3. mcwitt opened it at 1:25 p.m. (ET). It makes the ragged all-to-all transport the hero's default again, with fp32 master weights held on the GPU, undoing the fallback taken in [#8884](https://github.com/marin-community/marin/pull/8884) after two attempts hung silently on one rack at eleven racks ([#8870](https://github.com/marin-community/marin/issues/8870)). The case for trying again is three changes since: one train-step executable per rank removed a duplicate registration ([#8911](https://github.com/marin-community/marin/pull/8911)), the kernel wheel moved to a rebuilt routing kernel ([#8976](https://github.com/marin-community/marin/pull/8976)), and the kernel-library upgrade replaces the scheduler that could leave a multiply's output unwritten under contention ([#8959](https://github.com/marin-community/marin/pull/8959), which merges first). mcwitt writes plainly that the hang's cause is still open, and names a fabric fault on particular NVL72 racks as the strongest lead — the same pattern as the September 5 NVLink failure on this run ([#8934](https://github.com/marin-community/marin/issues/8934)), which the cluster operator has not yet answered. One step is one-way: this run's checkpoints keep their fp32 master copy in host memory and the new mode reads them in and moves it to the GPU, but the reverse conversion is refused, so a rollback has to resume this run's own checkpoint tree rather than anything the replacement writes. Automated review has failed on the pull request twice, each time reporting that the commit it was asked to read does not exist.
infrathroughput
mcwitt 8959-hero-treatment
The matched benchmark that #8959 names as its own merge condition has finished, and the two arms came out the same on both halves of it. The treatment arm stopped at 11:21 a.m. (ET) at step 81,011 — the control's own stop, from the control's own step-80912 checkpoint, after the same 339,788,955,648 tokens. Both arms logged the same 100 steps, 80,912 through 81,011. On loss they are the same to the fourth decimal: the first step after the restore is identical on both at 1.234894, and across the 100 steps the mean absolute difference is 0.0002, against a median move of 0.0205 from one step to the next — a hundredth of the curve's own movement. It does not grow: the treatment sits 0.00017 above the control on average over the first fifty steps and 0.00004 below over the second fifty, and the largest single gap, 0.00075, falls two steps from the end. That is what reordered floating-point arithmetic produces, not a change in how the model learns. On speed the arms are level: the median interval between logged steps is 16.14 seconds on the control and 16.17 on the treatment, and the whole span from first to last logged step is 1,607.5 seconds against 1,618.3. The treatment's slow tail is a little longer — 90th-percentile step 17.08 seconds against 16.56 — which is the same few-slow-steps difference this page reported when the arm was a third of the way through. So the evidence [#8959](https://github.com/marin-community/marin/pull/8959) asks for is in hand and it is clean. mcwitt has posted nothing to the pull request, which is still in draft, so the merge and the restart that would carry the upgrade into the hero stay held. A second treatment arm, [8959-d768-treatment](https://wandb.ai/marin-community/marin_moe/runs/8959-d768-treatment-20260909), started a minute later on a 1.6-billion-parameter model — a small shape, not the hero — and has logged no step yet. The hero kept training on its own eleven racks throughout.
infrakernels
hero-wd-gate-router-p02-step58k
The evaluation score the Discord argument was waiting on has landed, at about 10:30 a.m. (ET) — the time this page estimated, then withdrew, at its last read. At step 80,999 the equal-weighted average over the sixteen Paloma held-out text sets reached 2.23162, its lowest value on this run, below the 2.23502 of the step-77,999 score of the evening before. Thirteen of the sixteen sets fell. The argument is about four of them, the four that had been rising since the September 4 relaunch, and they split: gab fell from 3.38050 to 3.36716 and twitterAAE_HELM_fixed from 4.48101 to 4.47264, while ptb rose from 2.16294 to 2.18041 and wikitext_103 from 1.69934 to 1.70025. Measured instead against this run's first score, at step 59,999, three of the four now sit below where they started — gab (3.37914), ptb (2.18406) and wikitext_103 (1.70835) — and only twitterAAE_HELM_fixed (4.43192) remains above it. The set of four is therefore no longer rising as a group, which is what the reading that they had hit a noise floor predicted. Nobody has posted about the score. Note that this is the cap-lifted score, and the evaluation curve plotted on this page is the capacity-limited one, which is still frozen, so this point does not appear there.
evals
mcwitt rdx-split-latest-a
The spare-rack series did not end, and it has moved onto the weights the run holds now. This page reported the series over at its last read; two more jobs started after it. rdx-interleaved-latest-b ran from 6:20 to 6:39 a.m. (ET), and the linked rdx-split-latest-a from 6:44 to 7:02 a.m. Both carry the expert-routing backend rolled back on September 3, on one rack against the eleven the hero occupies, and both restore step-80183 — a checkpoint from this morning, not the step-54000 one every earlier job in the series used. Their recorded settings are identical to each other in every field W&B keeps, down to the same custom JAX plugin build; only the names separate them, and nothing on the record says what interleaved and split are. Their losses agree to four decimal places, so the two arms train the same. Their speeds do not: split took 16.83 seconds a step against interleaved's 17.32, a gap wider than the step-to-step spread inside either job. On both, the router discarded about two tokens in ten thousand, against about three in a hundred on the deployed build — the reduction this backend was adopted for, though these jobs run a batch a tenth the size of the hero's, which affects that number. Both were set to run to step 80,223 and neither reached it; W&B holds both in the crashed state, eight and ten steps short. None of it is written up.
mcwitt rdx-cudnn-e
The last of the overnight routing-backend jobs ended without training, and the spare rack has gone quiet. rdx-cudnn-e started at 5:59 a.m. (ET), one minute after rdx-cudnn-d finished, and W&B now holds it in the crashed state with no training step recorded: the only number it stored is the model's parameter count. That is the same empty ending rdx-cudnn-c had. This page reported the job as still running at its last read; it is not. The series is now complete. Five jobs carry the cudnn name, against the control [rdx-ctl-a](https://wandb.ai/marin-community/marin_moe/runs/rdx-ctl-a): rdx-cudnn-b and -d reached the control's own stop at step 54,059 and finished within 0.0001 of its train loss; rdx-cudnn-a crashed after eight minutes at step 54,006; and -c and -e logged nothing past the checkpoint they restored. Nothing on the record says why those three ended early. W&B holds no later job on that rack. mcwitt has posted none of this to the status log, to [#8870](https://github.com/marin-community/marin/issues/8870), to [#8959](https://github.com/marin-community/marin/pull/8959) or to Discord, so the expert-routing backend rolled back on September 3 still has no written verdict, and the restart that would carry it has no date. The hero trained through all of it on its own eleven racks.
epablation
mcwitt rdx-cudnn-d
A second cudnn arm reached the control's stop and landed on the other side of it, so the pair is level — and the 5% slowdown this page reported for the first arm was the cost of a profiler, not of the code being tested. rdx-cudnn-d started at 5:35 a.m. (ET) and finished clean at 5:57 a.m., at step 54,059 after 226,744,074,240 tokens: the same stop and the same tokens as rdx-ctl-a and rdx-cudnn-b. Final train losses are 1.27988 for the control, 1.27996 for rdx-cudnn-b and 1.27983 for rdx-cudnn-d, so the two cudnn arms sit 0.00008 above and 0.00005 below the control — a spread of 0.00013 on a loss near 1.28. Wall clock is 1,390.7 seconds for the control, 1,462.5 for rdx-cudnn-b and 1,341.0 for rdx-cudnn-d. W&B's stored configuration explains that split: rdx-cudnn-b runs the profiler — which records where a step's time goes, and costs time to do it — for three steps from step 54,003, and neither the control nor rdx-cudnn-d does. Correcting what this page said earlier: the record does say what the cudnn name marks. Every cudnn job installs three packages the control does not — a custom build of the JAX GPU plugin, taken from a branch of mcwitt's named ragged-dot after the grouped matrix multiply this model's expert layers use, plus NVIDIA's cuDNN and cuBLAS libraries pinned to versions 9.25.1.1 and 13.6.1.10 — while the control installs nothing extra and runs the deployed image. Two builds of that plugin appear: one in rdx-cudnn-a and -b, another in -c, -d and -e. The control also records an optimiser option, use_syrk, that no cudnn job records at all, which is what a different build would produce. Two of the remaining cudnn jobs failed: rdx-cudnn-a crashed after eight minutes, and rdx-cudnn-c, still running at the last read, ended in W&B's crashed state having logged no training step at all. rdx-cudnn-e started at 5:59 a.m. (ET) with the profiler on and is still running. mcwitt has posted nothing about any of these to the status log, to [#8870](https://github.com/marin-community/marin/issues/8870) or to Discord. The hero trained through all of it on its own eleven racks.
epablation
markhart0034 #hero-run-2026
The person who opened the argument over the curves says it is settled, and gives no number. At 5:21 a.m. (ET) markhart0034 writes “Glad it was noise”, and a minute later adds that they think they had also been looking at the train loss at the time. They opened this thread at 4:15 p.m. (ET) on September 7 by saying the training and evaluation losses both looked a little off since that morning’s restart, and allowing that it could be noise; two days of argument followed over whether four of the sixteen Paloma held-out text sets had stopped improving because no signal was left to learn, and over whether to replace the sixteen-set macro average the preregistered target was set against. The reply follows Percy Liang’s message two hours and eleven minutes earlier, which read the step-77,999 evaluation scores of eleven hours before as the run breaking out of its apparent stall, and it is the only reply to that message. Nothing is decided by it: Larry’s proposal to build a second aggregate alongside the macro average is still open, and the per-set check that would separate the competing explanations still has no date. On the training loss they name, the chart on this page shows no break at the September 7 restart.
evals
mcwitt rdx-cudnn-b
The routing-backend control run now has arms to compare against, and the first clean pair came out level. Three jobs named rdx-cudnn started on the same spare rack after rdx-ctl-a released it: one at 4:06 a.m. (ET) that stopped in W&B's crashed state after eight minutes at step 54,006; the linked one at 4:25 a.m. that ran to the control's own stop — step 54,059, after 226,744,074,240 tokens, the same work to the token — and finished clean at 4:49 a.m.; and a third at 5:00 a.m. that is still running. On that matched pair the final train losses were 1.27988 for the control and 1.27996 for the cudnn arm, a gap of 0.00008. The control took 1,391 seconds of wall clock and the cudnn arm 1,462, about 5% more, but that figure counts queueing, loading and compilation, which the night's kernel benchmarks showed can be larger than the training difference itself. W&B holds the same tags and the same model configuration for all four jobs, and mcwitt has posted nothing about them to the status log, to [#8870](https://github.com/marin-community/marin/issues/8870) or to Discord. A later read of the stored configuration did show what the cudnn name marks, and what the wall-clock gap above is: see the entry for [rdx-cudnn-d](https://wandb.ai/marin-community/marin_moe/runs/rdx-cudnn-d). Training was not touched; these run on a separate rack.
ablationep
Percy Liang #hero-run-2026
The first person to write about the turn in the evaluation scores says the run has broken out of the stall. At 3:10 a.m. (ET) Percy Liang writes that the Paloma evaluation loss had seemed to be stalling, and that the run seems to have broken out of that. He is reading what the step-77,999 score showed nine hours earlier: fifteen of the sixteen Paloma held-out text sets fell, and the macro average — the equal-weighted mean over those sets — reached its lowest value on this run. He names no set and gives no figure, and his message is the first in the channel in five and a half hours. Nobody has replied. It settles nothing in the argument still open there over whether to keep the sixteen-set average, and it does not touch training. Note that the evaluation curve on this page plots the capacity-limited score, which has logged no points on this run; the scores he reads are the cap-lifted ones.
evals
mcwitt rdx-ctl-a
mcwitt put the expert-routing backend that was rolled back on September 3 back on a machine, and it ran clean. A job named rdx-ctl-a started at 2:20 a.m. (ET), one minute after the last kernel benchmark released the rack, and finished at 2:43 a.m. It holds the whole 535-billion-parameter model on 64 GPUs — one rack, against the eleven the hero occupies — restores the same step-54000 checkpoint the September 2 backend swap started from, and carries the ragged all-to-all expert transport — the code path that moves each token to the expert chosen for it — that hung twice on that checkpoint. Its optimiser settings are the ones that checkpoint was trained with, without the gate and router weight decay the hero added on September 4, so this tests the transport rather than the recipe. It was configured to stop after 60 steps, and it stopped at step 54,059 in W&B's finished state, holding about 24.8% model FLOPs utilization throughout. That is not evidence the hang is fixed: both hangs took eleven racks, and the standup already records that later one-rack and eleven-rack attempts failed to reproduce them ([#8970](https://github.com/marin-community/marin/issues/8970)). The name marks this a control arm; no treatment run to compare it against has appeared, and nobody has written about it in the issues or in Discord. Every job on this rack since 5:52 p.m. (ET) on September 8, this one included, places its first task on s1wvxs64 — the node whose fabric link went down on September 7 and stopped the hero ([#8934](https://github.com/marin-community/marin/issues/8934)). The node is back in service and has carried about nine hours of work since. The hero was not touched.
epablation
mcwitt 8870-quack-main-static-r1
The overnight kernel-upgrade test ran its third arm, and on matched steps the three builds are the same on both halves of the gate. The arm is labelled static — QuACK 0.6.4, the kernel-library version #8959 upgrades to, run with a fixed work schedule rather than the dynamic one whose fix [#8959](https://github.com/marin-community/marin/pull/8959) is for. It started at 1:30 a.m. (ET) from the same frozen hero checkpoint at step 77,846 as the pair before it and stopped at 2:19 a.m. All three arms read the same batches: at step 77,850 their losses agree to within 0.00003. At step 77,927, the last step all three reached, the deployed build recorded 1.28414, the dynamic upgrade 1.28472 and the static variant 1.28478 — a spread of 0.0006 on a loss near 1.28. The speed half now reads the same way. Each arm logged its first step at a different point — 421, 377 and 375 seconds in, which is queueing, loading and compilation — and the 81 steps after that took 1,337, 1,336 and 1,334 seconds. So the 2.5% gap this page reported for the earlier pair is start-up time; the training itself differs by 0.2%. Separately, the night's failures are failures. Twelve jobs ran between 5:52 p.m. (ET) on September 8 and 2:19 a.m. on September 9. Ten of them trained; the other two logged nothing past the checkpoint they restored. Every training job but the two profiling ones was written to stop at step 78,846, a thousand steps past that checkpoint. The profiling pair, set to stop at 77,864, reached its stop and ended in W&B's finished state. The other eight ended in the crashed state between steps 77,863 and 78,004, none of them a sixth of the way through its planned thousand steps. Nothing on the record explains why. mcwitt has posted no result to #8959 or to [#8870](https://github.com/marin-community/marin/issues/8870), so the merge he gated on these tests is still held. The hero trained through the whole window on its own eleven racks.
throughputablation
mcwitt 8870-quack-main-r1
The kernel-upgrade benchmark produced its first pair that measures speed as well as loss, and the two builds came out close on both. Two jobs started at 12:18 and 12:49 a.m. (ET), one carrying the build now deployed and one carrying QuACK 0.6.4, the version whose scheduler fix [#8959](https://github.com/marin-community/marin/pull/8959) is for. Both stopped at step 77,927 after processing 326,853,722,112 tokens — the same work to the token, which none of the night's earlier pairs managed. Their final train losses were 1.28414 for the deployed build and 1.28472 for the upgrade, a gap of 0.0006 on a loss near 1.28. W&B recorded the deployed arm at 1,758 seconds of wall clock and the upgrade at 1,713, so the upgrade covered the same tokens about 45 seconds, or 2.5%, sooner. That figure counts start-up and compilation as well as training, and a single pair is not a throughput measurement. Both arms ended in W&B's crashed state, as most of the night's jobs did; nothing on the record says whether that is a fault or simply how these fixed-length benchmarks stop. A third arm, labelled static rather than control or upgrade, started at 1:30 a.m. (ET) and was still running at 2:25 a.m. mcwitt has posted no result to #8959 or to [#8870](https://github.com/marin-community/marin/issues/8870), so the merge he gated on these tests is still held. The hero kept training throughout.
throughputablation
mcwitt 8870-quack-main-upgrade-r1
mcwitt ran the tests that the kernel-library upgrade waits on, and on loss the two builds came out level. W&B records eight jobs between 8:27 p.m. (ET) on September 8 and 12:49 a.m. (ET) on September 9. Seven are training jobs of ten to thirty-three minutes; the eighth is a three-second record holding no steps. Every training job carries the hero's own parameter count and stops somewhere between steps 77,863 and 77,928 — the range the hero itself passed through on September 8 — which matches the plan mcwitt set in [#8959](https://github.com/marin-community/marin/pull/8959): matched pairs restored from one frozen hero checkpoint, so the test trains the production model rather than a small stand-in. Each job is labelled as the build now deployed or as QuACK 0.6.4, the version whose scheduler fix that pull request is for. The pair that compares cleanly is the one run with profiling on, which records where a step's time goes: both arms stopped on step 77,863, and their train losses differ by 0.00002. That answers the loss half of the gate mcwitt set on the merge. The speed half is not answered here — profiling costs time of its own, and no matched timing has been written down anywhere. Four of the seven training jobs end in W&B's crashed state and two in finished; nothing on the record says which of those endings are faults and which are how a fixed-length benchmark stops. The last job, the upgrade arm of a fresh pair started at 12:49 a.m. (ET), was still running when this page read W&B. mcwitt has posted no result to #8959 or to [#8870](https://github.com/marin-community/marin/issues/8870), so the gate is not closed. The hero kept training through the whole window.
throughputablation
Larry #hero-run-2026
Reads the run as still on course for its preregistered target loss, and hedges. Larry says that drawing a line from the end of one curve to the end of another leaves the run somewhat on track, so the projection — the preregistered number the scaling ladder set for this run — may still hold. He names neither curve, posts no figure the channel record keeps, and ends with "idk". This is a judgement about the target, not a measurement, and it is the second of two posts since the step-77,999 evaluation landed; neither names that score. Nobody has replied.
evals
ClassicLarry #8827
Posts the first written result from a checkpoint of the current run, and it shows the model failing to stop. At 9:18 p.m. (ET) Larry fed one step-78000 sample's own 300-token output back into the model and let it write until it filled the 4,096-token context, 3,757 new tokens at the low temperature all but one of the evening's jobs used. The model finished the arithmetic puzzle it had set up and reached the right answer, day 14, in about fifteen lines. It then restated its check of that answer roughly thirty-five times until the context ran out, cut off mid-sentence, and never wrote the answer in the form it had promised. Larry calls this a textbook low-temperature breakdown: at that setting the model puts nearly all its probability on one next word, so a loop has no way to end itself. He draws no link to the Paloma argument running in Discord, where both dlwh and Kaiyue-Wen had named temperature that afternoon. The issue collects these completions as a plain-language check on what the model has learned; nothing about it touches training.
evals
Sep 8
Larry #hero-run-2026
Answers dlwh's proposal to throw sets out with a split instead of a deletion. Larry says he likes having the suspect sets exist somewhere as a historical reference to the projection: the preregistered target was set against the sixteen-set macro average, so dropping sets from that average would break the comparison the target was written for. Instead he would build a new aggregate alongside it, tracked for decisions about intervening in the run. He does not say which sets the new aggregate would hold, or who would build it. It is a proposal; nothing is decided, and no one replied to it.
evals
ihodes #8970
Sets the week's plan for the run, and names the first thing on the record that would need another restart. mcwitt writes that he will merge the fixes found while diagnosing the September 4 hangs and try again to deploy the ragged all-to-all expert-parallel backend to production — the transport this run was moved to on September 2 and rolled back off on September 3 after it hung twice. No date is given, and the standup is the only place that plan is recorded. Larry's plan is to look at granular performance across the perplexity evaluations at token level, and at whether the scaling-law asymptote needs a better theory: a combination of two evaluations with different asymptotes does not extrapolate the way one aggregate does. That is the per-set work the Paloma argument has been waiting on. The standup also records the spare NVL72 rack as down, which blocks GB200 kernel work; Larry had said he would examine the suspect Paloma sets once a twelfth rack was free, beyond the eleven this run occupies. Rafal Wojdyla and Mark Muchane hold the hero on-call this week. Nothing in the standup refers to the evaluation score logged thirteen minutes earlier.
planning
hero-wd-gate-router-p02-step58k
All four rising Paloma sets turned down, which is the shape one of the three competing explanations predicted. At step 77,999, logged at 5:58 p.m. (ET), the macro average over the sixteen Paloma held-out text sets fell to 2.2350 from 2.2432 at step 74,999. That single drop is three times the largest of the five before it and takes the average to its lowest value on this run. Fifteen of the sixteen sets fell; only mc4 rose, by 0.0001. Every set that had been rising since September 4 fell: ptb (2.2000 to 2.1629) and wikitext_103 (1.7364 to 1.6993) by far the most, and both now sit below where they stood on September 4; gab (3.3928 to 3.3805) and twitterAAE_HELM_fixed (4.5021 to 4.4810) fell too but remain above their September 4 values. A step up followed by a smooth decline is what Kaiyue-Wen predicted from the weight-decay change, and what dlwh said the team could wait to see. It is not what a noise floor predicts: a floor holds a set level, it does not let one recover. One score settles nothing, and nobody has posted since it landed — the last message in the channel came ninety minutes before it. The uncheatable suite went the other way for the first time since September 4, from 1.6722 to 1.6726, a change too small to read as anything. These are the cap-lifted scores; the capacity-limited key this page plots still has no points on this run, so the evaluation curve above stays frozen.
evals
dlwh #hero-run-2026
Narrows the morning’s proposal from replacing the run’s headline evaluation number to dropping particular sets out of it. dlwh writes that he would be open to throwing out the Twitter set Larry had just described, along with gab and 4chan, and perhaps a few others. Two of the three — twitterAAE_HELM_fixed and gab — are among the four Paloma sets whose loss has risen since September 4; 4chan is not, and he gives no reason for including it. Nothing is decided. The sixteen-set macro average is still what the run reports and what the evaluation curve on this page summarises, and dropping sets from it would change the number the preregistered target was set against.
Larry #hero-run-2026
Names what the manual scoring costs to run. Answering dlwh, Larry says the autoregressive generation loop he used to push prompts through the model ran on one GB200 rack. This run is placed on eleven racks, and Larry has said he will examine the suspect Paloma sets once a twelfth is free. The figure covers his own prompt runs only. dlwh’s token-by-token check is a different job and still needs a hook added inside the training process at a restart.
Larry #hero-run-2026
Describes what is inside one of the suspect sets, which gives the formatting-mismatch explanation its first concrete support. Larry says the Paloma Twitter set — twitterAAE_HELM_fixed, posts written in African-American English — is made of documents about ten tokens long containing random emojis. Very short text full of unusual characters is the kind dlwh had guessed the training mixture holds too little of. Larry does not say whether the other suspect sets look similar. dlwh proposed dropping this set half an hour later.
Larry #hero-run-2026
Separates the training from the evaluation argument running beside it, and finds the training sound. Larry says the train loss still looks very good to him, posting a curve whose midpoint he identifies as the weight-decay swap — the September 4 relaunch that put this run on its current optimiser settings. He expects no intervention beyond bookkeeping on the evaluations and on how the scaling-law asymptote is modelled. The claim is about the training curve alone; the Paloma dispute concerns what the run reports, not how it trains.
Larry #hero-run-2026
Answers how the per-token scoring was run, and the answer sets a condition on the test dlwh proposed. Larry says his prompts went through a manual autoregressive generation loop — he drove the model himself rather than having training call the scorer. So there is no hook in the training job to reuse, and running the log-probability check over the Paloma subsets means adding one at the next restart, as dlwh had guessed. Nothing schedules that restart; the run is training normally and has no reason to stop.
evals
mcwitt #8959 open
Puts up the first named piece of the fix set that the week's restart plan waits on. mcwitt upgrades QuACK, the library holding the grouped matrix-multiply kernels this run's expert layers use, from 0.6.1 to 0.6.4 for an upstream scheduling fix: the old scheduler can leave parts of a grouped multiply's output unwritten when work contends for the GPU, and GB200 tests reproduced that corruption. He states plainly that its link to the September hangs in #8870 is unproved. He opened it as a draft at 2:22 p.m. (ET) and marked it ready at 6:06 p.m. (ET), and holds the merge until a full-rack training smoke test and matched twenty-minute benchmarks, both restored from one frozen hero checkpoint, show what the upgrade does to loss and to speed. The standup ([#8970](https://github.com/marin-community/marin/issues/8970)) names merging these fixes as the step before retrying the expert-parallel backend on this run, which is the only change on the record that would need a restart.
incidentthroughput
mcwitt #code-review
Puts the fix for the frozen evaluation up for review, and says the team does not intend to switch that evaluation back on. mcwitt posts #8911 and describes it as the fix for the issue that caused the capacity-limited evaluator to be disabled: an inconsistent inferred sharding makes JAX compile two separate programs, one for the first training step and one for the rest, and the two conflict over NCCL memory-window registration. He adds that the team does not need to re-enable the capacity-limited evaluator and that the PR does not do it. He wants it merged for correctness, and because a similar NCCL-implicated hang is still under investigation in the ragged all-to-all backend. This is the first posted statement of intent about the evaluator whose curve this page plots: it means that curve is likely to stay frozen even after #8911 lands.
evals
Kaiyue-Wen #hero-run-2026
Offers a third explanation for the rising sets and, unlike the others, a prediction that the existing curves can settle. Kaiyue-Wen writes that the weight decay this run added acts on some modules much like a raised learning rate, which would shift the model’s effective temperature — how sharply it concentrates probability on its top guesses. That predicts a particular shape: loss on the affected sets steps up to a higher level once, then resumes falling smoothly. They say this may be what the curves already show “if you squint.” No one has checked the per-set history against the prediction.
evals
dlwh #hero-run-2026
Proposes an alternative to the noise-floor explanation, and a way to test it that the run cannot currently perform. dlwh writes that much of Paloma is formatted unusually and the training mixture may hold too little text of that kind, so the model would be mismatched to those sets rather than out of headroom — which would make replacing the macro average the wrong response. He suggests running his log-probability visualization, an existing tool that scores a model token by token across sample documents (#853), over the Paloma subsets, expecting the loss to look “super spiky” on formatting tokens if he is right. He asks whether the scoring runs from a callback and whether he must add one at the next restart. On Kaiyue-Wen’s competing prediction he notes they could simply wait and see whether it holds.
evals
Kaiyue-Wen #hero-run-2026
Sharpens the morning’s objection and names the observation that no noise floor explains. Kaiyue-Wen argues that a floor predicts the affected sets hold roughly level while the rest keep falling, so the macro average should still decline smoothly — and calls it concerning that it may not. What does not fit is that loss on the three sets is rising rather than flattening; a floor cannot produce that. They allow it could be bad luck. This is the first reply to the question they posed nine hours earlier.
evals
Larry #hero-run-2026
Answers the open question with per-set evidence and explains the apparent plateau. Larry reports that the three Paloma sets he had called suspect have been rising over the last ten thousand steps, and that this produces the impression of a plateau in the macro average — the equal-weighted mean over the sixteen sets that serves as the run’s headline evaluation number. A few sets moving the wrong way can hold that mean flat while the rest improve. He adds that the previous 67B-A2B run looked cleaner on the same three, reaching 4.40 on twitterAAE, a Paloma set of African-American English posts from Twitter; but on the other two it never got below where the 535B already sits, so the smaller model was not simply better on all three.
evals
Larry #hero-run-2026
Raises what a high noise floor would cost beyond the choice of evaluation. Larry notes that the asymptote of a scaling-law fit is meant to estimate the noise floor. The ladder that set this page’s target line fits the form L = 1.5 + A·C^-α (#8435), so 1.5 is that fit’s own claim about where the floor lies. If the real floor is far above 1.5, he argues, the problem is not merely which evaluation the run reports but the functional form of the fit — and that fit is what the preregistered target was extrapolated from.
evalsscaling-law
Kaiyue-Wen #hero-run-2026
Questions the noise-floor explanation proposed an hour and a half earlier. Kaiyue-Wen asks whether the run’s headline evaluation number should plateau at all when only some of its sixteen Paloma held-out text sets do: the macro average gives every set equal weight, so the sets that keep improving should keep pulling it down after a few stop. The point bears on dlwh’s proposal to replace that average, because a macro average that is still falling is not itself evidence of a noise floor. Nobody has replied. The most recent per-set numbers, logged at 1:26 a.m. (ET), support the objection rather than a plateau: the macro average reached its lowest value on this run and twelve of the sixteen sets improved. The measure at issue is the one the run still logs, not the capacity-limited key this page’s evaluation curve plots, which has had no new point since September 4.
evals
dlwh #hero-run-2026
Proposes a cause for the suspect Paloma curves and a change to the run's headline evaluation number. dlwh writes that Will Held — willheld in the same channel, who posted the September 4 weight-decay notes — suspects the run has reached the noise floor on some of the sixteen Paloma held-out text sets: the score on those sets stops falling because what remains is variation in the text itself, which more training cannot remove. dlwh agrees, and says the team should drop the Paloma macro average for either the uncheatable suite, the second held-out suite the run scores alongside Paloma, or a Paloma micro average, which weights each set by its size instead of giving all sixteen equal weight. This is the measure the evaluation curve on this page summarises. Nothing is decided and no metric has changed. The reading fits the per-set numbers logged at 1:26 a.m. (ET): four of the sixteen Paloma sets have risen since September 4 and twelve have fallen, and all four were already rising before the September 7 restart. It answers part of what markhart0034 asked and what Larry replied to; Larry still plans to check the sets himself once a twelfth rack is free.
evals
hero-wd-gate-router-p02-step58k
The first evaluation since the September 7 restart, and the first per-set evidence on the question raised on Discord. At step 74,999, logged at 05:26:20Z, the macro average over the sixteen Paloma held-out text sets fell to 2.2432, its lowest value on this run, from 2.2458 at step 71,999 — the last score before the restart. The uncheatable suite the run reports alongside Paloma fell to 0.5296 bits per byte and has fallen at all six scores since September 4. Per set, twelve of the sixteen Paloma sets are lower than at the first of those six scores and four are higher: gab (3.3791 to 3.3928), ptb (2.1841 to 2.2000), twitterAAE_HELM_fixed (4.4319 to 4.5021) and wikitext_103 (1.7083 to 1.7364). Larry named three suspect sets without saying which; by this measure four have risen. The rise does not begin at the restart — all four were already rising through September 4 to 6 — and at this score twitterAAE_HELM_fixed and wikitext_103 fell back while gab and ptb rose. These are the cap-lifted (dropless) scores, the only ones the run logs; the capacity-limited key this page plots still has no points on this run, so the evaluation curve above stays frozen.
evals
Larry #hero-run-2026
Flags three of the Paloma held-out text sets as suspect, replying to markhart0034’s message above. Larry writes that some of the Paloma curves look wrong to him, and that the uncheatable suite — the seven extra held-out datasets the run reports alongside Paloma — is still falling cleanly. He will look into the three sets once a twelfth rack is back, perhaps Tuesday; the run itself is placed on eleven racks, and he does not say which rack is out or why. This bears on the evaluation curve on this page, which is a macro average over the Paloma sets.
evals
Sep 7
hero-wd-gate-router-p02-step58k
Resolves why the evaluation curve on this page stopped advancing: the run changed which held-out evaluation it logs, and evaluations themselves never stopped. This page plots eval/paloma/macro_loss, the measure taken with the per-expert capacity cap in force, as in training. The run has logged no point under that key since its history moved to the run id hero-wd-gate-router-p02-step58k on September 4, and the key is absent from its stored summary entirely. It has logged eval_dropless/paloma/macro_loss, the same evaluation with that cap lifted, on the same regular 3,000-step interval throughout — five points since the move, the most recent at step 71,999, shortly before this morning's outage. #8859 turned the capacity-limited evaluator off to work around #8861, in which the first training step after that evaluator ran failed every time on the ragged expert transport; the production run went back to the pooled-wave transport on September 3 and does not have that failure, so nothing posted records why the evaluator stays off for it. The two measures are not interchangeable: the run that preceded this one logged both, and the dropless value sat about 0.19 below the capacity-limited one at every shared step, including 2.2525 against 2.4430 at step 56,999. An earlier entry here reported that the run had logged no evaluation point at all since September 4. That was wrong. The training loss curve is unaffected and is current.
evals
markhart0034 #hero-run-2026
Opens the thread Larry answered later that evening, and questions the curves themselves. markhart0034 writes that the training loss and the evaluation loss both look a little off to him since that morning’s restart, and allows that it could be noise. He gives no numbers and names no measure. Against the loss chart on this page, the training curve shows no break at the restart: over the sampled points, the mean of the 2,400 steps since it sits above the mean of the 2,000 steps before it by less than half the scatter between neighbouring points, so the difference is within the noise he allows for. The curve has flattened, but it flattened through the first week of September, which predates the restart. The evaluation curve on this page has not moved since September 4 for the separate reason recorded below — the run changed which evaluation it logs. This message was not in the corpus when it was posted; an earlier entry here noted the thread had no recorded opening.
evalstraining-dynamics
hero-wd-gate-router-p02-step58k
Training restarted and the chart is moving again. W&B records a new attempt started at 13:59:05Z on host s1mwxs64, and the run state is running, so the cluster placed all eleven racks the job needs. The outage lasted three hours and fifty-eight minutes from the 10:01:27Z fabric fault. The attempt restored the step-72,183 checkpoint saved at 09:10:32Z and replayed about 160 steps without plotting them, because W&B does not re-plot steps at or below a resumed run's high-water mark. It cleared step 72,344, the point attempt 1 reached, between 14:40Z and 15:00Z, so every step it has logged since is new ground. Nothing has been posted to #8506 or #8934 about the recovery, so it is not recorded what released the rack-394 reservation the job was waiting on.
incident
loom-oa-dev #8934
Names the cause of the morning's stop and reports the recovery blocked at the cluster operator. At 10:01:27Z task 16, on node s1wvxs64 in rack dh1-394-14, logged Xid 149 NETIR_LINK_DOWN and Xid 154 Drain and Reset — the node's link into the GPU fabric went down and the driver reset the device. Fifteen retained disruption snapshots from rack 394 mark both fault conditions and a production reboot, and carry a reservation taint and a separate evict taint. Node-local agents came back on seventeen rack-394 nodes between 10:20:07Z and 10:28:49Z, but host reboot completion is unverified. Iris deleted the failed task and requeued the 176-task coscheduled group; at 10:55Z every task was still attempt 2 BUILDING with no start timestamp, and Kueue — which admits the job — could fit only nine of the run's eleven slices, each of which needs sixteen GB200 nodes in one NVL72 rack, because seventeen nodes still held the reservation. The latest confirmed complete resume checkpoint is step 72183, saved at 09:10:32Z; attempt 1 had reached step 72344, so a restart replays about 160 steps. Attempt 2 has not logged a checkpoint load or an optimizer step, so restore and resumption are both unverified. The asks to CoreWeave are to identify the Xid-149 source, confirm which seventeen nodes are reserved and whether each completed its reboot and health checks, finish the provider-side recovery, and advise whether to quarantine rack 394 from this run. Cluster permissions denied the investigation direct node reads, so boot-ID changes and exact taint membership are unknown; it made no changes to Iris or Kubernetes.
incident
hero-wd-gate-router-p02-step58k
Training stopped. W&B reported the run state as crashed by 10:20Z, and the last step in the sampled loss history is 72,333. The stop was written up at 11:00Z (#8934): a node in rack 394 lost its link into the GPU fabric at 10:01:27Z, and the job's own logs show it had reached step 72,344 by then. A new attempt started at 13:59:05Z, so the outage lasted just under four hours. It ended about forty hours of uninterrupted training since the September 5 restart.
incident
Larry Dial hero-wd-twitter-pertoken-72k
An attempt at the suspect Paloma sets that produced nothing to read. Larry Dial started a job named hero-wd-twitter-pertoken-72k at 6:04 a.m. (ET) on September 7. The name points at a per-token look — the score broken out token by token rather than averaged over the whole set — at twitterAAE_HELM_fixed, one of the sixteen Paloma held-out text sets, taken from around the hero run's step 72,000. W&B records it on 64 devices, carrying the same model-shape and data-mix tags as the hero run. It finished 375 seconds later having logged no metric and no history at all, so no number can be read off it. It started three minutes after the rack-394 fabric fault stopped the hero run on the same cluster; nothing posted records whether that is why it logged nothing, and nobody has referred to it in the status log or on Discord. It predates Larry's September 8 message saying he would look into the suspect sets once a twelfth rack is free, so it does not settle that question.
evals
Sep 5
mcwitt #8870
Prepares the QuACK upgrade that carries the upstream scheduler fix. The branch upgrade-quack-064 takes the corrected matrix-multiplication scheduler from upstream, and a follow-up commit adapts Marin's tensor views to the new library's batch-first calling convention. Nothing is deployed and the original hang stays unexplained; this only makes the fix available to test.
incident
mcwitt #8870
Argues that the day's NVLink fault is the best explanation this hang has had, and that it is hardware rather than code. The production run was on pooled wave, not the ragged transport, and it still stopped on one of the three racks the September 4 telemetry window had flagged. The two transports would report the same fault differently: pooled wave moves data through NCCL's host-driven path, where a fabric fault surfaces as a contained CUDA error and the driver tears the context down, while the ragged kernel spins in a barrier created with no abort flag, no watchdog and no way to raise an error, which is exactly the silence both September 2 attempts produced. Rack 10 supplied the day's fault and the window's 124 ms straggler; rack 8 was the September 2 attempt-0 stall node set and took 5.5 s on a fabric bind that takes half a second elsewhere. The September 2 attempt-1 hang, on rack 4, shows nothing in any sweep and may be a different event. The NCCL version skew between the hung wheel and every clean one remains the only code difference, but is now less likely, because the day's fault involved no ragged kernel at all. He asks for a pre-flight that reads NVLink error counters and times the first window bind per rack, so a bad rack is refused before the next ragged deployment.
incident
mcwitt #8934
Files the fabric faults against the cluster operator, separately from the training bug, because every fix is switch-side. Two NVL72 domains are implicated. On dh1-r137, which holds rack index 10 in every hero placement, the run completed step 65,069 at 17:19:31Z, stalled for fifteen minutes with all ranks blocked on the device, and at 17:34:36Z a node reported a contained NVLink fault; seven more nodes in the domain followed within half a second and every rack within seven seconds. The restart cost 125 steps and twenty minutes. GPU-side link counters did not move across the event, so the fault is switch-side or a non-link hardware error. On dh1-r129, one GPU's link has logged 505 recovery events over its life and four more while the run was on it — the only NVLink counter that moved anywhere in three read-only sweeps of 320 GPUs — and that domain's first multicast bind took 5.5 s against 0.44-0.63 s on the other ten racks, an excess of almost exactly five seconds, which reads as a fabric timeout that then succeeded. Placement has been deterministic since September 2, so the same hardware keeps landing under the same job. The asks are node-level fault logs, a diagnosis of the flapping link, and a per-rack health signal the scheduler can read before it places a 704-GPU job.
incident
Helw150 #8824
Closes the open-weight comparison. Every reference base model the team wanted has now been measured on the same two held-out text suites, and the run's forecast at 18T tokens lands ahead of all of them except Kimi K2 Base and the two DeepSeek V4 models. Helw150 says the result looks better than he expects and invites the issue to be reopened if further models are worth adding.
evals
loom-oa-dev #8506
The status-log record of the interruption, posted as the run came back. It links an incident record and files the stop under InfiniBand port errors; the analysis in #8934 three hours later names a contained NVLink fault on rack dh1-r137 instead.
incident
hero-wd-gate-router-p02-step58k
Training stopped logging at step 65,067 at 17:18Z and resumed at step 65,070 an hour later, at 18:19Z. It has run without a break since, and W&B reports the run state as running. The cause was written up the same evening: a contained NVLink fault on rack dh1-r137 stopped the job at 17:19:31Z and the scheduler restored the hourly checkpoint (#8934). The restart replayed 125 steps, and W&B suppresses steps at or below a resumed run's high-water mark, which is why the chart resumes at 65,070 rather than at the restored step.
incident
Helw150 #8824
The last three baselines finished on batch-priority H100s: DeepSeek V4 Pro at 0.6759 Paloma bits per byte, Mistral Large 3 at 0.7759 and Motif 3 at 0.8187, against the run's 0.7406 forecast. All six records cover the full suites, so no result rests on a partial evaluation.
evals
mcwitt #8870
Turns the scheduler lead into a measured defect. On four GPUs, the original scheduler left regions of its output unwritten in three of the tested cases whenever another process competed for the same GPU; the fixed version did not. Each case ran in a fresh process with its own compilation cache and checked the result against known inputs. An audit of the compiled kernels the hung run actually loaded found the unsafe code in four of nine cache entries from its first attempt, across three distinct binaries, and those objects predate the run — so the defect was present in what hung, not merely in the version string.
incident
mcwitt #8870
Opens a second line of inquiry into the hang, independent of the fabric. The commit that hung pinned a version of QuACK whose matrix-multiplication scheduler can keep cancelling work after it has already left its work loop, which NVIDIA's instruction specification does not permit; upstream replaced that path with a drain that checks each response. If the defect stops one rank's matrix multiply from finishing, the other ranks in its expert-parallel group would wait at the next barrier forever. The earlier argument that the transport's own copy loops are bounded does not cover a stall that happens before them, so this remains possible. The cause of #8870 is still unproved.
incident
Sep 4
mcwitt #8870
Reports the second eleven-rack telemetry window and the first measurement this hang has produced. The window ran 183 steps at the hero's exact shape with the barrier instrumented and did not hang. It did find an asymmetry between racks: on three of the eleven, one rank per event reached the pre-copy rendezvous 80-160 ms after its 63 clique peers, against 10 ms or less on the other eight racks. The late rank was a different rank on a different node each time, across six nodes, which argues against a single bad GPU or host. A long pre-copy wait means peers had not yet reached the routing kernel, so the delay sits upstream of the transfer rather than in it. 160 ms is three orders of magnitude short of the 45-second hang and nothing grew toward one, so whether this is the same phenomenon is open. The hero resumed from the forced step-61806 checkpoint at 23:35Z.
incident
Mooler0410 #8818
The outside group agrees with the decision to leave the first two layers alone. They repaired the same layers in their own model after its pre-training run finished, having found the problem midway through, and say it caused no fatal issues in that run.
training-dynamics
ClassicLarry #8818
Answers the outside group's report and rules out a mid-run fix. The hero's early attention layers are still learning, he argues, just managing their scale through the attention gate; but the near-zero gradients on the routed experts of layers 0 and 1 do suggest those experts contribute nothing. He judges the cost small: those two layers hold about 4% of the routed parameters, and doubling routed parameters is worth roughly 20% more training tokens near compute-optimal, so the loss is marginal, helped by the run's unusually large shared experts. The plan is to leave them for now and either remove them after pre-training or study them on saved checkpoints.
training-dynamics
mcwitt #8925
Files the scheduling gap that blocked a #8870 test. A production sixteen-node gang sat gated for over an hour behind two smaller batch jobs, because the cluster scheduler only considers evicting lower-priority work when quota is exhausted, and here the fleet had free GPUs while no single rack did. On an NVL72 cluster the rack is the unit that matters, so any gang larger than the best rack's free space can queue behind batch work while the cluster reports spare capacity. The fix wanted is quota tracked per rack rather than per fleet, which needs scheduler work that is still open upstream.
incident
mcwitt #8912
Collects the five reasons the September 2 hangs produced fifteen minutes of complete silence, so the next one is readable. The cluster's per-task records declare fields for GPU utilisation and memory but never write them — both were null in all 2,636 samples taken during the first hang — although that one number alone separates a kernel spinning from a kernel never launched, which is the first question any analysis of this failure has to answer. Enabling the progress diagnostic silently switched the watchdog off on every process except the first. Crash dumps are read from the environment once when a process starts, so they cannot be armed on a job that is already stuck, and a hang throws no exception to trigger one. The sharding dump was written to the pod and died with it. The diagnostic build that fixes the collection also changes the library version the device kernels compile against, which is a cost the issue records rather than hides.
incident
mcwitt #8911 open
Opens the change that would let the capacity-limited evaluator be switched back on — the evaluation this page’s curve plots, and the one the run has not logged since September 4. The training state’s step counter leaves setup and checkpoint restore laid out across the GPUs one way and comes back from the training step laid out another way. JAX keys its compilation cache on that layout, so it compiles the training step twice, and each compiled copy registers its own NCCL memory window over the same shared arena at the same address. Two windows are the condition #8861 fails on. The PR places the counter on the layout the training step returns, once the state is final, so there is one compiled step and one window per rank and a second one cannot shadow it. mcwitt records what it does not settle: the evaluator stays off on the ragged rungs under #8859, and running it between ragged training steps under this change is untested. The production run is on the pooled-wave transport and does not have that failure. Nothing posted records the PR merged.
evalsincident
mcwitt #8870
Two corrections that close the single-rack search. A one-rack test of the checkpointer's per-step broadcast ran clean for 164 steps but proves nothing: at one rack that broadcast collapses onto the same 64 ranks as the expert-parallel group and creates no distinct rank group, whereas at eleven racks it spans all 704. Because every reproduction so far ran without a checkpointer at all, that broadcast is the last per-step difference between the configuration that hung and the ones that did not, and it cannot be tested below eleven racks. He also retitled the issue to drop the claim that the hang follows a diagnostic watch step: the deployed build never compiles a separate watch step, and of the two hangs only one landed on a watch boundary.
incident
mcwitt #8870
Reports the overnight session on the hang and eliminates four more mechanisms, three by experiment. A grid sized to leave no spare room on the GPU ran 176 steps without stalling, so the kernel's threads are not starving each other. The shared compiler autotune cache cannot reach the training step, because this model's heavy matrix kernels are Marin's own rather than autotuned. Network writes cannot corrupt the barrier, since hardware rejects a write outside its registered region. And both routes by which ranks could build mismatched barriers are closed. With the kernel's copy loops shown to be bounded, the barrier is the only place left that can hang. Seven reproduction attempts returned nothing, and a single rack cannot reach the remaining space by construction, so closing this needs the failure caught in the act; the capture path is now armed by default.
incident
rjpower #8896 open
Gives each process its own compiler autotune directory on the node, after the crash that interrupted the run an hour into the relaunch. Three ranks sharing one directory lost their temporary files as the cache wrote them while checkpoint step 58,181 was saving; the remaining ranks aborted with them and the scheduler retried the 176-task gang. The cache stays enabled because turning it off once left a rack compiling for fifteen minutes without reaching a step. The change re-applies a fix written earlier for a smaller expert-parallel run.
incidentthroughput
Helw150 #8824
Kimi K2 Base is the first baseline to beat the run's forecast by a clear margin: 0.6754 Paloma bits per byte against the 0.7406 forecast, 8.8% lower, and lower on 14 of the 16 Paloma subsets. Both suites ran at full coverage.
evals
mcwitt #8506
The status-log record of the pause and the relaunch. The hero was cancelled at 23:14Z on September 3 to free its eleven racks for the #8870 debugging window; a checkpoint was forced through the training-control endpoint first, so the pause cost about one step instead of the roughly 44 minutes of work a cancel at a random point in the hourly checkpoint cycle would have thrown away. The window ran 235 steps across two jobs and did not reproduce the hang. The run was redeployed at 00:41Z under the new id hero-wd-gate-router-p02-step58k, from a clean checkout of main, carrying gate/router weight decay and the pooled-wave fallback and otherwise matching the paused run's recorded arguments field for field. It restored the checkpoint the pause wrote rather than the older permanent one, and took its first step at 00:50Z with the loss continuous across the break. The whole interruption cost about 96 minutes of the eleven racks.
incidentcheckpointing
hero-wd-gate-router-p02-step58k
The hero run's history moves to a new W&B run id. hero-wd-gate-router-p02-step58k starts logging at step 58,015, continuing the trajectory that hero-12d8b6f0-dee637 ended at when it was cancelled the evening before. The old run id stays in W&B marked crashed, because the job was cancelled rather than stopped cleanly, and it will record nothing further. The new run logs its evaluation under the dropless key only; the capacity-limited evaluation the old run also reported is no longer written.
evals
Sep 3
mcwitt #8890 merged
Points the hero launcher at the checkpoint the pause wrote instead of the older step-54000 one it still named. Relaunching from the stale pointer would have silently discarded about 4,000 steps, roughly a day of training on 704 GPUs. The checkpoint was forced immediately before the cancel and verified complete at 28,098 objects and 5.36 TB. It is marked temporary, which the PR notes is irrelevant to restoring it: both kinds are written by the same call in the same format, and the temporary flag is read only by the deletion policy.
checkpointing
mcwitt #8884 merged
Makes pooled wave the recipe's expert-routing transport again, so that no unplanned restart can bring the ragged backend back on its own. The live run was already on pooled wave after the September 2 rollback, but only because the rollback relaunched an older commit; this writes the choice into the recipe. The backend field also decides the rest of the configuration, so the run no longer needs Marin's patched runtime build. Against a paired one-rack measurement, pooled wave gives up very little compute efficiency but drops a much larger share of token assignments than the ragged transport would.
throughput
Mooler0410 #8818
An outside group reports a similar early-layer symptom from its own mixture-of-experts training, on 60B and 180B models with 768 experts per layer and eight active per token. Their routed experts in the first few expert layers stopped learning after a few thousand steps, while the shared expert stayed healthy and the loss, the evaluations and the expert load balance all looked normal. They trained with AdamW and weight decay, so it appeared as the weight norms in those layers falling toward zero — the opposite direction from the growing gate norm this thread tracks, but the same quiet early layers. They caution that the two setups share almost no components, and that their longest run stopped at 2T tokens, far short of this run's 18T target, so they offer the link as a possibility rather than a diagnosis. Write-up: alltoall.notion.site.
training-dynamics
hero-12d8b6f0-dee637
The hero run's logged history passed the step 54,232 it had stood at since the backend swap. W&B now records step 54,409, and the loss there sits on the run's earlier trajectory. The replay of steps the run had already trained is over and the loss chart is advancing again. The crossing came about an hour after the 02:40Z estimate; the run state stays running.
incident
mcwitt #8506
Rollback verified. The relaunched pooled-wave hero restored the step-54000 permanent checkpoint at 01:20Z and is training at about 17 s/step, with loss matching its own earlier trajectory step for step. Steps 54001-54262 are a replay of what the run had already trained before the swap, so its W&B history records no new rows until it passes 54262 around 02:40Z. No config change from before the swap; the next permanent checkpoint is step 60000.
incident
hero-12d8b6f0-dee637
W&B moved the hero run from finished back to running when the rolled-back job restored the step-54000 permanent checkpoint. The run's logged history still ends at step 54,232, the last row written before the backend swap, because W&B does not accept a step the run has already recorded. The loss chart therefore stays flat while the run retrains steps 54,001-54,262, and resumes only once it passes 54,262.
incident
mcwitt #8506
Rolled back the ragged all-to-all swap at 01:15Z after the new run hung twice, each time on one rack's expert-parallel domain with no error logged and the progress watchdog killing the gang after 15 minutes. The old code and the old run id were relaunched from the same step-54000 permanent checkpoint; the ragged run never wrote to that run's checkpoint tree or W&B history. Net cost of the trial: about three hours of the eleven racks.
incident
mcwitt #8870
Records the hang that ended the ragged all-to-all trial. Both times it struck the first plain train step after a diagnostic watch step, which the run inserts every tenth step. The plain and watch steps are two XLA executables that share one NCCL symmetric-memory arena and each register a window at its base. NCCL telemetry put one rack exactly three AllReduce operations behind the other ten, and Iris showed that rack's 16 tasks idling at half the CPU of the 160 ranks busy-polling, so the stall sits inside one expert-parallel domain rather than on a failed node. Neither a one-rack smoke on the same checkpoint nor the d768 ladder rung reproduced it.
incident
mcwitt #8861
Links #8870 as the same two-executables-one-arena design failing the other way. When the second executable has a different all-to-all footprint, as the capacity-limited eval does, the next train step raises ncclGroupEnd() failed: invalid argument; when the footprints are equal, as for the watch step, it hangs silently instead. So #8859, which runs only the dropless eval, removes the crash but not the hang.
incidentevals
mcwitt #8506
Deployment note for the backend swap. The pooled-wave hero was stopped at step 54262 at 22:27Z and the replacement launched at 23:26Z, taking its first step at 00:03Z after a 15-minute cold data-loader fetch and a 14-minute cold compile. Sets a go/no-go at step 54200 from a side-by-side 200-step trial report, and names the rollback: trigger_hero.sh at 04751985c7, resuming the old run from step-54000. The capacity-limited eval is off on the new run per #8859 and #8861.
incident
Sep 2
ClassicLarry #8868
Asks that the relaunch pin the step-54000 permanent checkpoint by path rather than resume from whatever the newest checkpoint is, so the handoff point from the old run to the new one is explicit and a temporary checkpoint written after step 54,000 cannot be picked up instead. mcwitt changed the flag to name a specific checkpoint six minutes later.
checkpointing
mcwitt #8868 open
Adds --initialize-from-checkpoint to the hero launcher, which names another run's checkpoint as a mandatory resume source. The ragged all-to-all deployment needs it: the relaunch continues hero-12d8b6f0-dee637 from its step-54000 checkpoint but must not append to that run's W&B history, because W&B rewind is not enabled for the org and the tracker rejects steps below the run's last logged step.
checkpointing
hero-ragged_a2a-ep-step54k
The ragged all-to-all replacement run started nine minutes after the old one was stopped, on the same eleven racks, the same d6144 shape and the same Harrier data mixture, resuming from the step-54000 permanent checkpoint under a fresh run id. It ended crashed after two attempts. Attempt 0 trained steps 54001-54070 with loss tracking the pooled-wave run at -0.006 nats, drop fraction falling from 3.4e-2 to under 1e-4, MFU rising 20.5 to 23.4% and 16.8 s/step against 18.5; step 54071 started at 00:23:30Z and never finished. Attempt 1 restored the same checkpoint, completed step 54000, then hung on step 54001.
incidentthroughput
hero-12d8b6f0-dee637
The hero run ended at step 54,232. W&B moved it from running to finished, which records a clean exit rather than a crash. This was the planned handoff: #8684 swaps the run to the ragged all-to-all expert-parallel backend at step 54,000, and the relaunch continues from that checkpoint under a new run id.
incident
ClassicLarry #8833
Holds the gate and router weight-decay change unmerged until the run restarts at step 55,000. The hero resumes by run id and picks up whatever the recipe says, so merging earlier would switch the decay on at the next unplanned restart instead of at the permanent checkpoint #8854 pins.
optimizer
mcwitt #8862 open
Points the hero-health alert and the scaling-ladder Paloma chart at the dropless evaluation's loss, because #8859 stops the capacity-limited evaluation from reporting on runs that use the ragged all-to-all backend.
evals
mcwitt #8861
New issue: on the ragged all-to-all expert-parallel backend at hero width, the first training step after a periodic evaluation fails on most of the 64 ranks with an NCCL error and the job shuts down. The evaluation reuses the training path, so it registers a shared-memory window of a different size over the same base address as the training step. The live run would hit this at its first evaluation after the step-54,000 backend swap, then crash-loop, because every retry resumes and reaches the same step.
evals
Larry #hero-run-2026
Answers the objection that decaying the attention gate would push the output-scaling job onto the normalization layers: posts the run's current RMSNorm gains, which vary less across depth than expected, so that parameter is not setting the overall output scale today.
training-dynamics
Kaiyue-Wen #hero-run-2026
States the main objection to the gate and router weight decay: if the decay shrinks the early-layer gate, the normalization gains in later layers may grow to compensate and recreate the same depth-dependent scale gap elsewhere. Accepts the change as low risk anyway, because the proposed decay is small.
optimizer
Larry #hero-run-2026
Explains why the attention gate is a worse home for output scaling than the normalization layer. The gate output averages under 0.05, so a token that landed on the axis the gate keys off would jump to the full multiplier and produce about twenty times the usual attention output. Expects the network to avoid that during pre-training, but notes a new vocabulary token added at random would not be safe.
training-dynamics
Kaiyue-Wen #hero-run-2026
Questions whether moving the scaling role onto the normalization layer is any better, and suggests a future architecture with an attention residual or gated highway connection would remove the depth-dependent scale difference rather than leave the attention gate to create it.
training-dynamics
mcwitt #8859 open
Turns the capacity-limited evaluator off on the ragged hero rungs and keeps the dropless one, which is the evaluation the reports and ladder figures already read. With the change a one-rack run restored from the hero's step-48000 checkpoint clears two evaluations and keeps training, and its Paloma loss matches the live run's step-47999 evaluation.
evals
Larry #code-review
Sets the schedule for the next restart: the expert-parallel backend change lands at step 54,000, and the extra permanent checkpoint at step 55,000 in #8854 gives the gate and router weight-decay swap a fixed comparison point. The regular permanent cadence would otherwise jump from step 54,000 straight to 60,000.
checkpointing
ClassicLarry #8854 open
Pins one extra permanent checkpoint at step 55,000, just past the point where the proposed gate and router weight decay would switch on. That gives a fixed comparison point between the decayed run and the undecayed one. The regular every-6,000-step schedule is unchanged.
checkpointing
ClassicLarry #8818
Outside precedent for the proposal: Microsoft's MAI-Base-1 report describes weight decay set per parameter group, weaker on attention and embedding weights than on the rest of the model.
optimizer
Larry #hero-run-2026
Lists ways a future run could handle the depth-dependent output scaling without the attention gate, including a normalized gate that centers all 48 head gates at one. Cautions against treating it as a fault at all while the loss and gradients look healthy.
training-dynamics
Larry #hero-run-2026
Plain-language summary of the attention-gate story for the channel: the model chooses to make its early layers contribute little to the output, and has co-opted the attention gate to do it by aligning all 48 head gates against the embeddings. Every layer is still learning and the first ten still matter most.
training-dynamics
Percy Liang #hero-run-2026
Argues a training statistic that does not hold across model scales deserves its own investigation, separate from the run. Asks how early the gate growth was detectable, whether the ladder suites show it, and whether the tracker should forecast more statistics than the loss and warn when one drifts off forecast.
training-dynamics
Larry #hero-run-2026
Answers the predictability objection: on the d1024 rung the decay barely moved the final loss, but the hero's attention gate is not following ladder dynamics, so the rungs cannot say whether decay pulls it closer to them or further away. A weak setting is the safer choice; he expects the gate norm to halve over roughly ten thousand steps.
optimizer
Kaiyue-Wen #hero-run-2026
First reply to the gate/router weight-decay proposal from someone other than its author: the change could help stability, but it may weaken how well the scaling ladder predicts the run's final performance.
optimizer
ClassicLarry #8833
Lowers the proposed weight decay from 0.05 to 0.02 after the checkpoint test showed how much the model depends on the attention gate. The pull request now anneals that 0.02 linearly to zero across the run, and the value can be raised later if needed.
optimizer
ClassicLarry #8818
Reads the scaling test as a reason not to collapse the gate norm quickly: the model relies heavily on the gate, and the loss curve and evaluations are both steady, so there is no cause to rush.
training-dynamics
ClassicLarry #8818
Scaled the attention-gate logits down on a step-48k checkpoint before the sigmoid. A 20 percent cut is nearly free, 40 percent degrades the model badly, and 60 percent or more collapses it, so the learned per-head gating is load-bearing.
ablation
Larry #hero-run-2026
Added a zoomed view to the public W&B report, so the recent stretch of the loss curve is readable without the whole run compressing it.
ClassicLarry #8833
Risk note on the gate/router weight decay: it could pull the gate down faster than the model adapts, which would show in the loss a day or two out; the fallback is to revert and retry weaker. A hundred-step trial on a step-48k checkpoint already shows both norms falling.
optimizer
ClassicLarry #8818
Final d1024 weight-decay ablation: decaying the router alongside the attention gate is consistently but slightly better than the baseline, and decaying the gate alone is indistinguishable from it. All differences sit inside evaluation noise, matching the d768 rung.
ablation
loom-oa-dev #8506
An InfiniBand port fault on host s55sxs64 made the node's link flap, and the training job lost it after step 50195. Iris restarted the job from the last checkpoint; the replacement attempt has clean hardware telemetry and needs no further action.
incident
Sep 1
hero-12d8b6f0-dee637
The training job logged its last step at 23:39 UTC and stopped sending heartbeats, so W&B moved the run to the crashed state. A replacement attempt picked the run back up within the hour and it is reporting again.
incident
ClassicLarry #8833 open
Turns on weight decay for the attention gate and MoE router by default in the hero recipe. The description now has the decayed continuation start as its own run, hero-wd-gate-router-p02-step54k, forking the step-54000 permanent checkpoint into a fresh run and tree rather than relaunching the hero in place, and states that it uses the pooled-wave transport that checkpoint was trained with. The optimizer state is unchanged, so a checkpoint trained without decay resumes cleanly.
optimizer
Larry #hero-run-2026
Proposes weight decay of 0.05 on the attention gate and router weights, scaled down to zero as the run progresses, and asks the channel for objections.
ClassicLarry #8818
Reads the ablations as all within noise, but still argues for constraining the norm, because the hero's attention-gate scale has already grown much further than on any ladder rung.
ClassicLarry #8818
Weight-decay ablations on the d768 and d1024 ladder rungs all land inside evaluation noise; a linear decay to zero is marginally the best of them.
ablation
Helw150 #8824
Nemotron 3 Super added to the comparison. The hero's final forecast is lower on 14 of the 16 Paloma subsets.
percyliang #8824
Cautions that the compared open-weight models may have trained on the evaluation sets, so the reported margins are a lower bound on how far ahead the run is.
evals
ClassicLarry #8818
The MoE router norm follows the same growth path as the attention gate, so light weight decay on both looks low risk and would keep them learnable later in the run.
ClassicLarry #8827
New issue collects prompt completions from three saved checkpoints, as a plain-language sanity check on what the model has learned so far.
ClassicLarry #8818
Measured the first-layer attention gate for every vocabulary token on a saved checkpoint. Only 546 of 128,256 tokens open any head, and those are mostly reserved tokens and undecodable byte fragments.
training-dynamics
ClassicLarry #8818
Summary of the gate investigation: output scale does not track a layer's impact, early attention is still used, and no immediate intervention is planned. Lists the conditions that would change that.
ClassicLarry #8818
Attention-only knockout on the saved checkpoint: zeroing attention in the first ten layers hurts the evaluation loss about twice as much as zeroing it in the last ten.
ablation
Helw150 #8824
New issue compares the hero's forecast quality with Nemotron 3 and Qwen3.5 base models in bits per byte, which is comparable across tokenizers where perplexity is not.
evals
Aug 31
ClassicLarry #8818
Layer-skip ablation: dropping any ten layers from the forward pass multiplies the evaluation loss, so the early layers are not redundant despite their small output norm.
ablation
Helw150 #8435
Published the clustering and classifier weights behind the Harrier data mixture on Hugging Face, so outside groups can reproduce how the mix was selected.
data
ClassicLarry #8818
The loss projection is still ahead of the scaling-ladder schedule. Per-submodule plots show the first twenty layers moving the residual stream very little.
ClassicLarry #8818
New issue: the attention gate's weight norm has grown far beyond the scaling-ladder trajectory, concentrated in the first five layers.
training-dynamics
Aug 28
mcwitt #8754
Context-parallel branches let the hero model shape train at 262K sequence length on one rack, restored from an early hero checkpoint. Router capacity is the open problem before a long-context cooldown.
long-context
Aug 27
loom-oa-dev #8506
Node s6htxs64 lost connectivity and the 176-task gang failed. Iris retried it automatically and training resumed from the last checkpoint. No software fault preceded the failure.
incident
Aug 25
mcwitt #8684 open
Swaps the in-flight hero run to the reworked ragged all-to-all expert-parallel backend from #8549, which dropped far fewer tokens at the same compute utilization in an A/B at hero shape. Needs a patched PJRT wheel, so the swap has to land at a planned restart. Also makes a resume that finds checkpoints but loads none of them fail loudly instead of silently restarting from random weights.
throughput
Aug 24
rjpower #8506
Loader-cache ablation finished: a 1 GB per-process decode cache costs almost no throughput, so the cache can stay small and give host memory back to checkpointing.
ablation
ravwojdyla #8506
A training task ran out of memory during checkpoint serialization and took the undersized coordinator with it. PR #8617 raises the coordinator to 4 GB and gives it a retry budget so one task failure cannot stop the job.
incident
rjpower #8506
Traced the leak: the checkpoint manager never clears completed TensorStore commit futures, so every save retains a staged shard per writer rank.
Aug 23
rjpower #8506
The coordinator ran out of memory while saving a checkpoint. The job was relaunched under a new coordinator instance carrying the memory fixes.
incident
Aug 22
rjpower #8506
Restarted on main with jemalloc after malloc_trim failed to stop the checkpoint-time host OOMs. Merged #8584 to abort stalled TensorStore reads that were blocking restores.
Aug 21
rjpower #8506
Restarted with bounded TensorStore restore parallelism, a 16 GB checkpoint offload cache, and a malloc_trim call after each save, to stop host OOMs recurring after checkpoints.
muchanem #8506
An uncorrectable NVLink error on node s9vrxs64 took down three tasks in the same rack.
incident
Aug 20
rjpower #8506
Host memory exhaustion recurs about every three hours. Tuning the XLA memory fraction and slop factor did not help.
incident
ravwojdyla-agent #8480 merged
Bounds the host memory a hero resume holds, after every resume attempt was killed by the container runtime once the checkpoint had loaded.
Aug 19
ClassicLarry #8435
Full model and training spec: 535B parameters with 23B active, 48 layers, expert parallelism inside each NVL72 rack, and a two-phase Harrier data mixture over 18T tokens.
spec