Marin 535B-A23B MoE

18T tokens · launched Aug 20 updated · auto-generated via LLM, using mumwelt and data from GitHub, Discord, and WandB
WandB report Run description Status log
71.8% · step 280,379 of 390,251
Aug 20 12.9T of 18T tokens Nov 9

Training has run without a fault since the hero run switched to 16,384-token inputs late Tuesday, October 13 (ET), up from 4,096 since launch and the first of two planned increases. The Open Athena team accepted the switch after it cleared every gate.

The share of tokens the expert router drops rose more than tenfold at the switch, from a very low base, as the switch's trial runs foresaw. Training and evaluation loss both stepped down at the switch, not because the model improved: a longer input gives each token more context, as the switch plan predicted. Model FLOPs utilization (MFU) rose as well, but only on paper: the figure shown here charges every layer for full attention while most layers attend only within a short window; #7934 merged Wednesday night (ET) to correct the count, too late for the build the run is on.

#10170 merged Friday evening (ET), the first of four changes that would cut the longer input's cost in speed and dropped tokens, also too late for that build; it spreads causal attention work evenly across the GPUs that split one sequence, which the next increase will need. The other three are still drafts: #10121 balances the same work across the GPUs that hold one batch, #10122 rewrites the step that stops attention crossing between documents, and #10145 would cut the dropped share at the 65,536-token second increase. Seventeen superseded checkpoints from finished phases were deleted to reclaim storage; #10180 reports that six checkpoints from the run's first weeks may mix two training steps' expert-router state, so they are not consistent snapshots.

Oct 17
ClassicLarry #10245 open
#10245 opened: the 17 superseded hero checkpoints have been deleted, so the rule that blocks deleting hero checkpoints goes back.
hero-runcheckpointinfrastructurestorage1024510218
ClassicLarry #10218 merged
#10218 merged: a temporary carve-out in the delete-protection rule let the Open Athena team thin 17 superseded hero checkpoints.
hero-runcheckpointinfrastructurestorage1021810180
loom-oa-dev #10233 open
#10233 opened: held-out evaluation would keep working when one sequence is split across GPUs, as the planned longer context lengths require.
hero-runevaluationsequence-length10233
ClassicLarry #10224 open
#10224 opened: the quality spot-check samples from hero checkpoints would be generated at a higher sampling temperature, after the low setting gave repetitive text.
hero-runevaluation10224
rjpower #10221 open
#10221 opened: temporary writes from every CoreWeave cluster would move to the scratch bucket that already holds the hero's rolling checkpoints.
hero-runcheckpointinfrastructurestorage10221
ClassicLarry #10218 open
#10218 opened: a temporary policy change would let the Open Athena team delete 17 superseded hero checkpoints from finished phases.
hero-runcheckpointinfrastructure1021810180
Oct 16
mcwitt #10170 merged
#10170 merged: context-parallel hero runs spread causal attention work evenly across context shards, which the planned longer context lengths require.
hero-runperformanceattentionsequence-length10170
mcwitt #10170 open
#10170 left draft after a rewrite: the zigzag now swaps query blocks inside the attention wrapper instead of reordering each stored sequence.
hero-runperformanceattentionsequence-length10170
loom-oa-dev #6494 open
#6494: the hero trains with grouped-query attention, not the latent attention or Kimi Delta Attention that the speed-up work was assumed to add.
hero-runattention6494
willheld #long-context
A proposal to finish an intermediate hero checkpoint on outside hardware was dropped; the hero is too large for the collaborators' spare capacity.
hero-runplanningcheckpoint
Oct 15
mcwitt-agent #10180 open
#10180 opened: six checkpoints from the hero's first weeks may mix two training steps' router state; later checkpoints behave as expected.
hero-runcheckpointmoe-router10180
ClassicLarry #10179 closed
#10179: a 67B ladder checkpoint beats Qwen3.5 on two public suites; the hero's rows in #8824 use a different eval split and packed documents.
hero-runscienceeval101798824
mcwitt #10170 open
#10170 draft opened: context-parallel hero runs would store each sequence zigzag across shards so every shard computes an equal share of causal attention.
hero-runperformanceattentionsequence-length10170draft
ihodes #10148 open
#10148 standup: Larry Dial plans to examine attention entropy on the hero during the context extension, the question the multiplier sweep left open.
hero-runplanningsequence-length101489615
mcwitt #10145 open
#10145 draft opened: ordering expert-MLP tokens inside each device, not across all of them, would cut dropped tokens at the next context length.
hero-runperformancemoe-routersequence-length1012110145draft
mcwitt #7934 merged
#7934 merged: the reported utilization figure now counts the short attention window most layers use instead of treating every layer as full attention.
hero-runmonitoringperformanceattention7934merged
bethehand #9874 open
#9874: a scan of a public code dataset found the lone Penn Treebank copy and a near-complete WikiText test set, leaving the other dips unexplained.
hero-runscienceevaldata-mix9874
Oct 14
mcwitt #10122 open
#10122 draft opened: the hero's attention kernel would evaluate the document mask in blocks and drop its separate zero-fill launches, paired with #10121.
hero-runperformanceattentionsequence-length1012110122draft
mcwitt #10121 open
#10121 draft opened: the hero train step would reorder each batch's rows so the GPUs holding it carry a similar share of attention work.
hero-runperformanceattentionsequence-length1012110122draft
ClassicLarry #9615 open
#9615: the Open Athena team confirmed the default attention multiplier on all layers, calling the setting barely sensitive at this context extension.
hero-runsciencesequence-length9615
bethehand #9615 open
#9615: a paired step-by-step reading told the attention-multiplier arms apart where the evaluation totals could not, and backs the default.
hero-runsciencesequence-lengtheval9615
bethehand #9977 open
#9977: the pre-registered forecast of the hero's final loss was scored at the context switch, and the run came in below its predicted range.
hero-runscienceforecasteval9977
mcwitt-agent #8506
The 16,384-token cutover passed every agreed gate and stays on as the hero, with the evaluation series stepping down at the switch by design.
hero-runsequence-lengthcutoverdeploy850610101
hero-ctx16k-step257k
The hero cut over to 16,384-token inputs as hero-ctx16k-step257k, restoring the step-257,553 checkpoint it paused at.
hero-runwandbcutoversequence-length96159689
mcwitt #10101 merged
#10101 merged: the hero switched to 16,384-token inputs, keeping the default multiplier because all four sweep arms landed within eval noise.
hero-runsequence-lengthcutoverdeployscience961510101merged
mcwitt-agent #10100 open
#10100 opened: the hero's completions sampler would read pre-switch checkpoints with the new attention multiplier, and nothing flags the mismatch.
hero-runmonitoringsequence-length1010010101
mcwitt-agent #8506
The attention-multiplier sweep ended after three arms with the default kept, and the hero will resume at 16,384 tokens instead of 4,096.
hero-runsciencesequence-lengthcutover85069615
ClassicLarry #9615 open
#9615: a 67B probe plot and an attention-entropy artifact argue for raising the multiplier only on the layers that read the whole sequence.
hero-runsciencesequence-length9615
mcwitt #10024 merged
#10024 merged: W&B metric conversion now bounds its in-flight host copies, so hero processes stop accumulating the threads that slowed stack dumps.
hero-runmonitoringperformance10024merged
Oct 13
mcwitt-agent #8506
Training stopped at step 257,625 and the eleven racks went to wave one of the 16,384-token attention-multiplier sweep.
hero-runsequence-lengthcutover85069615
mcwitt-agent #9615 open
#9615: a proposed sweep to pick the attention-score multiplier before the longer-context switch would pause the hero, and the rack count is still undecided.
hero-runsciencesequence-length9615
mcwitt #9689 merged
#9689 merged: the launch script can now set the input length, attention and capacity values a context switch needs, the last of three changes.
hero-runsequence-lengthdeploy96159689merged
mcwitt-agent #9689 open
#9689 now records every context length the hero has left, answering the review that it held only one switch while the run plan has two.
hero-runsequence-lengthdeploy96159689
mcwitt #10086 merged
#10086 merged: the layers that attend to the whole sequence get their own attention-score multiplier, which the longer-context trial arms will vary.
hero-runsequence-lengthmodel-spec961510086merged
mcwitt #9890 merged
#9890 merged: the hero's evaluation and compute figures stay continuous across a relaunch, the second of three changes for the context switch.
hero-runsequence-lengtheval96159890merged
bethehand #9615 open
#9615 settled the default attention multiplier for the longer-context arms and the rule that arms be compared step by step on one seed.
hero-runsciencesequence-lengtheval9615
mcwitt #9888 merged
#9888 merged: each data source keeps its place across a switch to a longer input length, so the hero neither repeats nor skips text.
hero-runsequence-lengthdata-mix96159888merged
ClassicLarry #9615 open
#9615: the Open Athena team accepted dropping the two noisiest evaluation subsets but declined to fix the decision rule for the context arms in advance.
hero-runsciencesequence-lengtheval9615
bethehand #9615 open
A proposal on #9615 would fix the tie rule for the longer-context attention arms before the eleven-rack trials, judged without the two noisiest eval subsets.
hero-runsciencesequence-lengtheval9615
loom-oa-dev #10049 merged
#10049 merged: the hero alert runbook now explains how an empty telemetry query hides an active run and when a retry leaves no alert.
hero-runmonitoringdocs1004810049merged
Oct 11
loom-oa-dev #10049 open
#10049 opened: an empty thirty-minute telemetry query can read an active hero run as silent, so the operator runbook is corrected.
hero-runmonitoringdocs1004810049open
loom-oa-dev #10048 open
#10048 opened: a failed hero attempt and its retry can pass without a critical alert, since the stall rule restarts its clock for the replacement.
hero-runmonitoringfailure-mode10048open
Oct 10
mcwitt #10024 open
#10024 opened: #9600's all-at-once metric copies leave background threads behind in every hero process, which is why periodic stack dumps slowed the step.
hero-runmonitoringperformancefailure-mode10024open
Oct 9
mcwitt #9998 merged
#9998 merged: periodic Kubernetes thread dumps now read each rank's stacks without pausing it, so sweeps no longer stall the training step.
hero-runmonitoringperformance9998merged
mcwitt #9998 open
#9998 opened: periodic Kubernetes thread dumps pause every rank on the hero's new build and cost throughput, so a non-blocking dump is drafted.
hero-runmonitoringperformancefailure-mode9998draft
mcwitt #9990 open
#9990 proposes a one-rack eval check before each hero cutover, because the paired trial ends before the new run's first eval.
hero-rundeploycutovereval88619990open
mcwitt-agent #9615 open
#9615 re-ran its 8,192- and 16,384-token context arms on the hero's new build; both still train and the long-context cost is unchanged.
hero-runsciencesequence-lengthperformance961598349960
mcwitt-agent #8506 open
The new step-time build passed its 200-step paired trial against the old run on the same batches and stays on as the hero.
hero-rundeploycutoverperformance99608506
hero-mfu30-step231k
The hero cut over to hero-mfu30-step231k on #9960's step-time build, restoring the step-231,464 handoff checkpoint after the old run finished its control window.
hero-runwandbdeploycutoverperformance97089960
mcwitt-agent #9689 open
A review of #9689 found the hero launcher records only one context switch, while the run plan has two, and proposed a checked-in history.
hero-runsequence-lengthdeploy96898435
mcwitt #9960 merged
#9960 merged: hero handoff to a new run built on the six merged step-time changes, forking at step 231,464.
hero-rundeploycutoverperformance97089960merged
bethehand #9874
#9874: an audit of the hero's public evals ties most Paloma macro noise to the Penn Treebank and WikiText splits, and proposes dropping both.
hero-runscienceevaldata-mix9874
bethehand #9977 open
#9977 opened: an independent pre-registered forecast of the hero's final loss, with 80 and 95 percent prediction intervals.
hero-runscienceforecasteval9977open
mcwitt #9958 merged
#9958 merged: the hero launch drops two memory settings that abort the new build during compilation, clearing the first step of #9960's handoff.
hero-rundeploy983199609958merged
Oct 8
mcwitt #9960 open
#9960 drafted: hand the hero to a new run built on the six merged step-time changes, after a forced checkpoint and a paired trial.
hero-rundeploycutoverperformance97089960draft
mcwitt #9958 open
#9958 opened to drop two memory settings from the hero launch script, which otherwise abort the new build during compilation.
hero-rundeploy98319958open
mcwitt #9834 merged
#9834 merged: the hero's expert backward fuses the activation gradient into one kernel and stops storing recomputed values, the sixth of #9708's eight step-time changes.
hero-runperformance97089834merged
mcwitt #9833 merged
#9833 merged: the hero now takes its router gradient from the expert kernel, dropping a return transport, the fifth of #9708's eight step-time changes.
hero-runperformance97089833merged
mcwitt #9832 merged
#9832 merged: the hero's ragged expert transport no longer writes rows nothing reads, the fourth of #9708's eight step-time changes.
hero-runperformance97089832merged
mcwitt #9831 merged
#9831 merged: the hero's compiler stops recomputing values already in host memory and stops overlapping expert transfers, the third of #9708's eight step-time changes.
hero-runperformance97089831merged
Oct 7
mcwitt #9830 merged
#9830 merged: the hero router now selects its top-k experts in a Triton kernel, the second of #9708's eight step-time changes to land.
hero-runperformance97089830merged
mcwitt-agent #9879
Testing #9879's trace-timed compile on one hero rack hit a compiler crash and then an out-of-memory failure, so a rollout needs a preflight check.
hero-runperformance9879
mcwitt #9836 closed
#9836 closed unmerged while still in draft, the second of #9708's eight hero step-time pull requests to be dropped.
hero-runperformance97089836closed
mcwitt #9829 merged
#9829 merged: hero short convolutions now run in a Triton kernel that reads each tensor once, the first of #9708's eight step-time changes.
hero-runperformance97089829merged
rjpower #9721 merged
#9721 merged: a node whose storage requests keep failing is now pulled out of service before it receives more hero training work.
hero-runinfrastructureincidentstorage9721merged
mcwitt #9888 open
#9888 and #9890 opened to carry out #9615's context switch: hold each data source's place in the shuffle and keep evals at 4,096 tokens.
hero-runsciencesequence-lengthdata-mix961598889890open
mcwitt #9835 closed
#9835 closed unmerged and #9836 sent back to draft, leaving six of the eight hero step-time pull requests under review.
hero-runperformance970898359836closed
ClassicLarry #9874
Scanning the hero's training stream found Penn Treebank eval text verbatim before the strong checkpoint, so that Paloma split is contaminated.
hero-runscienceevaldata-mix9874
mcwitt-agent #9879 open
#9879 opened to compile the hero with timings measured from its own trace, a step-time gain the production run does not yet use.
hero-runperformance9879open
ClassicLarry #9874 open
#9874 opened to investigate why one hero checkpoint scored far better than its neighbors on Paloma's Penn Treebank text, with overlapping training data suspected.
hero-runscienceevaldata-mix9874
rjpower #9721
Review on #9721 set the node-quarantine check to run on the hero's own cluster and to release a machine once its storage recovers.
hero-runinfrastructureincidentstorage9721
Oct 6
loom-oa-dev #8506
Kubernetes deleted one of the run's 176 training tasks from a node it marked unusable, stopping the rest; the scheduler requeued them.
hero-runcrashrestartincident8506
mcwitt #9708 closed
#9708 closed unmerged: its hero step-time kernel and scheduling changes now live in the eight smaller pull requests #9829 through #9836.
hero-runperformance9708closed
mcwitt #9829 open
#9708's hero step-time speed-up was split into eight pull requests, #9829 through #9836, one per kernel or scheduling change.
hero-runperformance97089829open
ClassicLarry #9776 merged
#9776 merged: the hero's checkpoint prompt bank now scores every retained checkpoint on a passage of Paloma's Penn Treebank text.
hero-runevalcheckpoint9776merged
dlwh #9460 merged
#9460 merged: the Hero pipeline prototype saves and resumes model, optimizer and router state, matched bit for bit against an uninterrupted run.
hero-runcheckpointpipeline-parallel9460merged
dlwh #9708
A reviewer asked on #9708 to split the hero step-time speed-up into about eight pull requests, one per kernel or scheduling change.
hero-runperformance9708
Oct 5
ClassicLarry #9776 open
#9776 opened: the hero's checkpoint prompt bank would add a passage of Paloma's Penn Treebank text, scoring every retained checkpoint on it.
hero-runevalcheckpoint9776open
ravwojdyla-agent #9745 merged
#9745 merged: the hero's hourly completion sampler now keeps two sampling jobs active, so the backlog of unsampled checkpoints can clear.
hero-runevalcheckpoint9745merged
rjpower #9720 merged
#9720 merged: an operator's stop for a handed-off task now reaches the cluster running it, which before could leave a hung hero trainer alive.
hero-runinfrastructureincident9720merged
Oct 4
ravwojdyla-agent #9745 open
#9745 opened: the hero's hourly completion sampler would keep two jobs active, after a backlog left a long stretch of saved checkpoints unsampled.
hero-runevalcheckpoint9745open
loom-oa-dev #9729 open
#9729 opened: the hero's eval-regression alert would watch UncheatableEval loss instead of the noisier Paloma macro loss.
hero-runmonitoringeval9729open
Oct 3
David Hall hero-dropless-refit-new-20261003-aa4pdwep
David Hall logged forecast-only fits of the hero's end-of-run loss under the new and old data mixtures, with the loss floors refitted.
heroevaldata-mixforecast
rjpower #9721 open
#9721 opened: the scheduler would probe each machine's storage and stop sending work to one whose requests keep failing, the fault behind Saturday's stop.
hero-runinfrastructureincidentstorage9721open
rjpower #9720 open
#9720 opened: a stop request for a handed-off task would reach the cluster running it, which before could leave a stuck training process alive.
hero-runinfrastructureincident9720open
loom-oa-dev #8506
Triage on #8506 blamed Saturday's stop on cloud storage refusing one process's read of an earlier checkpoint during a save, hanging the rest.
hero-runcrashincidentcheckpoint8506
hero-fa4sm100-nomask-step146k
Run stopped after step 197,999 when a checkpoint save failed reading an earlier checkpoint; a later attempt restored and resumed training.
hero-runwandbcrash
Helw150 #9615
Two owners settled #9615's context-length handoff: at each change, skip the rest of every data source's current shuffle window.
hero-runsciencesequence-lengthdata-mix9615
mcwitt-agent #9636
An owner posted the OLMo-core comparison on #9636: the run's own expert kernels already match or beat Ai2's, and six changes were adopted.
hero-runscienceperformance9636
mcwitt #9708 open
#9708 draft opened: MoE kernel and scheduling changes that speed up the hero's training step on one rack, superseding #9481.
hero-runperformance9708draft
Oct 2
loom-oa-dev #9705 merged
#9705 merged: the scheduler no longer deletes a replacement task's configuration early, the race that kept ten hero tasks out of Friday's restarts.
hero-runinfrastructureincident9705merged
loom-oa-dev #9700 merged
#9700 merged: hero alerts now cover only the production launcher's jobs, after a similarly named test job triggered a page.
hero-runmonitoringincident9700merged
ClassicLarry #9615
An owner on #9615 suggested permuting sequences at the eventual maximum context length now, so the next switch to 65,536 tokens avoids reshuffling.
hero-runsciencesequence-lengthdata-mix9615
ClassicLarry #9615
An owner on #9615 backed skipping each data source to its next shuffle window at the context switch, deferring the final call to two others.
hero-runsciencesequence-lengthdata-mix9615
ClassicLarry #9615
A proposal on #9615 would fix shuffle blocks in tokens, give each data-mix stage its own context length, and skip sources to window boundaries.
hero-runsciencesequence-lengthdata-mix9615
mcwitt #9615
On #9615 an owner found no checkpoint step gives a clean context-length handoff; the simplest route still repeats or drops tens of billions of tokens.
hero-runsciencesequence-lengthdata-mix9615
mcwitt #9689 open
The context-switch schedule fix moved to #9689, reopened on the same branch after #9639 was closed unmerged.
hero-runsciencesequence-length9689open
loom-oa-dev #8506
Owners posted the incident record for Friday's stop on #8506: restart attempts timed out in cluster coordination and the initiating cause is unconfirmed.
hero-runcrashincident8506
asartran #9637 merged
#9637 merged: a run now records the commit it launched from and whether the tree was dirty, a W&B field the hero left empty.
hero-runmonitoringprovenance9637merged
hero-fa4sm100-nomask-step146k
Run stopped after step 193,760 with no error logged, the third stop in six days; a second restart attempt resumed training.
hero-runwandbcrash
dlwh #9662 open
#9662 draft opened: a pipeline-parallel launcher for the hero recipe on an upstream runtime, passing a small-scale gate before rack-scale validation.
hero-runthroughputparallelism9662open
Oct 1
ClassicLarry #9615
An owner on #9615 preferred skipping data to repeating it at the context switch, and asked whether a divisible checkpoint step allows a clean handoff.
hero-runsciencesequence-lengthdata-mix9615
ClassicLarry #9615
On #9615 an owner asked for the context trials on eleven racks, with evals logged under the existing field, and a global-only attention-scaling arm.
hero-runsciencesequence-lengthattention9615
mcwitt-agent #9639 open
#9639 draft opened: hold tokens per update constant across a context-length switch, keep a fixed 4K evaluation, and carry cumulative token and FLOP counts.
hero-runsciencesequence-length9639open
mcwitt-agent #9180
#9180 found that deferring allocator page purges did not cut save cost on one rack, and one rack does not reproduce the production slowdown.
hero-runcheckpointthroughputscience9180
mcwitt-agent #9615
On #9615 an audit of the context-length switch found that holding tokens per update fixed preserves the hero's schedules but not its exact data stream.
hero-runsciencesequence-lengthdata-mix9615
asartran #9637 open
#9637 opened: record a run's launch commit and dirty state in W&B, a field the hero's runs have left empty.
hero-runmonitoringprovenance9637open
dlwh #9636
#9636 opened: benchmark OLMo-core's MoE kernels against the hero's expert path for throughput gains that keep the current precision and routing.
hero-runthroughputkernelsmoe9636
loom-oa-dev #9620 open
#9620 opened: keep hero alerts firing while a replacement process restores, a gap that Wednesday's stop exposed.
hero-runmonitoringincident9620open
ClassicLarry #9615
On #9615 an owner warned that a context-length change must also update the learning-rate, weight-decay, data-mix and eval schedules, and progress metrics.
hero-runsciencesequence-length9615
ClassicLarry #9615
On #9615 an owner proposed jumping to 16,384-token sequences instead of 8,192, and testing attention scaling coefficients for 100 steps each to choose one.
hero-runsciencesequence-length9615
mcwitt-agent #9615
#9615 measured the hero's planned move to 8,192-token sequences on one rack: slightly fewer tokens per second, several times more router drops.
hero-runsciencesequence-lengthrouting9615
loom-oa-dev #8506
Owners traced Wednesday's stop on #8506 to a checkpoint save that some trainers never joined; the logs do not say why.
hero-runcrashcheckpoint8506
Sep 30
hero-fa4sm100-nomask-step146k
Run stopped after step 184,960 and resumed training on an automatic restart from a restored checkpoint.
hero-runwandbcrashrestart
yonromai #9584 merged
#9584 merged: a pinned hero checkpoint can now be exported to Hugging Face BF16 files for inference.
hero-runcheckpointinference9584merged
mcwitt #9600 merged
#9600 merged: W&B metric logging now starts every copy off the GPUs before reading any, the host cost #9205 attributed.
hero-runthroughputtelemetry9600merged
mcwitt #9600 open
#9600 opened: start every logged metric's copy off the GPUs at once instead of one at a time, the cost #9205 attributed.
hero-runthroughputtelemetry9600open
yonromai #9584 open
#9584 draft opened: export one pinned hero checkpoint to Hugging Face BF16 files for inference, reusing shards it has already verified.
hero-runcheckpointinference9584
Sep 29
mcwitt-agent #9205
#9205 attributed the hero's time outside the step timer to host copies of logged router metrics and an every-step wait to agree on saving.
hero-runthroughputtelemetry9205
rjpower #9479
Owner asked on #9479 when the job reached a hero rack, saying placement there should be impossible while the hero is training.
hero-runschedulingnvlinkiris9479
dlwh #9205
#9205 re-measured on the current build: time outside the step timer persists, and existing metrics cannot say which phase spends it.
hero-runthroughputtelemetry9205
yonromai #9479
On #9479 an engineer reported a job landing on a hero training rack and asked whether Iris should prevent that placement.
hero-runschedulingnvlinkiris9479
rjpower #9479
Owner answered #9479: named-rack pinning is acceptable, but rack sharing needs no fix because higher-priority jobs displace lower ones.
hero-runschedulingnvlinkiris9479
Sep 28
mcwitt #9415 merged
#9415 merged: the hero evaluation-regression alert now reads evaluation history from W&B, so it has input for the first time.
hero-runmonitoringeval9415merged
loom-oa-dev #9486 merged
#9486 merged: the hero loss-spike alert runbook now keeps its baseline reading and scopes the query to one cluster and run.
hero-runmonitoringalerting9486merged
Sep 27
mcwitt-agent #8317
Overlap tuning on #8317 stopped: #9481 is the practical scheduling limit on one rack, and every further change measured worse.
hero-runthroughputexpert-parallelprofiling8317
loom-oa-dev #9486 open
#9486 opened: correct the hero loss-spike alert query so it keeps its baseline and stops merging clusters that share a run ID.
hero-runmonitoringalerting9486open
loom-oa-dev #8506
Owners posted the incident record for this morning's stall on #8506: what initiated it is still unknown, and no GPU fault is established.
hero-runcrash8506
hero-fa4sm100-nomask-step146k
Run stalled after step 164,937 with no error logged, and resumed on a third attempt after two restarts failed to start training.
hero-runwandbcrashrestart
Sep 26
mcwitt #9481 closed
#9481 opened: pipeline the ragged expert transport's two chunks and feed XLA a measured latency profile, so more communication hides under compute.
hero-runthroughputexpert-paralleloverlap9481open
yonromai #9479
#9479 asks whether Iris should let a job require a named NVLink rack, after an audit found batch jobs on three hero racks.
hero-runschedulingnvlinkiris9479
Sep 25
rjpower #9395 merged
#9395 merged: the deletion block on hero checkpoints now applies in one bucket only, so copies held in the others can be cleaned up.
hero-runcheckpointstorage9395merged
dlwh #9460 open
#9460 draft opened: the Hero pipeline prototype can now save and resume model, optimizer and router state, verified on two bounded probes.
hero-runpipeline-parallelcheckpointseqlen9460open
dlwh #9278 merged
#9278 merged: the hero Muon optimizer can now keep expert-stack updates sharded when the stack needs no padding.
hero-runoptimizerexpert-parallel9278merged
Sep 24
mcwitt #9427 merged
#9427 merged: hero cutovers now hand off from a permanent checkpoint, and temporary checkpoints left by replaced runs expire in three days.
hero-runcheckpointstorage9427merged
mcwitt-agent #8317
Overlap re-measurement on #8317 finds most remaining exposed communication comes from a per-layer chunk barrier in the ragged expert transport.
hero-runthroughputexpert-parallelprofiling8317
mcwitt #9427 open
#9427 opened: hand hero cutovers off from a permanent checkpoint, expire replaced runs' resume checkpoints sooner, and check storage headroom before deploying.
hero-runcheckpointstorage9427open
mcwitt #9402 merged
#9402 merged: the run-status dashboard now projects the finish from the last 1,000 steps instead of the whole-life step rate.
hero-runmonitoring9402merged
mcwitt #9415 open
#9415 opened: the hero's evaluation-regression alert has never had data, because its evaluations land on steps the metrics feed skips.
hero-runmonitoringeval9415open
mcwitt-agent #8506
Owners posted the storage postmortem on #8506: a write suspension from an exceeded object-store quota caused both of Wednesday night's checkpoint stalls.
hero-runstoragecheckpointcrash8506
hero-fa4sm100-nomask-step146k
Run stopped again at step 146,731 with no hang signature, restarted within minutes and rolled back to the step 146,355 save.
hero-runwandbcrashrestart8870
rjpower #9395 open
#9395 opened: narrow the rule that blocks hero checkpoint deletion to a single bucket, after it blocked a routine cleanup.
hero-runcheckpointstorage9395open
#8870
On #8870 the Open Athena team traced the hang to rank 0 blocked in an S3 checkpoint-manifest write; why it blocked is unresolved.
hero-rungithubhangcheckpoint8870
Sep 23
Run hung at step 146,583 on the new build, was killed by the step watchdog and auto-restarted from the hourly save at step 146,355.
mcwitt-agent #8506
Owners posted GO on #8506: the new build passed its paired trial, faster with matching loss, and the hero stays on it.
hero-rundeploy85069378933293339266
mcwitt-agent #8506
Cutover announced on #8506: old run cancelled and hero-fa4sm100-nomask-step146k launched from step 146,139 on the native SM100 FA4 mask-free build.
hero-rundeployrestartcheckpoint85069378
mcwitt #9378 merged
#9378 merged: hero handoff to native SM100 FA4 attention, mask-free ragged MLP and coordinated garbage collection, forking at step 146,139.
hero-rundeployrestartcheckpoint9378merged
mcwitt #9333 merged
#9333 merged: mask-free ragged expert MLP, removing three unused tail masks from the expert layer.
hero-rundeploythroughput933393789374
ravwojdyla-agent #9387 merged
#9387 merged: hero checkpoint sample completions now finish, after a model change discarded every result and the job outran its own timeout.
hero-runevaluationmonitoring9387merged
mcwitt #9378 open
#9378 opened: plan to hand the hero off from the August build to main, with a one-rack test showing the new build trains faster.
hero-rundeployrestartthroughput9378936693339332
mcwitt #9366 merged
#9366 merged: repaired the job-launch path a hero restart would use after GPU wheels lost their executable bit.
hero-rundeploytooling936693409378
mcwitt #9374 open
#9374 opened as draft: expert-parallel speed-up split so #9333 keeps only the tail-mask removal and the symmetric buffers wait for more testing.
hero-runexpert-parallelthroughput93749333
mcwitt #9332 merged
#9332 merged: native SM100 FA4 attention kernels landed on main, not yet in the running hero build.
hero-runattentionthroughput9332
Sep 22
yonromai #8435
Router-precision investigation on #8435 closed, recommending the hero keep its current router arithmetic; the more accurate variant was slightly slower per step.
hero-runrouting8435experiment
yonromai #9352 open
#9352: evaluating a checkpoint at its stop step skips a pending router-bias update, so scores may reflect a different routing state than training would use.
hero-runcheckpointevaluation9352
mcwitt #9346 merged
#9346 merged: fix for the hero's length-1 batch mesh axis, which failed a sharding check and blocked #9332.
hero-runattentionsharding93469332
mcwitt #9333 open
#9333 opened: removing unused tail masks in the expert MLP improved throughput on one rack; symmetric buffers alone showed no gain.
hero-runexpert-parallelthroughput9333
mcwitt #9332 open
#9332 opened: attention backend using FlashAttention-4's native Blackwell kernels, selected in the hero config, with an expected single-digit throughput gain.
hero-runattentionthroughput9332
ihodes #9324
#9324 team plan: merge and deploy coordinated memory cleanup and SM100-native attention kernels to the hero for a modest throughput gain.
hero-runthroughputdeployment932493329266
mcwitt #9266 merged
#9266 merged: coordinated memory cleanup, validated on one rack with checkpoints and evals, though the hero launcher config does not yet enable it.
hero-runstep-timecheckpoint9266
Sep 21
hero-router-pref-fixed-eval-qb-step126020-01a0c066-a2
Router-precision A/B on a hero checkpoint, repeated on each arm: the held-out quality gap reproduces and favours the precision the hero already uses.
hero-runwandbrouterdiagnostic
hero-main-step121638
Checkpoint metadata fix passed its second test: the first hourly save after a permanent milestone committed and the run kept training.
hero-runwandbcheckpointpage-feed
Sep 19
x-itm #8870
Reviewer argues cluster-launch control is the key discriminator for the hang and proposes a three-arm test matrix on production geometry.
hero-rungithubhang8870
mcwitt #8870
Owners verify permanent checkpoint 126,000 wrote cleanly under the metadata fix and attribute the September 19 stop to a contained CUDA error on one GPU.
hero-rungithubcrashcheckpoint8870
dlwh #9283 open
#9283 draft opened: context parallelism for the Hero pipeline mesh, aimed at the planned sequence-length extension, not the running job.
hero-runlong-contextcontext-parallel9283
hero-main-step121638
Run stopped after step 124,714, the first fault on the main build, and auto-restarted from the hourly save at step 124,642.
hero-runwandbcrashincident
9277
#9277 feasibility report: the full 535B model trained at sequence length 65,536 on 192 H100s with pipeline parallelism, with unresolved NaNs in unsynchronized runs.
hero-runlong-contextpipeline-parallelh100
mcwitt #9268 merged
#9268 opened: give forked runs their own loss-history sample budget so the dashboard loss curve for the hero fork stops lagging.
hero-runpage-fidelity9268
Sep 18
mcwitt #9266 open
#9266 opened: enable coordinated garbage collection every 100 steps for the hero run, the fix proposed for the recurring wall-clock slow stretches.
hero-runstep-time9266
Owners posted the go decision on #8506: hero cutover to the new build accepted after the full 200-step overlap, child run left running.
Planned cutover to new W&B run hero-main-step121638 on a build carrying #9159 and #9215, restored from the step-121,638 hourly save.
hero-runcutoverwandb915992159242
mcwitt #9159 merged
mcwitt answers on #9159 that pretraining uses continuous compaction with no padding, so padding-aware MoE dispatch should be a no-op for the hero.
hero-runmoerouting9159
yonromai #9246 closed
#9246 closed unmerged: forward-reference tooling stays experimental, and future vLLM and Megatron parity PRs carry only per-backend regression tests.
hero-runcheckpoint9246evaluation
mcwitt #8870
mcwitt confirms on #8870 the stop was another permanent-checkpoint metadata deletion failure; the automatic retry restored step 120,000 and resumed.
hero-runcheckpointstorage-policy88709215
hero-nopdl-step108k
Run stopped at step 120,206 on a permanent-checkpoint metadata delete failure and auto-restarted from the step-120,000 save, replaying 206 steps.
hero-runwandbcheckpoint92158506
mcwitt #8870
mcwitt attributes the stop to an InfiniBand port failure rather than the #8870 silent GPU hang; the automatic retry resumed from checkpoint.
hero-runhanginfinibandretraction88709183
Run stalled after step 118,836 with GPUs pegged idle, was killed by the watchdog and restarted in place from the hourly save.
ravwojdyla #8827
Prompt completions from saved hero checkpoints now publish to a self-updating report with multiple samples per prompt and reference-completion scores.
hero-runcompletions88279244
Sep 17
dlwh #9231 open
#9231 opened: W&B run forking so a hero cutover can continue one chart instead of starting an unrelated run.
hero-runwandb9231levanter
closed
#9182 closed as deprioritized; delaying memory-page release showed no step-time gain, leaving #9224 the only open fix for slow stretches.
mcwitt #9224 open
#9224 opened: opt-in coordinated garbage collection across ranks, proposed as the first code fix for the slow stretches.
hero-runstep-time92059224grug
9159
#9159 review turned to portability: dlwh objected that TPUs cannot do the float64 capacity calculation the pull request now uses.
hero-runmoe9159review
loom-oa-dev #9215 open
#9215 opened: write remote checkpoint metadata directly to its final object so no sibling is copied and deleted under the protected prefix.
hero-runcheckpointlevanter92158506
mcwitt-agent #8506
Follow-up traces the refused delete to #8974, which switched checkpoint metadata saves to an atomic rename under the protected prefix.
hero-runcheckpointcrash85068974
mcwitt-agent #8506
Status log posts the cause: the bucket deletion policy protecting permanent hero checkpoints refused the step-114,000 save's metadata cleanup, ending the attempt.
hero-runcheckpointcrash85068870
hero-nopdl-step108k
Run stopped after step 114,198 on a checkpoint error and restarted in place from the step-114,000 save, replaying about 200 steps.
hero-runcrashrestartreplaycheckpoint887091838506
Sep 16
mcwitt-agent #9159 open
mcwitt-agent profiled #9159 and found its step-time cost comes from extra kernel launches, not GPU work; a fix followed the same evening.
heroroutingthroughput
mcwitt-agent #9159 open
#9159 H100 benchmark: skipping padding positions in expert routing changes loss only within noise but runs slightly slower; PR stays open.
herorouting
#9205
#9205 root cause posted: staggered full Python garbage collections stall ranks at the per-step broadcast; fix proposed to collect on all ranks together.
hero-runthroughput92059183
#9205
#9205 opened to diagnose the recurring slow stretches between logged steps, which also appear on the control run and so predate #9183.
hero-runthroughput920591839180
ClassicLarry 9148
ClassicLarry stood down #9148: total gradient norm looks stable, having moved to a new normal after the switch to near-dropless expert parallelism.
hero9148grad-norm
hero-nopdl-step108k
First held-out eval since the #9126 mixture swap landed and does not separate the old and new mixtures.
hero9126914891838870eval
mcwitt-agent 8506
Status log: #9183 hang-fix build clear of hangs since the step 108,778 handoff; owners set about 5,000 clean steps as the bar.
hero850691838870
8506#issuecomment-5690222518
Go call posted on the status log: the hang-fix build stays, having matched the control's loss and beaten its throughput over the trial window.
herohang887091839179gate8506
Sep 15
hero-nopdl-step108k
Mixture run stopped by plan after step 108,981 and the hang-fix trial hero-nopdl-step108k restarted from the step-108778 checkpoint.
herohang887091799183cutover
mcwitt-agent #8506
Status log cutover note deploys the PDL-off QuACK launchers #9183 and the new PJRT wheel #9179 as hero-nopdl-step108k with a written pass/fail gate.
hero850691799183cutovergate
hero-uncheatable-floor14-new-from30-75c36f
Will Held posted an end-of-run forecast for the new mixture, crediting the swap with a larger uncheatable-eval gain than any earlier method.
heromixtureforecast9126
Helw150 #9126
Helw150 posted the final d1536 ladder result on #9126, confirming the new mixture's forecast speedup with Wikipedia the only regressing Paloma subset.
heromixture9126ladder
mcwitt #9183 open
#9183 turns off programmatic dependent launch on the three QuACK grouped matmul launchers as a mitigation for the #8870 hang.
hero8870hangquack
mcwitt-agent #9180
#9180 opened: a 64-GPU reproduction traces the hero's checkpoint-time step stalls to the allocator returning freed pages immediately.
herocheckpointjemallocthroughput
hero-mix-996f4891-step108k
The #8870 hang stopped the new mixture run after step 108,710, the fifth on this build, and it auto-restarted from the last hourly save.
herohang8870mixture
mcwitt-agent #8870
Offline decode of the crash dump narrows the #8870 hang to a deadlocked thread block in QuACK's grouped matmul kernel and proposes disabling PDL.
hero8870hangquack
Larry #hero-run-2026
Larry posted in Discord that the model rebalanced its experts within ten steps of the swap and the new data is slightly harder to predict.
heromixturetraining-dynamics
willheld #hero-run-2026
willheld announced in Discord that the hero was rebooting with new sampling weights, linking the mixture log, the #9126 ladder comparison, and the small-scale swarm.
heromixturedata-mix
hero-ragged_a2a-nccl2307-ep-step81k
Run stopped after step 108,257 for the planned mixture swap and restarted as hero-mix-996f4891-step108k from the step-108000 checkpoint with the #9126 mixture.
herodata-mixrestart
Helw150 #9162
#9162 applies the mixture swap by restoring the permanent step-108000 checkpoint into a new run after the current job is stopped by hand.
hero-ragged_a2a-nccl2307-ep-step81k
The #8870 silent hang stopped the run for the fourth time after step 107,502 and it auto-restarted with nothing replayed.
herohangrestartgrad-norm
mcwitt #8870
mcwitt posted a September 15 incident report on #8870 covering the recurring hang, a successful capture of the stuck peer, and offline findings.
Will Held hero-uncheatable-joint-floor14-swap-20260914
Will Held's ten further forecast runs show forecasting methods disagree on the uncheatable eval by far more than the swap benefit they measure.
heroevaldata-mixforecast
Will Held hero-forecast-swap-adjusted-20260914
Will Held logged forecast-only runs projecting the hero's end-of-run Paloma and uncheatable eval if the #9126 mixture swap goes ahead.
heroevaldata-mixforecast
Sep 14
Helw150 #9162
#9162 opened: the #9126 data swap restores the step-108,000 checkpoint into a new run with the evaluated mixture; Larry approved.
ClassicLarry #8827
Larry Dial posted written completions from a recent hero checkpoint: code and factual prompts answer cleanly, some open-ended prompts loop on one phrase.
herocheckpoint
mcwitt-agent #9148
Investigation in #9148 clears the new routing kernel, traces the gradient-norm climb to the training path, and recommends no change.
herograd-norm
ClassicLarry #9148
Larry Dial opened #9148 to track the gradient-norm growth; no intervention unless the norm reaches 5 and the loss diverges.
herograd-norm
ravwojdyla-agent #9147 open
#9147 opened: would automate written samples from the hero's permanent checkpoints on a daily schedule; not yet in service.
herocheckpoint
mcwitt #8870
#8870 update: capture report inventories the GPU communication paths the stall runs through; root cause still unresolved.
herohang
hero-ragged_a2a-nccl2307-ep-step81k
Silent hang #8870 stopped the hero a third time on the NCCL 2.30.7 build after step 103,004; it auto-restarted from the hourly save.
herohangrestart
AUG-LIN0-2L-DENSE-MUONH-X600-HIGH-2P2-d512-600x-lr2.2
The 2.2 learning-rate probe above the two-layer grid finished and lost to 1.7, closing the sweep's two-layer tree with the grid unchanged.
lr-sweep
Two-layer learning-rate sweep wave finished: 1.4 won the four shorter lengths and 1.7 the longest, so the best rate drifts up with length.
lr-sweep
Sep 13
Six-layer learning-rate sweep probe above the tested grid placed third at its longest length, so the grid does not need widening.
lr-sweep
Helw150 h100-mix25-20260912-d1536
All three narrower mixture-swap ladder rungs finished ahead of their August controls; the largest rung now trains past its control's crash point.
ladderdata-mixturemilestone
WhenWen AUG-LIN0-6L-DENSE-MUONH-020 / -005
Six-layer learning-rate sweep closed all twenty cells with the two fastest tested rates finishing near-level across the five run lengths.
ladderlr-sweepmilestone
Mixture-swap ladder's 4B rung finished matched to August's: the new mixture leads without the router capacity limit but trails with it.
WhenWen moe-lgr-9110-larry-d768-muonh
#9110 corrected-routing test finished at both widths and ended level with its controls, so the correction showed no gain.
dialroutingmilestone
Larry #hero-run-2026
Larry opened a Discord thread on the hero's rising gradient norm, calling it benign and naming a rollback trigger.
herotraining-dynamicsgradient-norm
mcwitt #8870
#8870 silent-hang issue got a revised diagnosis comment linking to an internal incident note, shortly after the restart.
herohang
mcwitt hero-ragged_a2a-nccl2307-ep-step81k
Run silently hung at step 95,972 and restarted in place nineteen minutes later from the last hourly save.
herocrashhang
Sep 12
Helw150 9126
#9126 opened: asks the team to judge the mixture-swap ladder against the original scaling ladder before the hero data swap planned for September 14.
ladderdata-mixhero
WhenWen AUG-LIN0-6L-DENSE-MUONH
#7856 learning-rate sweep moves from one-layer to six-layer stand-in models, one per run length, at the rates the Muon tree picked.
ladderlr-sweepmilestone
WhenWen moe-lgr-9110-larry-d512
#9110 routing test restarted against matched controls since stored baselines could not attribute the gap; both runs crashed and were relaunched on a newer commit.
experimentmoeroutingcrash
held h100-mix25-20260912-d768
Correction: read without token dropping, the new scaling ladder's two finished widths beat their August counterparts by well over seed noise.
ladderdata-mixcorrection
WhenWen AUG-LIN0-1L-DENSE-SGDMH-X600-HIGH-1-d512-600x-lr1.1
Nesterov-momentum sweep family also settles one step above its grid at the longest run length; the climb turns back at the next rate.
ladderlr-sweepmilestone
WhenWen AUG-LIN0-1L-DENSE-MUONH-X600-HIGH-1P4-d512-600x-lr1.4
Muon sweep family settles one step above its grid at the longest run length, matching its pick at the run length below.
ladderlr-sweepmilestone
WhenWen AUG-LRC-1L-SGDH-X600-HIGH-2-d512-600x-lr1.7
Flat-rate MoE sweep tree settles on the top rate of its original grid at the longest run length; higher probes all lost.
ladderlr-sweepmilestone
WhenWen #9110 — first gate fails at d512 open
#9110 verdict: the corrected expert routing fails its first gate at d512, so the wider models it could have unlocked will not be built.
experimentmoeroutingcorrection
WhenWen #7856 LR sweep — 600x above-grid probes
First above-grid results at the longest sweep length: dense plain-gradient loss keeps falling past the grid while the flat-rate MoE tree stays level.
ladderlr-sweepmilestone
held h100-mix25-20260912-d512/d768/d1024/d1536
Four-width H100 scaling ladder started on the mixture the fitted mixture model picks, to fit a power law across widths for the hero.
ladderdata-mixscaling-ladder
WhenWen h100-d512-mixprior wave 5
Fifth data-mixture wave complete: one mixture's four seeds spread wider than the eight distinct mixtures, so the wave ranks nothing.
ladderdata-mixmilestone
larrydial dynact-latent-* d512 fa4fix
Attention-latent width probes finished: scores again rank by parameter count rather than shape.
ladderablation
larrydial dynact-dual-updown-d512
Expert-design test finished on H100: arms rank by parameter count rather than design, since no arm is size-matched to the baseline.
laddermoeh100milestone
Sep 11
mcwitt lb0911moebared3
Six one-rack tests finished: the compiled GPU plugin wheel costs a little speed and nothing in loss.
herobenchone-rackmilestone
mcwitt #9118 open
#9118 opened: short convolution made to work with sequences split across GPUs, part of the long-context phase planned for the end of the run.
long-context
mcwitt #9119 open
#9119 opened: splits sequences across GPUs so the hero shape can train at 262,144 tokens for the planned long-context phase.
long-context
WhenWen h100-d512-mixprior — seed repeats
Seed reruns of one data mixture spread wider than the differences between mixtures, so the wave's mixture ranking is within noise.
ladderdata-mixmilestone
WhenWen #7856
WhenWen cancelled the falling-rate sweep because it decayed to a twentieth rather than zero, relaunched it, and added a momentum optimiser.
ladderlrcorrection
WhenWen #9110
#9110 checks in a learned-gate router change, with tests reproducing the hero's current behaviour when switched off, and submits a trial run.
ladderrouter
Larry #hero-run-2026
Larry answered that unit-norm router weights are worth trying but the warning signs behind the proposal do not apply to the hero.
routerstability
Sheng Zha #hero-run-2026
Sheng Zha proposed four router changes for stability: unit-norm router weights, noisy or sampled routing, and mean-centred router bias.
routerstability
Sep 10
mcwitt #9086
#9086 asks for per-link fabric error counters to be collected, since the failed link had degraded for days with no stored record.
incidenthardwaremonitoring
mcwitt #8870
mcwitt's audit of the GPU transport source found no deadlock in the path the hero uses, so the hang remains unexplained.
hangdebugging
mcwitt #9077
#9077 proposes replacing the hand-written expert kernels with a compiler operation; a one-rack test matched on loss but two of four runs hung.
kernelshang
held h100-d512-mixprior × 20
All twenty data-mixture ladder rungs finished with held-out scores in a narrow band, so no mixture weighting stood out.
ladderdata
hero-ragged_a2a-nccl2307-ep-step81k
Run died at step 83,486 on an uncorrectable NVLink error in rack 9 and auto-restarted; mcwitt calls it evidence the hangs start in the fabric.
infraepincident
hero-ragged_a2a-nccl2307-ep-step81k
Run hung at step 83,306, the first silent hang on the NCCL 2.30.7 build, and auto-restarted from the hourly save.
infraepincident
mcwitt #8506
mcwitt verified the new checkpoint layout: a one-rack job restored the hero's own save and trained cleanly, and that save is pinned as a fallback.
infracheckpointsep
Sep 9
mcwitt #8506
mcwitt ruled the redeploy a pass: the released #9062 build reproduces the trial run and beats the control, making it the run of record.
infraep
mcwitt #8506
mcwitt redeployed the hero from main after #9062 merged the ragged all-to-all build, launching a new W&B run from the pinned step-81716 checkpoint.
infraepincident
mcwitt #8506
mcwitt rules the ragged trial a pass: 200 steps with no hang, lower loss, far fewer dropped tokens, and higher throughput than the pooled-wave control.
infraepthroughput
mcwitt #9062 merged
#9062 merged: pins the GPU compiler runtime to the NCCL 2.30.7 wheel and relaunches the hero under a new run id from step 81,716.
infrakernelsep
#8506
Four hangs in four attempts on four different racks; attempt six launched with a PJRT wheel built against matching NCCL 2.30.7 headers.
infraepkernelsincident
mcwitt #8506
New run hung at step 81,759 with one rack at half CPU load; mcwitt cancelled the coordinator and relaunched the same run id.
infraep
mcwitt #8506
Cutover done: old run cancelled at step 81,919 and hero-ragged_a2a-ep-step81k launched from the step-81,716 checkpoint carrying #8959 and #9043.
infraep
mcwitt #8506
Plan posted to stop the run today and relaunch on the ragged backend from a forced, pinned checkpoint with a 200-step matched control window.
infraincidentthroughput
mcwitt #9043 open
#9043 opened: makes ragged all-to-all the hero's default transport again with fp32 master weights on GPU, undoing the #8884 fallback.
infrathroughput
Sep 8
Larry #hero-run-2026
Larry proposes keeping the sixteen-set Paloma average as the historical reference for the target and building a new aggregate for intervention decisions.
evals
ihodes #8970
#8970 standup: merge the hang-diagnosis fixes and retry the ragged all-to-all backend in production; Larry to examine per-token Paloma performance.
planning
mcwitt #code-review
mcwitt puts #8911 up for review and says the team does not intend to re-enable the capacity-limited evaluator.
evals
Kaiyue-Wen #hero-run-2026
Kaiyue-Wen proposes the new weight decay shifted the model's effective temperature, predicting a one-time step up in affected sets then smooth decline.
evals
Larry #hero-run-2026
Larry notes a high noise floor would challenge the scaling-law fit's assumed asymptote, and so the functional form behind the preregistered target.
evalsscaling-law
dlwh #hero-run-2026
dlwh suggests some Paloma sets have hit their noise floor and proposes replacing the macro average with the uncheatable suite or a micro average.
evals
Larry #hero-run-2026
Larry flags three Paloma held-out sets as suspect while the uncheatable suite keeps falling cleanly, and will investigate once a twelfth rack is free.
evals
Sep 7
hero-wd-gate-router-p02-step58k
Run crashed at step 72,344 when a node's fabric link went down; a new attempt restarted about four hours later from the last checkpoint.
incident
Sep 5
mcwitt #8934
#8934 filed against the cluster operator: two NVL72 domains show switch-side NVLink faults and a flapping link, with a per-rack health signal requested.
incident
Helw150 #8824
Open-weight comparison closed: the run's forecast at 18T tokens lands ahead of every reference base model except Kimi K2 Base and DeepSeek V4.
evals
hero-wd-gate-router-p02-step58k
Run stopped at step 65,069 on a contained NVLink fault on one rack and restarted an hour later from the hourly checkpoint.
incident
mcwitt #8870
QuACK matrix-multiply scheduler defect measured: under GPU contention it leaves output regions unwritten, and the unsafe code was in kernels the hung run loaded.
incident
Sep 4
ClassicLarry #8818
Decision on #8818: leave the near-dead routed experts in layers 0 and 1 for now, and remove or study them after pre-training.
training-dynamics
mcwitt #8911 open
#8911 opened: root cause of #8861 is the step counter changing layout, making JAX compile the training step twice and register two NCCL windows.
evalsincident
rjpower #8896 open
#8896 opened after a crash an hour into the relaunch: ranks sharing one compiler autotune directory lost temporary files during a checkpoint save.
incidentthroughput
mcwitt #8506
Hero paused to free racks for #8870 debugging, then relaunched as hero-wd-gate-router-p02-step58k from main with gate and router weight decay and pooled wave.
incidentcheckpointing
Sep 3
mcwitt #8890 merged
#8890 merged: points the hero launcher at the checkpoint the pause wrote instead of the stale step-54,000 one, saving about a day of training.
checkpointing
mcwitt #8884 merged
#8884 merged: makes pooled wave the recipe's expert-routing transport again so an unplanned restart cannot bring the ragged backend back.
throughput
Mooler0410 #8818
An outside group reports a similar quiet-early-layer symptom in their own MoE training, where routed experts in early layers stopped learning.
training-dynamics
mcwitt #8506
Ragged all-to-all swap rolled back after the new run hung twice on one rack's expert-parallel domain; old code relaunched from the step-54,000 checkpoint.
incident
mcwitt #8870
#8870 opened: records the silent hang that ended the ragged trial, with one rack three AllReduce operations behind the other ten and no error logged.
incident
Sep 2
hero-12d8b6f0-dee637
Hero run stopped cleanly at step 54,232 for the planned swap to the ragged all-to-all backend, relaunched under a new run id from step 54,000.
incident
mcwitt #8861
#8861 opened: on the ragged backend the first training step after an evaluation fails with an NCCL error because two executables register conflicting memory windows.
evals
Kaiyue-Wen #hero-run-2026
Main objection to the weight decay: shrinking the early-layer gate may push later normalization gains to recreate the scale gap; accepted as low risk anyway.
optimizer
mcwitt #8859 open
#8859 opened: turns the capacity-limited evaluator off on ragged hero rungs and keeps the dropless one, which the reports already read.
evals
Larry #code-review
Restart schedule set: the expert-parallel backend swap lands at step 54,000 and #8854 pins a permanent checkpoint at 55,000 for the weight-decay swap.
checkpointing
Percy Liang #hero-run-2026
Percy argues a statistic that drifts off the ladder deserves its own investigation and asks whether the tracker should forecast more statistics than loss.
training-dynamics
Kaiyue-Wen #hero-run-2026
First objection to the gate and router weight decay: it may help stability but could weaken how well the scaling ladder predicts final performance.
optimizer
ClassicLarry #8833
Proposed gate and router weight decay lowered from 0.05 to 0.02, annealed linearly to zero, after the checkpoint test showed heavy reliance on the gate.
optimizer
ClassicLarry #8818
Scaling the attention-gate logits down on a checkpoint: a small cut is nearly free but larger cuts collapse the model, so gating is essential.
ablation
loom-oa-dev #8506
An InfiniBand port fault on one host made its link flap and stopped the run after step 50,195; Iris restarted from the last checkpoint.
incident
Sep 1
ClassicLarry #8833 open
#8833 opened: turns on weight decay for the attention gate and MoE router, to start as a new run forked from the step-54,000 checkpoint.
optimizer
ClassicLarry #8818
Weight-decay ablations on the d768 and d1024 ladder rungs all land inside evaluation noise; linear decay to zero is marginally best.
ablation
percyliang #8824
Percy cautions that the compared open-weight models may have trained on the evaluation sets, so reported margins are a lower bound.
evals
ClassicLarry #8818
The MoE router norm follows the same growth path as the attention gate, so light weight decay on both looks low risk.
ClassicLarry #8818
Gate investigation summary: output scale does not track a layer's impact, early attention is still used, and no immediate intervention is planned.
Helw150 #8824
#8824 opened: compares the hero's forecast with Nemotron 3 and Qwen3.5 base models in bits per byte, comparable across tokenizers.
evals
Aug 31
ClassicLarry #8818
Layer-skip ablation: dropping any ten layers from the forward pass sharply raises eval loss, so the early layers are not redundant.
ablation
Helw150 #8435
Clustering and classifier weights behind the Harrier data mixture published on Hugging Face so others can reproduce the mix selection.
data
ClassicLarry #8818
#8818 opened: the attention gate's weight norm has grown far beyond the scaling-ladder trajectory, concentrated in the first five layers.
training-dynamics
Aug 28
mcwitt #8754
#8754: context-parallel branches train the hero shape at 262K sequence length on one rack; router capacity is the open problem before long-context cooldown.
long-context
Aug 27
loom-oa-dev #8506
Node lost connectivity and the 176-task gang failed; Iris retried it and training resumed from the last checkpoint.
incident
Aug 25
mcwitt #8684 open
#8684 opened: swap the hero to the ragged all-to-all expert-parallel backend at a planned restart, and fail loudly if no checkpoint loads.
throughput
Aug 24
rjpower #8506
Loader-cache ablation: a small per-process decode cache costs almost no throughput, so host memory can go back to checkpointing.
ablation
ravwojdyla #8506
A training task OOMed during checkpoint serialization and took the coordinator with it; #8617 raises coordinator memory and adds a retry budget.
incident
rjpower #8506
Leak traced: the checkpoint manager never clears completed TensorStore commit futures, so every save retains a staged shard per writer rank.
Aug 23
rjpower #8506
The coordinator ran out of memory while saving a checkpoint; the job was relaunched under a new coordinator carrying the memory fixes.
incident
Aug 22
rjpower #8506
Restarted on main with jemalloc after malloc_trim failed to stop checkpoint-time host OOMs; #8584 merged to abort stalled TensorStore reads.
Aug 21
rjpower #8506
Restarted with bounded TensorStore restore parallelism, a checkpoint offload cache, and malloc_trim after each save to stop host OOMs after checkpoints.
muchanem #8506
An uncorrectable NVLink error on one node took down three tasks in the same rack.
incident
Aug 20
rjpower #8506
Host memory exhaustion recurs about every three hours; tuning the XLA memory fraction and slop factor did not help.
incident
ravwojdyla-agent #8480 merged
#8480 merged: bounds host memory held by a hero resume, after every resume attempt was killed by the container runtime once the checkpoint loaded.
Aug 19
ClassicLarry #8435
Founding spec posted on #8435: 535B-parameter MoE with 23B active, 48 layers, rack-local expert parallelism, two-phase Harrier mix over 18T tokens.
spec