Skip to content

[MoE] cleanup aux loss - #518

Merged
mayank31398 merged 18 commits into
mainfrom
aux
Sep 25, 2026
Merged

mayank31398 merged 18 commits into
mainfrom
aux

Conversation

@mayank31398

@mayank31398 mayank31398 commented Sep 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary by CodeRabbit

  • Training
    • MoE router auxiliary loss and z-loss are tracked separately, with configurable loss weights and expert-frequency metrics.
    • Training metrics are organized into distinct categories for losses, parameter norms, and gradient norms.
    • Metric logging now handles differently shaped values and omits selected statistics from log messages.
  • Configuration
    • Updated pretraining and research configurations no longer specify the router auxiliary-loss setting.

mayank31398 and others added 6 commits September 20, 2026 02:26
Signed-off-by: Mayank Mishra <mayank31398@gmail.com>
Signed-off-by: Mayank Mishra <mayank31398@gmail.com>
Signed-off-by: Mayank Mishra <mayank31398@gmail.com>
Signed-off-by: Mayank Mishra <mayank31398@gmail.com>
@coderabbitai

coderabbitai Bot commented Sep 21, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 35715e62-6229-4a97-92e6-4d321ad4426e

📥 Commits

Reviewing files that changed from the base of the PR and between 34a6659 and 0ffd769.

📒 Files selected for processing (58)
  • configs/pretraining-examples/apple.yml
  • configs/pretraining-examples/nvidia-1.yml
  • configs/pretraining-examples/nvidia-2.yml
  • configs/pretraining-examples/nvidia-3.yml
  • configs/pretraining-examples/tpu.yml
  • configs/pretraining-examples/trainium.yml
  • configs/research/cross-layer-attention/base.yml
  • configs/research/cross-layer-attention/cla.yml
  • configs/research/ladder-residual/1b-base.yml
  • configs/research/ladder-residual/1b-ladder.yml
  • configs/research/ladder-residual/1b-parallel.yml
  • configs/research/ladder-residual/3b-base.yml
  • configs/research/ladder-residual/3b-ladder.yml
  • configs/research/ladder-residual/3b-parallel.yml
  • configs/research/m2rnn/400m-normal/gated-deltanet.yml
  • configs/research/m2rnn/400m-normal/gru.yml
  • configs/research/m2rnn/400m-normal/hybrid-gated-deltanet-m2rnn-1l.yml
  • configs/research/m2rnn/400m-normal/hybrid-gated-deltanet-m2rnn-3l.yml
  • configs/research/m2rnn/400m-normal/hybrid-gated-deltanet.yml
  • configs/research/m2rnn/400m-normal/hybrid-m2rnn.yml
  • configs/research/m2rnn/400m-normal/hybrid-mamba2-m2rnn-1l.yml
  • configs/research/m2rnn/400m-normal/hybrid-mamba2-m2rnn-3l.yml
  • configs/research/m2rnn/400m-normal/hybrid-mamba2.yml
  • configs/research/m2rnn/400m-normal/m2rnn.yml
  • configs/research/m2rnn/400m-normal/mamba2.yml
  • configs/research/m2rnn/400m-normal/rnn.yml
  • configs/research/m2rnn/400m-normal/softmax-attention.yml
  • configs/research/m2rnn/7b-normal/gated-deltanet.yml
  • configs/research/m2rnn/7b-normal/gru.yml
  • configs/research/m2rnn/7b-normal/hybrid-gated-deltanet-m2rnn-1l.yml
  • configs/research/m2rnn/7b-normal/hybrid-gated-deltanet-m2rnn-5l.yml
  • configs/research/m2rnn/7b-normal/hybrid-gated-deltanet.yml
  • configs/research/m2rnn/7b-normal/hybrid-m2rnn.yml
  • configs/research/m2rnn/7b-normal/hybrid-mamba2-m2rnn-1l.yml
  • configs/research/m2rnn/7b-normal/hybrid-mamba2-m2rnn-5l.yml
  • configs/research/m2rnn/7b-normal/hybrid-mamba2.yml
  • configs/research/m2rnn/7b-normal/m2rnn.yml
  • configs/research/m2rnn/7b-normal/mamba2.yml
  • configs/research/m2rnn/7b-normal/rnn.yml
  • configs/research/m2rnn/7b-normal/softmax-attention.yml
  • lm_engine/training/constants.py
  • lm_engine/training/logging_utils/__init__.py
  • lm_engine/training/logging_utils/metrics_tracking.py
  • lm_engine/training/logging_utils/tracking.py
  • lm_engine/training/loss.py
  • lm_engine/training/metrics.py
  • lm_engine/training/mixins/dense/main.py
  • lm_engine/training/model_config.py
  • lm_engine/training/model_wrapper/base.py
  • lm_engine/training/model_wrapper/pretraining.py
  • lm_engine/training/modeling_utils/io.py
  • lm_engine/training/modeling_utils/mlp_blocks/__init__.py
  • lm_engine/training/modeling_utils/mlp_blocks/moe/config.py
  • lm_engine/training/modeling_utils/mlp_blocks/moe/module.py
  • lm_engine/training/train.py
  • lm_engine/training/train_utils.py
  • tests/dcp/train.yml
  • tests/params_group/training_config.yml
 __________________________________________________________
< This abstraction leaks more than a sieve in a rainstorm. >
 ----------------------------------------------------------
  \
   \   (\__/)
       (•ㅅ•)
       /   づ
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Signed-off-by: Mayank Mishra <mayank31398@gmail.com>
Signed-off-by: Mayank Mishra <mayank31398@gmail.com>
Signed-off-by: Mayank Mishra <mayank31398@gmail.com>
Signed-off-by: Mayank Mishra <mayank31398@gmail.com>
Signed-off-by: Mayank Mishra <mayank31398@gmail.com>
Signed-off-by: Mayank Mishra <mayank31398@gmail.com>
Signed-off-by: Mayank Mishra <mayank31398@gmail.com>
Signed-off-by: Mayank Mishra <mayank31398@gmail.com>
Signed-off-by: Mayank Mishra <mayank31398@gmail.com>
Signed-off-by: Mayank Mishra <mayank31398@gmail.com>
Signed-off-by: Mayank Mishra <mayank31398@gmail.com>
Signed-off-by: Mayank Mishra <mayank31398@gmail.com>
@mayank31398
mayank31398 marked this pull request as ready for review September 25, 2026 07:37
@mayank31398
mayank31398 merged commit a9c6e2b into main Sep 25, 2026
1 of 3 checks passed
@mayank31398
mayank31398 deleted the aux branch September 25, 2026 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant