Expose negate_key_metric in FedAvg recipes - #4953
Conversation
|
/build |
Greptile SummaryThis PR adds a
Confidence Score: 5/5Safe to merge — the change is a straightforward bool parameter forwarded uniformly through all recipe wrappers to IntimeModelSelector, which already supports the flag. All seven recipe wrappers consistently receive and forward negate_key_metric via super().init(). The base FedAvgRecipe validates and stores the flag, BaseFedJob threads it into IntimeModelSelector, and six new unit tests verify the full pass-through for every framework variant. No behavioral changes occur when the default False is used. Files Needing Attention: No files require special attention. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A["Recipe user sets\nnegate_key_metric=True"] --> B["Framework-specific recipe\n(PT / NumPy / Sklearn / KMeans / SVM / TF)"]
B --> C["Base FedAvgRecipe\nnvflare/recipe/fedavg.py"]
C --> D["BaseFedJob\nnvflare/job_config/base_fed_job.py"]
D -->|"key_metric provided\n(model_selector not supplied)"| E["IntimeModelSelector\n(key_metric, negate_key_metric)"]
D -->|"custom model_selector supplied"| F["Custom FLComponent\n(negate_key_metric ignored)"]
E --> G["Model comparison:\nvalue x -1 if negate_key_metric\nfor lower-is-better metrics"]
Reviews (5): Last reviewed commit: "Address negate metric recipe review comm..." | Re-trigger Greptile |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4953 +/- ##
==========================================
+ Coverage 62.83% 62.85% +0.02%
==========================================
Files 995 995
Lines 97868 97870 +2
==========================================
+ Hits 61496 61520 +24
+ Misses 36372 36350 -22
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
/build |
a12889d to
945570a
Compare
|
/build |
945570a to
98a994f
Compare
|
/build |
## Summary - Add `nvflare.client.hf.patch()` for HuggingFace `Trainer` / TRL `SFTTrainer` federated rounds. - Support train, evaluate, and submit-model tasks with rank-0 Client API ownership, DDP coordination, checkpoint-backed Trainer state restore, budgeted round execution, metrics capture, PEFT adapter scope, and bf16-to-numpy-safe output conversion. - Add a clean `examples/advanced/hf_client_api` Qwen SFT/PEFT example using the default `flare.patch(trainer)` API and a standard `prepare_data.py` data-prep step, while leaving the legacy `examples/advanced/llm_hf` example unchanged. - Add the HF Client API design doc as implemented PR documentation and keep only one `3rdParty/packaging.LICENSE.txt` file for the optional `packaging` dependency. - Add focused fake and real dependency contract tests, including transformers min/latest and TRL 0.18/latest CI jobs. ## Testing - `python -m pytest tests/unit_test/app_opt/hf -q` - `python -m pytest tests/unit_test/recipe/fedavg_recipe_test.py -q` - `python -m py_compile examples/advanced/hf_client_api/client.py examples/advanced/hf_client_api/model.py examples/advanced/hf_client_api/prepare_data.py examples/advanced/hf_client_api/job.py` - `python examples/advanced/hf_client_api/client.py --help` - `python examples/advanced/hf_client_api/prepare_data.py --help` - `python examples/advanced/hf_client_api/prepare_data.py --n_clients 2 --data_root /private/tmp/nvflare_hf_client_api_qwen_data_prep` - `python examples/advanced/hf_client_api/job.py --help` - `python examples/advanced/hf_client_api/job.py --export_config --job_dir /private/tmp/nvflare_hf_client_api_qwen_job_prepared --data_root /private/tmp/nvflare_hf_client_api_qwen_data_prep` - Verified exported launcher config uses `python3 -u custom/client.py ...` rather than an absolute source path under `custom/` - `git diff --check upstream/main...HEAD` - `./runtest.sh -s` - Earlier in this PR: `./runtest.sh -l` ## Review Notes - The new Qwen example is validated locally through compile/help/export checks. The README now follows the standard hello-world example layout. Full simulation requires downloading the Qwen checkpoint and installing HF/TRL example dependencies. - FedAvg recipe `negate_key_metric` pass-through was split into separate PR #4953. This HF PR keeps the Qwen example independent by disabling recipe best-model selection with `key_metric=""`. - The TRL contract test constructs real CPU `SFTTrainer` instances and applies `flare.patch()`, but does not run a real TRL train/send round. The train/send path is covered by real `transformers.Trainer` and fake-contract tests. - `train_with_evaluation` hard-fail policy currently depends on existing Client API config. The new HF example explicitly calls `trainer.evaluate()` before `trainer.train()` each round; recipe-level pass-through can be added later if we want every recipe user to get a hard failure when pre-train eval is omitted.
98a994f to
5bb0df4
Compare
Summary
negate_key_metricpass-through to the base FedAvg recipe and framework-specific FedAvg recipe wrappers.IntimeModelSelectorso recipe users can select lower-is-better metrics such as losses without reporting a separate negated metric from clients.Testing
python -m pytest tests/unit_test/recipe/fedavg_recipe_test.py -qgit diff --check