This is the code repository for the paper BenHAR: Benchmarking Self-Supervised Learning for Generalizable Sensor-based Activity Recognition, a large-scale benchmark for evaluating the generalization capability of self-supervised sensor-based human activity recognition (HAR) models.
-
Large-scale multi-dataset integration: Curates and standardizes a large-scale (~258K samples, ~430 hours) sensor-based HAR benchmark that is multi-source (14 widely used datasets), diverse (300+ subjects, 11 countries, 13 body locations, and 2 device types), and multimodal (accelerometer and gyroscope).
-
Cross-dataset generalization benchmark:
- Evaluates four representative self-supervised learning paradigms, including 1 recognition method, 2 reconstruction methods, 3 contrastive methods, and 2 hybrid methods.
- Benchmarks 12 architecture combinations consisting of 3 encoders (CNN, ResNet, Transformer) and 4 classifiers (CNN, GRU, MLP, Transformer).
-
Analysis of data scale and domain shift factors:
- Investigates the impact of unlabeled data scale, labeled data scale, and irrelevant activity data scale on generalization performance.
- Studies the impact of cross-subject, cross-device, and cross-location domain shifts on generalization performance.
Create the environment using requirements.txt:
pip install -r requirements.txt| Dataset | Country | # Subject | # Activity | Device | # Location | Download |
|---|---|---|---|---|---|---|
| UCI | Italy | 30 | 6 | Custom | 1 | Link |
| HHAR | Denmark | 9 | 6 | Custom | 2 | Link |
| Shoaib | The Netherlands | 10 | 7 | Custom | 4 | Link |
| Motion | England | 24 | 6 | Custom | 1 | Link |
| DSADS | Turkey | 8 | 18 | Research | 3 | Link |
| USC-HAD | United States | 14 | 12 | Research | 1 | Link |
| KU-HAR | Bangladesh | 90 | 18 | Custom | 1 | Link |
| PAMAP2 | Germany | 9 | 18 | Research | 3 | Link |
| TNDA-HAR | China | 50 | 8 | Research | 5 | Link |
| Mhealth | Spain | 10 | 12 | Research | 2 | Link |
| WISDM | United States | 51 | 18 | Custom | 2 | Link |
| RealWorld | Germany | 15 | 8 | Custom | 7 | Link |
| HARSense | Meghalaya | 12 | 6 | Custom | - | Link |
| UT-Complex | The Netherlands | 10 | 13 | Custom | 2 | Link |
All datasets are publicly available. Please download and extract the datasets into the datasets/ directory, and name each dataset folder according to the dataset names listed in the table.
Run the following command to standardize all datasets:
bash scripts/preprocess.shThe preprocessing scripts for each dataset can be found in the preprocess/ folder.
All datasets are standardized into non-overlapping 6-second windows with a sampling rate of 20 Hz.
More details are provided in the paper.
| Method | Paradigm | Venue | Link |
|---|---|---|---|
| BioBankSSL | Recognition | npj Digital Medicine 2024 | Link |
| LIMU-BERT | Reconstruction | SenSys 2021 | Link |
| CRT | Reconstruction | TNNLS 2023 | Link |
| TS-TCC | Contrastive | IJCAL 2021 | Link |
| TS2Vec | Contrastive | AAAI 2022 | Link |
| FOCAL | Contrastive | NeurIPS 2023 | Link |
| SimMTM | Hybrid | NeurIPS 2023 | Link |
| CrossHAR | Hybrid | IMWUT 2024 | Link |
The 14 datasets are partitioned into five dataset-level folds. Models are trained on four folds and evaluated on the held-out fold.
bash scripts/run_dataset_5fold.shExample: BioBankSSL with transformer encoder and CNN classifier using accelerometer + gyroscope:
MODE=BioBankSSL \
ENCODER_BACKBONE=transformer \
CLASSIFIER_PREFIX=cnn \
INPUT_CHANNELS=6 \
bash scripts/run_dataset_5fold.shThis setting changes the amount of unlabeled source data used to pretrain the SSL encoder. The downstream classifier still uses the full labeled source split unless CLASSIFIER_TRAIN_SUBSET_RATE is changed.
Use TRAIN_SUBSET_RATE to control the fraction of source training data used for pretraining:
MODE=BioBankSSL \
ENCODER_BACKBONE=transformer \
CLASSIFIER_PREFIX=cnn \
INPUT_CHANNELS=6 \
TRAIN_SUBSET_RATE=0.25 \
CLASSIFIER_TRAIN_SUBSET_RATE=1.0 \
bash scripts/run_dataset_5fold.shThis setting evaluates how much labeled source data is needed after SSL pretraining. The encoder is pretrained with the full unlabeled source split, while the downstream classifier uses only a fraction of labeled source embeddings.
Use CLASSIFIER_TRAIN_SUBSET_RATE to control the labeled classifier training fraction:
MODE=BioBankSSL \
ENCODER_BACKBONE=transformer \
CLASSIFIER_PREFIX=cnn \
INPUT_CHANNELS=6 \
TRAIN_SUBSET_RATE=1.0 \
CLASSIFIER_TRAIN_SUBSET_RATE=0.1 \
bash scripts/run_dataset_5fold.shThis setting studies whether adding extra unlabeled data from activity categories that are not part of the main benchmark helps or hurts representation learning.
Use DATA_OTHER_SUBSET_RATE to control the fraction of data_other/ added to encoder pretraining.
MODE=BioBankSSL \
ENCODER_BACKBONE=transformer \
CLASSIFIER_PREFIX=cnn \
INPUT_CHANNELS=6 \
TRAIN_SUBSET_RATE=1.0 \
CLASSIFIER_TRAIN_SUBSET_RATE=1.0 \
DATA_OTHER_ROOT=./data_other \
DATA_OTHER_SUBSET_RATE=0.5 \
bash scripts/run_dataset_5fold.sh# Subject shift: train and test subjects are disjoint.
bash scripts/run_cross_subject.sh
# Device shift: custom-grade devices vs. research-grade devices.
bash scripts/run_cross_device.sh
# Body-location shift: train on one location group and test on another.
TRAIN_LOCATION=trunk \
TEST_LOCATION=upper \
bash scripts/run_cross_location.sh