Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .flake8
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[flake8]
extend-ignore = E501
exclude = .git,.venv,.direnv,__pycache__,.pytest_cache
61 changes: 61 additions & 0 deletions .github/workflows/pr-testing-suite.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
name: PR Testing Suite

on:
pull_request:
branches:
- main

jobs:
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"

- name: Install Dependencies
run: pip install -r act/requirements.txt

- name: Run Flake8
run: flake8 act/tests act/main.py act/utils/pipe.py act/utils/comparison_utils.py

test:
name: Tests and Coverage
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"

- name: Install Dependencies
run: pip install -r act/requirements.txt

- name: Run Pytest with Coverage
run: pytest -q --cov=act.utils.pipe --cov-report=term-missing --cov-fail-under=90

smoke-e2e:
name: Smoke E2E
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"

- name: Install Dependencies
run: pip install -r act/requirements.txt

- name: Run Smoke Integration Test
run: pytest -q act/tests/test_pipeline_smoke.py::test_pipeline_smoke_mocked_dependencies
8 changes: 4 additions & 4 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
.direnv/
.envrc.local
code/tests/dsmc/
code/tests/gt3x/
code/tests/sleep/
code/tests/non-wear/
act/tests/dsmc/
act/tests/gt3x/
act/tests/sleep/
act/tests/non-wear/

# Python bytecode and caches
__pycache__/
Expand Down
22 changes: 11 additions & 11 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,30 +1,30 @@
# Repository Guidelines

## Project Structure & Module Organization
- Primary code lives in `code/`; `main.py` orchestrates the pipeline via `utils.pipe.Pipe`.
- `code/utils/` contains data movement, QC, and plotting helpers; keep new helpers modular and reuse logging.
- `code/core/` houses GGIR-facing R scripts and the `GG` Python wrapper; treat these as the source of truth for summary metrics.
- `code/tests/` stores QA notebooks, sample CSVs, and plotting scripts grouped by study focus; add new fixtures beside the notebooks they support.
- Artifacts land in `code/res/data.json` and repo-level `logs/`; avoid committing large raw exports.
- Primary code lives in `act/`; `main.py` orchestrates the pipeline via `utils.pipe.Pipe`.
- `act/utils/` contains data movement, QC, and plotting helpers; keep new helpers modular and reuse logging.
- `act/core/` houses GGIR-facing R scripts and the `GG` Python wrapper; treat these as the source of truth for summary metrics.
- `act/tests/` stores QA notebooks, sample CSVs, and plotting scripts grouped by study focus; add new fixtures beside the notebooks they support.
- Artifacts land in `act/res/data.json` and repo-level `logs/`; avoid committing large raw exports.

## Build, Test, and Development Commands
- `python -m venv .venv && source .venv/bin/activate` – create a local env (Python 3.11).
- `pip install -r code/requirements.txt` – install runtime deps; add extras here only when they are runtime-critical.
- `pip install -r act/requirements.txt` – install runtime deps; add extras here only when they are runtime-critical.
- `python -m code.main 1 "$BOOST_TOKEN" vosslnx` – run the ingest + GGIR pipeline; system flag may be `vosslnxft`, `local`, or `argon`.
- `python code/tests/gt3x/plots.py` – regenerate GT3X diagnostic plots and CSV summaries; confirm paths before running.
- `python act/tests/gt3x/plots.py` – regenerate GT3X diagnostic plots and CSV summaries; confirm paths before running.
- `bash cron.sh` – mirrors production cron behaviour; ensure credentials and git remotes are safe before invoking.

## Coding Style & Naming Conventions
- Follow PEP 8 (4-space indents, snake_case for modules/functions, PascalCase for classes).
- Use `logging` (see `code/main.py`) instead of print for new runtime diagnostics.
- Use `logging` (see `act/main.py`) instead of print for new runtime diagnostics.
- File outputs should match `sub-####_ses-#_accel.csv` naming as enforced in `Save._determine_location`.
- Keep tokens and secrets in environment variables; never hard-code inside scripts or configs.

## Testing Guidelines
- Prefer pytest modules named `test_*.py` under `code/tests/`; reuse the existing subfolders to group by modality.
- Prefer pytest modules named `test_*.py` under `act/tests/`; reuse the existing subfolders to group by modality.
- Lightweight assertions should mock filesystem paths rather than writing to `/mnt`; see `utils.save` for patterns to isolate.
- For manual QA, refresh notebooks (`code/tests/*/*.ipynb`) after dataset changes and export key figures to git-tracked PNG/CSV in the same folder.
- Before submitting, run the pipeline in dry-run mode (use a low `daysago` and a sandbox token) and inspect `code/res/data.json`.
- For manual QA, refresh notebooks (`act/tests/*/*.ipynb`) after dataset changes and export key figures to git-tracked PNG/CSV in the same folder.
- Before submitting, run the pipeline in dry-run mode (use a low `daysago` and a sandbox token) and inspect `act/res/data.json`.

## Commit & Pull Request Guidelines
- Use short, present-tense commit subjects (for example: `Sync GGIR exports`, `Harden RDSS sync error handling`); history (`3c432e9`) favours concise summaries.
Expand Down
73 changes: 33 additions & 40 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,29 +3,31 @@
An automation stack for synchronizing raw actigraphy exports, routing them through GGIR, and publishing QC plots for the BOOST observational and intervention studies. The service pulls IDs from REDCap, reconciles them with RDSS file drops, mirrors curated files to the LSS project hierarchy, and runs the GGIR + QC suite end to end.

## Table of Contents
- [Features](#features)
- [Repository Layout](#repository-layout)
- [Prerequisites](#prerequisites)
- [Quick Start](#quick-start)
- [Running the Pipeline](#running-the-pipeline)
- [Configuration](#configuration)
- [Testing & QA](#testing--qa)
- [Automation & Cron Support](#automation--cron-support)
- [Troubleshooting Tips](#troubleshooting-tips)
- [Contributing](#contributing)
- [Roadmap](#roadmap)
- [License](#license)
- [Boost Actigraphy Processing Pipeline](#boost-actigraphy-processing-pipeline)
- [Table of Contents](#table-of-contents)
- [Features](#features)
- [Repository Layout](#repository-layout)
- [Prerequisites](#prerequisites)
- [Quick Start](#quick-start)
- [Running the Pipeline](#running-the-pipeline)
- [Configuration](#configuration)
- [Testing \& QA](#testing--qa)
- [Automation \& Cron Support](#automation--cron-support)
- [Troubleshooting Tips](#troubleshooting-tips)
- [Contributing](#contributing)
- [Roadmap](#roadmap)
- [License](#license)

## Features
- Mirrors RDSS accelerometer files into LSS study folders using deterministic naming (`sub-####_ses-#_accel.csv`).
- Executes GGIR (3.2.6) via bundled R scripts and captures QC metrics per subject/session.
- Generates interactive activity composition plots (Plotly/Matplotlib) and summary tables for downstream dashboards.
- Exposes a Python CLI (`code/main.py`) that orchestrates symlink creation, ingest, GGIR, QC, and group visualizations.
- Exposes a Python CLI (`act/main.py`) that orchestrates symlink creation, ingest, GGIR, QC, and group visualizations.
- Provides cron wrappers to keep remote environments synchronized with `main` and to ship new outputs automatically.

## Repository Layout
```
code/
act/
core/ # GGIR-facing R scripts and conda env spec
utils/ # ingest, QC, plotting, symlink helpers
tests/ # QA notebooks, sample fixtures, exploratory plots
Expand All @@ -37,7 +39,7 @@ AGENTS.md # contributor workflow guide

## Prerequisites
- Python 3.11 (see `pyproject.toml`).
- R 4.3 with GGIR 3.0+; `code/core/environment.yml` captures a conda environment that works on VossLab Linux.
- R 4.3 with GGIR 3.0+; `act/core/environment.yml` captures a conda environment that works on VossLab Linux.
- Access to RDSS (`/mnt/nfs/rdss/vosslab/Repositories/Accelerometer_Data`) and LSS project mounts.
- REDCap API token with access to report `43327` (store in an environment variable, e.g. `BOOST_TOKEN`).
- Git credentials for fetching/pushing when using the cron wrappers.
Expand All @@ -51,10 +53,10 @@ cd boost-act
# 2. Python environment
python -m venv .venv
source .venv/bin/activate
pip install -r code/requirements.txt
pip install -r act/requirements.txt

# 3. Optional: create R/GGIR env (Linux)
conda env create -f code/core/environment.yml
conda env create -f act/core/environment.yml
conda activate act-newer
```

Expand All @@ -70,21 +72,26 @@ python -m code.main <daysago> $BOOST_TOKEN <system>
2. Match REDCap IDs to RDSS filenames (`utils.comparison_utils`).
3. Copy curated CSVs into the correct LSS project folders (`utils.save`).
4. Call GGIR through `core/acc_new.R` and execute QC/plotting (`utils.qc`, `utils.group`).
5. Write a subject manifest to `code/res/data.json`.
5. Write a subject manifest to `act/res/data.json`.

For ad-hoc diagnostics, re-run plot generation with `python code/tests/gt3x/plots.py` (requires adjusting the hard-coded file path).
For ad-hoc diagnostics, re-run plot generation with `python act/tests/gt3x/plots.py` (requires adjusting the hard-coded file path).

## Configuration
- Edit `code/utils/pipe.py` if new deployment targets or mounts are added.
- Update `code/core/acc_new.R` to tweak GGIR parameters or derivative paths.
- Edit `act/utils/pipe.py` if new deployment targets or mounts are added.
- Update `act/core/acc_new.R` to tweak GGIR parameters or derivative paths.
- Place credentials in the environment or a secure secrets manager; never commit tokens.
- Logging defaults to INFO via `logging.basicConfig` in `code/main.py`; adjust the level for verbose runs.
- Logging defaults to INFO via `logging.basicConfig` in `act/main.py`; adjust the level for verbose runs.

## Testing & QA
- Lightweight Python tests can be added under `code/tests/<area>/test_*.py` and executed with `pytest`.
- QA notebooks in `code/tests/*/*.ipynb` document exploratory checks; rerun them after major data or script changes.
- Run local checks with the same commands used by CI:
- `flake8 act/tests act/main.py act/utils/pipe.py act/utils/comparison_utils.py`
- `pytest -q --cov=act.utils.pipe --cov-report=term-missing --cov-fail-under=90`
- `pytest -q act/tests/test_pipeline_smoke.py::test_pipeline_smoke_mocked_dependencies`
- CI runs on pull requests targeting `main` and fails when lint, tests, coverage (`< 90%` for scoped modules), or smoke integration checks fail.
- Add or update tests under `act/tests/test_*.py` for Python-only validation; keep QA notebooks in `act/tests/*/*.ipynb` for manual exploratory checks.
- `utils.qc` aggregates results into `logs/GGIR_QC_errs.csv`; inspect this file to confirm expected wear-time and calibration checks.
- Use sandbox tokens and the `local` system flag to validate changes without touching production mounts.
- Use sandbox tokens and mocked/local paths for tests; do not require `/mnt` mounts or real secrets.
- Detailed testing contributor guidance lives in `act/docs/TESTING.md`.

## Automation & Cron Support
- `cron.sh` bootstraps the conda env, pulls latest `main`, runs the pipeline (`daysago=1`, production token), and pushes any resulting artifacts.
Expand All @@ -93,7 +100,7 @@ For ad-hoc diagnostics, re-run plot generation with `python code/tests/gt3x/plot

## Troubleshooting Tips
- **Missing symlinks:** run `python -c "from code.utils.mnt import create_symlinks; create_symlinks('../mnt', system='argon')"` (swap `system` as needed) and confirm mount availability.
- **GGIR failures:** check the console output and logs under `code/core/` or R’s stderr; ensure the conda env includes GGIR dependencies.
- **GGIR failures:** check the console output and logs under `act/core/` or R’s stderr; ensure the conda env includes GGIR dependencies.
- **REDCap mismatches:** `utils.comparison_utils.ID_COMPARISONS` logs duplicate IDs; review its stdout and `AGENTS.md` for remediation steps.
- **Permission errors:** verify the executing user can read RDSS and write to the LSS target directories.

Expand All @@ -107,17 +114,3 @@ Contributions are welcome! Start by reviewing `AGENTS.md` for code style, testin

## License
No license has been specified yet. Until a license is published, usage is limited to collaborators with explicit permission from the maintainers.

<<<<<<< HEAD
/mnt/lss/Projects/BOOST/InterventionStudy/3-experiment/data/act-int-test/derivatives/GGIR-3.2.6/sub-8020/accel/ses-3/output_ses-3/results/part5_personsummary_MM_L44.8M100.6V428.8_T5A5.csv
/mnt/lss/Projects/BOOST/ObservationalStudy/3-experiment/data/act-obs-test/derivatives/GGIR-3.2.6/sub-7168/accel/ses-1/output_ses-1/results/part5_personsummary_MM_L44.8M100.6V428.8_T5A5.csv
=======
/mnt/lss/Projects/BOOST/InterventionStudy/3-experiment/data/act-int-test/derivatives/GGIR-3.2.6-test-ncp-sleep2/sub-8020/accel/ses-3/output_ses-3/results/part5_personsummary_MM_L44.8M100.6V428.8_T5A5.csv
/mnt/lss/Projects/BOOST/ObservationalStudy/3-experiment/data/act-obs-test/derivatives/GGIR-3.2.6-test-ncp-sleep2/sub-7168/accel/ses-1/output_ses-1/results/part5_personsummary_MM_L44.8M100.6V428.8_T5A5.csv

```sh
cp /mnt/rdss/VossLab/Repositories/Accelerometer_Data/1168\ \(2025-08-14\)RAW.csv \
/mnt/lss/Projects/BOOST/InterventionStudy/3-experiment/data/act-int-test/sub-8019/accel/ses-3
```

>>>>>>> 77463126661d2d8dd46654410d811d2b3d50bf5f
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
2 changes: 1 addition & 1 deletion code/core/accel.R → act/core/accel.R
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/usr/bin/env rscript

#usage:
# cd /path/to/code/func/
# cd /path/to/act/func/
# chmod +x accel.r
# ./accel.r --project_dir="path/to/project" --project_deriv_dir="path/to/derivatives" --files="file1.csv, file2.csv, file3.csv" --verbose

Expand Down
File renamed without changes.
File renamed without changes.
15 changes: 8 additions & 7 deletions code/core/gg.py → act/core/gg.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

logger = logging.getLogger(__name__)


class GG:
"""
Class to execute GGIR processing for matched subject records.
Expand All @@ -18,8 +19,8 @@ def __init__(self, matched, intdir, obsdir, system):
obsdir (str): Path to the observational directory.
"""
self.matched = matched
self.INTDIR = intdir.rstrip('/') + '/'
self.OBSDIR = obsdir.rstrip('/') + '/'
self.INTDIR = intdir.rstrip("/") + "/"
self.OBSDIR = obsdir.rstrip("/") + "/"
self.DERIVATIVES = "derivatives/GGIR-3.2.6/" # Defined within the class
self.system = system

Expand All @@ -32,7 +33,7 @@ def run_gg(self):
from code.utils.qc import QC

for project_dir in [self.INTDIR, self.OBSDIR]:
command = f"Rscript code/core/acc_new.R --project_dir {project_dir} --deriv_dir {self.DERIVATIVES}"
command = f"Rscript act/core/acc_new.R --project_dir {project_dir} --deriv_dir {self.DERIVATIVES}"

try:
# Execute the command in a new subprocess
Expand All @@ -43,7 +44,7 @@ def run_gg(self):
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
bufsize=1,
universal_newlines=True
universal_newlines=True,
)

# Stream output line-by-line
Expand All @@ -59,10 +60,10 @@ def run_gg(self):
logger.info("GGIR completed successfully for %s.", project_dir)

# Determine project type for QC ('int' for internal, 'obs' for observational)
if project_dir.rstrip('/') == self.INTDIR.rstrip('/'):
project_type = 'int'
if project_dir.rstrip("/") == self.INTDIR.rstrip("/"):
project_type = "int"
else:
project_type = 'obs'
project_type = "obs"

# Run QC for this project
logger.info("Starting QC pipeline for %s project.", project_type)
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
78 changes: 78 additions & 0 deletions act/docs/TESTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# TESTING.md

Testing guidance for BOOST Actigraphy Python changes.

## Scope
- Primary automated coverage targets Python orchestration and utility paths in `act/`.
- GGIR/R is not required for PR smoke checks and should be mocked/stubbed in tests.
- CI-safe tests must avoid live mounts, production data paths, and real API tokens.

## Local Commands
Run these from repo root:

```bash
flake8 act/tests act/main.py act/utils/pipe.py act/utils/comparison_utils.py
pytest -q --cov=act.utils.pipe --cov-report=term-missing --cov-fail-under=90
pytest -q act/tests/test_pipeline_smoke.py::test_pipeline_smoke_mocked_dependencies
```

If you use the local nix environment:

```bash
nix develop
flake8 act/tests act/main.py act/utils/pipe.py act/utils/comparison_utils.py
pytest -q --cov=act.utils.pipe --cov-report=term-missing --cov-fail-under=90
pytest -q act/tests/test_pipeline_smoke.py::test_pipeline_smoke_mocked_dependencies
```

## CI Expectations
- Workflow file: `.github/workflows/pr-testing-suite.yml`.
- Trigger: `pull_request` to `main` only.
- Required jobs:
- `lint`
- `test` (coverage fail-under 90 on scoped modules)
- `smoke-e2e`

## Test Layout
- Put tests in `act/tests/` and name files `test_*.py`.
- Prefer deterministic unit/integration tests over notebook-driven checks for CI.
- Keep test data small and explicit.

## Shared Fixtures
- Shared fixtures live in `act/tests/conftest.py`.
- Current fixture groups:
- Temporary roots for `int`, `obs`, and `rdss` paths.
- Filename/path factories for `sub-####_ses-#_accel.csv`.
- Signature example fixtures for known-good and mismatch cases.
- Reuse existing fixtures before creating new ones to reduce duplication.

## Save Edge Cases
When adding save logic tests:
- Use temporary directories (`tmp_path`) and local fixture roots.
- Cover both path styles:
- intervention (`int`)
- observational (`obs`)
- Validate deterministic session outputs:
- session numbers stay contiguous for the tested scenario
- generated file names follow `sub-####_ses-#_accel.csv`
- destination paths include expected `accel/ses-#` structure
- For error paths, mock file operations (for example `shutil.copy`) and assert graceful continuation or expected failure semantics.

## Smoke E2E Constraints
- Smoke tests must stay Python-only and fast.
- Mock external boundaries:
- REDCap-dependent ingest calls
- GGIR/R invocation paths
- filesystem cleanup paths when needed
- Do not require:
- `/mnt` or network storage mounts
- real REDCap tokens
- full GGIR execution

## Contributor Notes
AGENTS-aligned expectations:
- Keep commit subjects short and present tense.
- Keep test commits focused by checkpoint or behavior slice.
- Name tests by behavior (`test_<unit>_<expected_outcome>`).
- Add one clear assertion group per behavior being guarded.
- Run local lint/tests before opening a PR to keep CI feedback focused on regressions.
File renamed without changes.
Loading