Skip to content

Update URM Test Runner #568

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Kartik Nema (kartnema) wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Update URM Test Runner #568

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 27 additions & 5 deletions Runner/suites/Performance/userspace-resource-manager/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,17 @@ export URM_REQUIRE_COMMON_FILES="InitConfig.yaml PropertiesConfig.yaml Resources
export URM_REQUIRE_TEST_FILES="InitConfig.yaml PropertiesConfig.yaml ResourcesConfig.yaml SignalsConfig.yaml TargetConfig.yaml ExtFeaturesConfig.yaml Baseline.yaml"
```

### 3) Test test nodes
`/etc/urm/tests/nodes` must exist and be non‑empty for **`/usr/bin/UrmIntegrationTests`** and **`/usr/bin/UrmComponentTests`**. If missing/empty → **SKIP only that suite**.
### 3) Test nodes
The runner resolves the test-nodes source directory in the following priority order:

1. **`URM_TEST_NODES_DIR`** – explicit operator override (takes precedence over everything).
2. **`URM_CONFIG_DIR`** – legacy compatibility fallback (same variable honoured by the config roots).
3. **`/var/lib/urm/tests/nodes`** – runtime/writable location, used only when the directory exists **and is non-empty**.
4. **`/usr/share/urm/tests/nodes`** – package-installed default (Yocto and Debian).

Before tests run, the resolved nodes are **copied** into `/run/urm/tests/nodes` — the path hardcoded in the test binaries — so that URM can write to them during testing without modifying the read-only package-installed source. The copied directory is removed automatically on exit, interrupt, or termination.

The resolved source directory must exist and be non-empty for **`/usr/bin/UrmIntegrationTests`** and **`/usr/bin/UrmComponentTests`**. If missing, empty, or the copy fails → **SKIP only that suite**.

### 4) Base tools
Requires: `awk`, `grep`, `date`, `printf`. If missing → **overall SKIP**.
Expand Down Expand Up @@ -100,8 +109,11 @@ Per‑suite default timeouts (if helper is present):
## Environment overrides

- `SERVICE_NAME`: systemd unit to check (default: `urm.service`)
- `URM_CONFIG_DIR`: root of config tree (default: `/etc/urm`)
- `URM_REQUIRE_COMMON_FILES`, `URM_REQUIRE_TEST_FILES`: *space‑separated* filenames that must exist in `common/` / `tests/` respectively to treat that tree as present.
- `URM_CONFIG_DIR`: **legacy** compatibility override – sets the root for `common/`, `tests/configs/`, and `tests/nodes/` when the per-root variables below are not set (default: unset; built-in defaults are `/etc/urm`, `/usr/share/urm`, and `/usr/share/urm` respectively).
- `URM_COMMON_CONFIG_DIR`: root for `common/` configs (default: `/etc/urm`). Takes precedence over `URM_CONFIG_DIR`.
- `URM_TESTS_CONFIG_DIR`: root for `tests/configs/` (default: `/usr/share/urm`). Takes precedence over `URM_CONFIG_DIR`.
- `URM_TEST_NODES_DIR`: explicit root for `tests/nodes/` (default: auto-resolved; see [Test nodes](#3-test-nodes) above). Takes precedence over the automatic candidate search.
- `URM_REQUIRE_COMMON_FILES`, `URM_REQUIRE_TEST_FILES`: *space‑separated* filenames that must exist in `common/` / `tests/configs/` respectively to treat that tree as present.

---

Expand All @@ -122,11 +134,21 @@ List suites and presence coverage:
./run.sh --list
```

Use a different config root:
Use a different config root (legacy, applies to both common and tests-config roots):
```bash
URM_CONFIG_DIR=/opt/rt/etc ./run.sh
```

Override individual roots:
```bash
URM_COMMON_CONFIG_DIR=/opt/rt/etc/urm URM_TESTS_CONFIG_DIR=/opt/rt/usr/share/urm ./run.sh
```

Point to a custom test-nodes directory:
```bash
URM_TEST_NODES_DIR=/opt/rt/usr/share/urm ./run.sh
```

---

## Exit status
Expand Down
76 changes: 71 additions & 5 deletions Runner/suites/Performance/userspace-resource-manager/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,13 +71,15 @@ if command -v flock >/dev/null 2>&1; then
echo "$TESTNAME SKIP" >"$RES_FILE"
exit 0
fi
lock_flock=1
trap 'exec 9>&-' EXIT INT TERM
else
if ! mkdir "$LOCKDIR" 2>/dev/null; then
log_warn "Another ${TESTNAME} run is active or stale fallback lockdir exists: $LOCKDIR"
echo "$TESTNAME SKIP" >"$RES_FILE"
exit 0
fi
lock_flock=0
trap 'rmdir "$LOCKDIR" 2>/dev/null || true' EXIT INT TERM
fi

Expand Down Expand Up @@ -275,10 +277,50 @@ else
fi

# ---------- Config preflight (check both common/ and tests/) ----------
URM_CONFIG_DIR="${URM_CONFIG_DIR:-/etc/urm}"
COMMON_CONFIGS_DIR="$URM_CONFIG_DIR/common"
TEST_CONFIGS_DIR="$URM_CONFIG_DIR/tests/configs"
TEST_NODES_DIR="$URM_CONFIG_DIR/tests/nodes"
# Resolution order for each root (first match wins):
# 1. Explicit per-root override (URM_COMMON_CONFIG_DIR / URM_TESTS_CONFIG_DIR)
# 2. Legacy URM_CONFIG_DIR (compatibility fallback for existing callers)
# 3. Built-in default
#
# For test-nodes the candidate order is:
# 1. Explicit URM_TEST_NODES_DIR override
# 2. Legacy URM_CONFIG_DIR (compatibility fallback for existing callers)
# 3. /var/lib/urm — runtime/writable location, only when non-empty
# 4. /usr/share/urm — package-installed location (default)

# Resolve common-config root
if [ -n "${URM_COMMON_CONFIG_DIR:-}" ]; then
common_root="$URM_COMMON_CONFIG_DIR"
elif [ -n "${URM_CONFIG_DIR:-}" ]; then
common_root="$URM_CONFIG_DIR"
else
common_root="/etc/urm"
fi

# Resolve tests-config root
if [ -n "${URM_TESTS_CONFIG_DIR:-}" ]; then
tests_root="$URM_TESTS_CONFIG_DIR"
elif [ -n "${URM_CONFIG_DIR:-}" ]; then
tests_root="$URM_CONFIG_DIR"
else
tests_root="/usr/share/urm"
fi

# Resolve test-nodes root: explicit override > legacy URM_CONFIG_DIR > populated runtime dir > package dir
if [ -n "${URM_TEST_NODES_DIR:-}" ]; then
nodes_root="$URM_TEST_NODES_DIR"
elif [ -n "${URM_CONFIG_DIR:-}" ]; then
nodes_root="$URM_CONFIG_DIR"
elif [ -d "/var/lib/urm/tests/nodes" ] && \
[ "$(find "/var/lib/urm/tests/nodes" -mindepth 1 -maxdepth 1 -type f 2>/dev/null | wc -l | awk '{print $1}')" -gt 0 ]; then
nodes_root="/var/lib/urm"
else
nodes_root="/usr/share/urm"
fi

COMMON_CONFIGS_DIR="$common_root/common"
TEST_CONFIGS_DIR="$tests_root/tests/configs"
TEST_NODES_DIR="$nodes_root/tests/nodes"

COMMON_CONFIGS_OK=1
TEST_CONFIGS_OK=1
Expand Down Expand Up @@ -397,6 +439,30 @@ if [ -z "$TESTS" ]; then
exit 0
fi

# ---------- Stage test nodes into the hardcoded runtime path ----------
# UrmComponentTests and UrmIntegrationTests read nodes from the fixed path
# /run/urm/tests/nodes (hardcoded in the binaries). The package installs
# read-only node files under /usr/share/urm/tests/nodes (or /var/lib/urm),
# so we copy them to the writable runtime location before running the tests.
# Cleanup removes only the directory we created, and is registered with trap
# immediately so it runs on exit, interrupt, or termination.
RUNTIME_NODES_DIR="/run/urm/tests/nodes"
if [ "$TEST_NODES_OK" -eq 1 ]; then
mkdir -p "$RUNTIME_NODES_DIR"

@smuppand Srikanth Muppandam (smuppand) Sep 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mkdir -p accepts an existing /run/urm/tests/nodes, but this test does not establish that it created or exclusively owns that directory. A previous interrupted run, another service, or a manual diagnostic can leave nodes there; the subsequent copy merges its files with stale content and the exit trap later deletes the entire directory.
That can produce invalid test inputs or remove state outside this run. Require an absent/test-owned destination before staging, or snapshot and restore pre-existing state. The cleanup must remove only assets this invocation created.

if cp -r "$TEST_NODES_DIR/"* "$RUNTIME_NODES_DIR/"; then

@smuppand Srikanth Muppandam (smuppand) Sep 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The copy occurs before the cleanup trap is installed and its destination is not cleared or made atomic. If cp partially copies and then fails, TEST_NODES_OK=0 skips the suites but leaves a partial /run/urm/tests/nodes; a later run can merge with and execute against that stale partial set. Register a cleanup handler immediately after establishing owned staging state, and stage atomically, for example, copy to a private temporary directory, validate it, then publish it only when the fixed runtime path is safely owned.

log_info "[NODES] Staged test nodes from $TEST_NODES_DIR to $RUNTIME_NODES_DIR"
# Extend the existing trap to also clean up the staged nodes.
if [ "$lock_flock" -eq 1 ]; then
trap 'rm -rf "$RUNTIME_NODES_DIR"; exec 9>&-' EXIT INT TERM

@smuppand Srikanth Muppandam (smuppand) Sep 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The trap uses rm -rf "$RUNTIME_NODES_DIR" for every successful staging run, including when mkdir -p reused a pre-existing directory. This contradicts the comment that cleanup “removes only the directory we created.” Track ownership explicitly and fail/skip rather than deleting a path that was already present; otherwise an interrupted or concurrent system component can lose its runtime nodes.

else
trap 'rm -rf "$RUNTIME_NODES_DIR"; rmdir "$LOCKDIR" 2>/dev/null || true' EXIT INT TERM
fi
else
log_warn "[NODES] Failed to stage test nodes into $RUNTIME_NODES_DIR — suites requiring nodes will SKIP"
TEST_NODES_OK=0
fi
fi

# ---------- Execute ----------
PASS=0
FAIL=0
Expand Down Expand Up @@ -508,7 +574,7 @@ log_info "Overall counts: PASS=$PASS FAIL=$FAIL SKIP=$SKIP"
# - Else -> overall SKIP (everything skipped)
if [ "$FAIL" -gt 0 ]; then
echo "$TESTNAME FAIL" >"$RES_FILE"
exit 1
exit 1
fi
if [ "$PASS" -gt 0 ]; then
echo "$TESTNAME PASS" >"$RES_FILE"
Expand Down
Loading