Add DRP-AI/DRP-AI3 real-device test coverage for RZ V2L/V2H/V2N (NEW) - #2871
Open
rickwu666666 wants to merge 15 commits into
Open
rickwu666666 wants to merge 15 commits into
rickwu666666 wants to merge 15 commits into
Conversation
rickwu666666
added
the
ce-oem
rickwu666666
changed the title
Rename rz-drp-ai category/job/test-plan to the generation-neutral rz-ai-accelerator umbrella, add has_rz_drp_ai3 manifest, and add two new jobs: a driver-level device-node smoke test (V2L+V2H/V2N) and an app-level DRP-AI3 object-counter test using the upstream TEST input source for headless/unattended runs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fully synthetically generated (script-drawn shapes), no photographic or copyrighted content; only needs to be a decodable image for the gstreamer TEST pipeline since pass/fail is crash/CPU-fallback based, not detection accuracy. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Real-hardware verification on V2L (CID 202507-36926) confirmed /dev/drp1 does not exist on DRP-AI (single-core) devices - it is DRP-AI3-only. The previous single OR-gated driver-nodes job checked for /dev/drp1 unconditionally, so it always failed on V2L. Split into driver-nodes-drp-ai (V2L) and driver-nodes-drp-ai3 (V2H/V2N), each gated on its own manifest entry and checking only the device nodes confirmed present on that generation. Verified on real hardware: - V2L (202507-36926): driver-nodes-drp-ai PASS, inference PASS - V2H (202503-36406): driver-nodes-drp-ai3 PASS
…nite TEST loop Two real bugs found via V2H (CID 202503-36406) hardware verification: 1. object-counter's TEST mode is a live-display demo that never exits on its own (30fps imagefreeze loop, waits for a keypress). The job now runs it under `timeout -s INT 30` and inspects the captured output/exit code instead of expecting the process to terminate. 2. checkbox-ce-oem is itself a strict-confined snap and exports its own GST_PLUGIN_PATH/GST_PLUGIN_SYSTEM_PATH/GST_PLUGIN_SCANNER into every job's environment. That leaked into and hijacked the rzv-ai-applications subprocess, making GStreamer fail to find even core elements like "filesrc" - a checkbox-side environment leak, not an upstream rzv-ai-applications packaging defect as initially suspected. Fixed by unsetting those vars before invoking object-counter so it falls back to its own snap's plugin path. Verified on real V2H hardware: job now passes reliably, including a combined run alongside driver-nodes-drp-ai3.
- Add DRP_AI_TEST_SNAP environ so the same job body works for both rzv-ai-applications (V2H) and rzv-ai-applications-v2n (V2N), which share the same DRP-AI3 driver/command syntax and only differ by snap name. V2L's inference job is untouched (single snap, no variable needed). - Switch the object-counter model from COCO to animal so the 'dog' class is available, and change the success criterion from 'command ran without [ERROR]' to actually requiring '[INFO] dog detected.' in the captured output. - Fix a false-negative bug: object-counter's detection line isn't flushed when stdout isn't a tty (always true for a checkbox shell job), so it never reached our captured output. Fixed by spawning the command through Python's pty module (pty.spawn), which forces glibc line-buffering in the child without needing an interactive terminal. - Replace the synthetic placeholder test.png with a real, CC0-licensed dog photo (see test.png.LICENSE for provenance) so the model has something it can actually recognize. Verified via sideload on real hardware for both generations: - V2H (CID 202503-36406, DRP_AI_TEST_SNAP=rzv-ai-applications): job passed, dog correctly detected. - V2N (CID 202602-38383, DRP_AI_TEST_SNAP=rzv-ai-applications-v2n): job passed, dog correctly detected. (Required starting the ubuntu-frame snap's headless Wayland compositor for root, which object-counter needs even in TEST/static-image mode.) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Real V2N hardware (CID 202602-38383) does not expose /dev/drp1, unlike V2H (CID 202503-36406). Confirmed against Renesas' official product pages: RZ/V2H integrates an extra standalone "Dynamically reconfigurable processor (DRP)" IP block alongside DRP-AI3, which RZ/V2N does not have. This is an expected hardware difference, not a driver bug, so the job now checks /dev/drp1 only if present and does not fail when it is absent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Swaps the CC0-licensed stock photo for an AI-generated dog image (640x480 RGB PNG, center-cropped to 4:3 to match the object-counter TEST-mode pipeline's expected input resolution), removing the provenance/license note since AI-generated content carries no third-party copyright. Re-verified on real V2H hardware: the drp-ai3-object-counter job still passes and correctly logs '[INFO] dog detected.' with the new image. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
rickwu666666
force-pushed
the
add-rz-drp-ai3-object-counter-test
branch
from
September 17, 2026 01:43
88f7c9b to
5eeae20
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2871 +/- ##
=======================================
Coverage 61.36% 61.36%
=======================================
Files 500 500
Lines 50420 50420
Branches 8856 8856
=======================================
Hits 30941 30941
Misses 18633 18633
Partials 846 846 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Fix the invalid mktemp template and document the required DRP_AI_TEST_SNAP configuration.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds real-device DRP-AI and DRP-AI3 coverage for Renesas RZ V2L, V2H, and V2N boards.
Changes:
- Replaces the legacy DRP-AI definitions with unified accelerator plans and categories.
- Adds generation-specific driver-node checks and DRP-AI3 object-counter testing.
- Adds DRP-AI3 manifest gating and configurable application snap support.
File summaries
| File | Summary |
|---|---|
contrib/checkbox-ce-oem/checkbox-provider-ce-oem/units/rz/rz-test-plan.pxu |
Routes RZ plans to accelerator coverage. |
contrib/checkbox-ce-oem/checkbox-provider-ce-oem/units/rz/drp-ai3-manifest.pxu |
Adds DRP-AI3 capability gating. |
contrib/checkbox-ce-oem/checkbox-provider-ce-oem/units/rz/drp-ai3-jobs.pxu |
Adds the DRP-AI3 object-counter test. |
contrib/checkbox-ce-oem/checkbox-provider-ce-oem/units/rz/drp-ai-test-plan.pxu |
Removes the superseded test plan. |
contrib/checkbox-ce-oem/checkbox-provider-ce-oem/units/rz/drp-ai-jobs.pxu |
Removes the superseded job. |
contrib/checkbox-ce-oem/checkbox-provider-ce-oem/units/rz/drp-ai-category.pxu |
Removes the superseded category. |
contrib/checkbox-ce-oem/checkbox-provider-ce-oem/units/rz/ai-accelerator-test-plan.pxu |
Defines the unified accelerator plan. |
contrib/checkbox-ce-oem/checkbox-provider-ce-oem/units/rz/ai-accelerator-jobs.pxu |
Adds accelerator driver-node and inference jobs. |
contrib/checkbox-ce-oem/checkbox-provider-ce-oem/units/rz/ai-accelerator-category.pxu |
Defines the accelerator category. |
Review details
Suppressed comments (3)
contrib/checkbox-ce-oem/checkbox-provider-ce-oem/units/rz/drp-ai3-jobs.pxu:29
mktemprequires at least three consecutiveXcharacters in the template on Ubuntu. This fixed.logtemplate therefore fails before the object-counter command runs, so the new DRP-AI3 test cannot pass; use a randomized template instead.
output_file="$(mktemp /var/tmp/renesas-rz-drp-ai3.log)"
contrib/checkbox-ce-oem/checkbox-provider-ce-oem/units/rz/drp-ai3-jobs.pxu:28
- Because this command is not covered by
set -e, a failed copy is ignored. If an old$SNAP_REAL_HOME/test.pngremains, the application can test stale input and report a false pass; otherwise the failure is only detected indirectly. Check thecpstatus and exit with a clear failure before launching the sample.
cp "$PLAINBOX_PROVIDER_DATA/rz-ai-accelerator/test.png" "$SNAP_REAL_HOME/test.png"
contrib/checkbox-ce-oem/checkbox-provider-ce-oem/units/rz/drp-ai3-jobs.pxu:30
- This writes to a fixed file in the user's real home and the EXIT trap unconditionally removes it. If
$SNAP_REAL_HOME/test.pngexisted before the job, the test overwrites it and permanently deletes it; preserve and restore an existing file (and handle concurrent sessions) instead of always removing the path.
trap 'rm -f "$output_file" "$SNAP_REAL_HOME/test.png"' EXIT
- Files reviewed: 9/10 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
rickwu666666
marked this pull request as draft
September 17, 2026 02:03
- Add randomized XXXXXX suffix to mktemp templates in rz-ai-accelerator/inference and rz-ai-accelerator/drp-ai3-object-counter jobs; Ubuntu's mktemp requires at least three consecutive X's and was rejecting the previous fixed filenames, causing the jobs to fail before the sample apps even started. - Check the exit status of the test.png copy in rz-ai-accelerator/drp-ai3-object-counter and fail loudly instead of silently continuing with a possibly stale image on copy failure. - Document the DRP_AI_TEST_SNAP environ in contrib/checkbox-ce-oem/README.md so integrators know the valid snap names per board (V2H vs V2N). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
rickwu666666
marked this pull request as ready for review
September 17, 2026 02:51
…ject-counter - Drop the bundled data/rz-ai-accelerator/test.png asset to avoid shipping binary images inside checkbox-ce-oem. - Remove the cp step and PLAINBOX_PROVIDER_DATA environ from rz-ai-accelerator/drp-ai3-object-counter; the job now checks that $SNAP_REAL_HOME/test.png already exists and fails with a clear message if it is missing, instead of copying a bundled copy there. - No longer delete $SNAP_REAL_HOME/test.png on exit, since it is now a tester-supplied file rather than one owned by the job. - Document the required image format/content (640x480, 24-bit RGB PNG with a clearly visible dog) and path in the job _purpose and in contrib/checkbox-ce-oem/README.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
rickwu666666
marked this pull request as draft
September 17, 2026 03:32
Contributor
Author
rickwu666666
marked this pull request as ready for review
September 17, 2026 15:17
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds real-hardware DRP-AI/DRP-AI3 test coverage for Renesas RZ boards
(V2L/V2H/V2N) to
checkbox-provider-ce-oem, per the spec attached toKOTO-774 (
KOTO-774-drp-ai-test-spec.md).Split the driver-level device-node smoke test into two
generation-specific jobs:
driver-nodes-drp-ai(V2L, single-coreDRP-AI) and
driver-nodes-drp-ai3(V2H/V2N, DRP-AI3), instead of oneOR-gated job, because manifest.has_rz_drp_ai / has_rz_drp_ai3 gate
different node sets.
expose
/dev/drp1, unlike V2H. Confirmed against Renesas' officialproduct pages — RZ/V2H has an extra standalone "Dynamically
reconfigurable processor (DRP)" IP block alongside DRP-AI3 that
RZ/V2N lacks. This is an expected chip-level difference, not a bug,
so
/dev/drp1is now checked only if present and its absence doesnot fail the job.
Added a single, merged
drp-ai3-object-counterjob (V2H/V2N) builton the upstream Q08 Object Counter sample app, using a new
DRP_AI_TEST_SNAPenviron so integrators can point at either board'sdifferently-named AI-application snap (
rzv-ai-applications/rzv-ai-applications-v2n) while sharing one identical command/argsset.
TESTmode against a static, AI-generated dogphoto (
data/test.png, no third-party copyright), and asserts success only when theapplication actually logs
[INFO] dog detected.— not merely thatthe command exits 0.
ptymodule)because plain stdout redirection never flushed the detection log
line even after 60s, while a pty (forcing glibc's automatic
line-buffering) surfaced it within ~2s consistently on real
hardware.
GST_PLUGIN_PATHfrom the snapenvironment before invoking the app, which otherwise broke the
GStreamer pipeline.
Both jobs are included in
units/rz/ai-accelerator-test-plan.pxu(rz-ai-accelerator-automated).Verification note: the two touched jobs (
driver-nodes-drp-ai3,drp-ai3-object-counter) were sideloaded and run with a scratch,manifest-stripped copy of the provider (manifest gate only removed
in the scratch copy, never in the committed source) and run scoped
via
--exactto just those two jobs, rather than the wholerz-ai-accelerator-automatedtest plan -- running the full planagainst a manifest-stripped sideload also pulls in the unrelated,
V2L-only
inferencejob on V2H/V2N (its manifest gate is strippedtoo), which would not be representative of production behavior.
driver-nodes-drp-ai(V2L) was validated separately on real V2Lhardware (CID 202507-36926) during spec drafting.
Resolved issues
Resolves KOTO-774.
Documentation
Tests