Skip to content

Add OBOM: IAM access graph extraction from IaC (--include-obom) - #34

Open
jthDEV wants to merge 4 commits into
mainfrom
obom
Open

Add OBOM: IAM access graph extraction from IaC (--include-obom)#34
jthDEV wants to merge 4 commits into
mainfrom
obom

Conversation

@jthDEV

@jthDEV jthDEV commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds --include-obom to ts-scan scan: extracts an OBOM (Ownership Bill of
Materials — an IAM access graph, {principal, resource, actions[], effect, grantedVia}) from any CloudFormation/SAM or Terraform found in the local
scan sources.

This is evidence ts-tm-agent
currently has none of for its threat-model trust-boundary work — today its
trust zones are inferred from SBOM/dependency signals alone. Full background
in src/ts_scan/analyse/obom.py's module docstring.

Why a vendored subset instead of a checkov dependency

pip install checkov directly conflicts with ts-scan's own pins on three
libraries — importlib-metadata, cyclonedx-python-lib, packageurl-python
— the latter two core to ts-scan's own SBOM/PURL generation. Investigated
whether checkov's full dependency footprint (~50 packages: boto3,
cloudsplaining, detect_secrets, aiohttp, policy_sentry, …) was actually
needed for graph-building specifically — it wasn't. All of it turned out to
be pulled in through unused import couplings (e.g. cfn_utils.py imports
checkov's entire IAM check registry just for one unrelated function's unused
default-parameter value; graph-building never calls that function).

analyse/_obom_vendor/checkov/ is a trimmed, patched copy (Apache-2.0,
NOTICE.md documents provenance and every change made) containing only the
~25 files graph-building genuinely needs, plus 23 stub modules (a
unittest.mock.MagicMock behind a module-level __getattr__) replacing
subsystems that are imported on that path but never called. One real code
change: common/util/json_utils.py's CustomJSONEncoder is genuinely used
(Terraform variable serialization) and got local unreachable sentinel classes
for four cross-subsystem isinstance() branches instead of a blanket stub.

Resulting new dependencies (the obom extra): bc-python-hcl2,
bc-jsonpath-ng, networkx, dpath, asteval — none conflict with
ts-scan's existing pins.

Verification

  • Output verified byte-identical to a real (non-vendored, non-trimmed)
    checkov-backed extraction, against two fixtures: a real-world SAM template
    (CloudFormation: 15 edges / 3 unresolved) and a hand-written Terraform
    fixture using a different structural IAM pattern (1 edge / 1 unresolved) —
    both before and after trimming, and again through the actual installed
    ts-scan[obom] package and the --include-obom CLI flag end to end.
  • Regression: existing test suite (26 tests) passes unchanged; a plain
    ts-scan scan run (no --include-obom) produces identical output to
    before this change.

Design notes / open questions for review

  • --include-obom runs as an independent second pass, not folded into the
    dependency-scan output — IaC isn't tied to a specific package-manager
    DependencyScan, so OBOM is its own JSON artifact (--obom-output,
    defaults to alongside --output or stdout) rather than a new field on the
    existing scan format. Open to a different shape if there's a preference.
  • Only local Path sources are examined for OBOM; a file:// URL source in
    sources is not resolved back to a path for this pass.
  • CloudFormation/SAM and Terraform only for now (Terraform confirmed to
    reuse the same trimming approach cleanly — same checkov.*.graph_manager
    abstraction). Other checkov-supported IaC front-ends (Kubernetes, ARM,
    Bicep, Dockerfile) not attempted.
  • The vendored subset needs a manual re-trim if checkov's graph-builder
    internals change upstream in a way that breaks it — process documented in
    _obom_vendor/NOTICE.md.

Test plan

  • pip install -e ".[obom]" resolves cleanly
  • ts-scan scan --include-obom <dir-with-sam-template> — matches known-good edge/unresolved counts
  • ts-scan scan --include-obom <dir-with-terraform> — matches known-good edge/unresolved counts
  • ts-scan scan <dir> (no flag) — output unchanged from before this PR
  • pytest tests/ — 26/26 pass

jthDEV added 4 commits August 27, 2026 20:02
Ports the extraction logic from the scan2graph proof of concept
(jthDEV/checkov, feature/scan2graph-extraction) into
ts_scan/analyse/obom.py, mirroring analyse/deepscan.py's optional-
dependency pattern (is_checkov_installed / require_checkov /
CheckovNotInstalledError, lazy import). Verified the module imports
correctly and reports checkov's absence correctly with checkov not
installed.

Declares a new "obom" extra (checkov>=3.3) in pyproject.toml -- but
`pip install -e ".[obom]"` currently fails to resolve: checkov 3.3.x
pins importlib-metadata<8.0, cyclonedx-python-lib<8.0, and
packageurl-python<0.14 -- all three conflict with ts-scan's own pins,
and the latter two (cyclonedx-python-lib, packageurl-python) are core
to ts-scan's actual SBOM/PURL generation, not safely downgradable
without verifying real API compatibility first. Not attempted here.

Not wired into the `scan` CLI command yet -- see the
--include-obom discussion; needs the packaging conflict resolved
(most likely: run checkov-based extraction in an isolated subprocess/
venv rather than as an in-process ts-scan dependency) before that's
meaningful to add.
… dependency

pip install -e ".[obom]" previously failed: checkov 3.3.x pins
importlib-metadata<8.0, cyclonedx-python-lib<8.0, packageurl-python<0.14,
all conflicting with ts-scan's own pins -- the latter two on libraries
core to ts-scan's actual SBOM/PURL generation, not safely downgradable
on a guess.

Investigated why checkov needed so much in the first place: it didn't.
graph_manager.py alone pulled in ~50 packages (boto3, cloudsplaining,
detect_secrets, aiohttp, policy_sentry, ...) purely through unused
import couplings -- e.g. cfn_utils.py importing checkov's whole IAM
check registry for one unrelated function's unused default-parameter
value. None of that is on the path build_graph_from_source_directory()
actually exercises.

analyse/_obom_vendor/checkov/ is the trimmed result (ported from
jthDEV/checkov@feature/scan2graph-extraction, see that branch's
vendor_src/ commit for the full trimming process): the ~25 real files
graph-building needs, 23 stub modules (MagicMock via module __getattr__)
replacing subsystems that are imported but never called on this path,
and one hand-edit (json_utils.py's CustomJSONEncoder is genuinely used
for Terraform variable serialization -- its four cross-subsystem
isinstance() branches got local unreachable sentinel classes instead of
a blanket stub). Verified byte-identical output against both scan2graph
fixtures (CloudFormation: 15 edges/3 unresolved; Terraform: 1 edge/1
unresolved) both via the vendored source directly and via a real
`pip install -e ".[obom]"` of ts-scan itself. Existing test suite (26
tests) still passes.

obom.py: require_checkov() now prepends _obom_vendor/ to sys.path and
imports the vendored copy, instead of importing a pip-installed
checkov. Same call shape, so nothing calling it needs to change.

pyproject.toml: "obom" extra now lists the vendored subset's own real
(non-conflicting) dependencies -- bc-python-hcl2, bc-jsonpath-ng,
networkx, dpath, asteval -- not checkov.

Not wired into the `scan` CLI command yet.
Adds --include-obom (extracts an OBOM -- IAM access graph -- from any
CloudFormation/SAM/Terraform found in the local scan sources, via
analyse/obom.py's vendored graph-builder) and --obom-output (defaults
to alongside --output, or stdout) to `ts-scan scan`.

Runs as an independent second pass, not folded into the dependency
scan: IaC sources aren't tied to a specific package-manager
DependencyScan, so this is its own artifact (a flat {edges, unresolved}
JSON), not a new field grafted onto the existing scan format. Only
local Path sources are examined; a `file://` URL source is not
resolved back to a path for this pass.

Verified end-to-end against both scan2graph fixtures (ts-tm-agent's own
template.yaml: 15 edges/3 unresolved; the hand-written Terraform
fixture: 1 edge/1 unresolved) via the real `ts-scan scan` CLI, plus a
regression run without --include-obom to confirm the existing scan
output is unaffected. Full existing test suite (26 tests) still passes.
@jthDEV
jthDEV requested a review from gr-markin August 27, 2026 18:41

@gr-markin gr-markin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't merge the OBOM functionality in this form into the ts-scan repo, first of all it's a massive amount of code, approx. 300 files alone for OBOM analysis that doesn't have anything in common with the "ts-scan", I would suggest to move everything belonging to OBOM to a separate repo and publish it as a separate Python package. If is's really required to have a flag in the ts-scan, then the ts-scan[obom] option should require this package and make a call into it, but again, I don't see any reason to integrate it into the ts-scan as it's completely distinct from the input as well as from the output, i.e. in order to apply an OBOM scan I need to install an extra option and use an extra flag, so, I could actually install a "ts-obom" package and do "ts-obom scan"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants