Conversation
Ports the extraction logic from the scan2graph proof of concept (jthDEV/checkov, feature/scan2graph-extraction) into ts_scan/analyse/obom.py, mirroring analyse/deepscan.py's optional- dependency pattern (is_checkov_installed / require_checkov / CheckovNotInstalledError, lazy import). Verified the module imports correctly and reports checkov's absence correctly with checkov not installed. Declares a new "obom" extra (checkov>=3.3) in pyproject.toml -- but `pip install -e ".[obom]"` currently fails to resolve: checkov 3.3.x pins importlib-metadata<8.0, cyclonedx-python-lib<8.0, and packageurl-python<0.14 -- all three conflict with ts-scan's own pins, and the latter two (cyclonedx-python-lib, packageurl-python) are core to ts-scan's actual SBOM/PURL generation, not safely downgradable without verifying real API compatibility first. Not attempted here. Not wired into the `scan` CLI command yet -- see the --include-obom discussion; needs the packaging conflict resolved (most likely: run checkov-based extraction in an isolated subprocess/ venv rather than as an in-process ts-scan dependency) before that's meaningful to add.
… dependency pip install -e ".[obom]" previously failed: checkov 3.3.x pins importlib-metadata<8.0, cyclonedx-python-lib<8.0, packageurl-python<0.14, all conflicting with ts-scan's own pins -- the latter two on libraries core to ts-scan's actual SBOM/PURL generation, not safely downgradable on a guess. Investigated why checkov needed so much in the first place: it didn't. graph_manager.py alone pulled in ~50 packages (boto3, cloudsplaining, detect_secrets, aiohttp, policy_sentry, ...) purely through unused import couplings -- e.g. cfn_utils.py importing checkov's whole IAM check registry for one unrelated function's unused default-parameter value. None of that is on the path build_graph_from_source_directory() actually exercises. analyse/_obom_vendor/checkov/ is the trimmed result (ported from jthDEV/checkov@feature/scan2graph-extraction, see that branch's vendor_src/ commit for the full trimming process): the ~25 real files graph-building needs, 23 stub modules (MagicMock via module __getattr__) replacing subsystems that are imported but never called on this path, and one hand-edit (json_utils.py's CustomJSONEncoder is genuinely used for Terraform variable serialization -- its four cross-subsystem isinstance() branches got local unreachable sentinel classes instead of a blanket stub). Verified byte-identical output against both scan2graph fixtures (CloudFormation: 15 edges/3 unresolved; Terraform: 1 edge/1 unresolved) both via the vendored source directly and via a real `pip install -e ".[obom]"` of ts-scan itself. Existing test suite (26 tests) still passes. obom.py: require_checkov() now prepends _obom_vendor/ to sys.path and imports the vendored copy, instead of importing a pip-installed checkov. Same call shape, so nothing calling it needs to change. pyproject.toml: "obom" extra now lists the vendored subset's own real (non-conflicting) dependencies -- bc-python-hcl2, bc-jsonpath-ng, networkx, dpath, asteval -- not checkov. Not wired into the `scan` CLI command yet.
…ovenance and changes (Apache-2.0 §4b/c)
Adds --include-obom (extracts an OBOM -- IAM access graph -- from any
CloudFormation/SAM/Terraform found in the local scan sources, via
analyse/obom.py's vendored graph-builder) and --obom-output (defaults
to alongside --output, or stdout) to `ts-scan scan`.
Runs as an independent second pass, not folded into the dependency
scan: IaC sources aren't tied to a specific package-manager
DependencyScan, so this is its own artifact (a flat {edges, unresolved}
JSON), not a new field grafted onto the existing scan format. Only
local Path sources are examined; a `file://` URL source is not
resolved back to a path for this pass.
Verified end-to-end against both scan2graph fixtures (ts-tm-agent's own
template.yaml: 15 edges/3 unresolved; the hand-written Terraform
fixture: 1 edge/1 unresolved) via the real `ts-scan scan` CLI, plus a
regression run without --include-obom to confirm the existing scan
output is unaffected. Full existing test suite (26 tests) still passes.
gr-markin
left a comment
There was a problem hiding this comment.
I wouldn't merge the OBOM functionality in this form into the ts-scan repo, first of all it's a massive amount of code, approx. 300 files alone for OBOM analysis that doesn't have anything in common with the "ts-scan", I would suggest to move everything belonging to OBOM to a separate repo and publish it as a separate Python package. If is's really required to have a flag in the ts-scan, then the ts-scan[obom] option should require this package and make a call into it, but again, I don't see any reason to integrate it into the ts-scan as it's completely distinct from the input as well as from the output, i.e. in order to apply an OBOM scan I need to install an extra option and use an extra flag, so, I could actually install a "ts-obom" package and do "ts-obom scan"
Summary
Adds
--include-obomtots-scan scan: extracts an OBOM (Ownership Bill ofMaterials — an IAM access graph,
{principal, resource, actions[], effect, grantedVia}) from any CloudFormation/SAM or Terraform found in the localscan sources.
This is evidence ts-tm-agent
currently has none of for its threat-model trust-boundary work — today its
trust zones are inferred from SBOM/dependency signals alone. Full background
in
src/ts_scan/analyse/obom.py's module docstring.Why a vendored subset instead of a
checkovdependencypip install checkovdirectly conflicts with ts-scan's own pins on threelibraries —
importlib-metadata,cyclonedx-python-lib,packageurl-python— the latter two core to ts-scan's own SBOM/PURL generation. Investigated
whether checkov's full dependency footprint (~50 packages: boto3,
cloudsplaining, detect_secrets, aiohttp, policy_sentry, …) was actually
needed for graph-building specifically — it wasn't. All of it turned out to
be pulled in through unused import couplings (e.g.
cfn_utils.pyimportscheckov's entire IAM check registry just for one unrelated function's unused
default-parameter value; graph-building never calls that function).
analyse/_obom_vendor/checkov/is a trimmed, patched copy (Apache-2.0,NOTICE.mddocuments provenance and every change made) containing only the~25 files graph-building genuinely needs, plus 23 stub modules (a
unittest.mock.MagicMockbehind a module-level__getattr__) replacingsubsystems that are imported on that path but never called. One real code
change:
common/util/json_utils.py'sCustomJSONEncoderis genuinely used(Terraform variable serialization) and got local unreachable sentinel classes
for four cross-subsystem
isinstance()branches instead of a blanket stub.Resulting new dependencies (the
obomextra):bc-python-hcl2,bc-jsonpath-ng,networkx,dpath,asteval— none conflict withts-scan's existing pins.
Verification
checkov-backed extraction, against two fixtures: a real-world SAM template
(CloudFormation: 15 edges / 3 unresolved) and a hand-written Terraform
fixture using a different structural IAM pattern (1 edge / 1 unresolved) —
both before and after trimming, and again through the actual installed
ts-scan[obom]package and the--include-obomCLI flag end to end.ts-scan scanrun (no--include-obom) produces identical output tobefore this change.
Design notes / open questions for review
--include-obomruns as an independent second pass, not folded into thedependency-scan output — IaC isn't tied to a specific package-manager
DependencyScan, so OBOM is its own JSON artifact (--obom-output,defaults to alongside
--outputor stdout) rather than a new field on theexisting scan format. Open to a different shape if there's a preference.
Pathsources are examined for OBOM; afile://URL source insourcesis not resolved back to a path for this pass.reuse the same trimming approach cleanly — same
checkov.*.graph_managerabstraction). Other checkov-supported IaC front-ends (Kubernetes, ARM,
Bicep, Dockerfile) not attempted.
internals change upstream in a way that breaks it — process documented in
_obom_vendor/NOTICE.md.Test plan
pip install -e ".[obom]"resolves cleanlyts-scan scan --include-obom <dir-with-sam-template>— matches known-good edge/unresolved countsts-scan scan --include-obom <dir-with-terraform>— matches known-good edge/unresolved countsts-scan scan <dir>(no flag) — output unchanged from before this PRpytest tests/— 26/26 pass