bug: empty flag values can record COMPLIANT verdicts and silently disable assert gates

[JonJagger]

Preflight

• I searched existing issues and didn't find a duplicate.

What happened?

Problem

When a shell variable used as a flag value is unset, the CLI does not fail. In
several commands it records a compliant verdict, or exits 0 on a
non-compliant one.

kosli attest override --new-compliance-status "$STATUS" ...   # $STATUS unset

is currently identical to --new-compliance-status=true. The artifact is
overridden to compliant, and the command exits 0.

All of the following are reproduced below, on a binary built from f260dee.

Command and flag	An erased value causes	Direction
attest override --new-compliance-status ""	flips the artifact to compliant (flag is declared false)	wrong verdict
attest generic --compliant ""	records is_compliant: true	wrong verdict
assert artifact --dry-run ""	non-compliant artifact exits 0	gate passes
evaluate ... --no-assert $UNSET	policy DENIED, exits 0	gate passes
create flow -t ""	drops a required attestation from the template	rule weakened
create flow --template-file ""	reverts to the deprecated default template	rule weakened
attach-policy -e ""	attaches nothing, exits 0, sends no request	environment ungoverned
assert artifact --flow/--environment/--policy ""	query parameter dropped, assertion unscoped	wrong question asked

|

Two mechanisms

Booleans invert. A boolean flag consumes no value token, so pflag sets it
from the flag's mere presence - and presence means true regardless of the
declared default. An empty value therefore does not fall back to the default, it
inverts any flag declared false. This is why --new-compliance-status ""
yields true even though cmd/kosli/attestOverride.go:119 declares it false.

To be explicit about what is not being reported: a bare --dry-run meaning
true is correct pflag behaviour and is not a bug. The defect is only the
two-token form, where the user wrote a value and it was erased. That form
matters because this repo deliberately supports it - normalizeBoolFlagArgs
exists precisely because people write --dry-run false.

Slices and scalars drop. pflag's StringSlice.Set runs the value through
a CSV reader, and readAsCSV("") returns nothing, so an empty element vanishes
leaving no trace. Empty scalars are omitted from request payloads and query
strings by x != "" tests scattered through the command implementations.

Why the quoted and unquoted forms differ

• --flag "" leaves a stray empty positional argument. Commands whose arg
  validator rejects extra positionals catch it by accident; commands using
  CustomMaximumNArgs(1), which is every attest* command, accept it.
• --flag $UNSET leaves nothing at all - the shell removes the word. This
  form is silent on every command without exception.

Why it matters

Kosli's compliance reporting is asymmetric. Reporting something non-compliant
when it is actually compliant is acceptable; reporting something compliant when
it is actually non-compliant must never happen. The rows above are the second
kind.

Three properties make this worse than an ordinary bug:

• The output looks correct. The unquoted form leaves no residue anywhere.
  kosli attest generic --compliant $UNSET --name foo ... produces a payload
  identical to a deliberate compliant attestation, with the name and every other
  flag intact.
• The trigger is a routine CI accident - a secret that is not set on forks, a
  variable defined only on one matrix leg, a renamed variable.
• Two rows corrupt the rule, not one verdict. create flow -t "" and
  --template-file "" change what the flow requires. Every artifact that passes
  through that flow afterwards is judged against the weakened template, long
  after the run that caused it.

attest decision is the counter-example that shows the intended pattern: it
declares --compliant as false and requires cmd.Flags().Changed("compliant")
(cmd/kosli/attestDecision.go:111-112), so an erased value is an error rather
than a verdict.

Reproduce

Verified 2026-08-12, binary built from f260dee. The attest and create
reproductions use --dry-run so nothing is sent; the assert and evaluate
reproductions use a throwaway token against read-only endpoints, and a local
Rego file.

Override flips to compliant:

$ kosli attest override --new-compliance-status "" --fingerprint <sha> \
    --name foo --reason audit --original-attestation-type generic \
    --flow f --trail t --dry-run
    "new_compliance_status": true

$ kosli attest override --new-compliance-status false ... --dry-run
    "new_compliance_status": false

Generic attestation records compliant:

$ kosli attest generic --compliant "" --fingerprint <sha> --name foo \
    --flow f --trail t --dry-run
    "is_compliant": true

The unquoted form (--compliant $UNSET, i.e. the tokens --compliant --name foo) produces the same payload with no stray argument at all.

Assert gate passes a non-compliant artifact:

$ kosli assert artifact --fingerprint <sha> --flow f --dry-run "$FLAG"
Error: ...
[warning] Encountered an error but --dry-run is enabled. Exiting with 0 exit code.
$ echo $?
0

--dry-run is a boolean, so the empty value enables it, and dry-run converts
any error to exit 0 (cmd/kosli/main.go:198-201). The assert commands
signal non-compliance by returning an error
(cmd/kosli/assertArtifact.go:176-178 returns Artifact is not compliant), so
they land on that path.

Evaluate gate passes a denied policy:

$ kosli evaluate input --input-file in.json --policy deny.rego
RESULT:      DENIED
Error: policy denied: [always denied]          # exit 1

$ kosli evaluate input --input-file in.json --policy deny.rego --no-assert $UNSET
RESULT:      DENIED                            # exit 0

Flow template loses a required attestation:

$ kosli create flow myflow -t coverage -t unit-test --dry-run
    "template": ["coverage", "unit-test", "artifact"]

$ kosli create flow myflow -t "" -t unit-test --dry-run       # $COVERAGE unset
    "template": ["unit-test", "artifact"]

artifact is appended automatically, which masks the drop: the list is still
non-empty and still looks plausible.

Policy attached to nothing:

$ kosli attach-policy mypolicy -e prod --dry-run
... the request would have been sent to: .../environments/demo/prod/policies

$ kosli attach-policy mypolicy -e "" --dry-run
(no output, exit 0, no request)

$ kosli attach-policy mypolicy --dry-run
Error: required flag(s) "environment" not set

-e "" satisfies the required-flag check because Changed is true and
f.Value.String() is "[]", never "".

Why it matters

Kosli's compliance reporting is asymmetric. Reporting something non-compliant
when it is actually compliant is acceptable; reporting something compliant when
it is actually non-compliant must never happen. The rows above are the second
kind.

Three properties make this worse than an ordinary bug:

• The output looks correct. The unquoted form leaves no residue anywhere.
  kosli attest generic --compliant $UNSET --name foo ... produces a payload
  identical to a deliberate compliant attestation, with the name and every other
  flag intact.
• The trigger is a routine CI accident - a secret that is not set on forks, a
  variable defined only on one matrix leg, a renamed variable.
• Two rows corrupt the rule, not one verdict. create flow -t "" and
  --template-file "" change what the flow requires. Every artifact that passes
  through that flow afterwards is judged against the weakened template, long
  after the run that caused it.

attest decision is the counter-example that shows the intended pattern: it
declares --compliant as false and requires cmd.Flags().Changed("compliant")
(cmd/kosli/attestDecision.go:111-112), so an erased value is an error rather
than a verdict.

Reproduce

Verified 2026-08-12, binary built from f260dee. The attest and create
reproductions use --dry-run so nothing is sent; the assert and evaluate
reproductions use a throwaway token against read-only endpoints, and a local
Rego file.

Override flips to compliant:

$ kosli attest override --new-compliance-status "" --fingerprint <sha> \
    --name foo --reason audit --original-attestation-type generic \
    --flow f --trail t --dry-run
    "new_compliance_status": true

$ kosli attest override --new-compliance-status false ... --dry-run
    "new_compliance_status": false

Generic attestation records compliant:

$ kosli attest generic --compliant "" --fingerprint <sha> --name foo \
    --flow f --trail t --dry-run
    "is_compliant": true

The unquoted form (--compliant $UNSET, i.e. the tokens --compliant --name foo) produces the same payload with no stray argument at all.

Assert gate passes a non-compliant artifact:

$ kosli assert artifact --fingerprint <sha> --flow f --dry-run "$FLAG"
Error: ...
[warning] Encountered an error but --dry-run is enabled. Exiting with 0 exit code.
$ echo $?
0

--dry-run is a boolean, so the empty value enables it, and dry-run converts
any error to exit 0 (cmd/kosli/main.go:198-201). The assert commands
signal non-compliance by returning an error
(cmd/kosli/assertArtifact.go:176-178 returns Artifact is not compliant), so
they land on that path.

Evaluate gate passes a denied policy:

$ kosli evaluate input --input-file in.json --policy deny.rego
RESULT:      DENIED
Error: policy denied: [always denied]          # exit 1

$ kosli evaluate input --input-file in.json --policy deny.rego --no-assert $UNSET
RESULT:      DENIED                            # exit 0

Flow template loses a required attestation:

$ kosli create flow myflow -t coverage -t unit-test --dry-run
    "template": ["coverage", "unit-test", "artifact"]

$ kosli create flow myflow -t "" -t unit-test --dry-run       # $COVERAGE unset
    "template": ["unit-test", "artifact"]

artifact is appended automatically, which masks the drop: the list is still
non-empty and still looks plausible.

Policy attached to nothing:

$ kosli attach-policy mypolicy -e prod --dry-run
... the request would have been sent to: .../environments/demo/prod/policies

$ kosli attach-policy mypolicy -e "" --dry-run
(no output, exit 0, no request)

$ kosli attach-policy mypolicy --dry-run
Error: required flag(s) "environment" not set

-e "" satisfies the required-flag check because Changed is true and
f.Value.String() is "[]", never "".

Fix

The mechanism and the full slice plan are in kosli-dev/server#6070. In short,
three changes are needed because pflag destroys the evidence at three different
points:

• an argv-level check for booleans, since the value never reaches pflag;
• a Set-level custom pflag.Value for slices, since readAsCSV discards the
  empty element;
• the existing post-parse guard at cmd/kosli/root.go:407-417, widened to read
  SliceValue.GetSlice() and ungated from the required-flag annotation, for
  scalars and wholly-empty slices.

Two decisions are needed from the team rather than from the implementation:

• Should attest generic --compliant default to true? attest decision
  defaults to false and requires Changed. Aligning them is breaking, but the
  current default is what turns an erased value into a compliant attestation.
• Should the bare form of the gate booleans still be accepted? No
  implementation can distinguish --dry-run $UNSET from a deliberate bare
  --dry-run, because the shell removes the word before the process starts. The
  only fix for that form is to require --dry-run=true, for --dry-run,
  --no-assert, --compliant and --new-compliance-status.

Worth considering independently: dry-run currently swallows errors from the
assert and evaluate families. Dry-run means "write nothing", but those
commands only read, so swallowing their verdict turns a gate into a no-op even
when dry-run was genuinely intended.

Related

• kosli-dev/server#6070 - root cause, the full set of sixteen findings, and the
  slice plan. This issue is the compliance-affecting subset, split out because
  the title of #6070 ("attachments are silently dropped") does not convey that
  compliance verdicts are affected.

Steps to reproduce

See above

CLI version

See above

Environment

No response

Logs / output