Mutation Testing Results :: Dead Code + Missing Tests

[81reap]

results of running mutation testing on the rust codebase. I ran it over the entire codebase on my Ryzen 7 5800X3D ~3h (09:56 → 13:07). I left it running through the day as I was at work and these are the results.

files

• caught.txt
• lock.json
• missed.txt
• mutants.json

programmatic breakdowns

issues by file

[reap@jager:~/Desktop/SQLPage]$ awk -F: 'FNR==NR{m[$1]++;next}{c[$1]++}
  END{for(f in m) printf "%d\t%d\t%d\t%.0f%%\t%s\n", m[f], c[f], m[f]+c[f], 100*m[f]/(m[f]+c[f]), f}' \
  mutants.out/{missed,caught}.txt | sort -k1,1rn -k3,3rn

File	Missed	Caught	Viable	Survival
telemetry.rs	52	13	65	80%
template_helpers.rs	49	26	75	65%
render.rs	29	48	77	38%
webserver/database/sql/rewrite.rs	21	92	113	19%
webserver/database/sql_to_json.rs	21	23	44	48%
app_config.rs	18	73	91	20%
webserver/oidc.rs	18	54	72	25%
filesystem.rs	18	31	49	37%
webserver/http.rs	18	26	44	41%
webserver/lambda_http.rs	15	3	18	83%
webserver/database/mod.rs	12	7	19	63%
webserver/database/execute_queries.rs	10	34	44	23%
webserver/database/csv_import.rs	9	25	34	26%
webserver/database/connect.rs	9	4	13	69%
webserver/database/sqlpage_functions/function_traits.rs	7	45	52	13%
webserver/database/error_highlighting.rs	7	22	29	24%
cli/commands.rs	6	0	6	100%
webserver/http_client.rs	5	4	9	56%
webserver/database/migrations.rs	5	2	7	71%
webserver/server_timing.rs	4	12	16	25%
templates.rs	4	7	11	36%
webserver/content_security_policy.rs	4	3	7	57%
webserver/http_request_info.rs	3	40	43	7%
webserver/error.rs	3	16	19	16%
file_cache.rs	3	13	16	19%
webserver/response_writer.rs	3	4	7	43%
webserver/database/sqlpage_functions/functions.rs	3	0	3	100%
webserver/database/blob_to_data_url.rs	2	28	30	7%
webserver/database/sql.rs	2	16	18	11%
webserver/database/sqlpage_functions/http_fetch_request.rs	2	2	4	50%
webserver/error_with_status.rs	2	1	3	67%
webserver/database/sqlpage_expr.rs	1	16	17	6%
webserver/database/sqlpage_functions/url_parameters.rs	1	14	15	7%
main.rs	1	5	6	17%
webserver/database/sql/statement.rs	1	1	2	50%
webserver/http_metrics.rs	1	0	1	100%

Files with zero iissues: single_or_vec.rs, dynamic_component.rs, webserver/routing.rs, webserver/request_variables.rs, lib.rs, utils.rs.

issue by group

[reap@jager:~/Desktop/SQLPage]$ jq -r --rawfile missed mutants.out/missed.txt \
  --argjson dead '["src/template_helpers.rs:525:9","src/template_helpers.rs:526:13",
     "src/webserver/database/mod.rs:76:9","src/webserver/error_with_status.rs:22:9",
     "src/webserver/error_with_status.rs:27:24",
     "src/webserver/database/sqlpage_functions/url_parameters.rs:132:9"]' \
  --argjson harness '["src/telemetry.rs","src/webserver/lambda_http.rs","src/cli/commands.rs","src/main.rs"]' \
  --arg dialect 'Mssql|MySql|Postgres|postgres|mssql|mysql|oracle|duckdb|Snowflake|sql server|sql_nonnull_to_json|decode_pg_range|decimal_to_json' '
  ($missed|split("\n")|map(select(length>0))|INDEX(.)) as $M |
  map(select($M[.name])) | map(. as $m | {cat:
      if   $dead    | index($m.name|split(": ")[0]) then "dead-code"
      elif $harness | index($m.file)                then "harness-gap"
      elif $m.name  | test($dialect)                then "dialect"
      else "logic" end})
  | group_by(.cat)[] | "\(.[0].cat)\t\(length)"' mutants.out/mutants.json
dead-code       7
dialect 33
harness-gap     74
logic   255

[81reap]

LLM Breakdown

since this is a large amount of issues, I also asked claude to break it up by type. I am not aware of a programmatic way to make this breakdown without the use of an LLM.

note :: while an LLM was used for this part it was only used to process the files attached at the top of this issue, the data in SQLPage/mutants.out was created from a deterministic program.

Harness gap (74)(maybe, up to maintainers if these are added benifits or not)

File	Survivors	Why nothing catches it
telemetry.rs	52	no test asserts on log or span output; 19 replace a writer or layer hook with (), 16 of those in logfmt
webserver/lambda_http.rs	15	only compiled/run under the lambda-web feature on AWS
cli/commands.rs	6	100% survival — no test invokes a subcommand
main.rs	1	process entry point

Dialect-gated (33)(no-op, further deep dive required)

The run used the default sqlite::memory:. These arms are unreachable without the matching driver.

File	Survivors	Example
webserver/database/sql_to_json.rs	21	delete match arm "UUID" | "UNIQUEIDENTIFIER", "JSONB", "INT8RANGE", matches!(db_type, Mssql(_)) guards
webserver/database/mod.rs	4	delete match arm "postgres" | "postgresql" in SupportedDatabase::from_dbms_name
filesystem.rs	3	delete match arm SupportedDatabase::Mssql in DbFsQueries::get_create_table_sql
webserver/database/csv_import.rs	2	delete match arm AnyConnectionKind::Postgres in run_csv_import
webserver/database/execute_queries.rs	2	delete match arm sqlx::any::AnyKind::Postgres in set_trace_context
webserver/database/sql/rewrite.rs	1	match guard self.database.database_type == SupportedDatabase::Mssql

Re-running with DATABASE_URL pointed at each backend would reclassify these; CI already has the matrix, but I do not have my dev machine properly setup to be able to re-run tests against these other DBs.

Dead code (7)(yes, we should clean up/refactor these)

Every item in #1389 shows up here independently.

Mutant	Item
template_helpers.rs:525:9 replace <impl CanHelp for H0>::call with Ok(Default::default())	E2
template_helpers.rs:526:13 delete match arm [] in <impl CanHelp for H0>::call	E2
database/mod.rs:76:9 replace SupportedDatabase::otel_name_from_kind with ""	E3
database/mod.rs:76:9 replace SupportedDatabase::otel_name_from_kind with "xyzzy"	E3
error_with_status.rs:22:9 replace <impl ResponseError for ErrorWithStatus>::status_code with Default::default()	E4
error_with_status.rs:27:24 replace == with != in error_response	E4
url_parameters.rs:132:9 replace <impl From<URLParameters> for String>::from with Default::default()	E5

Logic gaps worth a test (subset of the 255)(yes, this is just adding more tests which is always a good thing)

Each row was checked against the source; the last column is what a test would have to assert.

Location	Survivors	Behaviour left unpinned
render.rs HeaderContext::add_cookie	10	secure and http_only default to on, remove/max_age handling. The only cookie test (tests/sql_test_files/data/cookie.sql) reads a cookie; nothing sets one through the component.
template_helpers.rs plus/minus/sum	12	+→-, +→*, +=→-= all survive; no test exercises the arithmetic helpers
template_helpers.rs MarkdownConfig for AppConfig	4	allow_dangerous_html and allow_dangerous_protocol can each be hardcoded true or false unnoticed
oidc.rs validate_redirect_url	3	delete ! in !url.starts_with(redirect_uri) and returning "" both survive; the existing test covers is_safe_relative_redirect, not this wrapper
oidc.rs set_auth_cookie / verify_logout_params	6	expiry and redirect-count comparisons (>→>=, >→==)
sql/rewrite.rs recognize_sqlpage_function	5	every `
sql/rewrite.rs has_single_row_shape / emulated_function	8	same, plus delete match arm "concat"
app_config.rs default_*	12	defaults for max file size, ACME URL, markdown_allow_dangerous_protocol, system_root_ca_certificates, OIDC client id
app_config.rs validation & durations	6	AppConfig::validate, validate_smtp_host, cache_stale_duration_ms, deserialize_duration_seconds, resolve_timeout
error_highlighting.rs quote_source_with_highlight	4	+→- and >→>= on the caret column arithmetic
content_security_policy.rs	4	is_enabled can return a constant either way
http_client.rs default_system_root_ca_certificates_from_env	3	TLS trust-store selection
csv_import.rs CopyCsvOption::{quote,header,null,escape}	7	COPY option parsing
server_timing.rs ServerTiming::header_value	4	/→%, %→+ in the duration formatting
filesystem.rs io_error_status + is_path_missing_error guards	10	mapping ErrorKind to a status code, and the missing-path guards in read_file, modified_since, file_exists
file_cache.rs Cached::needs_check	1	<→<= on the staleness comparison
http_metrics.rs middleware call	1	>=→< on the status-class bucket
blob_to_data_url.rs detect_mime_type	2	magic-byte comparisons

[lovasoa]

Hi @81reap ! Did you review the results? Did you find anything that would be likely to cause problems? Before changing CI, I'd be keen to see actual interesting PRs that fix real issues this uncovered. 100% code coverage is not a goal, what I'd like to see is a realistic assessment of the risk for anything that is not covered. Most of the machine output above does not seem to have enough context to effectively prune the results.

[lovasoa]

Would you be ok with starting with small targeted PRs that add actually useful tests based on what you uncovered here?

[lovasoa]

Plus, looking at the output, I think your mutation testing exercised only the rust crate tests. I think a lot of what it reports is actually covered in end to end playwright tests (the best kind of tests), and in hurl tests from the./examples/ folder.