Inspect and correct workload learning
Use
Choose Learned workloads from the native app's gear menu or the phone dashboard navigation. Search by name, select a workload, and inspect its measured peaks, successful and incomplete run counts, variants and recent numeric measurements. A job's detail view links to its current learning profile.
Give a profile a useful name and save minimum RAM or CPU reservations when you know a job needs more headroom. Blank minimums return to automatic estimation. Sharing learning combines the original families under the selected target profile, keeping the larger reservations from both profiles. Separate a family when it should learn independently. Reset measurements only when the old evidence is obsolete; reset removes those measurements permanently while preserving your corrections.
Group estimates deliberately have no exact-variant successful-run count: they describe the conservative envelope a new variant can borrow. Each actual job still has its own prediction and confidence. Measured history, the group estimate and operator minimums are displayed separately. Run history and variant peaks can include multiple machines; each run identifies its machine with an opaque hash, while the group estimate uses the current machine only.
The CLI has the same inspection and correction behavior:
agent-heavy profiles list --json
-agent-heavy profiles list --query Tests --offset 0 --limit 100 --json
Use the returned profile ID with agent-heavy profiles show ID --json. Edit, merge, split and reset take the profile ID and a JSON object on private standard input. Include the latest revision from the detail response. Edit requires label, memoryFloorBytes and cpuFloorCores, with null clearing a minimum. Merge requires targetId; split requires family; reset requires confirm: true. The response is the saved profile's new detail and revision. JSON failures contain an error message and exit with status 125.
Contract
The native app, phone dashboard and CLI expose learned workload profiles: their names, member job families, variants, successful and incomplete observations, measured resource peaks and recent runs. When a local machine identity is available, profile details also show the current resource prediction and its confidence. Unknown telemetry remains unavailable rather than inventing an estimate.
Job families remain independent of linked-worktree checkout paths and source revisions. A profile groups one or more families for statistical learning. Automatic groups initially contain one family. Operator corrections can rename a profile, share learning between profiles, separate a family into its own profile, set minimum CPU/RAM reservations, or reset obsolete measurements. A job's details link to the profile used for its prediction.
Sharing learning changes future lookup for existing and future variants of those families; it does not rewrite a command or merge exact-variant success counts. Machine-specific evidence stays machine-specific. Family transfer remains conservative until an exact variant has enough recent successful evidence. Separating a family restores its independent history because measurements retain their original family and variant identities.
Minimum reservations are explicit operator floors applied above the statistical estimate. They do not fabricate successful measurements or reduce observed peaks. Clearing a floor returns to the learned envelope. Resetting measurements is explicitly confirmed, preserves naming/grouping corrections and returns affected workloads to conservative cold-start learning. Active jobs continue and can contribute new measurements when they finish.
Consistency and privacy
Every mutation carries the revision the operator reviewed. Stale edits are rejected and require refresh. Changes invalidate pending automatic admission decisions and are applied by the independent controller on its next fresh pass. Existing active work and explicit manual overrides are preserved. A controller pass using predictions from before an edit cannot publish a new automatic grant afterward.
Adaptive corrections require a fresh acknowledgement from an updated controller. A still-running older controller and older waiting helpers cannot safely interpret the new grouping, so finish the installation and wait for its controller before editing in Adaptive mode, or deliberately select Fixed scheduling. Installation reloads only the resource controller before marking the upgrade complete. A failed reload leaves installation visibly incomplete and can be retried; the independent queue daemon keeps active jobs running.
The inspection API uses bounded summaries and numeric history. Names come from the existing safe job descriptions or explicit operator labels; raw arguments, inline programs, environment values and credentials are not retained by this feature. Read-only listing does not create or migrate a database. Private SQLite storage, bounded history, ownership checks and existing authenticated/CSRF browser boundaries remain in force.
Interface contract
The CLI entry point is agent-heavy profiles. List and show return JSON when requested. Changes use an explicit expected revision; reset additionally requires confirmation. The native and phone interfaces call the same CLI-owned behavior.
List responses contain revision, profiles, total, offset and limit. Each profile summary contains id, label, memberCount, variantCount, successCount, incompleteCount, peakMemoryBytes, peakCpuCores, lastRunAt, memoryFloorBytes and cpuFloorCores. An empty collection is a normal cold-start state.
Detail responses contain revision, profile, members, variants, runs, prediction, predictionMachine and truncated. A member contains family, label, project and executable. A variant contains id, family, successCount, incompleteCount, lastRunAt, peakMemoryBytes and peakCpuCores. A run contains id, family, variant, machine, sampledAt, memoryBytes, cpuCores, cpuSeconds, durationSeconds and success. Run records with success false are labelled incomplete or failed rather than being counted as successful evidence.
Prediction uses the existing memoryBytes, cpuCores, durationSeconds, samples, confidence and source fields, or null when machine identity is unavailable. predictionMachine is the opaque machine identifier or null. truncated contains boolean members, variants and runs flags where detail limits apply. All identities are opaque hashes; presentation does not claim that a hash is a reproducible-build cache key.
Catalog pages contain at most 100 profiles. A detail view contains at most 256 original families, 100 variants and 30 recent runs. Truncation is explicit and the same detail limits apply to the CLI. Bounded numeric retention can leave an upper envelope after individual run records expire; displayed successful and incomplete counts describe retained run records. Explicit naming and grouping corrections are pinned and never silently evicted to make room for automatic metadata.
Both response forms include controllerAppliedRevision and controllerStatus. Saving a correction can succeed before the controller acknowledges it; refresh learning to see that acknowledgement. Queue polling does not replace drafts or fetch profile history. Destructive confirmations bind to the reviewed profile and revision; a refresh cancels an old confirmation.
\ No newline at end of file
+Inspect and correct workload learning
Use
Choose Learned workloads from the native app's gear menu or the phone dashboard navigation. Search by name, select a workload, and inspect its measured peaks, successful and incomplete run counts, variants and recent numeric measurements. A job's detail view links to its current learning profile.
Give a profile a useful name and save minimum RAM or CPU reservations when you know a job needs more headroom. Blank minimums return to automatic estimation. Sharing learning combines the original families under the selected target profile, keeping the larger reservations from both profiles. Separate a family when it should learn independently. Reset measurements only when the old evidence is obsolete; reset removes those measurements permanently while preserving your corrections.
Related workloads now share learning automatically within the same verified git repository, whatever subdirectory or linked worktree they ran in. Grouping needs a recognized tool (Go, JavaScript, Python, shell, Cargo, Swift or Make). Each workload also has an operation: tests, browser tests, native tests, race tests, full tests, build, lint, type check, dependencies, benchmark or combined checks, and other for mixed batches and anything unknown. Workloads with other share one group per repository and tool, so one heavy script raises the conservative estimate for the rest of that group. Simple env and fnm exec wrappers, *.sh scripts and inline sh -c or bash -c programs are recognized for new jobs. Opaque scripts can use their safe task descriptions. Unrecognized tools and workloads outside a verified git repository remain independent. Project names alone never establish a repository match. Only the category is stored; command arguments and inline program text are never retained. Automatic groups retain the largest measured peaks rather than averaging dissimilar runs down.
Group estimates deliberately have no exact-variant successful-run count: they describe the conservative envelope a new variant can borrow. Each actual job still has its own prediction and confidence. Measured history, the group estimate and operator minimums are displayed separately. Run history and variant peaks can include multiple machines; each run identifies its machine with an opaque hash, while the group estimate uses the current machine only.
The CLI has the same inspection and correction behavior:
agent-heavy profiles list --json
+agent-heavy profiles list --query Tests --offset 0 --limit 100 --json
+agent-heavy profiles list --project 'Agent Queue' --operation tests --json
+agent-heavy profiles list --other-projects --json
+agent-heavy profiles list --hierarchy --limit 50 --json
--project and --operation filter before paging; --other-projects selects the collapsed set described below and cannot be combined with --project. --hierarchy returns the project to operation overview and accepts only --limit. The authenticated HTTP equivalents are the project, operation, otherProjects=true and hierarchy=true query parameters on GET /api/profiles; invalid values are rejected.
Use the returned profile ID with agent-heavy profiles show ID --json. Edit, merge, split and reset take the profile ID and a JSON object on private standard input. Include the latest revision from the detail response. Edit requires label, memoryFloorBytes and cpuFloorCores, with null clearing a minimum. Merge requires targetId; split requires family; reset requires confirm: true. The response is the saved profile's new detail and revision. JSON failures contain an error message and exit with status 125.
An updated resource controller also performs one retroactive pass after its first successful collection. A failed pass is logged in the resource controller log and retried after ten minutes; it does not hold admissions, because history only improves grouping. Retained numeric runs are joined to private safe metadata without requiring original commands or surviving worktrees. A deleted checkout can still group when its old directory scope maps to exactly one live repository; a scope that maps to more than one repository stays unknown and independent. Every retained workload receives a stored classification, with other as the fallback. Missing metadata leaves a family independent. So does metadata that disagrees on a family's kind or verifies two different repositories, and later agreeing files cannot undo that; running in two subdirectories or worktrees of one repository is not a disagreement. Pinned profiles are never touched. The controller runs this backfill once, recorded as grouping version 2; an explicit profiles regroup --apply never marks it done. Running the command again is safe and can incorporate additional retained metadata. Known limit: the repository for a deleted checkout comes from the scope stored for its family, which is trusted. If an unrelated repository later reused that path, the family can be assigned the wrong repository; ungroup the workload to correct it.
Contract
The native app, phone dashboard and CLI expose learned workload profiles: their names, member job families, variants, successful and incomplete observations, measured resource peaks and recent runs. When a local machine identity is available, profile details also show the current resource prediction and its confidence. Unknown telemetry remains unavailable rather than inventing an estimate.
Job families remain independent of linked-worktree checkout paths and source revisions. A profile groups one or more families for statistical learning. Automatic groups combine compatible families; their job details identify the match as Automatic similar workloads. Operator corrections can rename a profile, share learning between profiles, separate a family into its own profile, set minimum CPU/RAM reservations, or reset obsolete measurements. Corrections pin those profiles so automatic grouping cannot undo them or add new members. A job's details link to the profile used for its prediction.
Sharing learning changes future lookup for existing and future variants of those families; it does not rewrite a command or merge exact-variant success counts. Machine-specific evidence stays machine-specific. Family transfer remains conservative until an exact variant has enough recent successful evidence. Separating a family restores its independent history because measurements retain their original family and variant identities.
Minimum reservations are explicit operator floors applied above the statistical estimate. They do not fabricate successful measurements or reduce observed peaks. Clearing a floor returns to the learned envelope. Resetting measurements is explicitly confirmed, preserves naming/grouping corrections and returns affected workloads to conservative cold-start learning. Active jobs continue and can contribute new measurements when they finish.
Consistency and privacy
Every mutation carries the revision the operator reviewed. Stale edits are rejected and require refresh. Changes invalidate pending automatic admission decisions and are applied by the independent controller on its next fresh pass. Existing active work and explicit manual overrides are preserved. A controller pass using predictions from before an edit cannot publish a new automatic grant afterward.
Adaptive corrections require a fresh acknowledgement from an updated controller. A still-running older controller and older waiting helpers cannot safely interpret the new grouping, so finish the installation and wait for its controller before editing in Adaptive mode, or deliberately select Fixed scheduling. Installation reloads only the resource controller before marking the upgrade complete. A failed reload leaves installation visibly incomplete and can be retried; the independent queue daemon keeps active jobs running.
The inspection API uses bounded summaries and numeric history. Names come from the existing safe job descriptions or explicit operator labels; raw arguments, inline programs, environment values and credentials are not retained by this feature. Read-only listing does not create or migrate a database. Private SQLite storage, bounded history, ownership checks and existing authenticated/CSRF browser boundaries remain in force.
Interface contract
The CLI entry point is agent-heavy profiles. List and show return JSON when requested. Changes use an explicit expected revision; reset additionally requires confirmation. The native and phone interfaces call the same CLI-owned behavior.
List responses contain revision, profiles, total, offset and limit. Each profile summary contains id, label, project, operation, memberCount, variantCount, successCount, incompleteCount, peakMemoryBytes, peakCpuCores, lastRunAt, memoryFloorBytes and cpuFloorCores. project is the most common member project, or Unknown project; operation is the shared operation, or other. An empty collection is a normal cold-start state.
The hierarchy response contains revision, projects, total and limit. Each project row has project, collapsed, workloadCount, groupCount, runCount and its operations, each with the same counts. Projects with two workloads or fewer collapse into one Other projects row with collapsed: true and a projectCount; filtering with --other-projects returns exactly the groups behind that row. At most 100 rows are returned in total, including the Other projects row, so limit counts it and total counts every project that is not collapsed.
Detail responses contain revision, profile, members, variants, runs, prediction, predictionMachine and truncated. A member contains family, label, project and executable. A variant contains id, family, successCount, incompleteCount, lastRunAt, peakMemoryBytes and peakCpuCores. A run contains id, family, variant, machine, sampledAt, memoryBytes, cpuCores, cpuSeconds, durationSeconds and success. Run records with success false are labelled incomplete or failed rather than being counted as successful evidence.
Prediction uses the existing memoryBytes, cpuCores, durationSeconds, samples, confidence and source fields, or null when machine identity is unavailable. predictionMachine is the opaque machine identifier or null. truncated contains boolean members, variants and runs flags where detail limits apply. All identities are opaque hashes; presentation does not claim that a hash is a reproducible-build cache key.
Catalog pages contain at most 100 profiles. A detail view contains at most 256 original families, 100 variants and 30 recent runs. Truncation is explicit and the same detail limits apply to the CLI. Bounded numeric retention can leave an upper envelope after individual run records expire; displayed successful and incomplete counts describe retained run records. Explicit naming and grouping corrections are pinned and never silently evicted to make room for automatic metadata.
Both response forms include controllerAppliedRevision and controllerStatus. Saving a correction can succeed before the controller acknowledges it; refresh learning to see that acknowledgement. Queue polling does not replace drafts or fetch profile history. Destructive confirmations bind to the reviewed profile and revision; a refresh cancels an old confirmation.
\ No newline at end of file
diff --git a/docs/learning.md b/docs/learning.md
index f45f83b..5772544 100644
--- a/docs/learning.md
+++ b/docs/learning.md
@@ -14,7 +14,7 @@ Bulk merging is one transaction and one learning revision. It keeps the destinat
Give a profile a useful name and save minimum RAM or CPU reservations when you know a job needs more headroom. Blank minimums return to automatic estimation. Sharing learning combines the original families under the selected target profile, keeping the larger reservations from both profiles. Separate a family when it should learn independently. Reset measurements only when the old evidence is obsolete; reset removes those measurements permanently while preserving your corrections.
-Related workloads now share learning automatically within the same repository and repository-relative working directory. Grouping recognizes the tool and operation, with separate categories for broad, race, native and browser tests. Simple `env` and `fnm exec` wrappers are recognized for new jobs. Opaque scripts can use their safe task descriptions; mixed or unclassified batches remain independent. Project names alone never establish a repository match. Automatic groups retain the largest measured peaks rather than averaging dissimilar runs down.
+Related workloads now share learning automatically within the same verified git repository, whatever subdirectory or linked worktree they ran in. Grouping needs a recognized tool (Go, JavaScript, Python, shell, Cargo, Swift or Make). Each workload also has an operation: tests, browser tests, native tests, race tests, full tests, build, lint, type check, dependencies, benchmark or combined checks, and `other` for mixed batches and anything unknown. Workloads with `other` share one group per repository and tool, so one heavy script raises the conservative estimate for the rest of that group. Simple `env` and `fnm exec` wrappers, `*.sh` scripts and inline `sh -c` or `bash -c` programs are recognized for new jobs. Opaque scripts can use their safe task descriptions. Unrecognized tools and workloads outside a verified git repository remain independent. Project names alone never establish a repository match. Only the category is stored; command arguments and inline program text are never retained. Automatic groups retain the largest measured peaks rather than averaging dissimilar runs down. History keeps no command line, so an older job started through `env` or `fnm` counts as shell or JavaScript; new jobs are classified by what the wrapper runs.
Agent session IDs, per-call IDs and authentication socket locations no longer reset exact-variant learning. Source revisions are recorded as freshness context instead of creating a different exact variant on every commit. Arguments, resource controls, setup fingerprints and executable identity continue to distinguish variants. These changes affect learning only; commands and their execution environments are unchanged.
@@ -25,9 +25,12 @@ The CLI has the same inspection and correction behavior:
```sh
agent-heavy profiles list --json
agent-heavy profiles list --query Tests --offset 0 --limit 100 --json
+agent-heavy profiles list --project 'Agent Queue' --operation tests --json
+agent-heavy profiles list --other-projects --json
+agent-heavy profiles list --hierarchy --limit 50 --json
```
-Use the returned profile ID with `agent-heavy profiles show ID --json`. To inspect one original workload, use `agent-heavy profiles show ID --family FAMILY --json`. To search or page all members, use `agent-heavy profiles show ID --members-query "tests" --members-offset 0 --members-limit 256 --json`. Member search matches safe names, project, executable and opaque family identity. Family inspection and member search/paging are separate reads. Edit, merge, combine, split and reset take the profile ID and a JSON object on private standard input. Include the latest `revision` from the detail response. Edit requires `label`, `memoryFloorBytes` and `cpuFloorCores`, with null clearing a minimum. Merge requires `targetId`; split requires `family`; reset requires `confirm: true`. Bulk `combine` takes the destination profile ID and `{ "revision": 7, "profileIds": ["SOURCE_GROUP_ID"], "families": ["WORKLOAD_FAMILY_ID"] }` on standard input, replacing the placeholders with returned opaque identities. Both arrays are required, either may be empty, and their combined size must be 1 to 32. The authenticated HTTP equivalent is POST `/api/profiles/DESTINATION_ID/combine` with the same JSON object and existing CSRF protection. The response is the saved profile's new detail and revision. JSON failures contain an error message and exit with status 125.
+`--project` and `--operation` filter before paging; `--other-projects` selects the collapsed set described below and cannot be combined with `--project`. `--hierarchy` returns the project to operation overview and accepts only `--limit`. The authenticated HTTP equivalents are the `project`, `operation`, `otherProjects=true` and `hierarchy=true` query parameters on GET `/api/profiles`; invalid values are rejected. Use the returned profile ID with `agent-heavy profiles show ID --json`. To inspect one original workload, use `agent-heavy profiles show ID --family FAMILY --json`. To search or page all members, use `agent-heavy profiles show ID --members-query "tests" --members-offset 0 --members-limit 256 --json`. Member search matches safe names, project, executable and opaque family identity. Family inspection and member search/paging are separate reads. Edit, merge, combine, split and reset take the profile ID and a JSON object on private standard input. Include the latest `revision` from the detail response. Edit requires `label`, `memoryFloorBytes` and `cpuFloorCores`, with null clearing a minimum. Merge requires `targetId`; split requires `family`; reset requires `confirm: true`. Bulk `combine` takes the destination profile ID and `{ "revision": 7, "profileIds": ["SOURCE_GROUP_ID"], "families": ["WORKLOAD_FAMILY_ID"] }` on standard input, replacing the placeholders with returned opaque identities. Both arrays are required, either may be empty, and their combined size must be 1 to 32. The authenticated HTTP equivalent is POST `/api/profiles/DESTINATION_ID/combine` with the same JSON object and existing CSRF protection. The response is the saved profile's new detail and revision. JSON failures contain an error message and exit with status 125.
Preview retroactive grouping without changing the database:
@@ -41,7 +44,7 @@ The response gives `revision`, `profilesBefore`, `profilesAfter`, `familiesGroup
agent-heavy profiles regroup --apply --json < regroup-request.json
```
-An updated resource controller also performs one retroactive pass after its first successful collection. Retained numeric runs are joined to private safe metadata without requiring original commands or surviving worktrees. A deleted checkout can still group within its original directory; the algorithm does not guess which repository owned it. Missing metadata and conflicting classifications leave families independent. Running the command again is safe and can incorporate additional retained metadata.
+An updated resource controller also performs one retroactive pass after its first successful collection. A failed pass is logged in the resource controller log and retried after ten minutes; it does not hold admissions, because history only improves grouping. Retained numeric runs are joined to private safe metadata without requiring original commands or surviving worktrees. A deleted checkout can still group when its old directory scope maps to exactly one live repository; a scope that maps to more than one repository stays unknown and independent. Every registered workload receives a stored classification, with `other` as the fallback. Tool and operation come from the registered workload alone (its legacy kind if it has one, else its executable and title), never from per-file metadata, so file order cannot change them. The repository is the one its surviving checkouts verify. Checkouts that verify two repositories leave it unknown, and the family stays independent with no alias fallback. A family with no verified checkout takes the one repository of its stored scope. Different directories, labels or missing kinds never block sharing. A stored `other/other` can be replaced once by a real classification; any other stored tool or operation stays. Pinned profiles are never touched. One unreadable or corrupt metadata file, a corrupt sidecar or a full registry skips that record and never holds the backfill. Adding tool or operation vocabulary needs a new grouping version so existing history is classified again. The controller runs this backfill once, recorded as grouping version 2; an explicit `profiles regroup --apply` never marks it done. Running the command again is safe and can incorporate additional retained metadata. Known limit: the repository for a deleted checkout comes from the scope stored for its family, which is trusted. If an unrelated repository later reused that path, the family can be assigned the wrong repository; ungroup the workload to correct it.
## Contract
@@ -69,7 +72,9 @@ The inspection API uses bounded summaries and numeric history. Names come from t
The CLI entry point is agent-heavy profiles. List and show return JSON when requested. Changes use an explicit expected revision; reset additionally requires confirmation. The native and phone interfaces call the same CLI-owned behavior.
-List responses contain revision, profiles, total, offset and limit. Each profile summary contains id, label, memberCount, variantCount, successCount, incompleteCount, peakMemoryBytes, peakCpuCores, lastRunAt, memoryFloorBytes and cpuFloorCores. An empty collection is a normal cold-start state.
+List responses contain revision, profiles, total, offset and limit. Each profile summary contains id, label, project, operation, memberCount, variantCount, successCount, incompleteCount, peakMemoryBytes, peakCpuCores, lastRunAt, memoryFloorBytes and cpuFloorCores. `project` is the most common member project, or `Unknown project`; `operation` is the shared operation, or `other`. An empty collection is a normal cold-start state.
+
+The hierarchy response contains revision, projects, total and limit. Each project row has project, collapsed, workloadCount, groupCount, runCount and its operations, each with the same counts. Projects with two workloads or fewer collapse into one `Other projects` row with `collapsed: true` and a `projectCount`; filtering with `--other-projects` returns exactly the groups behind that row. At most 100 rows are returned in total, including the `Other projects` row, so `limit` counts it and `total` counts every project that is not collapsed.
Detail responses contain revision, profile, members, variants, runs, prediction, predictionMachine and truncated. A member contains family, label, project, executable and measurements. Its measurements contain variantCount, successCount, incompleteCount, peakMemoryBytes, peakCpuCores and lastRunAt for that member's full retained history. memberPage contains total, offset, limit and query. An inspectedFamily response contains exactly that member and filters variants and runs before applying their limits; profile and prediction remain explicitly group summaries. A variant contains id, family, successCount, incompleteCount, lastRunAt, peakMemoryBytes and peakCpuCores. A run contains id, family, variant, machine, sampledAt, memoryBytes, cpuCores, cpuSeconds, durationSeconds and success. Run records with success false are labelled incomplete or failed rather than being counted as successful evidence.
diff --git a/internal/backend/inspection.go b/internal/backend/inspection.go
index c9f64d6..fcead81 100644
--- a/internal/backend/inspection.go
+++ b/internal/backend/inspection.go
@@ -132,6 +132,7 @@ func MakeDescription(g *Gate, argv []string, directory, label string) (Object, e
if kind := profileCommandKind(argv, title, executable); kind != "" {
result["learningKind"] = kind
}
+ result["learningTool"], result["learningOperation"] = profileClassify(argv, title, executable)
return result, nil
}
func payloadLabel(payload Object) string {
@@ -248,12 +249,23 @@ func DescribeTask(g *Gate, task Object, state string) (Object, error) {
result["title"] = DisplayText(label, 120)
}
if value != "" {
- if !jsonEqual(g.DescriptionCache[value], result) {
- if err := CacheDescription(g, value, result); err != nil {
+ // The classification is known only at submission, so the cached file keeps
+ // it; the returned description stays limited to the display fields.
+ persisted := Object{}
+ for k, v := range result {
+ persisted[k] = v
+ }
+ for _, key := range []string{"learningTool", "learningOperation"} {
+ if v, ok := description[key]; ok {
+ persisted[key] = v
+ }
+ }
+ if !jsonEqual(g.DescriptionCache[value], persisted) {
+ if err := CacheDescription(g, value, persisted); err != nil {
return nil, err
}
}
- g.DescriptionCache[value] = result
+ g.DescriptionCache[value] = persisted
}
return result, nil
}
diff --git a/internal/backend/learning.go b/internal/backend/learning.go
index 02d406a..af9478d 100644
--- a/internal/backend/learning.go
+++ b/internal/backend/learning.go
@@ -97,7 +97,23 @@ func learningController(g *Gate) (string, any, string) {
}
return machine, applied, reason
}
+
+// ProfileCatalogOptions selects a catalog page, or the project hierarchy.
+type ProfileCatalogOptions struct {
+ Offset, Limit int
+ Query string
+ Filter ProfileFilter
+ Hierarchy bool
+}
+
func LearningInspect(g *Gate, identity string, offset, limit int, query string, detailOptions ...ProfileDetailOptions) (Object, error) {
+ detail := ProfileDetailOptions{MemberLimit: 256}
+ if len(detailOptions) > 0 {
+ detail = detailOptions[0]
+ }
+ return learningInspect(g, identity, ProfileCatalogOptions{Offset: offset, Limit: limit, Query: query}, detail)
+}
+func learningInspect(g *Gate, identity string, catalog ProfileCatalogOptions, options ProfileDetailOptions) (Object, error) {
machine, applied, status := learningController(g)
store, e := NewProfileStore(g.State, true)
if e != nil {
@@ -109,13 +125,14 @@ func LearningInspect(g *Gate, identity string, offset, limit int, query string,
if e = profileID(identity); e != nil {
return nil, e
}
- options := ProfileDetailOptions{MemberLimit: 256}
- if len(detailOptions) > 0 {
- options = detailOptions[0]
- }
result, e = store.Inspect(identity, machine, options)
+ } else if catalog.Hierarchy {
+ if catalog.Offset != 0 || catalog.Query != "" || catalog.Filter.any() {
+ return nil, fmt.Errorf("The project hierarchy takes no page, search or filter")
+ }
+ result, e = store.Hierarchy(catalog.Limit)
} else {
- result, e = store.Catalog(offset, limit, query)
+ result, e = store.CatalogFiltered(catalog.Offset, catalog.Limit, catalog.Query, catalog.Filter)
}
if e != nil {
return nil, e
@@ -354,6 +371,8 @@ func LearningCLI(gate *Gate, args []string, input io.Reader, output, errors io.W
}
identity, query := "", ""
offset, limit := 0, 100
+ catalog := ProfileCatalogOptions{}
+ seenCatalogOptions := map[string]bool{}
detailOptions := ProfileDetailOptions{MemberLimit: 256}
seenDetailOptions := map[string]bool{}
apply := false
@@ -405,9 +424,22 @@ func LearningCLI(gate *Gate, args []string, input io.Reader, output, errors io.W
}
return fail(fmt.Errorf("Invalid workload regroup argument"))
}
+ if operation == "list" && (arg == "--other-projects" || arg == "--hierarchy") {
+ if seenCatalogOptions[arg] {
+ return fail(fmt.Errorf("Duplicate workload profile option"))
+ }
+ seenCatalogOptions[arg] = true
+ catalog.Filter.OtherProjects = catalog.Filter.OtherProjects || arg == "--other-projects"
+ catalog.Hierarchy = catalog.Hierarchy || arg == "--hierarchy"
+ continue
+ }
if operation == "list" {
key, value, attached := strings.Cut(arg, "=")
- if containsString([]string{"--offset", "--limit", "--query"}, key) {
+ if containsString([]string{"--offset", "--limit", "--query", "--project", "--operation"}, key) {
+ if (key == "--project" || key == "--operation") && seenCatalogOptions[key] {
+ return fail(fmt.Errorf("Duplicate workload profile option"))
+ }
+ seenCatalogOptions[key] = true
if !attached {
i++
if i >= len(args) {
@@ -428,6 +460,15 @@ func LearningCLI(gate *Gate, args []string, input io.Reader, output, errors io.W
}
case "--query":
query = value
+ case "--project", "--operation":
+ if value == "" {
+ return fail(fmt.Errorf("Invalid workload %s filter", strings.TrimPrefix(key, "--")))
+ }
+ if key == "--project" {
+ catalog.Filter.Project = value
+ } else {
+ catalog.Filter.Operation = value
+ }
}
continue
}
@@ -438,6 +479,10 @@ func LearningCLI(gate *Gate, args []string, input io.Reader, output, errors io.W
}
return fail(fmt.Errorf("Invalid workload profile argument"))
}
+ // Presence decides, as in the HTTP route: a default-valued page is still a page.
+ if catalog.Hierarchy && (seenCatalogOptions["--offset"] || seenCatalogOptions["--query"]) {
+ return fail(fmt.Errorf("The project hierarchy takes no page, search or filter"))
+ }
if operation != "list" && operation != "regroup" && identity == "" {
return fail(fmt.Errorf("A workload profile identity is required"))
}
@@ -452,7 +497,8 @@ func LearningCLI(gate *Gate, args []string, input io.Reader, output, errors io.W
result, e = LearningRegroup(gate, apply, request)
}
} else if operation == "list" || operation == "show" {
- result, e = LearningInspect(gate, identity, offset, limit, query, detailOptions)
+ catalog.Offset, catalog.Limit, catalog.Query = offset, limit, query
+ result, e = learningInspect(gate, identity, catalog, detailOptions)
} else {
var request Object
request, e = ReadLearningInput(input)
diff --git a/internal/backend/profiles_classification.go b/internal/backend/profiles_classification.go
new file mode 100644
index 0000000..f0a1729
--- /dev/null
+++ b/internal/backend/profiles_classification.go
@@ -0,0 +1,453 @@
+package backend
+
+import (
+ "fmt"
+ "path/filepath"
+ "regexp"
+ "sort"
+ "strings"
+)
+
+// The classification lives beside learning_similarity rather than in it: old
+// clients insert three values into that table and reject any kind they do not
+// know, and they share this database. Nothing here is written to it.
+const profileClassificationSchema = `CREATE TABLE IF NOT EXISTS learning_classification (family TEXT PRIMARY KEY,repository TEXT,tool TEXT NOT NULL,operation TEXT NOT NULL);
+CREATE INDEX IF NOT EXISTS learning_classification_key ON learning_classification(repository,operation,tool);`
+
+// The similarity version that marks the repository-wide classification backfill.
+const profileClassificationVersion = 2
+
+var profileOperations = []string{"tests", "browser-tests", "native-tests", "race-tests", "full-tests", "build", "lint", "typecheck", "dependencies", "benchmark", "checks", "other"}
+
+// ProfileOperations is the fixed operation vocabulary, for adapters that
+// validate it without opening the database.
+func ProfileOperations() []string { return append([]string{}, profileOperations...) }
+
+var profileTools = []string{"python", "shell", "javascript", "go", "cargo", "swift", "make", "gmake", "other"}
+
+// A class is shared only when it is verified (repository) and recognized (tool).
+// Everything else is stored for browsing only.
+type profileClass struct{ repository, tool, operation string }
+
+func (k profileClass) valid() bool {
+ return (k.repository == "" || profileHash.MatchString(k.repository)) && containsString(profileTools, k.tool) && containsString(profileOperations, k.operation)
+}
+
+func (k profileClass) shareable() bool {
+ return k.repository != "" && k.tool != "other"
+}
+
+func (k profileClass) kind() string { return k.tool + ":" + k.operation }
+
+func hasProfileClassification(c profileSQL) (bool, error) {
+ row, e := profileRow(c, "SELECT 1 FROM sqlite_master WHERE type='table' AND name='learning_classification'")
+ return row != nil, e
+}
+
+// profileToolOf extends the legacy tool vocabulary for classification only, so
+// the kinds written for old clients keep their meaning.
+// A wrapper is never a tool itself: what it runs decides.
+func profileToolOf(executable string) string {
+ if profileWrapper(executable) {
+ return ""
+ }
+ if tool := profileToolClass(executable); tool != "" {
+ return tool
+ }
+ if strings.HasSuffix(filepath.Base(executable), ".sh") {
+ return "shell"
+ }
+ return ""
+}
+
+func profileWrapper(executable string) bool {
+ return containsString([]string{"env", "fnm"}, filepath.Base(executable))
+}
+
+// profileUnwrap strips simple env and fnm wrappers without interpreting shell
+// programs. It reports false for an fnm invocation it cannot unwrap.
+func profileUnwrap(argv []string) ([]string, bool) {
+ command := argv
+ for len(command) > 0 {
+ switch filepath.Base(command[0]) {
+ case "env":
+ command = command[1:]
+ for len(command) > 0 && (command[0] == "--" || (!strings.HasPrefix(command[0], "-") && strings.Contains(command[0], "="))) {
+ command = command[1:]
+ }
+ case "fnm":
+ if len(command) < 2 || command[1] != "exec" {
+ return command, false
+ }
+ separator := -1
+ for i, arg := range command[2:] {
+ if arg == "--" {
+ separator = i + 2
+ break
+ }
+ }
+ if separator < 0 {
+ return command, false
+ }
+ command = command[separator+1:]
+ default:
+ return command, true
+ }
+ }
+ return command, true
+}
+
+const profileInlineLimit = 8192
+
+var profileInlineSeparators = regexp.MustCompile(`[;|&\n(){}]`)
+var profileInlineAssignment = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*=`)
+
+// profileInlineOperation reads the category of a `sh -c` program and keeps
+// nothing else. It splits on shell separators without honoring quotes, which
+// can only misclassify a command, never expose it. Mixed programs return "".
+func profileInlineOperation(command []string) string {
+ if len(command) < 3 || profileToolClass(command[0]) != "shell" {
+ return ""
+ }
+ program := ""
+ for i := 1; i < len(command)-1; i++ {
+ arg := command[i]
+ if strings.HasPrefix(arg, "--") {
+ continue
+ }
+ if !strings.HasPrefix(arg, "-") {
+ return "" // A script path, not an inline program.
+ }
+ if strings.Contains(arg, "c") {
+ program = command[i+1]
+ break
+ }
+ if strings.HasSuffix(arg, "o") {
+ i++ // -o and clusters like -euo take an option name.
+ }
+ }
+ program = program[:min(len(program), profileInlineLimit)]
+ operation := ""
+ for _, segment := range profileInlineSeparators.Split(program, -1) {
+ tokens := []string{}
+ for _, token := range strings.Fields(segment) {
+ tokens = append(tokens, strings.Trim(token, `'"`))
+ }
+ for len(tokens) > 0 && (containsString([]string{"do", "then", "else", "if", "while", "until", "time", "exec", "!"}, tokens[0]) || profileInlineAssignment.MatchString(tokens[0])) {
+ tokens = tokens[1:]
+ }
+ // The builtin test mentions "test" without testing anything.
+ if len(tokens) == 0 || tokens[0] == "test" {
+ continue
+ }
+ found := ""
+ if kind := profileCommandKind(tokens, "", filepath.Base(tokens[0])); kind != "" {
+ _, found, _ = strings.Cut(kind, ":")
+ } else if unwrapped, ok := profileUnwrap(tokens); ok {
+ summary, _ := SafeSummary(unwrapped)
+ found = profileOperation(summary)
+ }
+ if found == "" {
+ continue
+ }
+ if operation != "" && operation != found {
+ return ""
+ }
+ operation = found
+ }
+ return operation
+}
+
+// profileClassify returns the tool and operation of a new job. Sources in
+// order: the legacy kind (which already covers the safe summary and wrappers),
+// the summary and title for tools the legacy vocabulary skips, then the inline
+// program. Anything mixed or unknown is "other".
+func profileClassify(argv []string, title, executable string) (string, string) {
+ tool, operation := "", ""
+ command, ok := profileUnwrap(argv)
+ if profileWrapper(executable) {
+ // The wrapper's name must not stand in for the tool it runs.
+ executable = ""
+ if ok && len(command) > 0 {
+ executable = command[0]
+ }
+ }
+ if kind := profileCommandKind(argv, title, executable); kind != "" {
+ tool, operation, _ = strings.Cut(kind, ":")
+ } else {
+ if len(command) > 0 {
+ tool = profileToolOf(command[0])
+ }
+ if tool == "" {
+ tool = profileToolOf(executable)
+ }
+ summary, _ := SafeSummary(command)
+ if operation = profileOperation(summary); operation == "" {
+ operation = profileOperation(title)
+ }
+ if operation == "" {
+ operation = profileInlineOperation(command)
+ }
+ }
+ return profileDefault(tool), profileDefault(operation)
+}
+
+func profileDefault(value string) string {
+ if value == "" {
+ return "other"
+ }
+ return value
+}
+
+// A repository digest never includes the relative directory.
+type profileIdentity struct{ scope, repository string }
+
+// profileHistory is what the retroactive pass learns from retained metadata.
+type profileHistory struct {
+ // descriptions holds what registering a family that only has runs needs. A
+ // learning scope and kind appear only when every file of the family agrees,
+ // because they become the pair old clients read from learning_similarity.
+ descriptions map[string]Object
+ // aliases maps a v1 scope to the one repository it belongs to. An empty
+ // value marks a scope seen in two repositories, which is ambiguous.
+ aliases map[string]string
+ // repositories maps a family to the one repository its live checkouts
+ // verify, or "" when they verify two. A family with no entry verified none.
+ repositories map[string]string
+}
+
+func profileAddAlias(aliases map[string]string, scope, repository string) {
+ if scope == "" || repository == "" {
+ return
+ }
+ if prior, seen := aliases[scope]; seen && prior != repository {
+ repository = ""
+ }
+ aliases[scope] = repository
+}
+
+// profileStoredClass reads a stored class. A row that is missing or that the
+// vocabulary no longer accepts is unclassified, so it never merges.
+func profileStoredClass(row Object) (profileClass, bool) {
+ if row == nil || row["tool"] == nil {
+ return profileClass{}, false
+ }
+ class := profileClass{text(row["repository"]), text(row["tool"]), text(row["operation"])}
+ return class, class.valid()
+}
+
+// profileValidLegacy returns a learning_similarity row old clients wrote, or nil
+// when it is absent or holds a scope or kind this version cannot read.
+func profileValidLegacy(row Object) Object {
+ if row == nil || !profileHash.MatchString(text(row["scope"])) || !validProfileSimilarityKind(text(row["kind"])) {
+ return nil
+ }
+ return row
+}
+
+// improvedBy fills what a stored class lacks: a repository, or a tool and
+// operation that were both unknown.
+func (k profileClass) improvedBy(known profileClass) profileClass {
+ if k.tool == "other" && k.operation == "other" {
+ k.tool, k.operation = known.tool, known.operation
+ }
+ if k.repository == "" {
+ k.repository = known.repository
+ }
+ return k
+}
+
+// profileBackfillClass classifies a registered family. Tool and operation come
+// from the member alone, so file order and metadata cannot change them: the
+// legacy kind, then the stored executable and title. The repository is the one
+// the family's checkouts verify, else the one its legacy scope belongs to.
+func profileBackfillClass(member Object, kind, scope string, history *profileHistory) profileClass {
+ family := text(member["family"])
+ tool, operation, ok := strings.Cut(kind, ":")
+ if !ok {
+ tool, operation = profileHistoricalTool(text(member["executable"])), profileOperation(text(member["title"]))
+ }
+ class := profileClass{tool: profileDefault(tool), operation: profileDefault(operation)}
+ if repository, verified := history.repositories[family]; verified {
+ class.repository = repository
+ } else {
+ if scope == "" {
+ scope = text(history.descriptions[family]["learningScope"])
+ }
+ class.repository = history.aliases[scope]
+ }
+ return class
+}
+
+// shortcut: history keeps no command line, so a wrapper counts as its usual runtime and a job it ran under another tool is grouped by that runtime, upgrade if retained metadata ever records the wrapped tool.
+func profileHistoricalTool(executable string) string {
+ switch filepath.Base(executable) {
+ case "env":
+ return "shell"
+ case "fnm":
+ return "javascript"
+ }
+ return profileToolOf(executable)
+}
+
+// profileBackfillClasses returns the class every listed family has or would
+// receive. An existing class wins; a missing repository may be filled in.
+// Preview and apply both call this before apply writes any legacy row.
+func profileBackfillClasses(c profileSQL, history *profileHistory, members []Object) (map[string]profileClass, error) {
+ stored, legacy := map[string]Object{}, map[string]Object{}
+ if has, e := hasProfileClassification(c); e != nil {
+ return nil, e
+ } else if has {
+ rows, e := profileRows(c, "SELECT * FROM learning_classification")
+ if e != nil {
+ return nil, e
+ }
+ for _, row := range rows {
+ stored[text(row["family"])] = row
+ }
+ }
+ if has, e := hasProfileSimilarity(c); e != nil {
+ return nil, e
+ } else if has {
+ rows, e := profileRows(c, "SELECT * FROM learning_similarity")
+ if e != nil {
+ return nil, e
+ }
+ for _, row := range rows {
+ legacy[text(row["family"])] = profileValidLegacy(row)
+ }
+ }
+ result := map[string]profileClass{}
+ for _, member := range members {
+ family := text(member["family"])
+ old := legacy[family]
+ class := profileBackfillClass(member, text(old["kind"]), text(old["scope"]), history)
+ if prior, ok := profileStoredClass(stored[family]); ok {
+ class = prior.improvedBy(class)
+ }
+ result[family] = class
+ }
+ return result, nil
+}
+
+// profileStoreClass keeps what is stored except for the two fills improvedBy
+// describes, so the SQL below must change with it.
+func profileStoreClass(c profileSQL, family string, class profileClass) error {
+ if !class.valid() {
+ return fmt.Errorf("Invalid workload classification")
+ }
+ var repository any
+ if class.repository != "" {
+ repository = class.repository
+ }
+ if e := profileExec(c, "INSERT OR IGNORE INTO learning_classification VALUES (?,?,?,?)", family, repository, class.tool, class.operation); e != nil {
+ return e
+ }
+ if class.tool != "other" || class.operation != "other" {
+ if e := profileExec(c, "UPDATE learning_classification SET tool=?,operation=? WHERE family=? AND tool='other' AND operation='other'", class.tool, class.operation, family); e != nil {
+ return e
+ }
+ }
+ if repository == nil {
+ return nil
+ }
+ return profileExec(c, "UPDATE learning_classification SET repository=? WHERE family=? AND repository IS NULL", repository, family)
+}
+
+type profileMemberClass struct {
+ family, profile, project string
+ pinned, classified bool
+ class profileClass
+}
+
+type profileMerge struct {
+ class profileClass
+ project string
+ target string
+ sources []string
+}
+
+// profilePlanMerges is the single grouping rule. Only unpinned profiles whose
+// members all share one shareable class merge, into the lowest profile id.
+// It never splits a profile.
+func profilePlanMerges(members []profileMemberClass) []profileMerge {
+ type candidate struct {
+ class profileClass
+ project string
+ ok bool
+ }
+ profiles := map[string]*candidate{}
+ sort.Slice(members, func(i, j int) bool {
+ if members[i].profile != members[j].profile {
+ return members[i].profile < members[j].profile
+ }
+ return members[i].family < members[j].family
+ })
+ for _, member := range members {
+ entry := profiles[member.profile]
+ if entry == nil {
+ entry = &candidate{class: member.class, project: member.project, ok: !member.pinned}
+ profiles[member.profile] = entry
+ }
+ entry.ok = entry.ok && member.classified && member.class.shareable() && member.class == entry.class
+ }
+ byClass := map[profileClass][]string{}
+ for id, entry := range profiles {
+ if entry.ok {
+ byClass[entry.class] = append(byClass[entry.class], id)
+ }
+ }
+ merges := []profileMerge{}
+ for class, ids := range byClass {
+ if len(ids) < 2 {
+ continue
+ }
+ sort.Strings(ids)
+ merges = append(merges, profileMerge{class: class, project: profiles[ids[0]].project, target: ids[0], sources: ids[1:]})
+ }
+ sort.Slice(merges, func(i, j int) bool { return merges[i].target < merges[j].target })
+ return merges
+}
+
+func profileApplyMerges(c profileSQL, merges []profileMerge) (bool, error) {
+ for _, merge := range merges {
+ for _, source := range merge.sources {
+ if e := profileExec(c, "UPDATE learning_members SET profile=? WHERE profile=?", merge.target, source); e != nil {
+ return false, e
+ }
+ if e := profileExec(c, "DELETE FROM learning_profiles WHERE id=? AND NOT EXISTS(SELECT 1 FROM learning_members WHERE profile=?)", source, source); e != nil {
+ return false, e
+ }
+ }
+ if e := profileExec(c, "UPDATE learning_profiles SET label=?,updated=? WHERE id=? AND pinned=0", profileSimilarityLabel(merge.class.kind(), merge.project), resourceNow(), merge.target); e != nil {
+ return false, e
+ }
+ }
+ return len(merges) > 0, nil
+}
+
+// profileGroupSimilar groups by the stored classification. A non-nil class
+// limits the work to profiles holding a family of that class, which is all a
+// single new observation can affect.
+func profileGroupSimilar(c profileSQL, only *profileClass) (bool, error) {
+ has, e := hasProfileClassification(c)
+ if e != nil || !has {
+ return false, e
+ }
+ query := "SELECT m.family,m.profile,m.project,p.pinned,k.repository,k.tool,k.operation FROM learning_members m JOIN learning_profiles p ON p.id=m.profile LEFT JOIN learning_classification k ON k.family=m.family"
+ args := []any{}
+ if only != nil {
+ query += " WHERE m.profile IN (SELECT m2.profile FROM learning_members m2 JOIN learning_classification k2 ON k2.family=m2.family WHERE k2.repository=? AND k2.tool=? AND k2.operation=?)"
+ args = append(args, only.repository, only.tool, only.operation)
+ }
+ rows, e := profileRows(c, query, args...)
+ if e != nil {
+ return false, e
+ }
+ members := make([]profileMemberClass, 0, len(rows))
+ for _, row := range rows {
+ class, ok := profileStoredClass(row)
+ members = append(members, profileMemberClass{family: text(row["family"]), profile: text(row["profile"]), project: text(row["project"]), pinned: metric(row["pinned"]) != 0, classified: ok, class: class})
+ }
+ return profileApplyMerges(c, profilePlanMerges(members))
+}
diff --git a/internal/backend/profiles_classification_test.go b/internal/backend/profiles_classification_test.go
new file mode 100644
index 0000000..25b9627
--- /dev/null
+++ b/internal/backend/profiles_classification_test.go
@@ -0,0 +1,1887 @@
+package backend
+
+import (
+ "bytes"
+ "encoding/json"
+ "fmt"
+ "math/rand"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "reflect"
+ "regexp"
+ "strings"
+ "testing"
+ "time"
+)
+
+func runGit(t *testing.T, directory string, args ...string) {
+ t.Helper()
+ cmd := exec.Command("git", append([]string{"-C", directory, "-c", "user.name=Fixture", "-c", "user.email=fixture@example.invalid", "-c", "commit.gpgsign=false"}, args...)...)
+ cmd.Env = []string{"PATH=" + os.Getenv("PATH"), "HOME=" + t.TempDir(), "GIT_CONFIG_GLOBAL=/dev/null", "GIT_CONFIG_SYSTEM=/dev/null"}
+ if out, e := cmd.CombinedOutput(); e != nil {
+ t.Fatalf("git %v: %v\n%s", args, e, out)
+ }
+}
+
+// gitRepository creates a private repository with one commit and a subdirectory.
+func gitRepository(t *testing.T) string {
+ t.Helper()
+ root, e := filepath.EvalSymlinks(t.TempDir())
+ if e != nil {
+ t.Fatal(e)
+ }
+ runGit(t, root, "init", "-q")
+ runGit(t, root, "commit", "-q", "--allow-empty", "-m", "fixture")
+ if e = os.MkdirAll(filepath.Join(root, "sub", "deep"), 0700); e != nil {
+ t.Fatal(e)
+ }
+ return root
+}
+
+// observeIn observes a family the way the controller does: from metadata that
+// names only a directory.
+func observeIn(t *testing.T, store *ProfileStore, name, directory, title, executable string) Object {
+ t.Helper()
+ identity := similarityIdentity(name)
+ description := profileSimilarityDescription(Object{"title": title, "project": filepath.Base(directory), "executable": executable, "directory": directory}, map[string]profileIdentity{})
+ if e := store.Observe(identity, description); e != nil {
+ t.Fatal(e)
+ }
+ return identity
+}
+
+func matchID(t *testing.T, store *ProfileStore, identity Object) string {
+ t.Helper()
+ match, e := store.Match(identity)
+ if e != nil {
+ t.Fatal(e)
+ }
+ return text(match["id"])
+}
+
+func storedClass(t *testing.T, store *ProfileStore, identity Object) (profileClass, bool) {
+ t.Helper()
+ var class profileClass
+ var found bool
+ if e := store.connection(false, func(c profileSQL) error {
+ row, e := profileRow(c, "SELECT * FROM learning_classification WHERE family=?", identity["family"])
+ class, found = profileStoredClass(row)
+ return e
+ }); e != nil {
+ t.Fatal(e)
+ }
+ return class, found
+}
+
+// observeClassified observes a family the way a submission does: with the
+// repository, tool and operation already known.
+func observeClassified(t *testing.T, store *ProfileStore, name, project, title, executable, repository string) Object {
+ t.Helper()
+ identity := similarityIdentity(name)
+ description := Object{"title": title, "executable": executable, "learningTool": profileDefault(profileToolOf(executable)), "learningOperation": profileDefault(profileOperation(title))}
+ if project != "" {
+ description["project"] = project
+ }
+ if repository != "" {
+ description["learningRepository"] = profileDigest(repository)
+ }
+ if e := store.Observe(identity, description); e != nil {
+ t.Fatal(e)
+ }
+ return identity
+}
+
+func familyHasLegacyRow(t *testing.T, store *ProfileStore, identity Object) bool {
+ t.Helper()
+ found := false
+ if e := store.connection(false, func(c profileSQL) error {
+ row, e := profileRow(c, "SELECT 1 FROM learning_similarity WHERE family=?", identity["family"])
+ found = row != nil
+ return e
+ }); e != nil {
+ t.Fatal(e)
+ }
+ return found
+}
+
+func TestProfileClassifyTable(t *testing.T) {
+ const sentinel = "SENTINEL_SECRET_TOKEN"
+ for _, entry := range []struct {
+ argv []string
+ title string
+ tool, operation string
+ }{
+ {[]string{"go", "test", "./internal/first"}, "Go tests", "go", "tests"},
+ {[]string{"go", "test", "-race", "./..."}, "Tests", "go", "race-tests"},
+ {[]string{"env", "GOMAXPROCS=2", "go", "test", "./x"}, "Anything", "go", "tests"},
+ {[]string{"fnm", "exec", "--using=26", "--", "pnpm", "test"}, "Anything", "javascript", "tests"},
+ {[]string{"./scripts/check-all.sh"}, "check-all.sh", "shell", "checks"},
+ {[]string{"/repo/run-build.sh"}, "Build", "shell", "build"},
+ {[]string{"env", "FOO=1", "./tool"}, "Lint", "other", "lint"},
+ {[]string{"env", "FOO=1", "mystery-tool"}, "Lint", "other", "lint"},
+ {[]string{"env", "X=1", "xcodebuild", "build"}, "Build", "other", "build"},
+ {[]string{"env", "X=1", "go", "test", "./x"}, "Tests", "go", "tests"},
+ {[]string{"fnm", "exec", "--", "mystery-tool"}, "Tests", "other", "tests"},
+ {[]string{"fnm", "exec", "--", "node", "x"}, "Tests", "javascript", "tests"},
+ {[]string{"fnm", "use", "20"}, "Tests", "other", "tests"},
+ {[]string{"bash", "/private/batch.sh"}, "Issue 19 browser regression tests", "shell", "browser-tests"},
+ {[]string{"xcodebuild", "build"}, "Build", "other", "build"},
+ {[]string{"mystery"}, "Unclassified batch 123", "other", "other"},
+ {[]string{"bash", "-c", "go test ./internal/a"}, "Shell batch", "shell", "tests"},
+ {[]string{"sh", "-lc", "cd web && " + sentinel + "=1 npm run lint"}, "Shell batch", "shell", "lint"},
+ {[]string{"bash", "-ec", "go test -race ./... ; go test -race ./..."}, "Shell batch", "shell", "race-tests"},
+ {[]string{"bash", "-c", "go test ./... && go vet ./..."}, "Shell batch", "shell", "other"},
+ {[]string{"bash", "-euo", "pipefail", "-c", "go test ./internal/a"}, "Shell batch", "shell", "tests"},
+ {[]string{"bash", "-o", "pipefail", "-c", "go build ./..."}, "Shell batch", "shell", "build"},
+ {[]string{"bash", "-eu", "-o", "pipefail", "-c", "go build ./..."}, "Shell batch", "shell", "build"},
+ {[]string{"bash", "-c", "test -f foo && echo yes"}, "Shell batch", "shell", "other"},
+ {[]string{"bash", "-c", "[ -f test ] && echo yes"}, "Shell batch", "shell", "other"},
+ {[]string{"bash", "-c", "test -f .env && go build ./..."}, "Shell batch", "shell", "build"},
+ {[]string{"bash", "-c", "echo " + sentinel}, "Shell batch", "shell", "other"},
+ {[]string{"bash", "script.sh", "-c", "go test"}, "Shell batch", "shell", "other"},
+ {[]string{"bash", "-c", "opaque"}, "Build and test all modules", "shell", "other"},
+ {[]string{"bash", "-c", "for d in a b; do go test ./$d; done"}, "Shell batch", "shell", "tests"},
+ {[]string{"python3.14", "-c", "opaque"}, "Issue 20 regression tests", "python", "tests"},
+ } {
+ _, executable := SafeSummary(entry.argv)
+ tool, operation := profileClassify(entry.argv, entry.title, executable)
+ if tool != entry.tool || operation != entry.operation {
+ t.Errorf("%v / %s: got %s %s, want %s %s", entry.argv, entry.title, tool, operation, entry.tool, entry.operation)
+ }
+ if !(profileClass{"", tool, operation}).valid() {
+ t.Errorf("%v: classification %s %s is outside the vocabulary", entry.argv, tool, operation)
+ }
+ encoded, _ := json.Marshal(Object{"tool": tool, "operation": operation})
+ if strings.Contains(string(encoded), sentinel) {
+ t.Errorf("%v: arguments reached the classification", entry.argv)
+ }
+ }
+}
+
+func TestProfileInlineProgramIsBounded(t *testing.T) {
+ padding := strings.Repeat("echo padding; ", profileInlineLimit)
+ if got := profileInlineOperation([]string{"bash", "-c", padding + "go test ./..."}); got != "" {
+ t.Fatalf("A command beyond the scan bound was classified: %q", got)
+ }
+ if got := profileInlineOperation([]string{"bash", "-c", "go test ./a; " + padding}); got != "tests" {
+ t.Fatalf("The bounded prefix was not scanned: %q", got)
+ }
+}
+
+func TestProfileVocabularyMatchesLegacyKinds(t *testing.T) {
+ for _, tool := range profileTools {
+ for _, operation := range profileOperations {
+ legal := tool != "other" && operation != "other"
+ if validProfileSimilarityKind(tool+":"+operation) != legal {
+ t.Errorf("%s:%s: legacy validity disagrees with the classification vocabulary", tool, operation)
+ }
+ }
+ }
+}
+
+func TestProfileClassificationNeverStoresRawArguments(t *testing.T) {
+ const sentinel = "SENTINEL_SECRET_TOKEN"
+ g := testGate(t)
+ argv := []string{"bash", "-c", "API_KEY=" + sentinel + " go test ./..."}
+ description, e := MakeDescription(g, argv, t.TempDir(), "")
+ if e != nil {
+ t.Fatal(e)
+ }
+ encoded, _ := json.Marshal(description)
+ if strings.Contains(string(encoded), sentinel) || description["learningOperation"] != "full-tests" || description["learningTool"] != "shell" {
+ t.Fatal("The description leaked the program or missed its operation", description)
+ }
+ store, e := NewProfileStore(g.State, false)
+ if e != nil {
+ t.Fatal(e)
+ }
+ identity := similarityIdentity("secret")
+ if e = store.Observe(identity, profileSimilarityDescription(description, map[string]profileIdentity{})); e != nil {
+ t.Fatal(e)
+ }
+ database, _ := os.ReadFile(store.path)
+ if bytes.Contains(database, []byte(sentinel)) {
+ t.Fatal("The database retained inline program text")
+ }
+ if class, ok := storedClass(t, store, identity); !ok || class.operation != "full-tests" || class.tool != "shell" {
+ t.Fatal("The inline operation was not stored", class)
+ }
+}
+
+func TestProfileDescriptionSurvivesTaskDescription(t *testing.T) {
+ g := testGate(t)
+ description, e := MakeDescription(g, []string{"bash", "-c", "go test ./..."}, t.TempDir(), "")
+ if e != nil {
+ t.Fatal(e)
+ }
+ token := strings.Repeat("a", 32)
+ if e = CacheDescription(g, token, description); e != nil {
+ t.Fatal(e)
+ }
+ g.DescriptionCache = map[string]Object{}
+ described, e := DescribeTask(g, Object{"token": token}, "running")
+ if e != nil {
+ t.Fatal(e)
+ }
+ if _, leaked := described["learningOperation"]; leaked {
+ t.Fatal("The status description exposes the classification")
+ }
+ stored, e := ReadPrivate(filepath.Join(g.Metadata, token+".json"))
+ if e != nil || stored["learningOperation"] != "full-tests" || stored["learningTool"] != "shell" {
+ t.Fatal("A status poll discarded the submission-time classification", stored, e)
+ }
+}
+
+func TestProfileRepositoryIdentityIgnoresDirectoryAndWorktree(t *testing.T) {
+ repository := gitRepository(t)
+ worktree := filepath.Join(t.TempDir(), "linked")
+ runGit(t, repository, "worktree", "add", "-q", worktree, "-b", "linked")
+ worktree, _ = filepath.EvalSymlinks(worktree)
+ scopeRoot, repositoryRoot := profileDirectoryIdentity(repository)
+ scopeSub, repositorySub := profileDirectoryIdentity(filepath.Join(repository, "sub"))
+ scopeLinked, repositoryLinked := profileDirectoryIdentity(worktree)
+ if repositoryRoot == "" || repositoryRoot != repositorySub || repositoryRoot != repositoryLinked {
+ t.Fatal("Subdirectories or linked worktrees changed the repository", repositoryRoot, repositorySub, repositoryLinked)
+ }
+ if scopeRoot == scopeSub || scopeRoot != scopeLinked || scopeRoot != profileDirectoryScope(repository) {
+ t.Fatal("The v1 scope changed", scopeRoot, scopeSub, scopeLinked)
+ }
+ if _, repositoryOther := profileDirectoryIdentity(gitRepository(t)); repositoryOther == repositoryRoot {
+ t.Fatal("Two repositories share an identity")
+ }
+ if scope, repository := profileDirectoryIdentity(t.TempDir()); scope == "" || repository != "" {
+ t.Fatal("A plain directory has a repository", scope, repository)
+ }
+ if scope, repository := profileDirectoryIdentity("relative"); scope != "" || repository != "" {
+ t.Fatal("A relative path was identified")
+ }
+}
+
+func TestProfileGroupsSubdirectoriesAndSeparatesEverythingElse(t *testing.T) {
+ store := fixtureProfileStore(t)
+ repository, other := gitRepository(t), gitRepository(t)
+ plain := t.TempDir()
+ root := observeIn(t, store, "root", repository, "Go tests", "go")
+ sub := observeIn(t, store, "sub", filepath.Join(repository, "sub"), "Go tests", "go")
+ deep := observeIn(t, store, "deep", filepath.Join(repository, "sub", "deep"), "Go tests", "go")
+ if matchID(t, store, root) != matchID(t, store, sub) || matchID(t, store, root) != matchID(t, store, deep) {
+ t.Fatal("Subdirectories of one repository do not share a group")
+ }
+ apart := map[string]Object{
+ "another repository": observeIn(t, store, "other repository", other, "Go tests", "go"),
+ "another operation": observeIn(t, store, "build", repository, "Go build", "go"),
+ "another tool": observeIn(t, store, "pnpm", repository, "Tests", "pnpm"),
+ "not a repository": observeIn(t, store, "plain", plain, "Go tests", "go"),
+ "second plain": observeIn(t, store, "plain two", plain, "Go tests", "go"),
+ "unrecognized tool": observeIn(t, store, "xcode", repository, "Tests", "xcodebuild"),
+ "second unrecognized": observeIn(t, store, "xcode two", repository, "Tests", "xcodebuild"),
+ "other operation": observeIn(t, store, "other one", repository, "Mystery batch", "go"),
+ "other in a repo": observeIn(t, store, "other elsewhere", other, "Mystery batch", "go"),
+ }
+ seen := map[string]string{matchID(t, store, root): "grouped"}
+ for name, identity := range apart {
+ id := matchID(t, store, identity)
+ if prior, duplicate := seen[id]; duplicate {
+ t.Errorf("%s shares a group with %s", name, prior)
+ }
+ seen[id] = name
+ }
+ if class, ok := storedClass(t, store, apart["not a repository"]); !ok || class.repository != "" || class.tool != "go" || class.operation != "tests" {
+ t.Fatal("A plain directory family was not classified for display", class, ok)
+ }
+ if class, ok := storedClass(t, store, apart["unrecognized tool"]); !ok || class.tool != "other" || class.operation != "tests" {
+ t.Fatal("An unrecognized tool was not stored", class, ok)
+ }
+ if class, _ := storedClass(t, store, apart["other operation"]); class.operation != "other" {
+ t.Fatal("An unrecognized operation was not stored as other", class)
+ }
+ if second := observeIn(t, store, "other two", repository, "Mystery batch", "go"); matchID(t, store, second) != matchID(t, store, apart["other operation"]) {
+ t.Fatal("Workloads with an unknown operation do not share within a repository and tool")
+ }
+}
+
+func TestProfileRepositoryFillsLaterButNeverChanges(t *testing.T) {
+ store := fixtureProfileStore(t)
+ repository, other := gitRepository(t), gitRepository(t)
+ identity := observeIn(t, store, "first", t.TempDir(), "Go tests", "go")
+ if class, ok := storedClass(t, store, identity); !ok || class.repository != "" {
+ t.Fatal("Unverified family has a repository", class)
+ }
+ description := profileSimilarityDescription(Object{"title": "Go build", "project": "p", "executable": "go", "directory": repository}, map[string]profileIdentity{})
+ if e := store.Observe(identity, description); e != nil {
+ t.Fatal(e)
+ }
+ class, _ := storedClass(t, store, identity)
+ if class.repository == "" || class.operation != "tests" {
+ t.Fatal("A later observation did not fill the repository only", class)
+ }
+ filled := class.repository
+ description = profileSimilarityDescription(Object{"title": "Go tests", "project": "p", "executable": "go", "directory": other}, map[string]profileIdentity{})
+ if e := store.Observe(identity, description); e != nil {
+ t.Fatal(e)
+ }
+ if class, _ = storedClass(t, store, identity); class.repository != filled {
+ t.Fatal("An observed repository was overwritten")
+ }
+}
+
+func TestProfileOperatorCorrectionsSurviveRepositoryGrouping(t *testing.T) {
+ store := fixtureProfileStore(t)
+ repository := gitRepository(t)
+ a := observeIn(t, store, "a", repository, "Go tests", "go")
+ b := observeIn(t, store, "b", filepath.Join(repository, "sub"), "Go tests", "go")
+ revision, _ := store.Revision()
+ if _, e := store.Split(matchID(t, store, a), text(a["family"]), revision); e != nil {
+ t.Fatal(e)
+ }
+ c := observeIn(t, store, "c", filepath.Join(repository, "sub", "deep"), "Go tests", "go")
+ if matchID(t, store, a) == matchID(t, store, b) || matchID(t, store, a) == matchID(t, store, c) {
+ t.Fatal("Automatic grouping reopened an operator split")
+ }
+ // A pinned profile is neither a source nor a target of a later merge.
+ pinned := matchID(t, store, a)
+ revision, _ = store.Revision()
+ if e := store.Edit(pinned, revision, "Mine", 4*GiB, nil); e != nil {
+ t.Fatal(e)
+ }
+ d := observeIn(t, store, "d", repository, "Go tests", "go")
+ if matchID(t, store, d) == pinned || matchID(t, store, d) == matchID(t, store, b) || matchID(t, store, d) != matchID(t, store, c) {
+ t.Fatal("A pinned profile joined an automatic group", matchID(t, store, d))
+ }
+}
+
+func TestProfileCompatibleWithOldClients(t *testing.T) {
+ store := fixtureProfileStore(t)
+ repository := gitRepository(t)
+ observeIn(t, store, "tests", repository, "Go tests", "go")
+ observeIn(t, store, "other", repository, "Mystery batch", "go")
+ observeIn(t, store, "unrecognized", repository, "Tests", "xcodebuild")
+ columns := map[string]int{"learning_state": 2, "learning_profiles": 7, "learning_members": 6, "learning_resets": 2, "learning_retained": 7, "learning_similarity": 3, "learning_similarity_state": 2}
+ if e := store.connection(true, func(c profileSQL) error {
+ for table, want := range columns {
+ rows, e := profileRows(c, "PRAGMA table_info("+table+")")
+ if e != nil || len(rows) != want {
+ t.Errorf("%s changed shape: %d columns, want %d (%v)", table, len(rows), want, e)
+ }
+ }
+ rows, e := profileRows(c, "SELECT * FROM learning_similarity")
+ if e != nil {
+ return e
+ }
+ for _, row := range rows {
+ if !profileHash.MatchString(text(row["scope"])) || !validProfileSimilarityKind(text(row["kind"])) {
+ t.Errorf("An old client would reject %v", row)
+ }
+ }
+ other := similarityIdentity("other")
+ if row, e := profileRow(c, "SELECT 1 FROM learning_similarity WHERE family=?", other["family"]); e != nil || row != nil {
+ t.Error("A family classified other has a similarity row", e)
+ }
+ // Old clients insert exactly three values and group with the old plan.
+ if e = profileExec(c, "INSERT OR IGNORE INTO learning_similarity VALUES (?,?,?)", profileDigest("old family"), profileDigest("old scope"), "go:tests"); e != nil {
+ return e
+ }
+ rows, e = profileRows(c, "SELECT * FROM learning_similarity")
+ for _, row := range rows {
+ if !validProfileSimilarityKind(text(row["kind"])) {
+ t.Errorf("An invalid kind appeared: %v", row)
+ }
+ }
+ return e
+ }); e != nil {
+ t.Fatal(e)
+ }
+}
+
+func TestProfileOldClientQueuePlanLoadsNewDatabase(t *testing.T) {
+ g := testGate(t)
+ store, e := NewProfileStore(g.State, false)
+ if e != nil {
+ t.Fatal(e)
+ }
+ repository := gitRepository(t)
+ for _, entry := range []struct{ name, title, executable string }{{"a", "Go tests", "go"}, {"b", "Mystery batch", "go"}, {"c", "Tests", "xcodebuild"}} {
+ observeIn(t, store, entry.name, repository, entry.title, entry.executable)
+ }
+ if _, e = LearningRegroup(g, false, nil); e != nil {
+ t.Fatal("The preview rejected a database holding new rows", e)
+ }
+ if e = store.connection(false, func(c profileSQL) error {
+ rows, e := profileRows(c, "SELECT kind FROM learning_similarity")
+ for _, row := range rows {
+ if !validProfileSimilarityKind(text(row["kind"])) {
+ t.Errorf("Old clients would reject %v", row)
+ }
+ }
+ return e
+ }); e != nil {
+ t.Fatal(e)
+ }
+}
+
+func TestProfileRegroupBackfillAliasesDeletedWorktreesAndKeepsAmbiguousUnknown(t *testing.T) {
+ g := testGate(t)
+ store, e := NewProfileStore(g.State, false)
+ if e != nil {
+ t.Fatal(e)
+ }
+ repository := gitRepository(t)
+ worktree := filepath.Join(t.TempDir(), "linked")
+ runGit(t, repository, "worktree", "add", "-q", worktree, "-b", "linked")
+ worktree, _ = filepath.EvalSymlinks(worktree)
+ if e = os.MkdirAll(filepath.Join(worktree, "sub"), 0700); e != nil {
+ t.Fatal(e)
+ }
+ liveScope, liveRepository := profileDirectoryIdentity(filepath.Join(repository, "sub"))
+ deletedScope, _ := profileDirectoryIdentity(filepath.Join(worktree, "sub"))
+ if liveScope != deletedScope {
+ t.Fatal("Fixture: the worktree should share its relative scope")
+ }
+ // Three old families: the live checkout, a deleted worktree sharing its
+ // scope, and one in a directory that never verified.
+ live, retired, stranger := similarityIdentity("live"), similarityIdentity("retired"), similarityIdentity("stranger")
+ for i, identity := range []Object{live, retired, stranger} {
+ if e = store.Observe(identity, Object{"title": "Go regression tests", "project": "fixture", "executable": "go"}); e != nil {
+ t.Fatal(e)
+ }
+ recordProfile(t, store, fmt.Sprintf("%032x", i+1), identity, GiB, 1, true, profileZero(), 0)
+ }
+ if e = store.connection(true, func(c profileSQL) error {
+ for _, entry := range []struct {
+ identity Object
+ scope string
+ }{{live, liveScope}, {retired, deletedScope}, {stranger, profileDigest("elsewhere")}} {
+ if e := profileExec(c, "INSERT INTO learning_similarity VALUES (?,?,?)", entry.identity["family"], entry.scope, "go:tests"); e != nil {
+ return e
+ }
+ }
+ return nil
+ }); e != nil {
+ t.Fatal(e)
+ }
+ mustWrite(t, filepath.Join(g.Metadata, fmt.Sprintf("%032x.json", 1)), Object{"title": "Go regression tests", "project": "fixture", "executable": "go", "directory": filepath.Join(repository, "sub")})
+ runGit(t, repository, "worktree", "remove", "--force", worktree)
+ preview, e := LearningRegroup(g, false, nil)
+ if e != nil || metric(preview["profilesBefore"]) != 3 || metric(preview["profilesAfter"]) != 2 || metric(preview["familiesGrouped"]) != 2 {
+ t.Fatal("The preview missed the deleted worktree", preview, e)
+ }
+ saved, e := LearningRegroup(g, true, Object{"revision": preview["revision"]})
+ if e != nil {
+ t.Fatal(e)
+ }
+ catalog, _ := store.Catalog(0, 100, "")
+ if metric(catalog["total"]) != metric(preview["profilesAfter"]) || metric(saved["revision"]) != metric(preview["revision"])+1 {
+ t.Fatal("Preview and apply disagree", preview, catalog)
+ }
+ for identity, want := range map[string]string{text(live["family"]): liveRepository, text(retired["family"]): liveRepository, text(stranger["family"]): ""} {
+ class, ok := storedClass(t, store, Object{"family": identity})
+ if !ok || class.repository != want || class.tool != "go" || class.operation != "tests" {
+ t.Errorf("%s: classified %+v", identity[:8], class)
+ }
+ }
+ if matchID(t, store, live) != matchID(t, store, retired) || matchID(t, store, live) == matchID(t, store, stranger) {
+ t.Fatal("Aliased families did not group, or the stranger joined")
+ }
+ if e := store.connection(false, func(c profileSQL) error {
+ row, e := profileRow(c, "SELECT COUNT(*) AS n FROM learning_members WHERE family NOT IN (SELECT family FROM learning_classification)")
+ if e == nil && metric(row["n"]) != 0 {
+ t.Errorf("%v members have no classification", row["n"])
+ }
+ return e
+ }); e != nil {
+ t.Fatal(e)
+ }
+}
+
+// A family's stored scope can differ from the one its checkout yields today, and
+// running in two subdirectories of one repository is not a disagreement.
+func TestProfileRegroupLearnsRepositoryFromStoredScopeAndSubdirectories(t *testing.T) {
+ g := testGate(t)
+ store, e := NewProfileStore(g.State, false)
+ if e != nil {
+ t.Fatal(e)
+ }
+ repository := gitRepository(t)
+ if e = os.MkdirAll(filepath.Join(repository, "sub"), 0700); e != nil {
+ t.Fatal(e)
+ }
+ _, want := profileDirectoryIdentity(repository)
+ seed, retired, roaming := similarityIdentity("seed"), similarityIdentity("retired"), similarityIdentity("roaming")
+ description := Object{"title": "Go tests", "project": "fixture", "executable": "go"}
+ stale := profileDigest("a scope no checkout yields today")
+ for i, identity := range []Object{seed, retired, roaming} {
+ if e = store.Observe(identity, description); e != nil {
+ t.Fatal(e)
+ }
+ recordProfile(t, store, fmt.Sprintf("%032x", i+1), identity, GiB, 1, true, profileZero(), 0)
+ }
+ recordProfile(t, store, fmt.Sprintf("%032x", 4), roaming, GiB, 1, true, profileZero(), 0)
+ recordProfile(t, store, fmt.Sprintf("%032x", 5), roaming, GiB, 1, true, profileZero(), 0)
+ if e = store.connection(true, func(c profileSQL) error {
+ for _, identity := range []Object{seed, retired} {
+ if e := profileExec(c, "INSERT INTO learning_similarity VALUES (?,?,?)", identity["family"], stale, "go:tests"); e != nil {
+ return e
+ }
+ }
+ return nil
+ }); e != nil {
+ t.Fatal(e)
+ }
+ for token, directory := range map[int]string{1: repository, 3: repository, 4: filepath.Join(repository, "sub"), 5: repository} {
+ mustWrite(t, filepath.Join(g.Metadata, fmt.Sprintf("%032x.json", token)), Object{"title": "Go tests", "project": "fixture", "executable": "go", "directory": directory})
+ }
+ if _, e = LearningRegroup(g, true, Object{"revision": 0}); e != nil {
+ t.Fatal(e)
+ }
+ for name, identity := range map[string]Object{"seed": seed, "retired": retired, "roaming": roaming} {
+ if class, ok := storedClass(t, store, identity); !ok || class.repository != want {
+ t.Errorf("%s: classified %+v", name, class)
+ }
+ }
+ if matchID(t, store, seed) != matchID(t, store, retired) || matchID(t, store, seed) != matchID(t, store, roaming) {
+ t.Fatal("Families of one repository did not share a group")
+ }
+ // Two directories are not a conflict, but they name no single legacy scope,
+ // and a later file agreeing with the first cannot bring one back.
+ if familyHasLegacyRow(t, store, roaming) {
+ t.Fatal("A family seen in two directories received a legacy scope")
+ }
+}
+
+func TestProfileAliasAmbiguityAndConflicts(t *testing.T) {
+ aliases := map[string]string{}
+ profileAddAlias(aliases, "scope", "repository-a")
+ profileAddAlias(aliases, "scope", "repository-a")
+ if aliases["scope"] != "repository-a" {
+ t.Fatal("A consistent alias was lost")
+ }
+ profileAddAlias(aliases, "scope", "repository-b")
+ profileAddAlias(aliases, "scope", "repository-a")
+ if aliases["scope"] != "" {
+ t.Fatal("An ambiguous alias resolved to a repository")
+ }
+ profileAddAlias(aliases, "", "repository-a")
+ profileAddAlias(aliases, "plain", "")
+ if _, ok := aliases[""]; ok || len(aliases) != 1 {
+ t.Fatal("An unverified directory taught an alias", aliases)
+ }
+ history := &profileHistory{descriptions: map[string]Object{}, aliases: map[string]string{"known": "repo", "doubtful": ""}, repositories: map[string]string{"moved": "repo", "both": ""}}
+ member := func(family string) Object { return Object{"family": family, "title": "Go tests", "executable": "go"} }
+ for _, entry := range []struct {
+ family, scope, want string
+ }{{"a", "known", "repo"}, {"b", "doubtful", ""}, {"c", "unseen", ""}, {"moved", "unseen", "repo"}, {"both", "doubtful", ""}} {
+ class := profileBackfillClass(member(entry.family), "go:tests", entry.scope, history)
+ if class.repository != entry.want || class.tool != "go" || class.operation != "tests" {
+ t.Errorf("%s: %+v", entry.family, class)
+ }
+ }
+ // Operation: the legacy kind, then the title, then the executable's tool.
+ if class := profileBackfillClass(Object{"family": "x", "title": "Build", "executable": "make"}, "go:lint", "", history); class.tool != "go" || class.operation != "lint" {
+ t.Error("The legacy kind did not win", class)
+ }
+ if class := profileBackfillClass(Object{"family": "x", "title": "Build", "executable": "make"}, "", "", history); class.tool != "make" || class.operation != "build" {
+ t.Error("The title and executable were ignored", class)
+ }
+ if class := profileBackfillClass(Object{"family": "x", "title": "Shell batch", "executable": "bash"}, "", "", history); class.tool != "shell" || class.operation != "other" {
+ t.Error("Only the member decides tool and operation", class)
+ }
+}
+
+func TestProfileMetadataNamingTwoRepositoriesLeavesFamilyUnshared(t *testing.T) {
+ g := testGate(t)
+ store, e := NewProfileStore(g.State, false)
+ if e != nil {
+ t.Fatal(e)
+ }
+ first, second := gitRepository(t), gitRepository(t)
+ peer := observeIn(t, store, "peer", first, "Go tests", "go")
+ identity := similarityIdentity("moved")
+ if e = store.Observe(identity, Object{"title": "Go tests", "project": "fixture", "executable": "go"}); e != nil {
+ t.Fatal(e)
+ }
+ // The third file agrees with the first; one other repository still decides.
+ for i, directory := range []string{first, second, first} {
+ token := fmt.Sprintf("%032x", i+1)
+ recordProfile(t, store, token, identity, GiB, 1, true, profileZero(), 0)
+ mustWrite(t, filepath.Join(g.Metadata, token+".json"), Object{"title": "Go tests", "project": "fixture", "executable": "go", "directory": directory})
+ }
+ if _, e = LearningRegroup(g, true, Object{"revision": 0}); e != nil {
+ t.Fatal(e)
+ }
+ if class, ok := storedClass(t, store, identity); !ok || class.repository != "" || class.operation != "tests" {
+ t.Fatal("Metadata naming two repositories produced a repository", class, ok)
+ }
+ if matchID(t, store, identity) == matchID(t, store, peer) {
+ t.Fatal("A family in two repositories shared a group")
+ }
+}
+
+func TestProfileBackfillGateAndExplicitRegroup(t *testing.T) {
+ g := testGate(t)
+ store, e := NewProfileStore(g.State, false)
+ if e != nil {
+ t.Fatal(e)
+ }
+ version := func() float64 {
+ var v float64
+ if e := store.connection(false, func(c profileSQL) error {
+ row, e := profileRow(c, "SELECT version FROM learning_similarity_state WHERE id=1")
+ v = metric(row["version"])
+ return e
+ }); e != nil {
+ t.Fatal(e)
+ }
+ return v
+ }
+ repository := gitRepository(t)
+ for i := 0; i < 2; i++ {
+ identity := similarityIdentity(fmt.Sprint(i))
+ if e = store.Observe(identity, Object{"title": "Go regression tests", "project": "fixture", "executable": "go"}); e != nil {
+ t.Fatal(e)
+ }
+ token := fmt.Sprintf("%032x", i+1)
+ recordProfile(t, store, token, identity, GiB, 1, true, profileZero(), 0)
+ mustWrite(t, filepath.Join(g.Metadata, token+".json"), Object{"title": "Go regression tests", "project": "fixture", "executable": "go", "directory": filepath.Join(repository, []string{"sub", "sub/deep"}[i])})
+ }
+ // Pretend an earlier controller finished only the first backfill.
+ if e = store.connection(true, func(c profileSQL) error {
+ return profileExec(c, "UPDATE learning_similarity_state SET version=1 WHERE id=1")
+ }); e != nil {
+ t.Fatal(e)
+ }
+ if _, e = LearningRegroup(g, true, Object{"revision": 0}); e != nil || version() != 1 {
+ t.Fatal("An explicit regroup marked the controller backfill complete", version(), e)
+ }
+ if e = store.backfillSimilarity(g); e != nil || version() != 2 {
+ t.Fatal("The controller did not mark version 2", version(), e)
+ }
+ // A rerun would classify every registered member. Register one it could reach.
+ late := similarityIdentity("late")
+ if e = store.Observe(late, Object{}); e != nil {
+ t.Fatal(e)
+ }
+ if e = store.backfillSimilarity(g); e != nil || version() != 2 {
+ t.Fatal(e)
+ }
+ if _, ok := storedClass(t, store, late); ok {
+ t.Fatal("A completed backfill ran again")
+ }
+}
+
+func TestProfileRegroupRejectsStaleRevisionWithoutWriting(t *testing.T) {
+ g := testGate(t)
+ store, e := NewProfileStore(g.State, false)
+ if e != nil {
+ t.Fatal(e)
+ }
+ repository := gitRepository(t)
+ identity := similarityIdentity("x")
+ if e = store.Observe(identity, Object{"title": "Go tests", "project": "fixture", "executable": "go"}); e != nil {
+ t.Fatal(e)
+ }
+ recordProfile(t, store, strings.Repeat("1", 32), identity, GiB, 1, true, profileZero(), 0)
+ mustWrite(t, filepath.Join(g.Metadata, strings.Repeat("1", 32)+".json"), Object{"title": "Go tests", "project": "fixture", "executable": "go", "directory": repository})
+ before, _ := os.ReadFile(store.path)
+ if _, e = LearningRegroup(g, true, Object{"revision": 5}); e == nil {
+ t.Fatal("A stale revision was applied")
+ }
+ // The outer check races with other writers; the transaction must repeat it.
+ if _, e = store.regroup(&profileHistory{descriptions: map[string]Object{}, aliases: map[string]string{}}, 5, true); e == nil {
+ t.Fatal("A stale revision was applied inside the transaction")
+ }
+ if preview, e := LearningRegroup(g, false, nil); e != nil || preview["applied"] != false {
+ t.Fatal(preview, e)
+ }
+ if after, _ := os.ReadFile(store.path); !bytes.Equal(before, after) {
+ t.Fatal("A rejected or preview regroup wrote the database")
+ }
+}
+
+func TestProfileRegistryEvictionClearsClassification(t *testing.T) {
+ store := fixtureProfileStore(t)
+ identity := observeIn(t, store, "evictable", gitRepository(t), "Go tests", "go")
+ if e := store.connection(true, func(c profileSQL) error {
+ for i := 0; i < maxRegisteredFamilies; i++ {
+ family := profileDigest([]any{"filler", i})
+ if e := profileExec(c, "INSERT INTO learning_profiles VALUES (?,NULL,NULL,NULL,1,0,?)", family, 1e12); e != nil {
+ return e
+ }
+ if e := profileExec(c, "INSERT INTO learning_members VALUES (?,?,NULL,NULL,NULL,?)", family, family, 1e12); e != nil {
+ return e
+ }
+ }
+ if e := profileExec(c, "UPDATE learning_members SET updated=1 WHERE family=?", identity["family"]); e != nil {
+ return e
+ }
+ slot, e := profileRegistrySlot(c)
+ if e != nil || !slot {
+ t.Fatal("No slot was freed", slot, e)
+ }
+ for _, table := range []string{"learning_members", "learning_similarity", "learning_classification"} {
+ if row, e := profileRow(c, "SELECT 1 FROM "+table+" WHERE family=?", identity["family"]); e != nil || row != nil {
+ t.Errorf("%s kept an evicted family (%v)", table, e)
+ }
+ }
+ return nil
+ }); e != nil {
+ t.Fatal(e)
+ }
+}
+
+func TestProfileReadonlyOldDatabaseWithoutClassificationNeverMigrates(t *testing.T) {
+ store := fixtureProfileStore(t)
+ identity := observeIn(t, store, "old", gitRepository(t), "Go tests", "go")
+ if e := store.connection(true, func(c profileSQL) error {
+ return profileExec(c, "DROP TABLE learning_classification")
+ }); e != nil {
+ t.Fatal(e)
+ }
+ before, _ := os.ReadFile(store.path)
+ readonly, e := NewProfileStore(store.state, true)
+ if e != nil {
+ t.Fatal(e)
+ }
+ catalog, e := readonly.CatalogFiltered(0, 100, "", ProfileFilter{Operation: "tests"})
+ if e != nil || metric(catalog["total"]) != 0 {
+ t.Fatal("Unclassified families matched an operation filter", catalog, e)
+ }
+ if hierarchy, e := readonly.Hierarchy(10); e != nil || len(hierarchy["projects"].([]Object)) != 1 {
+ t.Fatal("The hierarchy needs the classification table", hierarchy, e)
+ }
+ if _, e = readonly.Detail(text(identity["family"]), ""); e != nil {
+ t.Fatal(e)
+ }
+ if after, _ := os.ReadFile(store.path); !bytes.Equal(before, after) {
+ t.Fatal("A read migrated the database")
+ }
+}
+
+func TestProfileCatalogFiltersAndHierarchy(t *testing.T) {
+ store := fixtureProfileStore(t)
+ add := func(name, project, title, executable, repository string) Object {
+ t.Helper()
+ return observeClassified(t, store, name, project, title, executable, repository)
+ }
+ // "big" has three families in two operations; "pair" two; "single" one;
+ // "unclassified" has no operation.
+ add("big tests one", "big", "Go tests", "go", "big")
+ add("big tests two", "big", "Go tests", "go", "big")
+ add("big build", "big", "Go build", "go", "big")
+ add("pair lint", "pair", "Lint", "pnpm", "pair")
+ add("single", "single", "Go tests", "go", "single")
+ add("mystery", "pair", "Mystery", "tool", "pair")
+ hierarchy, e := store.Hierarchy(100)
+ if e != nil {
+ t.Fatal(e)
+ }
+ projects := hierarchy["projects"].([]Object)
+ if len(projects) != 2 || projects[0]["project"] != "big" || projects[0]["collapsed"] != false || projects[1]["collapsed"] != true {
+ t.Fatalf("Unexpected hierarchy %v", projects)
+ }
+ if metric(projects[0]["workloadCount"]) != 3 || metric(projects[0]["groupCount"]) != 2 {
+ t.Fatalf("big counts %v", projects[0])
+ }
+ operations := projects[0]["operations"].([]Object)
+ if len(operations) != 2 || operations[0]["operation"] != "tests" || metric(operations[0]["workloadCount"]) != 2 || metric(operations[0]["groupCount"]) != 1 {
+ t.Fatalf("big operations %v", operations)
+ }
+ if projects[1]["project"] != "Other projects" || metric(projects[1]["workloadCount"]) != 3 || metric(projects[1]["projectCount"]) != 2 {
+ t.Fatalf("collapsed row %v", projects[1])
+ }
+ // A real project literally named like the collapsed row folds like any other.
+ add("lookalike", "Other projects", "Go tests", "go", "lookalike")
+ hierarchy, _ = store.Hierarchy(100)
+ rows := hierarchy["projects"].([]Object)
+ named := 0
+ for _, row := range rows {
+ if row["project"] == "Other projects" && row["collapsed"] == false {
+ named++
+ }
+ }
+ if named != 0 {
+ t.Fatal("A one-workload project named like the collapsed row was not folded")
+ }
+ list := func(filter ProfileFilter) Object {
+ t.Helper()
+ result, e := store.CatalogFiltered(0, 100, "", filter)
+ if e != nil {
+ t.Fatal(e)
+ }
+ return result
+ }
+ if r := list(ProfileFilter{Project: "big"}); metric(r["total"]) != 2 {
+ t.Fatal("Project filter", r)
+ }
+ if r := list(ProfileFilter{Project: "big", Operation: "build"}); metric(r["total"]) != 1 {
+ t.Fatal("Project and operation filter", r)
+ }
+ if r := list(ProfileFilter{Operation: "tests"}); metric(r["total"]) != 3 {
+ t.Fatal("Operation filter", r)
+ }
+ other := list(ProfileFilter{OtherProjects: true})
+ if metric(other["total"]) != 4 {
+ t.Fatal("The collapsed set does not match the hierarchy row", other)
+ }
+ for _, p := range other["profiles"].([]Object) {
+ if p["project"] == "big" {
+ t.Fatal("A named project leaked into the collapsed set")
+ }
+ }
+ if r := list(ProfileFilter{OtherProjects: true, Operation: "other"}); metric(r["total"]) != 1 {
+ t.Fatal("Collapsed set and operation filter", r)
+ }
+ if page, e := store.CatalogFiltered(1, 1, "", ProfileFilter{Project: "big"}); e != nil || len(page["profiles"].([]Object)) != 1 || metric(page["total"]) != 2 {
+ t.Fatal("Filters apply after paging", page, e)
+ }
+ if r, e := store.CatalogFiltered(0, 100, "build", ProfileFilter{Project: "big"}); e != nil || metric(r["total"]) != 1 {
+ t.Fatal("Search and filter do not combine", r, e)
+ }
+ if limited, _ := store.Hierarchy(1); len(limited["projects"].([]Object)) != 1 || metric(limited["total"]) != 1 {
+ t.Fatal("The hierarchy limit counts the other-projects row", limited)
+ }
+ for _, bad := range []ProfileFilter{{Operation: "typo"}, {Project: strings.Repeat("x", 121)}, {Project: "a\nb"}, {Project: " pad"}, {Project: "big", OtherProjects: true}} {
+ if _, e := store.CatalogFiltered(0, 100, "", bad); e == nil {
+ t.Errorf("Invalid filter accepted: %+v", bad)
+ }
+ }
+ for _, limit := range []int{0, 101, -1} {
+ if _, e := store.Hierarchy(limit); e == nil {
+ t.Errorf("Invalid hierarchy limit %d accepted", limit)
+ }
+ }
+}
+
+func TestProfileCLICatalogFilters(t *testing.T) {
+ g := testGate(t)
+ store, e := NewProfileStore(g.State, false)
+ if e != nil {
+ t.Fatal(e)
+ }
+ repository := gitRepository(t)
+ identity := similarityIdentity("cli")
+ description := profileSimilarityDescription(Object{"title": "Go tests", "project": "cli", "executable": "go", "directory": repository}, map[string]profileIdentity{})
+ if e = store.Observe(identity, description); e != nil {
+ t.Fatal(e)
+ }
+ run := func(args ...string) (int, Object) {
+ var output, stderr bytes.Buffer
+ code := LearningCLI(g, append([]string{"list", "--json"}, args...), strings.NewReader(""), &output, &stderr)
+ result := Object{}
+ _ = json.Unmarshal(output.Bytes(), &result)
+ return code, result
+ }
+ if code, r := run("--project", "cli", "--operation=tests"); code != 0 || metric(r["total"]) != 1 {
+ t.Fatal("Filtered list", code, r)
+ }
+ if code, r := run("--operation", "build"); code != 0 || metric(r["total"]) != 0 {
+ t.Fatal("Filtered list should be empty", code, r)
+ }
+ if code, r := run("--other-projects"); code != 0 || metric(r["total"]) != 1 {
+ t.Fatal("A one-workload project is in the collapsed set", code, r)
+ }
+ if code, r := run("--hierarchy"); code != 0 || len(r["projects"].([]any)) != 1 || r["controllerStatus"] == nil {
+ t.Fatal("Hierarchy", code, r)
+ }
+ for _, bad := range [][]string{
+ {"--operation", "nope"}, {"--project", ""}, {"--operation", ""}, {"--project", "a", "--project", "b"},
+ {"--project", "a", "--other-projects"}, {"--other-projects", "--other-projects"},
+ {"--hierarchy", "--project", "a"}, {"--hierarchy", "--query", "a"}, {"--hierarchy", "--offset", "1"},
+ {"--hierarchy", "--offset", "0"}, {"--hierarchy", "--query", ""}, {"--hierarchy", "--query="}, {"--hierarchy", "--offset=0"}, {"--hierarchy", "--other-projects"},
+ {"--hierarchy", "--limit", "101"}, {"--project"}, {"--kind", "go:tests"},
+ } {
+ if code, r := run(bad...); code != 125 || r["error"] == nil {
+ t.Errorf("%v accepted: %d %v", bad, code, r)
+ }
+ }
+}
+
+func TestProfilePreviewMatchesAppliedCatalog(t *testing.T) {
+ g := testGate(t)
+ store, e := NewProfileStore(g.State, false)
+ if e != nil {
+ t.Fatal(e)
+ }
+ repository := gitRepository(t)
+ for i := 0; i < 5; i++ {
+ identity := similarityIdentity(fmt.Sprint(i))
+ title := []string{"Go tests", "Go tests", "Go build", "Go tests", "Mystery"}[i]
+ if e = store.Observe(identity, Object{"title": title, "project": "fixture", "executable": "go"}); e != nil {
+ t.Fatal(e)
+ }
+ token := fmt.Sprintf("%032x", i+1)
+ recordProfile(t, store, token, identity, GiB, 1, i != 3, profileZero(), 0)
+ mustWrite(t, filepath.Join(g.Metadata, token+".json"), Object{"title": title, "project": "fixture", "executable": "go", "directory": filepath.Join(repository, []string{"", "sub", "", "sub/deep", ""}[i])})
+ }
+ revision, _ := store.Revision()
+ if e = store.Edit(text(similarityIdentity("1")["family"]), revision, "Pinned", nil, nil); e != nil {
+ t.Fatal(e)
+ }
+ preview, e := LearningRegroup(g, false, nil)
+ if e != nil {
+ t.Fatal(e)
+ }
+ saved, e := LearningRegroup(g, true, Object{"revision": preview["revision"]})
+ if e != nil {
+ t.Fatal(e)
+ }
+ catalog, _ := store.Catalog(0, 100, "")
+ if metric(catalog["total"]) != metric(preview["profilesAfter"]) || metric(preview["profilesBefore"])-metric(preview["profilesAfter"]) != 1 {
+ t.Fatal("Preview counts differ from the applied catalog", preview, catalog)
+ }
+ if !reflect.DeepEqual(saved["groups"], preview["groups"]) || saved["familiesGrouped"] != preview["familiesGrouped"] {
+ t.Fatal("Applied plan differs from the preview", saved, preview)
+ }
+}
+
+func TestProfilePlanMergesRules(t *testing.T) {
+ class := func(repository, tool, operation string) profileClass {
+ return profileClass{repository: repository, tool: tool, operation: operation}
+ }
+ repoA, repoB := profileDigest("repo a"), profileDigest("repo b")
+ member := func(profile string, pinned, classified bool, c profileClass) profileMemberClass {
+ return profileMemberClass{family: "family-" + profile, profile: profile, pinned: pinned, classified: classified, class: c}
+ }
+ for _, entry := range []struct {
+ name string
+ members []profileMemberClass
+ want int
+ }{
+ {"same class merges", []profileMemberClass{member("b", false, true, class(repoA, "go", "tests")), member("a", false, true, class(repoA, "go", "tests"))}, 1},
+ {"different repositories stay apart", []profileMemberClass{member("a", false, true, class(repoA, "go", "tests")), member("b", false, true, class(repoB, "go", "tests"))}, 0},
+ {"different operations stay apart", []profileMemberClass{member("a", false, true, class(repoA, "go", "tests")), member("b", false, true, class(repoA, "go", "lint"))}, 0},
+ {"different tools stay apart", []profileMemberClass{member("a", false, true, class(repoA, "go", "tests")), member("b", false, true, class(repoA, "cargo", "tests"))}, 0},
+ {"no repository stays alone", []profileMemberClass{member("a", false, true, class("", "go", "tests")), member("b", false, true, class("", "go", "tests"))}, 0},
+ {"unrecognized tool stays alone", []profileMemberClass{member("a", false, true, class(repoA, "other", "tests")), member("b", false, true, class(repoA, "other", "tests"))}, 0},
+ {"unknown operation shares", []profileMemberClass{member("a", false, true, class(repoA, "go", "other")), member("b", false, true, class(repoA, "go", "other"))}, 1},
+ {"pinned profile is untouched", []profileMemberClass{member("a", true, true, class(repoA, "go", "tests")), member("b", false, true, class(repoA, "go", "tests"))}, 0},
+ {"unclassified member blocks", []profileMemberClass{member("a", false, false, class(repoA, "go", "tests")), member("b", false, true, class(repoA, "go", "tests"))}, 0},
+ {"mixed profile is never split", []profileMemberClass{member("a", false, true, class(repoA, "go", "tests")), {family: "second", profile: "a", classified: true, class: class(repoA, "go", "lint")}, member("b", false, true, class(repoA, "go", "tests"))}, 0},
+ } {
+ merges := profilePlanMerges(entry.members)
+ if len(merges) != entry.want {
+ t.Errorf("%s: %d merges, want %d (%+v)", entry.name, len(merges), entry.want, merges)
+ }
+ }
+ merges := profilePlanMerges([]profileMemberClass{member("c", false, true, class(repoA, "go", "tests")), member("a", false, true, class(repoA, "go", "tests")), member("b", false, true, class(repoA, "go", "tests"))})
+ if len(merges) != 1 || merges[0].target != "a" || len(merges[0].sources) != 2 {
+ t.Fatal("Merging did not keep the lowest profile", merges)
+ }
+}
+
+func TestProfileUnwrapBehaviours(t *testing.T) {
+ for _, entry := range []struct {
+ argv []string
+ want []string
+ ok bool
+ }{
+ {[]string{"go", "test"}, []string{"go", "test"}, true},
+ {[]string{"env", "A=1", "B=2", "go", "test"}, []string{"go", "test"}, true},
+ {[]string{"/usr/bin/env", "--", "A=1", "go"}, []string{"go"}, true},
+ {[]string{"env", "A=1", "env", "B=2", "go"}, []string{"go"}, true},
+ {[]string{"env", "-i", "A=1", "go"}, []string{"-i", "A=1", "go"}, true},
+ {[]string{"env", "A=1"}, []string{}, true},
+ {[]string{"fnm", "exec", "--using=26", "--", "pnpm", "test"}, []string{"pnpm", "test"}, true},
+ {[]string{"env", "A=1", "fnm", "exec", "--", "env", "B=2", "node"}, []string{"node"}, true},
+ {[]string{"fnm", "use", "20"}, []string{"fnm", "use", "20"}, false},
+ {[]string{"fnm", "exec", "node"}, []string{"fnm", "exec", "node"}, false},
+ {[]string{"fnm"}, []string{"fnm"}, false},
+ {[]string{"go", "run", "A=1"}, []string{"go", "run", "A=1"}, true},
+ } {
+ got, ok := profileUnwrap(entry.argv)
+ if ok != entry.ok || !reflect.DeepEqual(append([]string{}, got...), entry.want) {
+ t.Errorf("%v: got %v %v, want %v %v", entry.argv, got, ok, entry.want, entry.ok)
+ }
+ }
+ if profileToolOf("env") != "" || profileToolOf("fnm") != "" || profileToolOf("/x/run.sh") != "shell" || profileToolOf("go") != "go" {
+ t.Fatal("A wrapper is a tool, or a script is not")
+ }
+ history := &profileHistory{}
+ for _, executable := range []string{"env", "fnm"} {
+ if class := profileBackfillClass(Object{"family": "x", "executable": executable}, "go:lint", "", history); class.tool != "go" || class.operation != "lint" {
+ t.Errorf("A stored legacy kind lost to the %s wrapper: %+v", executable, class)
+ }
+ }
+}
+
+func TestProfileInlineScanBoundIsExact(t *testing.T) {
+ // The bound is part of the behaviour, so it is spelled out here.
+ exact := strings.Repeat(" ", 8192-len("go test")) + "go test"
+ if got := profileInlineOperation([]string{"bash", "-c", exact}); got != "tests" {
+ t.Fatalf("A program ending at the bound was not scanned: %q", got)
+ }
+ if got := profileInlineOperation([]string{"bash", "-c", " " + exact}); got != "" {
+ t.Fatalf("A program crossing the bound was scanned: %q", got)
+ }
+}
+
+func TestProfileFirstObservationHonorsStoredLegacyKind(t *testing.T) {
+ store := fixtureProfileStore(t)
+ repository := gitRepository(t)
+ peer := observeIn(t, store, "peer", repository, "Go tests", "go")
+ // Registered by an older controller: a legacy kind, no classification yet.
+ old := similarityIdentity("old")
+ if e := store.Observe(old, Object{}); e != nil {
+ t.Fatal(e)
+ }
+ if e := store.connection(true, func(c profileSQL) error {
+ return profileExec(c, "INSERT INTO learning_similarity VALUES (?,?,?)", old["family"], profileDigest("old scope"), "go:full-tests")
+ }); e != nil {
+ t.Fatal(e)
+ }
+ description := profileSimilarityDescription(Object{"title": "Go tests", "project": "fixture", "executable": "go", "directory": repository}, map[string]profileIdentity{})
+ if e := store.Observe(old, description); e != nil {
+ t.Fatal(e)
+ }
+ if class, ok := storedClass(t, store, old); !ok || class.tool != "go" || class.operation != "full-tests" || class.repository == "" {
+ t.Fatal("The first classification ignored the stored kind", class, ok)
+ }
+ if matchID(t, store, old) == matchID(t, store, peer) {
+ t.Fatal("A full-tests family merged with a tests family")
+ }
+}
+
+func TestProfileCorruptOrphanMetadataDoesNotBlockBackfill(t *testing.T) {
+ g := testGate(t)
+ store, e := NewProfileStore(g.State, false)
+ if e != nil {
+ t.Fatal(e)
+ }
+ repository := gitRepository(t)
+ orphan := filepath.Join(g.Metadata, strings.Repeat("e", 32)+".json")
+ if e = os.WriteFile(orphan, []byte("{truncated"), 0600); e != nil {
+ t.Fatal(e)
+ }
+ // A readable orphan names a checkout but belongs to no family, so it teaches nothing.
+ mustWrite(t, filepath.Join(g.Metadata, strings.Repeat("f", 32)+".json"), Object{"title": "Go tests", "project": "fixture", "executable": "go", "directory": repository})
+ history, e := store.historicalSimilarityDescriptions(g)
+ if e != nil || len(history.aliases) != 0 || len(history.repositories) != 0 || len(history.descriptions) != 0 {
+ t.Fatal("An orphan file broke or taught the history", history, e)
+ }
+ if _, e = LearningRegroup(g, false, nil); e != nil {
+ t.Fatal(e)
+ }
+ if e = store.backfillSimilarity(g); e != nil {
+ t.Fatal("A corrupt orphan held the backfill", e)
+ }
+ // A corrupt file that belongs to a family is skipped like an orphan.
+ identity := similarityIdentity("owned")
+ if e = store.Observe(identity, Object{"title": "Go tests", "project": "fixture", "executable": "go"}); e != nil {
+ t.Fatal(e)
+ }
+ recordProfile(t, store, strings.Repeat("1", 32), identity, GiB, 1, true, profileZero(), 0)
+ if e = os.WriteFile(filepath.Join(g.Metadata, strings.Repeat("1", 32)+".json"), []byte("{truncated"), 0600); e != nil {
+ t.Fatal(e)
+ }
+ if _, e = LearningRegroup(g, false, nil); e != nil {
+ t.Fatal("A corrupt file with a family blocked the regroup", e)
+ }
+}
+
+func TestProfileMissingDirectoryIdentityNeverSpawnsGit(t *testing.T) {
+ bin, marker := t.TempDir(), filepath.Join(t.TempDir(), "spawned")
+ if e := os.WriteFile(filepath.Join(bin, "git"), []byte("#!/bin/sh\ntouch '"+marker+"'\nexit 1\n"), 0700); e != nil {
+ t.Fatal(e)
+ }
+ missing := filepath.Join(t.TempDir(), "deleted", "worktree")
+ wantScope, wantRepository := profileDirectoryIdentity(missing)
+ t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH"))
+ if _, e := os.Stat(marker); e == nil {
+ t.Fatal("Fixture: the reference computation should have used the real git")
+ }
+ identities := map[string]profileIdentity{}
+ for range 2 {
+ got := profileDescriptionIdentity(missing, identities)
+ if got.scope != wantScope || got.repository != wantRepository || got.repository != "" {
+ t.Fatal("A missing directory has a different identity", got, wantScope)
+ }
+ }
+ for _, directory := range []string{"", "relative/path"} {
+ if got := profileDescriptionIdentity(directory, identities); got != (profileIdentity{}) {
+ t.Fatalf("%q was identified: %v", directory, got)
+ }
+ }
+ if _, e := os.Stat(marker); e == nil {
+ t.Fatal("git was spawned for a directory that does not exist")
+ }
+ profileDescriptionIdentity(t.TempDir(), identities)
+ if _, e := os.Stat(marker); e != nil {
+ t.Fatal("Fixture: an existing directory should consult git")
+ }
+}
+
+func TestProfileOnlyTwoRepositoriesStayIndependentAndNothingElseBlocksSharing(t *testing.T) {
+ g := testGate(t)
+ store, e := NewProfileStore(g.State, false)
+ if e != nil {
+ t.Fatal(e)
+ }
+ repository := gitRepository(t)
+ peer := observeIn(t, store, "peer", repository, "Go tests", "go")
+ _, want := profileDirectoryIdentity(repository)
+ deleted := filepath.Join(t.TempDir(), "gone")
+ token := 0
+ describe := func(name, title string, directories ...string) Object {
+ identity := similarityIdentity(name)
+ if e := store.Observe(identity, Object{"title": "Go tests", "project": "fixture", "executable": "go"}); e != nil {
+ t.Fatal(e)
+ }
+ for _, directory := range directories {
+ token++
+ value := fmt.Sprintf("%032x", token)
+ recordProfile(t, store, value, identity, GiB, 1, true, profileZero(), 0)
+ mustWrite(t, filepath.Join(g.Metadata, value+".json"), Object{"title": title, "project": "fixture", "executable": "go", "directory": directory})
+ }
+ return identity
+ }
+ kinds := similarityIdentity("kinds")
+ if e = store.Observe(kinds, Object{"title": "Go tests", "project": "fixture", "executable": "go"}); e != nil {
+ t.Fatal(e)
+ }
+ for _, title := range []string{"Go tests", "Go build", "Go tests"} {
+ token++
+ value := fmt.Sprintf("%032x", token)
+ recordProfile(t, store, value, kinds, GiB, 1, true, profileZero(), 0)
+ mustWrite(t, filepath.Join(g.Metadata, value+".json"), Object{"title": title, "project": "fixture", "executable": "go", "directory": repository})
+ }
+ // Metadata that claims one repository cannot hide two checkouts of different ones.
+ claimed := similarityIdentity("claimed")
+ if e = store.Observe(claimed, Object{"title": "Go tests", "project": "fixture", "executable": "go"}); e != nil {
+ t.Fatal(e)
+ }
+ for _, directory := range []string{repository, gitRepository(t)} {
+ token++
+ value := fmt.Sprintf("%032x", token)
+ recordProfile(t, store, value, claimed, GiB, 1, true, profileZero(), 0)
+ mustWrite(t, filepath.Join(g.Metadata, value+".json"), Object{"title": "Go tests", "project": "fixture", "executable": "go", "directory": directory, "learningRepository": want})
+ }
+ subdirectories := describe("subdirectories", "Go tests", repository, filepath.Join(repository, "sub"), repository)
+ retired := describe("retired", "Go tests", deleted, repository)
+ if _, e = LearningRegroup(g, true, Object{"revision": 0}); e != nil {
+ t.Fatal(e)
+ }
+ // Titles that disagree do not block sharing: the label is not identity. Old
+ // clients still get no legacy row, because they would see two kinds.
+ if class, ok := storedClass(t, store, kinds); !ok || class.repository != want || class.operation != "tests" {
+ t.Fatal("Metadata that disagrees on its label blocked the repository", class)
+ }
+ if matchID(t, store, kinds) != matchID(t, store, peer) || familyHasLegacyRow(t, store, kinds) {
+ t.Fatal("A family with differing labels did not share, or received a conflicting legacy scope")
+ }
+ if class, ok := storedClass(t, store, claimed); !ok || class.repository != "" || matchID(t, store, claimed) == matchID(t, store, peer) {
+ t.Fatal("Checkouts of two repositories shared a group", class)
+ }
+ for name, identity := range map[string]Object{"subdirectories": subdirectories, "one deleted directory": retired} {
+ if class, ok := storedClass(t, store, identity); !ok || class.repository != want {
+ t.Errorf("%s: a directory difference counted as a conflict: %+v", name, class)
+ }
+ if matchID(t, store, identity) != matchID(t, store, peer) {
+ t.Errorf("%s did not share with its repository", name)
+ }
+ }
+}
+
+func TestProfileObservationFillsRepositoryAndRejectsInvalidLegacyKind(t *testing.T) {
+ store := fixtureProfileStore(t)
+ repository := gitRepository(t)
+ peer := observeIn(t, store, "peer", repository, "Go tests", "go")
+ _, want := profileDirectoryIdentity(repository)
+ // Same title: only the repository is new.
+ same := observeIn(t, store, "same", t.TempDir(), "Go tests", "go")
+ if class, _ := storedClass(t, store, same); class.repository != "" {
+ t.Fatal("Fixture: the repository should be unknown", class)
+ }
+ description := profileSimilarityDescription(Object{"title": "Go tests", "executable": "go", "directory": repository}, map[string]profileIdentity{})
+ if e := store.Observe(same, description); e != nil {
+ t.Fatal(e)
+ }
+ if class, _ := storedClass(t, store, same); class.repository != want || matchID(t, store, same) != matchID(t, store, peer) {
+ t.Fatal("A later observation with the same title did not fill the repository", class)
+ }
+ // A different tool stored first keeps its tool.
+ other := observeIn(t, store, "other", t.TempDir(), "Go tests", "go")
+ later := profileSimilarityDescription(Object{"title": "Tests", "executable": "pnpm", "directory": repository}, map[string]profileIdentity{})
+ if e := store.Observe(other, later); e != nil {
+ t.Fatal(e)
+ }
+ if class, _ := storedClass(t, store, other); class.repository != want || class.tool != "go" || class.operation != "tests" || matchID(t, store, other) != matchID(t, store, peer) {
+ t.Fatal("The repository fill-in changed the stored tool", class)
+ }
+ // A legacy kind no client could have written is treated as absent.
+ broken := similarityIdentity("broken")
+ if e := store.Observe(broken, Object{}); e != nil {
+ t.Fatal(e)
+ }
+ if e := store.connection(true, func(c profileSQL) error {
+ return profileExec(c, "INSERT INTO learning_similarity VALUES (?,?,?)", broken["family"], profileDigest("scope"), "ruby:deploy")
+ }); e != nil {
+ t.Fatal(e)
+ }
+ if e := store.Observe(broken, profileSimilarityDescription(Object{"title": "Go tests", "executable": "go", "directory": repository}, map[string]profileIdentity{})); e != nil {
+ t.Fatal("An invalid stored kind failed the observation", e)
+ }
+ if class, ok := storedClass(t, store, broken); !ok || class.tool != "go" || class.operation != "tests" {
+ t.Fatal("An invalid stored kind decided the class", class)
+ }
+}
+
+func TestProfileBackfillFillsRepositoryAndPreviewMatchesApply(t *testing.T) {
+ g := testGate(t)
+ store, e := NewProfileStore(g.State, false)
+ if e != nil {
+ t.Fatal(e)
+ }
+ repository := gitRepository(t)
+ _, want := profileDirectoryIdentity(repository)
+ // One family stored with an unknown repository, one already in the repository.
+ unverified, verified := similarityIdentity("unverified"), observeIn(t, store, "verified", repository, "Go tests", "go")
+ if e = store.Observe(unverified, profileSimilarityDescription(Object{"title": "Go tests", "project": "fixture", "executable": "go"}, map[string]profileIdentity{})); e != nil {
+ t.Fatal(e)
+ }
+ recordProfile(t, store, strings.Repeat("1", 32), unverified, GiB, 1, true, profileZero(), 0)
+ mustWrite(t, filepath.Join(g.Metadata, strings.Repeat("1", 32)+".json"), Object{"title": "Go tests", "project": "fixture", "executable": "go", "directory": repository})
+ if class, _ := storedClass(t, store, unverified); class.repository != "" || class.tool != "go" {
+ t.Fatal("Fixture: the family should be stored without a repository", class)
+ }
+ preview, e := LearningRegroup(g, false, nil)
+ if e != nil || metric(preview["profilesBefore"])-metric(preview["profilesAfter"]) != 1 {
+ t.Fatal("The preview ignored the repository the backfill fills in", preview, e)
+ }
+ saved, e := LearningRegroup(g, true, Object{"revision": preview["revision"]})
+ if e != nil {
+ t.Fatal(e)
+ }
+ if class, _ := storedClass(t, store, unverified); class.repository != want || class.operation != "tests" {
+ t.Fatal("The backfill did not fill the repository", class)
+ }
+ catalog, _ := store.Catalog(0, 100, "")
+ if metric(catalog["total"]) != metric(preview["profilesAfter"]) || !reflect.DeepEqual(saved["groups"], preview["groups"]) || matchID(t, store, unverified) != matchID(t, store, verified) {
+ t.Fatal("Preview and apply disagree", preview, saved, catalog)
+ }
+}
+
+func TestProfileRegroupLearnsAliasFromScopeComputedNow(t *testing.T) {
+ g := testGate(t)
+ store, e := NewProfileStore(g.State, false)
+ if e != nil {
+ t.Fatal(e)
+ }
+ repository := gitRepository(t)
+ scope, want := profileDirectoryIdentity(filepath.Join(repository, "sub"))
+ // The live family's stored scope teaches nothing; only its checkout does.
+ live, retired := similarityIdentity("live"), similarityIdentity("retired")
+ for i, identity := range []Object{live, retired} {
+ if e = store.Observe(identity, Object{"title": "Go tests", "project": "fixture", "executable": "go"}); e != nil {
+ t.Fatal(e)
+ }
+ recordProfile(t, store, fmt.Sprintf("%032x", i+1), identity, GiB, 1, true, profileZero(), 0)
+ }
+ if e = store.connection(true, func(c profileSQL) error {
+ if e := profileExec(c, "INSERT INTO learning_similarity VALUES (?,?,?)", live["family"], profileDigest("stale"), "go:tests"); e != nil {
+ return e
+ }
+ return profileExec(c, "INSERT INTO learning_similarity VALUES (?,?,?)", retired["family"], scope, "go:tests")
+ }); e != nil {
+ t.Fatal(e)
+ }
+ mustWrite(t, filepath.Join(g.Metadata, fmt.Sprintf("%032x.json", 1)), Object{"title": "Go tests", "project": "fixture", "executable": "go", "directory": filepath.Join(repository, "sub")})
+ if _, e = LearningRegroup(g, true, Object{"revision": 0}); e != nil {
+ t.Fatal(e)
+ }
+ if class, _ := storedClass(t, store, retired); class.repository != want {
+ t.Fatal("The deleted checkout was not recovered from a live scope", class)
+ }
+}
+
+func TestProfileEmptyObservationStoresNoClassificationAndEditClassifiesRunOnlyFamilies(t *testing.T) {
+ store := fixtureProfileStore(t)
+ empty := similarityIdentity("empty")
+ if e := store.Observe(empty, profileSimilarityDescription(Object{}, map[string]profileIdentity{})); e != nil {
+ t.Fatal(e)
+ }
+ if _, ok := storedClass(t, store, empty); ok {
+ t.Fatal("An empty description left a lasting classification")
+ }
+ // A later observation classifies it.
+ observeIn(t, store, "empty", gitRepository(t), "Go tests", "go")
+ if class, ok := storedClass(t, store, empty); !ok || class.tool != "go" || class.operation != "tests" {
+ t.Fatal("A later observation did not classify the family", class, ok)
+ }
+ runOnly := similarityIdentity("run only")
+ recordProfile(t, store, strings.Repeat("2", 32), runOnly, GiB, 1, true, profileZero(), 0)
+ revision, _ := store.Revision()
+ if e := store.Edit(text(runOnly["family"]), revision, "Mine", nil, nil); e != nil {
+ t.Fatal(e)
+ }
+ if class, ok := storedClass(t, store, runOnly); !ok || class.tool != "other" || class.operation != "other" {
+ t.Fatal("A family registered by an edit has no classification", class, ok)
+ }
+ if e := store.connection(false, func(c profileSQL) error {
+ row, e := profileRow(c, "SELECT (SELECT COUNT(*) FROM learning_members) AS members,(SELECT COUNT(*) FROM learning_classification) AS classes")
+ if e == nil && (row["members"] != row["classes"] || metric(row["members"]) != 2) {
+ t.Errorf("members and classifications differ: %v", row)
+ }
+ return e
+ }); e != nil {
+ t.Fatal(e)
+ }
+}
+
+func TestProfileInvalidStoredClassificationFailsEveryReadPath(t *testing.T) {
+ for name, set := range map[string]string{
+ "operation": "UPDATE learning_classification SET operation='deploy'",
+ "tool": "UPDATE learning_classification SET tool='ruby'",
+ "repository": "UPDATE learning_classification SET repository='not a digest'",
+ } {
+ g := testGate(t)
+ store, e := NewProfileStore(g.State, false)
+ if e != nil {
+ t.Fatal(e)
+ }
+ identity := observeIn(t, store, "x", gitRepository(t), "Go tests", "go")
+ if e = store.connection(true, func(c profileSQL) error { return profileExec(c, set) }); e != nil {
+ t.Fatal(e)
+ }
+ readonly, e := NewProfileStore(g.State, true)
+ if e != nil {
+ t.Fatal(e)
+ }
+ _, catalogError := readonly.Catalog(0, 100, "")
+ _, hierarchyError := readonly.Hierarchy(10)
+ _, detailError := readonly.Detail(text(identity["family"]), "")
+ _, previewError := LearningRegroup(g, false, nil)
+ for read, e := range map[string]error{"catalog": catalogError, "hierarchy": hierarchyError, "detail": detailError, "preview": previewError} {
+ if e == nil || !strings.Contains(e.Error(), "Invalid stored workload classification") {
+ t.Errorf("%s %s: %v", name, read, e)
+ }
+ }
+ }
+}
+
+func TestProfileSummaryProjectOperationAndMergeLabel(t *testing.T) {
+ store := fixtureProfileStore(t)
+ // The most common project wins, ties go to the lower name, and none is unknown.
+ if got := profileProject([]Object{{"project": "b"}, {"project": "a"}, {"project": "b"}}); got != "b" {
+ t.Fatal(got)
+ }
+ if got := profileProject([]Object{{"project": "b"}, {"project": "a"}}); got != "a" {
+ t.Fatal(got)
+ }
+ if got := profileProject([]Object{{"project": nil}, {}}); got != "Unknown project" {
+ t.Fatal(got)
+ }
+ observeClassified(t, store, "one", "x", "Go tests", "go", "repo")
+ observeClassified(t, store, "two", "y", "Go tests", "go", "repo")
+ observeClassified(t, store, "three", "y", "Go tests", "go", "repo")
+ observeClassified(t, store, "four", "", "Go build", "go", "elsewhere")
+ catalog, e := store.Catalog(0, 100, "")
+ if e != nil {
+ t.Fatal(e)
+ }
+ byProject := map[string]Object{}
+ for _, summary := range catalog["profiles"].([]Object) {
+ byProject[text(summary["project"])] = summary
+ }
+ if byProject["y"] == nil || metric(byProject["y"]["memberCount"]) != 3 || byProject["y"]["operation"] != "tests" || byProject["y"]["label"] != "y · go tests" {
+ t.Fatal("The merged group is mislabeled", byProject)
+ }
+ if byProject["Unknown project"] == nil || byProject["Unknown project"]["operation"] != "build" {
+ t.Fatal("A group without a project is not unknown", byProject)
+ }
+ // Members with different operations share no operation.
+ revision, _ := store.Revision()
+ if e = store.Merge(text(byProject["y"]["id"]), text(byProject["Unknown project"]["id"]), revision); e != nil {
+ t.Fatal(e)
+ }
+ catalog, _ = store.Catalog(0, 100, "")
+ if len(catalog["profiles"].([]Object)) != 1 || catalog["profiles"].([]Object)[0]["operation"] != "other" {
+ t.Fatal("A mixed group reported an operation", catalog)
+ }
+}
+
+func TestProfileHierarchyLimitCountsAndOrder(t *testing.T) {
+ store := fixtureProfileStore(t)
+ // Inserted out of order: names and sizes both decide the order.
+ var alpha Object
+ for i, entry := range []struct{ project, title, executable string }{
+ {"gamma", "Go tests", "go"}, {"gamma", "Go tests", "go"}, {"gamma", "Go tests", "go"},
+ {"beta", "Go tests", "go"}, {"beta", "Go build", "go"}, {"beta", "Go build", "go"},
+ {"alpha", "Go tests", "go"}, {"alpha", "Go tests", "go"}, {"alpha", "Go build", "go"}, {"alpha", "Lint", "pnpm"},
+ {"tiny", "Go tests", "go"},
+ } {
+ identity := observeClassified(t, store, fmt.Sprint("workload ", i), entry.project, entry.title, entry.executable, "repo "+entry.project)
+ if entry.project == "alpha" && alpha == nil {
+ alpha = identity
+ }
+ }
+ recordProfile(t, store, strings.Repeat("1", 32), alpha, GiB, 1, true, profileZero(), 0)
+ recordProfile(t, store, strings.Repeat("2", 32), alpha, GiB, 1, false, profileZero(), 0)
+ names := func(hierarchy Object) []string {
+ result := []string{}
+ for _, row := range hierarchy["projects"].([]Object) {
+ result = append(result, text(row["project"]))
+ }
+ return result
+ }
+ for limit, want := range map[int][]string{1: {"alpha"}, 2: {"alpha", "beta"}, 3: {"alpha", "beta", "gamma"}, 4: {"alpha", "beta", "gamma", "Other projects"}, 100: {"alpha", "beta", "gamma", "Other projects"}} {
+ hierarchy, e := store.Hierarchy(limit)
+ if e != nil || !reflect.DeepEqual(names(hierarchy), want) || metric(hierarchy["total"]) != 3 || metric(hierarchy["limit"]) != float64(limit) {
+ t.Fatalf("limit %d: %v %v", limit, names(hierarchy), e)
+ }
+ }
+ hierarchy, _ := store.Hierarchy(100)
+ rows := hierarchy["projects"].([]Object)
+ if metric(rows[0]["workloadCount"]) != 4 || metric(rows[0]["groupCount"]) != 3 || metric(rows[0]["runCount"]) != 2 || metric(rows[3]["projectCount"]) != 1 {
+ t.Fatal("Unexpected alpha counts", rows[0], rows[3])
+ }
+ operations := rows[0]["operations"].([]Object)
+ beta := rows[1]["operations"].([]Object)
+ if len(beta) != 2 || beta[0]["operation"] != "build" || metric(beta[0]["workloadCount"]) != 2 || metric(beta[0]["groupCount"]) != 1 || beta[1]["operation"] != "tests" {
+ t.Fatal("Operations are not ordered by workload count", beta)
+ }
+ if len(operations) != 3 || operations[0]["operation"] != "tests" || metric(operations[0]["runCount"]) != 2 || operations[1]["operation"] != "build" || metric(operations[1]["runCount"]) != 0 {
+ t.Fatal("Unexpected alpha operations", operations)
+ }
+ for _, limit := range []int{1, 2, 100} {
+ if hierarchy, _ = store.Hierarchy(limit); len(hierarchy["projects"].([]Object)) > limit {
+ t.Fatalf("limit %d returned %d rows", limit, len(hierarchy["projects"].([]Object)))
+ }
+ }
+}
+
+// shuffledHistory is the whole retroactive conclusion for one evidence order.
+func shuffledHistory(evidence []profileEvidence, scopes map[string]string, members []Object) (*profileHistory, map[string]profileClass, []profileMerge) {
+ history := profileAggregate(evidence, scopes)
+ classes, planned := map[string]profileClass{}, []profileMemberClass{}
+ for _, member := range members {
+ family := text(member["family"])
+ classes[family] = profileBackfillClass(member, "", scopes[family], history)
+ planned = append(planned, profileMemberClass{family: family, profile: family, project: "fixture", classified: true, class: classes[family]})
+ }
+ return history, classes, profilePlanMerges(planned)
+}
+
+func TestProfileClassificationIsIndependentOfEvidenceOrder(t *testing.T) {
+ repositoryA, repositoryB := profileDigest("repository a"), profileDigest("repository b")
+ scope := func(name string) string { return profileDigest("scope " + name) }
+ file := func(family, title, scopeName, kind string, identity profileIdentity) profileEvidence {
+ clean := Object{"title": title, "project": "fixture", "executable": "go"}
+ if kind != "" {
+ clean["learningScope"], clean["learningKind"] = scope(scopeName), kind
+ }
+ return profileEvidence{family, clean, identity}
+ }
+ live := func(name, repository string) profileIdentity { return profileIdentity{scope(name), repository} }
+ evidence := []profileEvidence{
+ file("missing", "Go tests", "a", "", live("a", repositoryA)),
+ file("kinds", "Go tests", "a", "go:tests", live("a", repositoryA)),
+ file("kinds", "Go tests", "a", "go:build", live("a", repositoryA)),
+ file("directories", "Go tests", "a", "go:tests", live("a", repositoryA)),
+ file("directories", "Go tests", "sub", "go:tests", live("sub", repositoryA)),
+ file("two", "Go tests", "a", "go:tests", live("a", repositoryA)),
+ file("two", "Go tests", "b", "go:tests", live("b", repositoryB)),
+ file("labels", "Go tests", "a", "go:tests", live("a", repositoryA)),
+ file("labels", "Go tests for issue 7", "a", "go:tests", live("a", repositoryA)),
+ file("deleted", "Go tests", "gone", "go:tests", live("gone", "")),
+ file("deleted", "Go tests", "gone", "go:tests", live("gone", "")),
+ }
+ scopes := map[string]string{"deleted": scope("gone"), "directories": scope("a")}
+ // A second checkout of repository A teaches the deleted scope its repository.
+ evidence = append(evidence, file("teacher", "Go tests", "gone", "go:tests", live("gone", repositoryA)))
+ members := []Object{}
+ for _, family := range []string{"missing", "kinds", "directories", "two", "labels", "deleted", "teacher"} {
+ members = append(members, Object{"family": family, "title": "Go tests", "executable": "go"})
+ }
+ wantHistory, wantClasses, wantMerges := shuffledHistory(evidence, scopes, members)
+ if wantClasses["two"].repository != "" || wantClasses["deleted"].repository != repositoryA || wantClasses["kinds"].repository != repositoryA || wantClasses["missing"].repository != repositoryA {
+ t.Fatal("Unexpected baseline classes", wantClasses)
+ }
+ if len(wantMerges) != 1 || len(wantMerges[0].sources) != 5 {
+ t.Fatal("The mixed fixture should merge every family except the two-repository one", wantMerges)
+ }
+ if _, ok := wantHistory.descriptions["kinds"]["learningKind"]; ok || wantHistory.descriptions["labels"]["title"] != "Go tests" || wantHistory.descriptions["labels"]["learningKind"] != "go:tests" {
+ t.Fatal("The legacy pair rule or the smallest description changed", wantHistory.descriptions)
+ }
+ random := rand.New(rand.NewSource(7))
+ for i := 0; i < 30; i++ {
+ random.Shuffle(len(evidence), func(a, b int) { evidence[a], evidence[b] = evidence[b], evidence[a] })
+ history, classes, merges := shuffledHistory(evidence, scopes, members)
+ if !reflect.DeepEqual(history, wantHistory) || !reflect.DeepEqual(classes, wantClasses) || !reflect.DeepEqual(merges, wantMerges) {
+ t.Fatalf("Shuffle %d changed the conclusion", i)
+ }
+ }
+}
+
+func TestProfileBackfillSkipsBadRecordsAndClassifiesTheRest(t *testing.T) {
+ g := testGate(t)
+ store, e := NewProfileStore(g.State, false)
+ if e != nil {
+ t.Fatal(e)
+ }
+ repository := gitRepository(t)
+ _, want := profileDirectoryIdentity(repository)
+ good := map[string]any{"title": "Go tests", "project": "fixture", "executable": "go", "directory": repository}
+ healthy := observeIn(t, store, "healthy", repository, "Go tests", "go")
+ recordProfile(t, store, strings.Repeat("1", 32), healthy, GiB, 1, true, profileZero(), 0)
+ mustWrite(t, filepath.Join(g.Metadata, strings.Repeat("1", 32)+".json"), good)
+ for i, mode := range []os.FileMode{0600, 0644} {
+ identity := similarityIdentity(fmt.Sprint("damaged ", i))
+ token := strings.Repeat(fmt.Sprint(i+2), 32)
+ recordProfile(t, store, token, identity, GiB, 1, true, profileZero(), 0)
+ path := filepath.Join(g.Metadata, token+".json")
+ if i == 0 {
+ if e = os.WriteFile(path, []byte("{truncated"), mode); e != nil {
+ t.Fatal(e)
+ }
+ } else {
+ mustWrite(t, path, good)
+ if e = os.Chmod(path, mode); e != nil {
+ t.Fatal(e)
+ }
+ }
+ }
+ // A readable file whose token has no run and a corrupt scheduling sidecar.
+ mustWrite(t, filepath.Join(g.Metadata, strings.Repeat("4", 32)+".json"), good)
+ if e = os.MkdirAll(filepath.Join(g.State, "workloads"), 0700); e != nil {
+ t.Fatal(e)
+ }
+ if e = os.WriteFile(filepath.Join(g.State, "workloads", strings.Repeat("4", 32)+".json"), []byte("{truncated"), 0600); e != nil {
+ t.Fatal(e)
+ }
+ if e = store.backfillSimilarity(g); e != nil {
+ t.Fatal("A bad record held the backfill", e)
+ }
+ if e = store.connection(false, func(c profileSQL) error {
+ row, e := profileRow(c, "SELECT version FROM learning_similarity_state WHERE id=1")
+ if metric(row["version"]) != profileClassificationVersion {
+ t.Fatal("The backfill was not marked complete", row)
+ }
+ return e
+ }); e != nil {
+ t.Fatal(e)
+ }
+ if class, ok := storedClass(t, store, healthy); !ok || class.repository != want {
+ t.Fatal("The healthy family was not classified", class, ok)
+ }
+}
+
+func TestProfileInvalidLegacyRowsAreAbsentEverywhere(t *testing.T) {
+ g := testGate(t)
+ store, e := NewProfileStore(g.State, false)
+ if e != nil {
+ t.Fatal(e)
+ }
+ repository := gitRepository(t)
+ for i, row := range [][2]string{{profileDigest("scope"), "ruby:deploy"}, {"not a scope", "go:tests"}} {
+ identity := similarityIdentity(fmt.Sprint("broken ", i))
+ if e = store.Observe(identity, Object{}); e != nil {
+ t.Fatal(e)
+ }
+ if e = store.connection(true, func(c profileSQL) error {
+ return profileExec(c, "INSERT INTO learning_similarity VALUES (?,?,?)", identity["family"], row[0], row[1])
+ }); e != nil {
+ t.Fatal(e)
+ }
+ if e = store.Observe(identity, profileSimilarityDescription(Object{"title": "Lint", "executable": "pnpm", "directory": repository}, map[string]profileIdentity{})); e != nil {
+ t.Fatal("An invalid legacy row failed an observation", e)
+ }
+ if class, ok := storedClass(t, store, identity); !ok || class.tool != "javascript" || class.operation != "lint" {
+ t.Fatal("An invalid legacy row decided the class", class)
+ }
+ if _, e = store.Match(identity); e != nil {
+ t.Fatal(e)
+ }
+ }
+ if _, e = LearningRegroup(g, false, nil); e != nil {
+ t.Fatal("An invalid legacy row failed the preview", e)
+ }
+ if e = store.backfillSimilarity(g); e != nil {
+ t.Fatal("An invalid legacy row held the backfill", e)
+ }
+}
+
+func TestProfileRunsOnlyFamilyInTwoDirectoriesIsRegisteredAndClassified(t *testing.T) {
+ g := testGate(t)
+ store, e := NewProfileStore(g.State, false)
+ if e != nil {
+ t.Fatal(e)
+ }
+ repository := gitRepository(t)
+ _, want := profileDirectoryIdentity(repository)
+ identity := similarityIdentity("runs only")
+ for i, directory := range []string{repository, filepath.Join(repository, "sub")} {
+ token := fmt.Sprintf("%032x", i+1)
+ recordProfile(t, store, token, identity, GiB, 1, true, profileZero(), 0)
+ mustWrite(t, filepath.Join(g.Metadata, token+".json"), Object{"title": "Go tests", "project": "fixture", "executable": "go", "directory": directory})
+ }
+ if _, e = LearningRegroup(g, true, Object{"revision": 0}); e != nil {
+ t.Fatal(e)
+ }
+ if class, ok := storedClass(t, store, identity); !ok || class.repository != want || class.tool != "go" || class.operation != "tests" {
+ t.Fatal("A family in two directories was not registered and classified", class, ok)
+ }
+ if familyHasLegacyRow(t, store, identity) {
+ t.Fatal("Two scopes produced a legacy pair")
+ }
+}
+
+func TestProfileEveryLegacyKindYieldsAValidClassAndInvalidClassesAreRefused(t *testing.T) {
+ for _, tool := range profileTools {
+ for _, operation := range profileOperations {
+ if kind := tool + ":" + operation; validProfileSimilarityKind(kind) {
+ if class := profileBackfillClass(Object{"family": "x"}, kind, "", &profileHistory{}); !class.valid() || class.kind() != kind {
+ t.Errorf("%s produced %+v", kind, class)
+ }
+ }
+ }
+ }
+ store := fixtureProfileStore(t)
+ if e := store.connection(true, func(c profileSQL) error {
+ return profileStoreClass(c, "x", profileClass{tool: "ruby", operation: "tests"})
+ }); e == nil {
+ t.Fatal("An invalid class was stored")
+ }
+}
+
+func TestProfileUnknownClassIsReplaceableOnce(t *testing.T) {
+ store := fixtureProfileStore(t)
+ repository := profileDigest("repository")
+ store.connection(true, func(c profileSQL) error {
+ for _, step := range []profileClass{{"", "other", "other"}, {"", "go", "tests"}, {repository, "javascript", "lint"}} {
+ if e := profileStoreClass(c, "x", step); e != nil {
+ t.Fatal(e)
+ }
+ }
+ row, e := profileRow(c, "SELECT * FROM learning_classification WHERE family='x'")
+ if class, ok := profileStoredClass(row); !ok || class != (profileClass{repository, "go", "tests"}) {
+ t.Fatal("The unknown class was not replaced exactly once, or the repository was not filled", class, e)
+ }
+ return e
+ })
+}
+
+func TestProfileOperationVocabularyMatchesDashboardSchema(t *testing.T) {
+ source, e := os.ReadFile(filepath.Join("..", "..", "remote", "static", "schema.mjs"))
+ if e != nil {
+ t.Fatal(e)
+ }
+ match := regexp.MustCompile(`operationSchema = z\.literal\(\[([^\]]*)\]`).FindSubmatch(source)
+ if match == nil {
+ t.Fatal("schema.mjs has no operationSchema literal list")
+ }
+ listed := []string{}
+ for _, word := range strings.Split(string(match[1]), ",") {
+ listed = append(listed, strings.Trim(strings.TrimSpace(word), `"`))
+ }
+ if !reflect.DeepEqual(listed, ProfileOperations()) {
+ t.Fatalf("schema.mjs operations %v differ from ProfileOperations %v", listed, ProfileOperations())
+ }
+}
+
+func TestProfileBackfillCountsHistoricalWrappersAsTheirRuntime(t *testing.T) {
+ for executable, tool := range map[string]string{"env": "shell", "/usr/bin/env": "shell", "fnm": "javascript", "go": "go", "run.sh": "shell", "mystery": "other"} {
+ member := Object{"family": "f", "executable": executable, "title": "Run tests"}
+ if class := profileBackfillClass(member, "", "", &profileHistory{}); class.tool != tool || class.operation != "tests" {
+ t.Fatalf("%s classified as %+v, want %s/tests", executable, class, tool)
+ }
+ }
+ // A new job shows what the wrapper really ran.
+ if tool, _ := profileClassify([]string{"env", "FOO=1", "mystery"}, "Run tests", "env"); tool != "other" {
+ t.Fatalf("observation of a wrapped unknown tool got %s", tool)
+ }
+}
+
+func TestProfileMetadataWithOnlyADirectoryPassesValidation(t *testing.T) {
+ clean := profileSimilarityDescription(Object{"directory": gitRepository(t)}, map[string]profileIdentity{})
+ if e := profileValidateSimilarityDescription(Object{}, clean); e != nil {
+ t.Fatal("Metadata origin accepted would hold every tick:", e)
+ }
+}
+
+func TestProfileObservationOfOldMetadataMatchesTheBackfill(t *testing.T) {
+ clean := profileSimilarityDescription(Object{"title": "Run tests", "executable": "env"}, map[string]profileIdentity{})
+ if clean["learningTool"] != "shell" || clean["learningOperation"] != "tests" {
+ t.Fatal("An old wrapper job classifies differently when a tick sees it first", clean)
+ }
+}
+
+func TestProfileAggregatePrefersDescribedMetadataOverBlankFields(t *testing.T) {
+ described := Object{"title": "Go tests", "project": "fixture", "executable": "go"}
+ for _, evidence := range [][]profileEvidence{{{family: "f", clean: described}, {family: "f", clean: Object{}}}, {{family: "f", clean: Object{}}, {family: "f", clean: described}}} {
+ if got := profileAggregate(evidence, nil).descriptions["f"]; got["title"] != "Go tests" || got["executable"] != "go" {
+ t.Fatal("A blank file registered the family", got)
+ }
+ }
+ // A sparse file loses even when its first fields sort earlier.
+ for _, sparse := range []Object{{"project": "a"}, {"project": "fixture", "title": "A"}, {"title": "A"}} {
+ evidence := []profileEvidence{{family: "f", clean: sparse}, {family: "f", clean: described}}
+ if got := profileAggregate(evidence, nil).descriptions["f"]; got["executable"] != "go" {
+ t.Fatal("A sparse file registered the family", sparse, got)
+ }
+ }
+}
+
+func TestProfileTwoRepositoriesNeverFallBackToAnAlias(t *testing.T) {
+ scope, repository := strings.Repeat("a", 64), strings.Repeat("b", 64)
+ history := &profileHistory{repositories: map[string]string{}, aliases: map[string]string{scope: repository}, descriptions: map[string]Object{"f": {"learningScope": scope}}}
+ if class := profileBackfillClass(Object{"family": "f"}, "", "", history); class.repository != repository {
+ t.Fatal("Fixture: an unverified family should use its alias", class)
+ }
+ history.repositories["f"] = ""
+ if class := profileBackfillClass(Object{"family": "f"}, "", "", history); class.repository != "" {
+ t.Fatal("A family verified in two repositories took an alias", class)
+ }
+}
+
+func TestProfileBackfillSkipsAFamilyWhenTheRegistryIsFull(t *testing.T) {
+ g := testGate(t)
+ store, e := NewProfileStore(g.State, false)
+ if e != nil {
+ t.Fatal(e)
+ }
+ if e = store.connection(true, func(c profileSQL) error {
+ for i := 0; i < maxRegisteredFamilies; i++ {
+ family := profileDigest([]any{"filler", i})
+ if e := profileExec(c, "INSERT INTO learning_profiles VALUES (?,NULL,NULL,NULL,1,0,?)", family, 1e12); e != nil {
+ return e
+ }
+ if e := profileExec(c, "INSERT INTO learning_members VALUES (?,?,NULL,NULL,NULL,?)", family, family, 1e12); e != nil {
+ return e
+ }
+ }
+ return profileExec(c, "UPDATE learning_profiles SET pinned=1")
+ }); e != nil {
+ t.Fatal(e)
+ }
+ identity := similarityIdentity("no room")
+ token := fmt.Sprintf("%032x", 1)
+ recordProfile(t, store, token, identity, GiB, 1, true, profileZero(), 0)
+ mustWrite(t, filepath.Join(g.Metadata, token+".json"), Object{"title": "Go tests", "project": "fixture", "executable": "go", "directory": gitRepository(t)})
+ if e = store.backfillSimilarity(g); e != nil {
+ t.Fatal("A full registry stopped the backfill:", e)
+ }
+ if e = store.connection(false, func(c profileSQL) error {
+ row, e := profileRow(c, "SELECT version FROM learning_similarity_state WHERE id=1")
+ if e == nil && metric(row["version"]) != profileClassificationVersion {
+ t.Error("The pass was not marked complete", row)
+ }
+ return e
+ }); e != nil {
+ t.Fatal(e)
+ }
+ if familyHasLegacyRow(t, store, identity) {
+ t.Fatal("A family with no registry slot gained a legacy row")
+ }
+ if e = store.connection(false, func(c profileSQL) error {
+ row, e := profileRow(c, "SELECT 1 FROM learning_members WHERE family=?", identity["family"])
+ if e == nil && row != nil {
+ t.Error("A family was registered past the cap")
+ }
+ return e
+ }); e != nil {
+ t.Fatal(e)
+ }
+}
+
+func TestControllerForgetsScopeAnswersAfterTheirLifetime(t *testing.T) {
+ c := &Controller{}
+ c.scopes()["checkout"] = profileIdentity{scope: "s"}
+ if _, kept := c.scopes()["checkout"]; !kept {
+ t.Fatal("A fresh answer was dropped")
+ }
+ c.scopesChecked = time.Now().Add(-learningScopeLifetime - time.Second)
+ if _, kept := c.scopes()["checkout"]; kept {
+ t.Fatal("A failed git probe would last for the controller's lifetime")
+ }
+}
+
+func TestFailedHistoryPassDoesNotHoldAdmissions(t *testing.T) {
+ g, _ := schedulerFixtureGate(t)
+ store, e := NewProfileStore(g.State, false)
+ if e != nil {
+ t.Fatal(e)
+ }
+ // A vocabulary this build does not know fails the catalog read the pass needs.
+ family := profileDigest([]any{"future"})
+ if e = store.connection(true, func(c profileSQL) error {
+ for _, statement := range []struct {
+ query string
+ args []any
+ }{
+ {"INSERT INTO learning_profiles VALUES (?,NULL,NULL,NULL,1,0,?)", []any{family, 1e12}},
+ {"INSERT INTO learning_members VALUES (?,?,NULL,NULL,NULL,?)", []any{family, family, 1e12}},
+ {"INSERT INTO learning_classification VALUES (?,NULL,'future','other')", []any{family}},
+ {"UPDATE learning_similarity_state SET version=1 WHERE id=1", nil},
+ } {
+ if e := profileExec(c, statement.query, statement.args...); e != nil {
+ return e
+ }
+ }
+ return nil
+ }); e != nil {
+ t.Fatal(e)
+ }
+ sampler := NewResourceSampler(g.State, &admissionFixtureBackend{pressure: "normal", processCount: map[int]uint64{}})
+ if e = serveScheduler(g, sampler, true); e != nil {
+ t.Fatal("A failed history pass stopped the controller:", e)
+ }
+ status, e := ReadPrivate(filepath.Join(g.State, "resource-status.json"))
+ if e != nil || status["host"] == nil || status["reason"] == "Resource collection failed; admissions held" {
+ t.Fatalf("A failed history pass held admissions: %v %v", status, e)
+ }
+ if e = store.connection(false, func(c profileSQL) error {
+ row, e := profileRow(c, "SELECT version FROM learning_similarity_state WHERE id=1")
+ if e == nil && metric(row["version"]) >= profileClassificationVersion {
+ t.Error("Fixture: the pass should have failed, not completed")
+ }
+ return e
+ }); e != nil {
+ t.Fatal(e)
+ }
+}
+
+func TestHistoryPassWaitsBeforeRetrying(t *testing.T) {
+ g := testGate(t)
+ store, e := NewProfileStore(g.State, false)
+ if e != nil {
+ t.Fatal(e)
+ }
+ family := profileDigest([]any{"future"})
+ if e = store.connection(true, func(c profileSQL) error {
+ if e := profileExec(c, "INSERT INTO learning_profiles VALUES (?,NULL,NULL,NULL,1,0,?)", family, 1e12); e != nil {
+ return e
+ }
+ if e := profileExec(c, "INSERT INTO learning_members VALUES (?,?,NULL,NULL,NULL,?)", family, family, 1e12); e != nil {
+ return e
+ }
+ if e := profileExec(c, "INSERT INTO learning_classification VALUES (?,NULL,'future','other')", family); e != nil {
+ return e
+ }
+ return profileExec(c, "UPDATE learning_similarity_state SET version=1 WHERE id=1")
+ }); e != nil {
+ t.Fatal(e)
+ }
+ c := &Controller{store: store}
+ start := time.Now()
+ if c.historyPass(g, start) {
+ t.Fatal("Fixture: the pass should fail")
+ }
+ retry := c.retryHistory
+ if !retry.Equal(start.Add(learningScopeLifetime)) {
+ t.Fatal("A failed pass set no retry time", retry)
+ }
+ if c.historyPass(g, start.Add(time.Minute)) || !c.retryHistory.Equal(retry) {
+ t.Fatal("The pass ran again before its retry time")
+ }
+ if e = store.connection(true, func(c profileSQL) error {
+ return profileExec(c, "DELETE FROM learning_classification WHERE family=?", family)
+ }); e != nil {
+ t.Fatal(e)
+ }
+ if c.historyPass(g, start.Add(time.Minute)) {
+ t.Fatal("A repaired store ran before its retry time")
+ }
+ if !c.historyPass(g, retry) {
+ t.Fatal("The pass did not retry at its retry time")
+ }
+}
diff --git a/internal/backend/profiles_learning.go b/internal/backend/profiles_learning.go
index df42e06..b34d6f9 100644
--- a/internal/backend/profiles_learning.go
+++ b/internal/backend/profiles_learning.go
@@ -23,11 +23,15 @@ func profileDataset(c profileSQL) (map[string]Object, map[string]*profileGroup,
for _, row := range rows {
profiles[text(row["id"])] = row
}
- rows, e = profileRows(c, "SELECT * FROM learning_members")
+ rows, e = profileMemberRows(c, "")
if e != nil {
return nil, nil, e
}
for _, row := range rows {
+ // Only the catalog fails on a classification it cannot interpret; grouping skips it.
+ if _, ok := profileStoredClass(Object{"repository": row["learningRepository"], "tool": row["learningTool"], "operation": row["learningOperation"]}); !ok && row["learningTool"] != nil {
+ return nil, nil, fmt.Errorf("Invalid stored workload classification")
+ }
members[text(row["family"])] = row
}
}
@@ -100,15 +104,78 @@ func profileSummary(profile Object, group *profileGroup) (Object, error) {
incomplete++
}
}
- return Object{"id": profile["id"], "label": profileGroupLabel(profile, group.members), "memberCount": len(group.members), "variantCount": len(variants), "successCount": successes, "incompleteCount": incomplete, "peakMemoryBytes": int64(peakMemory), "peakCpuCores": peakCPU, "lastRunAt": last, "memoryFloorBytes": profile["memory_floor"], "cpuFloorCores": profile["cpu_floor"]}, nil
+ return Object{"id": profile["id"], "label": profileGroupLabel(profile, group.members), "project": profileProject(group.members), "operation": profileGroupOperation(group.members), "memberCount": len(group.members), "variantCount": len(variants), "successCount": successes, "incompleteCount": incomplete, "peakMemoryBytes": int64(peakMemory), "peakCpuCores": peakCPU, "lastRunAt": last, "memoryFloorBytes": profile["memory_floor"], "cpuFloorCores": profile["cpu_floor"]}, nil
+}
+
+// ProfileFilter narrows the catalog before paging. OtherProjects selects the
+// projects the hierarchy collapses; it cannot be combined with Project.
+type ProfileFilter struct {
+ Project, Operation string
+ OtherProjects bool
+}
+
+func (f ProfileFilter) validate() error {
+ if f.Project != "" && (len([]rune(f.Project)) > 120 || !profilePrintable(f.Project) || strings.TrimSpace(f.Project) != f.Project) {
+ return fmt.Errorf("Invalid workload project filter")
+ }
+ if f.Operation != "" && !containsString(profileOperations, f.Operation) {
+ return fmt.Errorf("Invalid workload operation filter")
+ }
+ if f.OtherProjects && f.Project != "" {
+ return fmt.Errorf("Choose a project or the other projects, not both")
+ }
+ return nil
+}
+
+func (f ProfileFilter) any() bool { return f.Project != "" || f.Operation != "" || f.OtherProjects }
+
+// profileProject is the group's display name: its most common member project.
+func profileProject(members []Object) string {
+ counts := map[string]int{}
+ for _, member := range members {
+ if name := text(member["project"]); name != "" {
+ counts[name]++
+ }
+ }
+ best := ""
+ for name, count := range counts {
+ if count > counts[best] || count == counts[best] && name < best {
+ best = name
+ }
+ }
+ if best == "" {
+ return "Unknown project"
+ }
+ return best
+}
+
+// profileGroupOperation is the shared operation of all members, else "other".
+func profileGroupOperation(members []Object) string {
+ operation := ""
+ for _, member := range members {
+ value := text(member["learningOperation"])
+ if value == "" || operation != "" && value != operation {
+ return "other"
+ }
+ operation = value
+ }
+ return profileDefault(operation)
}
+
func (s *ProfileStore) Catalog(offset, limit int, query string) (Object, error) {
+ return s.CatalogFiltered(offset, limit, query, ProfileFilter{})
+}
+
+func (s *ProfileStore) CatalogFiltered(offset, limit int, query string, filter ProfileFilter) (Object, error) {
if offset < 0 || limit < 1 || limit > 100 {
return nil, fmt.Errorf("Invalid workload profile page")
}
if len([]rune(query)) > 120 || !profilePrintable(query) {
return nil, fmt.Errorf("Invalid workload profile search")
}
+ if e := filter.validate(); e != nil {
+ return nil, e
+ }
result := Object{"revision": 0, "profiles": []Object{}, "total": 0, "offset": offset, "limit": limit}
if s.empty {
return result, nil
@@ -122,13 +189,25 @@ func (s *ProfileStore) Catalog(offset, limit int, query string) (Object, error)
if e != nil {
return e
}
- entries := []Object{}
- needle := strings.TrimSpace(cases.Fold().String(query))
+ all := []Object{}
for id, profile := range profiles {
summary, e := profileSummary(profile, groups[id])
if e != nil {
return e
}
+ all = append(all, summary)
+ }
+ collapsed := map[string]bool{}
+ if filter.OtherProjects {
+ collapsed = profileCollapsedProjects(profileProjectTotals(all))
+ }
+ entries := []Object{}
+ needle := strings.TrimSpace(cases.Fold().String(query))
+ for _, summary := range all {
+ id := text(summary["id"])
+ if filter.Project != "" && text(summary["project"]) != filter.Project || filter.Operation != "" && text(summary["operation"]) != filter.Operation || filter.OtherProjects && !collapsed[text(summary["project"])] {
+ continue
+ }
if needle == "" || strings.Contains(cases.Fold().String(text(summary["label"])), needle) || strings.Contains(id, needle) {
entries = append(entries, summary)
}
@@ -146,6 +225,126 @@ func (s *ProfileStore) Catalog(offset, limit int, query string) (Object, error)
return result, e
}
+// A project with this many workloads or fewer folds into the other-projects row.
+const profileCollapseThreshold = 2
+const maxProfileHierarchyRows = 100
+
+type profileTotals struct{ workloads, groups, runs int }
+
+func (t *profileTotals) add(summary Object) {
+ t.workloads += int(metric(summary["memberCount"]))
+ t.groups++
+ t.runs += int(metric(summary["successCount"]) + metric(summary["incompleteCount"]))
+}
+
+func profileProjectTotals(summaries []Object) map[string]*profileTotals {
+ totals := map[string]*profileTotals{}
+ for _, summary := range summaries {
+ project := text(summary["project"])
+ if totals[project] == nil {
+ totals[project] = &profileTotals{}
+ }
+ totals[project].add(summary)
+ }
+ return totals
+}
+
+// profileCollapsedProjects is the one rule for which projects fold together,
+// shared by the hierarchy and by the filter that lists the folded set.
+func profileCollapsedProjects(totals map[string]*profileTotals) map[string]bool {
+ collapsed := map[string]bool{}
+ for project, total := range totals {
+ if total.workloads <= profileCollapseThreshold {
+ collapsed[project] = true
+ }
+ }
+ return collapsed
+}
+
+func profileHierarchyRow(project string, collapsed bool, summaries []Object) Object {
+ total := &profileTotals{}
+ byOperation := map[string]*profileTotals{}
+ for _, summary := range summaries {
+ operation := text(summary["operation"])
+ if byOperation[operation] == nil {
+ byOperation[operation] = &profileTotals{}
+ }
+ byOperation[operation].add(summary)
+ total.add(summary)
+ }
+ operations := []Object{}
+ for _, operation := range profileOperations {
+ if t := byOperation[operation]; t != nil {
+ operations = append(operations, Object{"operation": operation, "workloadCount": t.workloads, "groupCount": t.groups, "runCount": t.runs})
+ }
+ }
+ sort.SliceStable(operations, func(i, j int) bool {
+ return metric(operations[i]["workloadCount"]) > metric(operations[j]["workloadCount"])
+ })
+ return Object{"project": project, "collapsed": collapsed, "workloadCount": total.workloads, "groupCount": total.groups, "runCount": total.runs, "operations": operations}
+}
+
+// Hierarchy lists projects with their operations, busiest first, in at most
+// limit rows including the other-projects row. Projects over the limit are
+// counted in total but not listed. Counts describe the same
+// groups the project and operation filters return.
+func (s *ProfileStore) Hierarchy(limit int) (Object, error) {
+ if limit < 1 || limit > maxProfileHierarchyRows {
+ return nil, fmt.Errorf("Invalid workload project limit")
+ }
+ result := Object{"revision": 0, "projects": []Object{}, "total": 0, "limit": limit}
+ if s.empty {
+ return result, nil
+ }
+ e := s.connection(false, func(c profileSQL) error {
+ revision, e := profileRevision(c)
+ if e != nil {
+ return e
+ }
+ profiles, groups, e := profileDataset(c)
+ if e != nil {
+ return e
+ }
+ byProject := map[string][]Object{}
+ all := []Object{}
+ for id, profile := range profiles {
+ summary, e := profileSummary(profile, groups[id])
+ if e != nil {
+ return e
+ }
+ byProject[text(summary["project"])] = append(byProject[text(summary["project"])], summary)
+ all = append(all, summary)
+ }
+ totals := profileProjectTotals(all)
+ collapsed := profileCollapsedProjects(totals)
+ rows, folded := []Object{}, []Object{}
+ for project, summaries := range byProject {
+ if collapsed[project] {
+ folded = append(folded, summaries...)
+ } else {
+ rows = append(rows, profileHierarchyRow(project, false, summaries))
+ }
+ }
+ sort.Slice(rows, func(i, j int) bool {
+ if metric(rows[i]["workloadCount"]) != metric(rows[j]["workloadCount"]) {
+ return metric(rows[i]["workloadCount"]) > metric(rows[j]["workloadCount"])
+ }
+ return text(rows[i]["project"]) < text(rows[j]["project"])
+ })
+ total := len(rows)
+ if len(folded) > 0 {
+ other := profileHierarchyRow("Other projects", true, folded)
+ other["projectCount"] = len(collapsed)
+ rows = append(rows, other)
+ }
+ // The limit counts every row returned, and truncation drops the other-projects row first.
+ rows = rows[:min(limit, len(rows))]
+ result = Object{"revision": revision, "projects": rows, "total": total, "limit": limit}
+ return nil
+ })
+ return result, e
+}
+
type ProfileDetailOptions struct {
Family string
MemberOffset int
@@ -350,7 +549,13 @@ func profileEditable(c profileSQL, id string) (Object, *profileGroup, error) {
return nil, nil, fmt.Errorf("Workload profile was not found; refresh the list")
}
for _, member := range group.members {
- if _, e = profileRegister(c, text(member["family"]), nil, true); e != nil {
+ family := text(member["family"])
+ if _, e = profileRegister(c, family, nil, true); e != nil {
+ return nil, nil, e
+ }
+ // A family that only had runs is a member now, and every member is classified.
+ legacy := profileValidLegacy(Object{"scope": member["learningScope"], "kind": member["learningKind"]})
+ if e = profileStoreClass(c, family, profileBackfillClass(member, text(legacy["kind"]), text(legacy["scope"]), &profileHistory{})); e != nil {
return nil, nil, e
}
}
diff --git a/internal/backend/profiles_regroup.go b/internal/backend/profiles_regroup.go
index 1c3bb68..27cf717 100644
--- a/internal/backend/profiles_regroup.go
+++ b/internal/backend/profiles_regroup.go
@@ -9,24 +9,54 @@ import (
"syscall"
)
+// profileEvidence is what one metadata file says, read once. Everything the
+// retroactive pass concludes is a pure function of a slice of these.
+type profileEvidence struct {
+ family string
+ clean Object
+ identity profileIdentity
+}
+
// Historical execution payloads are disposable. Join retained numeric runs to
// their private safe metadata by hashed token, without retaining raw commands.
-func (s *ProfileStore) historicalSimilarityDescriptions(g *Gate) (map[string]Object, error) {
- tokens := map[string]string{}
+// One bad file or sidecar is skipped: only an unsafe directory, cancellation
+// and database errors stop the pass.
+func (s *ProfileStore) historicalSimilarityDescriptions(g *Gate) (*profileHistory, error) {
+ tokens, scopes := map[string]string{}, map[string]string{}
if !s.empty {
if e := s.connection(false, func(c profileSQL) error {
rows, e := profileRows(c, "SELECT token,family FROM runs")
+ if e != nil {
+ return e
+ }
for _, row := range rows {
tokens[text(row["token"])] = text(row["family"])
}
+ if has, e := hasProfileSimilarity(c); e != nil || !has {
+ return e
+ }
+ rows, e = profileRows(c, "SELECT * FROM learning_similarity")
+ for _, row := range rows {
+ if profileValidLegacy(row) != nil {
+ scopes[text(row["family"])] = text(row["scope"])
+ }
+ }
return e
}); e != nil {
return nil, e
}
}
+ evidence, e := profileReadEvidence(g, tokens)
+ if e != nil {
+ return nil, e
+ }
+ return profileAggregate(evidence, scopes), nil
+}
+
+func profileReadEvidence(g *Gate, tokens map[string]string) ([]profileEvidence, error) {
info, e := os.Lstat(g.Metadata)
if os.IsNotExist(e) {
- return map[string]Object{}, nil
+ return nil, nil
}
if e != nil {
return nil, e
@@ -39,9 +69,7 @@ func (s *ProfileStore) historicalSimilarityDescriptions(g *Gate) (map[string]Obj
if e != nil {
return nil, e
}
- descriptions := map[string]Object{}
- conflicts := map[string]bool{}
- scopes := map[string]string{}
+ identities, evidence := map[string]profileIdentity{}, []profileEvidence{}
for _, entry := range entries {
if e := g.operationContext().Err(); e != nil {
return nil, e
@@ -54,99 +82,121 @@ func (s *ProfileStore) historicalSimilarityDescriptions(g *Gate) (map[string]Obj
if family == "" {
identity, e := SchedulingIdentity(g, value)
if e != nil {
- return nil, e
+ continue
}
family = text(identity["family"])
}
- if family == "" || conflicts[family] {
+ if family == "" {
continue
}
description, e := ReadPrivate(filepath.Join(g.Metadata, entry.Name()))
- if os.IsNotExist(e) {
- continue // A concurrent normal history cleanup may retire a file.
- }
if e != nil {
- return nil, e
- }
- clean := profileSimilarityDescription(description, scopes)
- if clean["learningScope"] == nil {
- continue
- }
- if prior := descriptions[family]; prior != nil && (prior["learningScope"] != clean["learningScope"] || prior["learningKind"] != clean["learningKind"]) {
- delete(descriptions, family)
- conflicts[family] = true
continue
}
- descriptions[family] = clean
+ evidence = append(evidence, profileEvidence{family, profileSimilarityDescription(description, identities), profileDescriptionIdentity(text(description["directory"]), identities)})
}
- return descriptions, nil
+ return evidence, g.operationContext().Err()
}
-func profileRegroupPlan(c profileSQL, descriptions map[string]Object) (Object, error) {
- profiles, groups, e := profileDataset(c)
- if e != nil {
- return nil, e
+// profileAggregate concludes from all the evidence at once, so file order cannot
+// matter. Live metadata has no scope of its own, so each file that still
+// verifies as a checkout teaches aliases to its repository, from the v1 scope
+// computed now and from the scope stored for its family. A stored scope can
+// outlive its checkout and differ from a fresh one, and it is how deleted
+// worktrees are recovered.
+func profileAggregate(evidence []profileEvidence, scopes map[string]string) *profileHistory {
+ history := &profileHistory{descriptions: map[string]Object{}, aliases: map[string]string{}, repositories: map[string]string{}}
+ type pair struct{ scope, kind string }
+ pairs, chosen := map[string]map[pair]bool{}, map[string]string{}
+ for _, item := range evidence {
+ profileAddAlias(history.aliases, item.identity.scope, item.identity.repository)
+ // shortcut: a family's stored scope is trusted, so a path reused by an unrelated repository can assign the wrong repository, upgrade when operators report that more than rarely.
+ profileAddAlias(history.aliases, scopes[item.family], item.identity.repository)
+ profileAddAlias(history.repositories, item.family, item.identity.repository)
+ if item.clean["learningScope"] != nil {
+ if pairs[item.family] == nil {
+ pairs[item.family] = map[pair]bool{}
+ }
+ pairs[item.family][pair{text(item.clean["learningScope"]), text(item.clean["learningKind"])}] = true
+ }
+ // The smallest description registers the family, whatever the file order.
+ // Fewer blank fields come first, so a sparse file never beats a described one.
+ key, blanks := "", byte('0')
+ for _, name := range []string{"project", "title", "executable"} {
+ key += text(item.clean[name]) + "\x00"
+ if text(item.clean[name]) == "" {
+ blanks++
+ }
+ }
+ key = string(blanks) + key
+ if prior, seen := chosen[item.family]; !seen || key < prior {
+ chosen[item.family] = key
+ description := Object{}
+ for _, name := range []string{"title", "project", "executable"} {
+ if item.clean[name] != nil {
+ description[name] = item.clean[name]
+ }
+ }
+ history.descriptions[item.family] = description
+ }
}
- signatures := map[string]Object{}
- for family, description := range descriptions {
- signatures[family] = description
+ // The legacy table is shared with old clients, which reject unknown kinds, so
+ // a family gets a row only when its files name exactly one scope and kind.
+ for family, set := range pairs {
+ for each := range set {
+ if len(set) == 1 {
+ history.descriptions[family]["learningScope"], history.descriptions[family]["learningKind"] = each.scope, each.kind
+ }
+ }
}
- has, e := hasProfileSimilarity(c)
+ return history
+}
+
+func profileRegroupPlan(c profileSQL, history *profileHistory) (Object, map[string]profileClass, error) {
+ profiles, groups, e := profileDataset(c)
if e != nil {
- return nil, e
+ return nil, nil, e
}
- if has {
- rows, e := profileRows(c, "SELECT * FROM learning_similarity")
+ registered := map[string]bool{}
+ if has, e := hasProfileLearning(c); e != nil {
+ return nil, nil, e
+ } else if has {
+ rows, e := profileRows(c, "SELECT family FROM learning_members")
if e != nil {
- return nil, e
+ return nil, nil, e
}
for _, row := range rows {
- if !profileHash.MatchString(text(row["scope"])) || !validProfileSimilarityKind(text(row["kind"])) {
- return nil, fmt.Errorf("Invalid stored workload similarity")
- }
- signatures[text(row["family"])] = Object{"learningScope": row["scope"], "learningKind": row["kind"]}
+ registered[text(row["family"])] = true
}
}
- compatible := map[string][]Object{}
+ // Apply registers a family that only has runs from its description, so
+ // the preview classifies it from the same description.
+ members, owners := []Object{}, []string{}
for id, group := range groups {
- if metric(profiles[id]["pinned"]) != 0 {
- continue
- }
- key := ""
- valid := true
for _, member := range group.members {
- signature := signatures[text(member["family"])]
- scope, kind := text(signature["learningScope"]), text(signature["learningKind"])
- candidate := scope + ":" + kind
- if !profileHash.MatchString(scope) || !validProfileSimilarityKind(kind) || (key != "" && candidate != key) {
- valid = false
- break
+ if description := history.descriptions[text(member["family"])]; !registered[text(member["family"])] && description != nil {
+ member = Object{"family": member["family"], "title": description["title"], "project": description["project"], "executable": description["executable"]}
}
- key = candidate
- }
- if valid && key != "" {
- compatible[key] = append(compatible[key], profiles[id])
+ members, owners = append(members, member), append(owners, id)
}
}
- keys := []string{}
- for key := range compatible {
- keys = append(keys, key)
+ classes, e := profileBackfillClasses(c, history, members)
+ if e != nil {
+ return nil, nil, e
}
- sort.Strings(keys)
- merges, families := 0, 0
- proposed := []Object{}
- for _, key := range keys {
- entries := compatible[key]
- if len(entries) < 2 {
- continue
- }
- sort.Slice(entries, func(i, j int) bool { return text(entries[i]["id"]) < text(entries[j]["id"]) })
- target := text(entries[0]["id"])
- members, successes, incomplete := 0, 0, 0
- for _, entry := range entries {
- group := groups[text(entry["id"])]
- members += len(group.members)
- for _, run := range group.runs {
+ planned := make([]profileMemberClass, 0, len(members))
+ for i, member := range members {
+ family := text(member["family"])
+ planned = append(planned, profileMemberClass{family: family, profile: owners[i], project: text(member["project"]), pinned: metric(profiles[owners[i]]["pinned"]) != 0, classified: true, class: classes[family]})
+ }
+ merges := profilePlanMerges(planned)
+ grouped, proposed, removed := 0, []Object{}, 0
+ for _, merge := range merges {
+ ids := append([]string{merge.target}, merge.sources...)
+ count, successes, incomplete := 0, 0, 0
+ for _, id := range ids {
+ count += len(groups[id].members)
+ for _, run := range groups[id].runs {
if metric(run["success"]) == 1 {
successes++
} else {
@@ -154,20 +204,18 @@ func profileRegroupPlan(c profileSQL, descriptions map[string]Object) (Object, e
}
}
}
- member := groups[target].members[0]
- kind := text(signatures[text(member["family"])]["learningKind"])
- proposed = append(proposed, Object{"id": target, "label": profileSimilarityLabel(kind, text(member["project"])), "memberCount": members, "successCount": successes, "incompleteCount": incomplete})
- merges += len(entries) - 1
- families += members
+ proposed = append(proposed, Object{"id": merge.target, "label": profileSimilarityLabel(merge.class.kind(), merge.project), "memberCount": count, "successCount": successes, "incompleteCount": incomplete})
+ grouped += count
+ removed += len(merge.sources)
}
revision, e := profileRevision(c)
if e != nil {
- return nil, e
+ return nil, nil, e
}
- return Object{"revision": revision, "profilesBefore": len(profiles), "profilesAfter": len(profiles) - merges, "familiesGrouped": families, "groups": proposed, "applied": false}, nil
+ return Object{"revision": revision, "profilesBefore": len(profiles), "profilesAfter": len(profiles) - removed, "familiesGrouped": grouped, "groups": proposed, "applied": false}, classes, nil
}
-func (s *ProfileStore) regroup(descriptions map[string]Object, revision int, apply bool) (Object, error) {
+func (s *ProfileStore) regroup(history *profileHistory, revision int, apply bool) (Object, error) {
if s.empty {
return Object{"revision": 0, "profilesBefore": 0, "profilesAfter": 0, "familiesGrouped": 0, "groups": []Object{}, "applied": false}, nil
}
@@ -179,7 +227,8 @@ func (s *ProfileStore) regroup(descriptions map[string]Object, revision int, app
}
}
var e error
- result, e = profileRegroupPlan(c, descriptions)
+ var classes map[string]profileClass
+ result, classes, e = profileRegroupPlan(c, history)
if e != nil || !apply {
return e
}
@@ -194,26 +243,39 @@ func (s *ProfileStore) regroup(descriptions map[string]Object, revision int, app
}
}
keys := []string{}
- for family := range descriptions {
+ for family := range history.descriptions {
if families[family] {
keys = append(keys, family)
}
}
sort.Strings(keys)
for _, family := range keys {
- clean := descriptions[family]
- member, e := profileRegister(c, family, clean, true)
+ clean := history.descriptions[family]
+ member, e := profileRegister(c, family, clean, false)
if e != nil {
return e
}
if member == nil {
- return fmt.Errorf("Historical workload registration failed")
+ continue // The registry is full; this family stays unregistered.
}
- if e = profileExec(c, "INSERT OR IGNORE INTO learning_similarity VALUES (?,?,?)", family, clean["learningScope"], clean["learningKind"]); e != nil {
- return e
+ if text(clean["learningKind"]) != "" {
+ if e = profileExec(c, "INSERT OR IGNORE INTO learning_similarity VALUES (?,?,?)", family, clean["learningScope"], clean["learningKind"]); e != nil {
+ return e
+ }
+ }
+ }
+ members, e := profileRows(c, "SELECT family FROM learning_members ORDER BY family")
+ if e != nil {
+ return e
+ }
+ for _, member := range members {
+ if class, ok := classes[text(member["family"])]; ok {
+ if e = profileStoreClass(c, text(member["family"]), class); e != nil {
+ return e
+ }
}
}
- changed, e := profileGroupSimilar(c)
+ changed, e := profileGroupSimilar(c, nil)
if e != nil {
return e
}
@@ -234,12 +296,12 @@ func LearningRegroup(g *Gate, apply bool, request Object) (Object, error) {
if e != nil {
return nil, e
}
- descriptions, e := store.historicalSimilarityDescriptions(g)
+ history, e := store.historicalSimilarityDescriptions(g)
if e != nil {
return nil, e
}
if !apply {
- return store.regroup(descriptions, 0, false)
+ return store.regroup(history, 0, false)
}
revision, ok := schedulingInteger(request["revision"])
if !ok || revision < 0 || len(request) != 1 {
@@ -270,7 +332,7 @@ func LearningRegroup(g *Gate, apply bool, request Object) (Object, error) {
}
// Rotate grants through the same guarded path as an operator correction.
// Active ownership and manual overrides survive; pending grants do not.
- plan, e := store.regroup(descriptions, revision, false)
+ plan, e := store.regroup(history, revision, false)
if e != nil {
return nil, e
}
@@ -279,14 +341,14 @@ func LearningRegroup(g *Gate, apply bool, request Object) (Object, error) {
return nil, e
}
}
- return store.regroup(descriptions, revision, true)
+ return store.regroup(history, revision, true)
}
func (s *ProfileStore) backfillSimilarity(g *Gate) error {
needed := false
if e := s.connection(false, func(c profileSQL) error {
row, e := profileRow(c, "SELECT version FROM learning_similarity_state WHERE id=1")
- needed = metric(row["version"]) < 1
+ needed = metric(row["version"]) < profileClassificationVersion
return e
}); e != nil || !needed {
return e
@@ -302,6 +364,6 @@ func (s *ProfileStore) backfillSimilarity(g *Gate) error {
// Only the updated controller marks its backfill complete. An explicit CLI
// regroup can precede a runtime upgrade while old clients still add families.
return s.connection(true, func(c profileSQL) error {
- return profileExec(c, "UPDATE learning_similarity_state SET version=1 WHERE id=1")
+ return profileExec(c, "UPDATE learning_similarity_state SET version=? WHERE id=1", profileClassificationVersion)
})
}
diff --git a/internal/backend/profiles_similarity.go b/internal/backend/profiles_similarity.go
index 08c68c2..f4eceda 100644
--- a/internal/backend/profiles_similarity.go
+++ b/internal/backend/profiles_similarity.go
@@ -4,7 +4,6 @@ import (
"fmt"
"os"
"path/filepath"
- "sort"
"strings"
"unicode"
)
@@ -26,11 +25,20 @@ func profileMemberRows(c profileSQL, where string, args ...any) ([]Object, error
if e != nil {
return nil, e
}
- query := "SELECT m.* FROM learning_members m"
+ classified, e := hasProfileClassification(c)
+ if e != nil {
+ return nil, e
+ }
+ columns, joins := "m.*", ""
if has {
- query = "SELECT m.*,s.scope AS learningScope,s.kind AS learningKind FROM learning_members m LEFT JOIN learning_similarity s ON s.family=m.family"
+ columns += ",s.scope AS learningScope,s.kind AS learningKind"
+ joins += " LEFT JOIN learning_similarity s ON s.family=m.family"
}
- return profileRows(c, query+where, args...)
+ if classified {
+ columns += ",k.repository AS learningRepository,k.tool AS learningTool,k.operation AS learningOperation"
+ joins += " LEFT JOIN learning_classification k ON k.family=m.family"
+ }
+ return profileRows(c, "SELECT "+columns+" FROM learning_members m"+joins+where, args...)
}
func profileMember(c profileSQL, family string) (Object, error) {
@@ -42,8 +50,16 @@ func profileMember(c profileSQL, family string) (Object, error) {
}
func profileDirectoryScope(directory string) string {
+ scope, _ := profileDirectoryIdentity(directory)
+ return scope
+}
+
+// profileDirectoryIdentity returns the v1 scope (repository plus relative
+// directory) and the repository digest alone. The repository is empty unless
+// git verifies it; a basename or path pattern never establishes one.
+func profileDirectoryIdentity(directory string) (string, string) {
if directory == "" || !filepath.IsAbs(directory) {
- return ""
+ return "", ""
}
location := strings.Split(profileGit(directory, "rev-parse", "--path-format=absolute", "--show-toplevel", "--git-common-dir"), "\n")
if len(location) == 2 {
@@ -55,16 +71,20 @@ func profileDirectoryScope(directory string) string {
relative, inside := profileUnder(resolved, root)
if err == nil && inside {
dev, ino := profileStatIdentity(info)
- return profileDigest([]any{"learning-scope-v1", "git", dev, ino, relative})
+ return profileDigest([]any{"learning-scope-v1", "git", dev, ino, relative}), profileDigest([]any{"learning-repository-v1", dev, ino})
}
}
}
- // Deleted checkouts can still share evidence within their original directory.
- // Never infer repository identity from a project basename or a path pattern.
+ return profileMissingDirectoryIdentity(directory)
+}
+
+// Deleted checkouts can still share evidence within their original directory.
+// Never infer repository identity from a project basename or a path pattern.
+func profileMissingDirectoryIdentity(directory string) (string, string) {
if resolved, e := filepath.EvalSymlinks(directory); e == nil {
directory = resolved
}
- return profileDigest([]any{"learning-scope-v1", "directory", filepath.Clean(directory)})
+ return profileDigest([]any{"learning-scope-v1", "directory", filepath.Clean(directory)}), ""
}
func profileToolClass(executable string) string {
@@ -89,6 +109,15 @@ func profileSimilarityKind(title, executable string) string {
if tool == "" {
return ""
}
+ if operation := profileOperation(title); operation != "" {
+ return tool + ":" + operation
+ }
+ return ""
+}
+
+// profileOperation returns the one operation a title names, or "" when it names
+// none or several. It needs no recognized tool.
+func profileOperation(title string) string {
words := map[string]bool{}
for _, word := range strings.FieldsFunc(strings.ToLower(title), func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) }) {
words[word] = true
@@ -140,7 +169,7 @@ func profileSimilarityKind(title, executable string) string {
if len(operations) != 1 {
return ""
}
- return tool + ":" + operations[0]
+ return operations[0]
}
func validProfileSimilarityKind(kind string) bool {
@@ -151,34 +180,10 @@ func validProfileSimilarityKind(kind string) bool {
func profileCommandKind(argv []string, title, executable string) string {
// Recognize simple wrappers without interpreting or executing shell programs.
- command := argv
- for len(command) > 0 {
- switch filepath.Base(command[0]) {
- case "env":
- command = command[1:]
- for len(command) > 0 && (command[0] == "--" || (!strings.HasPrefix(command[0], "-") && strings.Contains(command[0], "="))) {
- command = command[1:]
- }
- case "fnm":
- if len(command) < 2 || command[1] != "exec" {
- return profileSimilarityKind(title, executable)
- }
- separator := -1
- for i, arg := range command[2:] {
- if arg == "--" {
- separator = i + 2
- break
- }
- }
- if separator < 0 {
- return profileSimilarityKind(title, executable)
- }
- command = command[separator+1:]
- default:
- goto unwrapped
- }
+ command, ok := profileUnwrap(argv)
+ if !ok {
+ return profileSimilarityKind(title, executable)
}
-unwrapped:
if len(command) > 0 {
if filepath.Base(command[0]) == "cargo" && len(command) > 1 && command[1] == "bench" {
return "cargo:benchmark"
@@ -223,84 +228,68 @@ func profileSimilarityLabel(kind, project string) string {
}
func profileRememberSimilarity(c profileSQL, family string, clean Object) (bool, error) {
- scope, kind := text(clean["learningScope"]), text(clean["learningKind"])
- if scope == "" || kind == "" {
- return false, nil
+ class := profileClass{text(clean["learningRepository"]), text(clean["learningTool"]), text(clean["learningOperation"])}
+ // A family that already has a legacy kind keeps it: the controller observes
+ // before it backfills, and a job description no longer carries that kind.
+ if has, e := hasProfileSimilarity(c); e != nil {
+ return false, e
+ } else if has {
+ row, e := profileRow(c, "SELECT scope,kind FROM learning_similarity WHERE family=?", family)
+ if e != nil {
+ return false, e
+ }
+ if legacy := profileValidLegacy(row); legacy != nil {
+ class.tool, class.operation, _ = strings.Cut(text(legacy["kind"]), ":")
+ }
}
// Once registered, a label change cannot reclassify measured history.
- if e := profileExec(c, "INSERT OR IGNORE INTO learning_similarity VALUES (?,?,?)", family, scope, kind); e != nil {
+ if scope, kind := text(clean["learningScope"]), text(clean["learningKind"]); scope != "" && kind != "" {
+ if e := profileExec(c, "INSERT OR IGNORE INTO learning_similarity VALUES (?,?,?)", family, scope, kind); e != nil {
+ return false, e
+ }
+ }
+ // A description with nothing in it cannot be classified yet. Backfill or a
+ // later observation will, rather than storing a lasting other/other.
+ if clean["learningTool"] == nil {
+ return false, nil
+ }
+ if e := profileStoreClass(c, family, class); e != nil {
return false, e
}
- stored, e := profileRow(c, "SELECT scope,kind FROM learning_similarity WHERE family=?", family)
+ row, e := profileRow(c, "SELECT repository,tool,operation FROM learning_classification WHERE family=?", family)
if e != nil {
return false, e
}
- return profileGroupSimilar(c, stored)
+ stored, ok := profileStoredClass(row)
+ if !ok || !stored.shareable() {
+ return false, nil
+ }
+ return profileGroupSimilar(c, &stored)
}
-// profileGroupSimilar is deterministic, idempotent and transactional. Pinned
-// profiles (including explicit splits and resets) are never automatic targets.
-func profileGroupSimilar(c profileSQL, signature ...Object) (bool, error) {
- query := `WITH eligible AS (SELECT m.profile,s.scope,s.kind FROM learning_members m JOIN learning_profiles p ON p.id=m.profile JOIN learning_similarity s ON s.family=m.family WHERE p.pinned=0`
- args := []any{}
- if len(signature) > 0 {
- query += " AND s.scope=? AND s.kind=?"
- args = append(args, signature[0]["scope"], signature[0]["kind"])
- }
- // Prove that every member of a profile has this signature once per profile,
- // rather than rescanning a large shared profile for each individual member.
- query += ` GROUP BY m.profile,s.scope,s.kind HAVING COUNT(*)=(SELECT COUNT(*) FROM learning_members all_members WHERE all_members.profile=m.profile)) SELECT m.family,m.profile,m.project,e.scope,e.kind FROM eligible e JOIN learning_members m ON m.profile=e.profile ORDER BY m.profile,m.family`
- rows, e := profileRows(c, query, args...)
- if e != nil {
- return false, e
- }
- groups := map[string][]Object{}
- for _, row := range rows {
- if !profileHash.MatchString(text(row["scope"])) || !validProfileSimilarityKind(text(row["kind"])) {
- return false, fmt.Errorf("Invalid stored workload similarity")
- }
- key := text(row["scope"]) + ":" + text(row["kind"])
- groups[key] = append(groups[key], row)
- }
- keys := make([]string, 0, len(groups))
- for key := range groups {
- keys = append(keys, key)
- }
- sort.Strings(keys)
- changed := false
- for _, key := range keys {
- members := groups[key]
- if len(members) < 2 {
- continue
- }
- target := text(members[0]["profile"])
- groupChanged := false
- for _, member := range members[1:] {
- source := text(member["profile"])
- if source == target {
- continue
- }
- if e = profileExec(c, "UPDATE learning_members SET profile=? WHERE family=?", target, member["family"]); e != nil {
- return false, e
- }
- // Do not delete an automatic profile while it still owns another family.
- if e = profileExec(c, "DELETE FROM learning_profiles WHERE id=? AND NOT EXISTS(SELECT 1 FROM learning_members WHERE profile=?)", source, source); e != nil {
- return false, e
- }
- changed = true
- groupChanged = true
- }
- if groupChanged {
- label := profileSimilarityLabel(text(members[0]["kind"]), text(members[0]["project"]))
- if e = profileExec(c, "UPDATE learning_profiles SET label=?,updated=? WHERE id=? AND pinned=0", label, resourceNow(), target); e != nil {
- return false, e
- }
+// profileDescriptionIdentity computes the directory's v1 scope and repository
+// once per checkout, keyed by device and inode so renames do not go stale.
+func profileDescriptionIdentity(directory string, identities map[string]profileIdentity) profileIdentity {
+ cacheKey := directory
+ info, statError := os.Stat(directory)
+ if statError == nil {
+ dev, ino := profileStatIdentity(info)
+ cacheKey = profileDigest([]any{directory, dev, ino})
+ }
+ identity, ok := identities[cacheKey]
+ if !ok {
+ if statError != nil && filepath.IsAbs(directory) {
+ // Nothing to verify, so do not spend a git process on it.
+ identity.scope, identity.repository = profileMissingDirectoryIdentity(directory)
+ } else {
+ identity.scope, identity.repository = profileDirectoryIdentity(directory)
}
+ identities[cacheKey] = identity
}
- return changed, nil
+ return identity
}
-func profileSimilarityDescription(description Object, scopes map[string]string) Object {
+func profileSimilarityDescription(description Object, identities map[string]profileIdentity) Object {
result := Object{}
for _, key := range []string{"title", "project", "executable"} {
limit := 80
@@ -315,38 +304,57 @@ func profileSimilarityDescription(description Object, scopes map[string]string)
if !validProfileSimilarityKind(kind) {
kind = profileSimilarityKind(text(result["title"]), text(result["executable"]))
}
- if kind == "" {
- return result
+ identity := profileDescriptionIdentity(text(description["directory"]), identities)
+ if scope := text(description["learningScope"]); profileHash.MatchString(scope) {
+ identity.scope = scope
+ }
+ if repository := text(description["learningRepository"]); profileHash.MatchString(repository) {
+ identity.repository = repository
}
- scope := text(description["learningScope"])
- if !profileHash.MatchString(scope) {
- directory := text(description["directory"])
- cacheKey := directory
- if info, e := os.Stat(directory); e == nil {
- dev, ino := profileStatIdentity(info)
- cacheKey = profileDigest([]any{directory, dev, ino})
+ if identity.scope != "" && kind != "" {
+ result["learningScope"], result["learningKind"] = identity.scope, kind
+ }
+ // New jobs carry the tool and operation seen at submission, where the
+ // arguments were available. Older metadata falls back to what survives.
+ tool, operation := text(description["learningTool"]), text(description["learningOperation"])
+ if !containsString(profileTools, tool) || !containsString(profileOperations, operation) {
+ if result["title"] == nil && result["project"] == nil && result["executable"] == nil {
+ return result // Nothing survives to classify.
+ }
+ tool, operation, _ = strings.Cut(kind, ":")
+ if tool == "" {
+ tool = profileHistoricalTool(text(result["executable"]))
}
- var ok bool
- scope, ok = scopes[cacheKey]
- if !ok {
- scope = profileDirectoryScope(directory)
- scopes[cacheKey] = scope
+ if operation == "" {
+ operation = profileOperation(text(result["title"]))
}
+ tool, operation = profileDefault(tool), profileDefault(operation)
}
- if scope != "" {
- result["learningScope"], result["learningKind"] = scope, kind
+ result["learningTool"], result["learningOperation"] = tool, operation
+ if identity.repository != "" {
+ result["learningRepository"] = identity.repository
}
return result
}
func profileValidateSimilarityDescription(clean, description Object) error {
scope, kind := text(description["learningScope"]), text(description["learningKind"])
- if scope == "" && kind == "" {
+ if scope != "" || kind != "" {
+ if !profileHash.MatchString(scope) || !validProfileSimilarityKind(kind) {
+ return fmt.Errorf("Invalid workload similarity description")
+ }
+ clean["learningScope"], clean["learningKind"] = scope, kind
+ }
+ repository, tool, operation := text(description["learningRepository"]), text(description["learningTool"]), text(description["learningOperation"])
+ if repository == "" && tool == "" && operation == "" {
return nil
}
- if !profileHash.MatchString(scope) || !validProfileSimilarityKind(kind) {
- return fmt.Errorf("Invalid workload similarity description")
+ if !(profileClass{repository, tool, operation}).valid() {
+ return fmt.Errorf("Invalid workload classification description")
+ }
+ if repository != "" {
+ clean["learningRepository"] = repository
}
- clean["learningScope"], clean["learningKind"] = scope, kind
+ clean["learningTool"], clean["learningOperation"] = tool, operation
return nil
}
diff --git a/internal/backend/profiles_similarity_test.go b/internal/backend/profiles_similarity_test.go
index 87af1a4..49b42ff 100644
--- a/internal/backend/profiles_similarity_test.go
+++ b/internal/backend/profiles_similarity_test.go
@@ -15,8 +15,11 @@ func similarityIdentity(name string) Object {
return Object{"family": profileDigest([]any{"family", name}), "variant": profileDigest([]any{"variant", name}), "context": profileDigest("context")}
}
-func similarityDescription(title, scope, kind string) Object {
- return Object{"title": title, "project": "fixture", "executable": "go", "learningScope": profileDigest(scope), "learningKind": kind}
+// similarityDescription describes a family observed in the named repository.
+func similarityDescription(title, repository, kind string) Object {
+ tool, operation, _ := strings.Cut(kind, ":")
+ return Object{"title": title, "project": "fixture", "executable": "go", "learningScope": profileDigest("scope:" + repository), "learningKind": kind,
+ "learningRepository": profileDigest(repository), "learningTool": tool, "learningOperation": operation}
}
func TestProfileSimilaritySharesConservativeLearningWithoutExactSuccesses(t *testing.T) {
@@ -157,7 +160,7 @@ func TestProfileSimilarityRetroactivePlanPreservesRunsAndIsIdempotent(t *testing
if e != nil {
t.Fatal(e)
}
- deletedDirectory := filepath.Join(t.TempDir(), "retired-checkout")
+ repository := gitRepository(t)
for i := 0; i < 4; i++ {
identity := similarityIdentity(fmt.Sprint(i))
token := fmt.Sprintf("%032x", i+1)
@@ -166,7 +169,7 @@ func TestProfileSimilarityRetroactivePlanPreservesRunsAndIsIdempotent(t *testing
t.Fatal(e)
}
recordProfile(t, store, token, identity, (i+1)*GiB, 1, i != 2, profileZero(), 0)
- mustWrite(t, filepath.Join(g.Metadata, token+".json"), Object{"title": title, "project": "fixture", "executable": "go", "directory": deletedDirectory})
+ mustWrite(t, filepath.Join(g.Metadata, token+".json"), Object{"title": title, "project": "fixture", "executable": "go", "directory": repository})
// No workload sidecar or original command is needed for retained runs.
}
pinned := similarityIdentity("3")
@@ -224,7 +227,7 @@ func TestProfileSimilarityRetroactivePlanPreservesRunsAndIsIdempotent(t *testing
t.Fatal(e)
}
recordProfile(t, store, newToken, newIdentity, GiB, 1, true, profileZero(), 0)
- mustWrite(t, filepath.Join(g.Metadata, newToken+".json"), Object{"title": "New regression tests", "project": "fixture", "executable": "go", "directory": deletedDirectory})
+ mustWrite(t, filepath.Join(g.Metadata, newToken+".json"), Object{"title": "New regression tests", "project": "fixture", "executable": "go", "directory": repository})
if e = store.backfillSimilarity(g); e != nil {
t.Fatal(e)
}
@@ -244,12 +247,13 @@ func TestProfileSimilarityRetroactivePlanPreservesRunsAndIsIdempotent(t *testing
func TestProfileSimilarityFailedMigrationRollsBackEveryMembership(t *testing.T) {
store := fixtureProfileStore(t)
- descriptions := map[string]Object{}
+ descriptions, repositories := map[string]Object{}, map[string]string{}
for i := 0; i < 3; i++ {
identity := similarityIdentity(fmt.Sprint(i))
if e := store.Observe(identity, Object{"title": "Tests", "project": "fixture", "executable": "go"}); e != nil {
t.Fatal(e)
}
+ repositories[text(identity["family"])] = profileDigest("repository")
descriptions[text(identity["family"])] = similarityDescription("Tests", "package", "go:tests")
}
if e := store.connection(true, func(c profileSQL) error {
@@ -257,7 +261,7 @@ func TestProfileSimilarityFailedMigrationRollsBackEveryMembership(t *testing.T)
}); e != nil {
t.Fatal(e)
}
- if _, e := store.regroup(descriptions, 0, true); e == nil {
+ if _, e := store.regroup(&profileHistory{descriptions: descriptions, aliases: map[string]string{}, repositories: repositories}, 0, true); e == nil {
t.Fatal("Injected migration interruption did not fail")
}
catalog, e := store.Catalog(0, 100, "")
diff --git a/internal/backend/profiles_store.go b/internal/backend/profiles_store.go
index 6420eb9..79eb39d 100644
--- a/internal/backend/profiles_store.go
+++ b/internal/backend/profiles_store.go
@@ -89,7 +89,7 @@ func NewProfileStore(state string, readonly bool) (*ProfileStore, error) {
return fmt.Errorf("Unsupported resource history schema")
}
if !readonly {
- if _, e := c.ExecContext(profileContext, profileDecisionSchema+profileLearningSchema+profileSimilaritySchema); e != nil {
+ if _, e := c.ExecContext(profileContext, profileDecisionSchema+profileLearningSchema+profileSimilaritySchema+profileClassificationSchema); e != nil {
return e
}
}
@@ -350,8 +350,10 @@ func profileRegistrySlot(c profileSQL) (bool, error) {
if e = profileExec(c, "DELETE FROM learning_members WHERE family=?", candidate["family"]); e != nil {
return false, e
}
- if e = profileExec(c, "DELETE FROM learning_similarity WHERE family=?", candidate["family"]); e != nil {
- return false, e
+ for _, table := range []string{"learning_similarity", "learning_classification"} {
+ if e = profileExec(c, "DELETE FROM "+table+" WHERE family=?", candidate["family"]); e != nil {
+ return false, e
+ }
}
e = profileExec(c, "DELETE FROM learning_profiles WHERE id=? AND NOT EXISTS(SELECT 1 FROM learning_members WHERE profile=?)", candidate["profile"], candidate["profile"])
return e == nil, e
@@ -399,7 +401,7 @@ func profileDescription(identity, description Object) (Object, error) {
return nil, fmt.Errorf("Invalid workload identity")
}
for key := range description {
- if !containsString([]string{"title", "project", "executable", "learningScope", "learningKind"}, key) {
+ if !containsString([]string{"title", "project", "executable", "learningScope", "learningKind", "learningRepository", "learningTool", "learningOperation"}, key) {
return nil, fmt.Errorf("Invalid safe workload description")
}
}
@@ -426,8 +428,12 @@ func profileDescriptionChanged(prior, clean Object) bool {
return true
}
}
- if prior["learningScope"] == nil && clean["learningScope"] != nil {
- return true
+ // A family without a stored classification, or with an unknown repository
+ // that is now verified, needs one observation.
+ for _, key := range []string{"learningScope", "learningTool", "learningRepository"} {
+ if prior[key] == nil && clean[key] != nil {
+ return true
+ }
}
return false
}
diff --git a/internal/backend/scheduler_controller.go b/internal/backend/scheduler_controller.go
index 81c8895..68b01de 100644
--- a/internal/backend/scheduler_controller.go
+++ b/internal/backend/scheduler_controller.go
@@ -23,7 +23,37 @@ type Controller struct {
runs map[string]Object
finalized map[string]bool
groups map[string]int
- learningScopes map[string]string
+ learningScopes map[string]profileIdentity
+ scopesChecked time.Time
+ retryHistory time.Time
+}
+
+// A failed git probe looks like "not a repository", so an answer must not last.
+const learningScopeLifetime = 10 * time.Minute
+
+// historyPass reports whether the history pass has finished. History only
+// improves grouping, so a failure must not hold admissions, and it waits before
+// retrying because a failed apply rotates pending grants.
+func (c *Controller) historyPass(g *Gate, now time.Time) bool {
+ if now.Before(c.retryHistory) {
+ return false
+ }
+ failure := c.store.backfillSimilarity(g)
+ if failure == nil {
+ return true
+ }
+ c.retryHistory = now.Add(learningScopeLifetime)
+ if g.operationContext().Err() == nil {
+ fmt.Fprintln(os.Stderr, "Learning history pass failed; retrying later:", failure)
+ }
+ return false
+}
+
+func (c *Controller) scopes() map[string]profileIdentity {
+ if c.learningScopes == nil || time.Since(c.scopesChecked) > learningScopeLifetime {
+ c.learningScopes, c.scopesChecked = map[string]profileIdentity{}, time.Now()
+ }
+ return c.learningScopes
}
func NewController(g *Gate, sampler *ResourceSampler, store *ProfileStore) (*Controller, error) {
@@ -40,7 +70,7 @@ func NewController(g *Gate, sampler *ResourceSampler, store *ProfileStore) (*Con
return nil, e
}
}
- return &Controller{gate: g, sampler: sampler, store: store, boot: boot, runs: map[string]Object{}, finalized: map[string]bool{}, groups: map[string]int{}, learningScopes: map[string]string{}}, nil
+ return &Controller{gate: g, sampler: sampler, store: store, boot: boot, runs: map[string]Object{}, finalized: map[string]bool{}, groups: map[string]int{}, learningScopes: map[string]profileIdentity{}}, nil
}
func (c *Controller) SampleRun(id int, value string, marker, host, prediction Object, finished bool) (Object, error) {
run := c.runs[value]
@@ -283,10 +313,7 @@ func (c *Controller) Tick() (Object, error) {
} else if e != nil {
return nil, e
}
- if c.learningScopes == nil {
- c.learningScopes = map[string]string{}
- }
- safe := profileSimilarityDescription(description, c.learningScopes)
+ safe := profileSimilarityDescription(description, c.scopes())
// Predictions are read together after every completion has been recorded.
prediction := Object{}
resources := Object{"prediction": prediction, "observed": nil, "decision": nil}
@@ -963,14 +990,11 @@ func serveScheduler(g *Gate, sampler *ResourceSampler, once bool) error {
regrouped := false
// Publish this controller's revision support before the first migration.
// The migration rotates pending grants; a fresh tick adopts its grouping.
- if e == nil && !backfilled {
- e = controller.store.backfillSimilarity(g)
- if e == nil {
- backfilled = true
- var revision int
- revision, e = controller.store.Revision()
- regrouped = metric(state["learningRevision"]) != float64(revision)
- }
+ if e == nil && !backfilled && controller.historyPass(g, started) {
+ backfilled = true
+ var revision int
+ revision, e = controller.store.Revision()
+ regrouped = metric(state["learningRevision"]) != float64(revision)
}
if ctx.Err() != nil {
return nil
diff --git a/internal/backend/scheduler_test.go b/internal/backend/scheduler_test.go
index b51dab1..71a6cb2 100644
--- a/internal/backend/scheduler_test.go
+++ b/internal/backend/scheduler_test.go
@@ -657,6 +657,9 @@ func TestControllerFinalizesJobFirstSeenFinished(t *testing.T) {
if e != nil {
t.Fatal(e)
}
+ if controller.scopesChecked.IsZero() {
+ t.Fatal("Tick bypassed the expiring identity cache")
+ }
// Samples of a finished job stay out of its evidence, but it still holds this capacity.
if held := metric(obj(obj(obj(state["jobs"])[value])["prediction"])["memoryBytes"]); held < 20<<30 {
t.Fatalf("Admission reserved %v bytes for a job holding 20 GiB", held)
diff --git a/internal/remote/server.go b/internal/remote/server.go
index efd8807..a2d414b 100644
--- a/internal/remote/server.go
+++ b/internal/remote/server.go
@@ -404,6 +404,9 @@ func (api *queueAPI) profileDetail(w http.ResponseWriter, r *http.Request, id st
api.invoke(w, r, append(arguments, "--json"), nil)
}
+// Kept in step with backend.ProfileOperations by a test; this package does not link the backend.
+var profileOperation = regexp.MustCompile(`^(tests|browser-tests|native-tests|race-tests|full-tests|build|lint|typecheck|dependencies|benchmark|checks|other)$`)
+
func (api *queueAPI) profiles(w http.ResponseWriter, r *http.Request) {
query, err := url.ParseQuery(r.URL.RawQuery)
if err != nil {
@@ -411,7 +414,7 @@ func (api *queueAPI) profiles(w http.ResponseWriter, r *http.Request) {
return
}
for key, values := range query {
- if key != "offset" && key != "limit" && key != "query" || len(values) != 1 {
+ if !map[string]bool{"offset": true, "limit": true, "query": true, "project": true, "operation": true, "otherProjects": true, "hierarchy": true}[key] || len(values) != 1 {
jsonError(w, 400, "Unknown profile query parameter")
return
}
@@ -428,7 +431,39 @@ func (api *queueAPI) profiles(w http.ResponseWriter, r *http.Request) {
jsonError(w, 400, "Invalid profile pagination or search")
return
}
- api.invoke(w, r, []string{"profiles", "list", "--offset", offset, "--limit", limit, "--query", search, "--json"}, nil)
+ for _, key := range []string{"otherProjects", "hierarchy"} {
+ if query.Has(key) && query.Get(key) != "true" {
+ jsonError(w, 400, "Invalid profile filter")
+ return
+ }
+ }
+ project, operation := query.Get("project"), query.Get("operation")
+ if query.Has("project") && (project == "" || utf8.RuneCountInString(project) > 120 || !printable(project) || strings.TrimSpace(project) != project) ||
+ query.Has("operation") && !profileOperation.MatchString(operation) || query.Has("project") && query.Has("otherProjects") {
+ jsonError(w, 400, "Invalid profile filter")
+ return
+ }
+ if query.Has("hierarchy") {
+ for key := range query {
+ if key != "hierarchy" && key != "limit" {
+ jsonError(w, 400, "The project hierarchy takes only a limit")
+ return
+ }
+ }
+ api.invoke(w, r, []string{"profiles", "list", "--hierarchy", "--limit", limit, "--json"}, nil)
+ return
+ }
+ arguments := []string{"profiles", "list", "--offset", offset, "--limit", limit, "--query", search}
+ if query.Has("project") {
+ arguments = append(arguments, "--project", project)
+ }
+ if query.Has("operation") {
+ arguments = append(arguments, "--operation", operation)
+ }
+ if query.Has("otherProjects") {
+ arguments = append(arguments, "--other-projects")
+ }
+ api.invoke(w, r, append(arguments, "--json"), nil)
}
func (api *queueAPI) profileAction(w http.ResponseWriter, r *http.Request, id, action string) {
diff --git a/internal/remote/server_test.go b/internal/remote/server_test.go
index d8e9c1c..6cd0058 100644
--- a/internal/remote/server_test.go
+++ b/internal/remote/server_test.go
@@ -14,6 +14,7 @@ import (
"syscall"
"testing"
+ "github.com/Dunky13/agent-job-queue/internal/backend"
"github.com/gorilla/csrf"
)
@@ -161,6 +162,20 @@ func TestAPIStrictQueueMutations(t *testing.T) {
}
}
+// The remote server validates operations without linking the backend, so the two
+// lists must not drift apart.
+func TestProfileOperationFilterMatchesBackend(t *testing.T) {
+ operations := backend.ProfileOperations()
+ if strings.Count(profileOperation.String(), "|")+1 != len(operations) {
+ t.Fatalf("%s does not list exactly %v", profileOperation, operations)
+ }
+ for _, operation := range operations {
+ if !profileOperation.MatchString(operation) {
+ t.Errorf("%s is rejected", operation)
+ }
+ }
+}
+
func TestAPIProfilesReadOnlyBounds(t *testing.T) {
f := newAPIFixture(t)
id := strings.Repeat("a", 64)
@@ -174,6 +189,27 @@ func TestAPIProfilesReadOnlyBounds(t *testing.T) {
if !reflect.DeepEqual(f.calls[0].args, []string{"profiles", "list", "--offset", "100", "--limit", "25", "--query", "Browser tests", "--json"}) || !reflect.DeepEqual(f.calls[1].args, []string{"profiles", "show", id, "--json"}) {
t.Fatalf("wrong CLI arguments: %#v", f.calls)
}
+ filtered := map[string][]string{
+ "project=my%20app&operation=race-tests": {"profiles", "list", "--offset", "0", "--limit", "100", "--query", "", "--project", "my app", "--operation", "race-tests", "--json"},
+ "otherProjects=true&operation=other": {"profiles", "list", "--offset", "0", "--limit", "100", "--query", "", "--operation", "other", "--other-projects", "--json"},
+ "hierarchy=true&limit=20": {"profiles", "list", "--hierarchy", "--limit", "20", "--json"},
+ "hierarchy=true": {"profiles", "list", "--hierarchy", "--limit", "100", "--json"},
+ }
+ for query, want := range filtered {
+ before := len(f.calls)
+ if w := f.request("GET", "/api/profiles?"+query, "", nil); w.Code != 200 || len(f.calls) != before+1 || !reflect.DeepEqual(f.calls[before].args, want) {
+ t.Fatalf("filter %s: %d %#v", query, w.Code, f.calls[before:])
+ }
+ }
+ calls := len(f.calls)
+ for _, query := range []string{"operation=nope", "operation=", "operation=Tests", "project=", "project=%20x", "project=" + strings.Repeat("x", 121), "project=a%0Ab", "project=a&otherProjects=true", "otherProjects=false", "otherProjects=1", "hierarchy=yes", "hierarchy=true&query=x", "hierarchy=true&offset=1", "hierarchy=true&project=a", "hierarchy=true&limit=101", "kind=go%3Atests", "project=a&project=b", "operation=xtests", "operation=testsx", "operation=tests%0A", "operation=tests%20", "operation=deploy"} {
+ if w := f.request("GET", "/api/profiles?"+query, "", nil); w.Code != 400 {
+ t.Fatalf("accepted %s: %d", query, w.Code)
+ }
+ }
+ if len(f.calls) != calls {
+ t.Fatal("invalid filter invoked backend")
+ }
for _, query := range []string{"limit=101", "limit=0", "offset=-1", "offset=1.2", "offset=00", "query=" + strings.Repeat("x", 121), "limit=10&limit=20", "command=run", "query=%00", "query=%invalid", "offset=9999999999", "limit=01", "limit="} {
if w := f.request("GET", "/api/profiles?"+query, "", nil); w.Code != 400 {
t.Fatalf("accepted %s: %d", query, w.Code)
@@ -184,7 +220,7 @@ func TestAPIProfilesReadOnlyBounds(t *testing.T) {
t.Fatal("accepted invalid profile detail")
}
}
- if len(f.calls) != 2 {
+ if len(f.calls) != 2+len(filtered) {
t.Fatal("invalid query invoked backend")
}
}
diff --git a/remote/static/schema.mjs b/remote/static/schema.mjs
index 0a71a2e..e343d36 100644
--- a/remote/static/schema.mjs
+++ b/remote/static/schema.mjs
@@ -86,11 +86,14 @@ export function eligible(job, action) {
return action === "cancel" && job.canCancel;
}
+const operationSchema = z.literal(["tests", "browser-tests", "native-tests", "race-tests", "full-tests", "build", "lint", "typecheck", "dependencies", "benchmark", "checks", "other"]);
const profileSchema = z.object({
id: identitySchema, label: labelSchema, memberCount: integer.nonnegative(), variantCount: integer.nonnegative(),
successCount: integer.nonnegative(), incompleteCount: integer.nonnegative(), peakMemoryBytes: integer.nonnegative(),
peakCpuCores: nonnegative, lastRunAt: nonnegative.nullable(), memoryFloorBytes: integer.positive().max(2 ** 50).nullable(),
cpuFloorCores: z.number().positive().max(1024).nullable(),
+ // Absent from daemons that predate project grouping.
+ project: z.string().min(1).max(120).optional(), operation: operationSchema.optional(),
});
const controllerFields = {
controllerAppliedRevision: integer.nonnegative().nullable(), controllerStatus: z.string().max(4096),
@@ -99,6 +102,14 @@ export const profileCatalogSchema = z.object({
revision: integer.nonnegative(), profiles: z.array(profileSchema).max(100), total: integer.nonnegative(),
offset: integer.nonnegative(), limit: integer.min(1).max(100), ...controllerFields,
});
+const hierarchyCounts = { workloadCount: integer.nonnegative(), groupCount: integer.nonnegative(), runCount: integer.nonnegative() };
+export const profileHierarchySchema = z.object({
+ revision: integer.nonnegative(), total: integer.nonnegative(), limit: integer.min(1).max(100), ...controllerFields,
+ projects: z.array(z.object({
+ project: z.string().min(1).max(120), collapsed: z.boolean(), projectCount: integer.positive().optional(), ...hierarchyCounts,
+ operations: z.array(z.object({ operation: operationSchema, ...hierarchyCounts })).max(32),
+ })).max(100),
+});
export const profileDetailSchema = z.object({
revision: integer.nonnegative(), profile: profileSchema, ...controllerFields,
members: z.array(z.object({ family: identitySchema, label: labelSchema, project: z.string().max(120), executable: z.string().max(120),
diff --git a/remote/static/schema.test.mjs b/remote/static/schema.test.mjs
index 07293db..d5340e2 100644
--- a/remote/static/schema.test.mjs
+++ b/remote/static/schema.test.mjs
@@ -110,3 +110,32 @@ test("accepts lane, lease and plugin fields and keeps older status valid", () =>
assert.equal(eligible({ ...lease, state: "queued", canStart: true }, "start"), true);
assert.equal(eligible(lease, "cancel"), true);
});
+
+test("learning schemas accept project grouping and still read older daemons", async () => {
+ const { profileCatalogSchema, profileHierarchySchema } = await import("./schema.mjs");
+ const controller = { controllerAppliedRevision: 3, controllerStatus: "Current" };
+ const profile = { id: "a".repeat(64), label: "Browser tests", memberCount: 1, variantCount: 1, successCount: 1, incompleteCount: 0,
+ peakMemoryBytes: 1, peakCpuCores: 1, lastRunAt: 1, memoryFloorBytes: null, cpuFloorCores: null };
+ const catalog = { ...controller, revision: 3, total: 2, offset: 0, limit: 100, profiles: [profile, { ...profile, project: "agent-queue", operation: "browser-tests" }] };
+ assert.deepEqual(profileCatalogSchema.parse(catalog), catalog);
+ for (const invalid of [{ project: "" }, { operation: "" }, { project: "x".repeat(121) }, { operation: 5 }]) {
+ assert.equal(profileCatalogSchema.safeParse({ ...catalog, profiles: [{ ...profile, ...invalid }] }).success, false);
+ }
+ const counts = { workloadCount: 3, groupCount: 2, runCount: 9 };
+ const hierarchy = { ...controller, revision: 3, total: 1, limit: 100, projects: [
+ { project: "agent-queue", collapsed: false, ...counts, operations: [{ operation: "tests", ...counts }] },
+ { project: "Other projects", collapsed: true, projectCount: 4, ...counts, operations: [] }] };
+ assert.deepEqual(profileHierarchySchema.parse(hierarchy), hierarchy);
+ assert.equal("extra" in profileHierarchySchema.parse({ ...hierarchy, extra: 1 }), false);
+ for (const invalid of [{ collapsed: "yes" }, { workloadCount: -1 }, { project: "" }, { operations: [{ operation: "tests", workloadCount: 1 }] }]) {
+ assert.equal(profileHierarchySchema.safeParse({ ...hierarchy, projects: [{ ...hierarchy.projects[0], ...invalid }] }).success, false);
+ }
+ for (const operation of ["deploy", "Tests", ""]) {
+ assert.equal(profileCatalogSchema.safeParse({ ...catalog, profiles: [{ ...profile, operation }] }).success, false);
+ assert.equal(profileHierarchySchema.safeParse({ ...hierarchy, projects: [{ ...hierarchy.projects[0], operations: [{ operation, ...counts }] }] }).success, false);
+ }
+ assert.equal(profileHierarchySchema.safeParse({ ...hierarchy, projects: Array(101).fill(hierarchy.projects[0]) }).success, false);
+ assert.equal(profileHierarchySchema.safeParse({ ...hierarchy, projects: Array(100).fill(hierarchy.projects[0]) }).success, true);
+ assert.equal(profileHierarchySchema.safeParse({ ...hierarchy, projects: Array(102).fill(hierarchy.projects[0]) }).success, false);
+ assert.equal(profileHierarchySchema.safeParse({ ...hierarchy, projects: undefined }).success, false);
+});