diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..51be1ac --- /dev/null +++ b/.dockerignore @@ -0,0 +1,9 @@ +.git +.sg +export +out +**/.terraform +**/.terraform.lock.hcl +*.tfstate +*.tfstate.* +*.tfvars diff --git a/.gitignore b/.gitignore index 1d638cc..7819528 100644 --- a/.gitignore +++ b/.gitignore @@ -16,6 +16,8 @@ eggs/ .eggs/ lib/ lib64/ +# The migrator's sourced shell libraries are not Python build output. +!scripts/lib/ parts/ sdist/ var/ @@ -146,6 +148,8 @@ crash.log # control as they are data points which are potentially sensitive and subject # to change depending on the environment. *.tfvars +# The init wizard keeps the previous file as terraform.tfvars.bak +*.tfvars.bak # Ignore override files as they are usually used to override resources locally and so # are not checked in @@ -163,4 +167,7 @@ out/* zip zip/* +# Migrator local cache + config (downloaded tool binaries, workflow-group map) +.sg/ + .DS_Store \ No newline at end of file diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..1a3d3f1 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,88 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## What this is + +A migration tool that extracts workloads from other IaC platforms and transforms them into StackGuardian Workflow definitions (`sg-payload.json`), ready for bulk import via [sg-cli](https://github.com/StackGuardian/sg-cli). Currently the only implemented source is **Terraform Cloud / Enterprise (TFC/TFE)**. + +There is no application code to build — the "engine" is Terraform itself. The migrator is a Terraform root module that uses the `tfe` provider to read workspaces and the `local`/`null` providers to write the output payload and pull state files. + +## Architecture + +The migration is a user-driven pipeline, not a single program: + +1. **Extract + transform** — `transformer/terraform-cloud/` is a Terraform root module. `terraform apply` reads TFC/TFE workspaces and emits one `/sg-payload..json` per TFC project, a `migration-summary.md`/`.json` report, and per-workspace `.tfstate` files when `exportStateFiles` is true. +2. **Enrich (variable sets)** — `scripts/enrich_variable_sets.sh` merges TFC Variable Set variables into the payloads via the TFC API (the provider can't enumerate sets). Resolves global/project/workspace scope + TFC precedence per workspace; non-sensitive only. +3. **Tuning** — adjust per-workspace differences (integration IDs, VCS auth, runners, approvers, version) via the `workspaceOverrides` variable and re-apply; or hand-edit the payload files. `example_payload.jsonc` is the annotated field reference. +4. **HCL→JSON conversion** — `scripts/convert_hcl_to_json.sh` rewrites HCL-string variable values in each payload to real JSON (one result line per file; details with `-v`). +5. **Validation** — `scripts/validate_payload.sh` checks each payload against `schema/sg-payload.schema.json` (downloads `yajsv`; yajsv's lines are re-rendered as ✓/✗ per file, raw with `-v`). +6. **Import** — `sg-cli workflow create --bulk`, run once per project file, each into its own workflow group. + +### Orchestration & tooling + +The five phases are wrapped by an orchestrator so users don't run them by hand: + +- `sg-migrate.sh` (host entrypoint, repo root) — runs `scripts/migrate.sh` **inside the Docker image** (`Dockerfile`), bind-mounting the repo at `/app` and forwarding `SG_API_TOKEN`/`SG_ORG`/`SG_BASE_URL`/`TFE_TOKEN`/`TF_TOKEN_*`. For TFC auth it prefers a long-lived `TFE_TOKEN` (env); otherwise it mounts `~/.terraform.d/credentials.tfrc.json` read-only. Runs natively instead when `--native`/`--local` is passed, `SG_NATIVE=1` is set, the command is `clean`/`completion`/help (or no command is given), or Docker is absent. Exports `SG_PROG` so `migrate.sh` shows `./sg-migrate.sh` in its usage/hints. +- `scripts/migrate.sh` — the actual orchestrator (sources `scripts/lib/*.sh`, see below). Subcommands `init|preflight|apply|enrich|convert|validate|import|triggers|checklist|all|clean|completion` (no command prints the help menu; `enrich` runs in `all` unless `--no-variable-sets`). Flags beyond the basics: `--dry-run` (import: plan only), `--fresh` (ignore run state), `--project SEG` / `--workspace NAME` (repeatable filters; apply maps `--workspace` to `-var workspacenames=[...]`, import works on a jq-filtered temp copy), `--skip-preflight`, `--no-secret-stubs`. Resolves each tool via `sg_resolve` (PATH first — the image installs them — else `sg_ensure_*` cache). `apply`/`all` first run `require_tfc_auth`, which resolves the token the tfe provider will use (`TFE_TOKEN`, `TF_TOKEN_`, or the `terraform login` file for `tfHostname`) and verifies it with `GET /api/v2/account/details`, failing fast on a missing or rejected (expired) credential. Runs `convert` and `import` in parallel (`--concurrency`, default 4), raises `terraform apply -parallelism`, and retries `sg-cli` imports with backoff (`sg_retry`, `SG_RETRIES`). Import resolves each project's workflow group, checks existence via the SG API, shows a plan (`exists`/`create`), prompts (skip with `-y`), **creates the missing `tfc-` groups via the API**, then imports. sg-cli exits 0 even when individual workflows fail, so `do_import` parses its output: a workflow rejected as above SG's managed Terraform ceiling (1.5.7, last MPL/FOSS release) is re-imported with `SGDefaultTerraformVersion` (read from `terraform.tfvars`; an explicit `null` there means the `terraformVersion` key is deleted instead, so the org's execution preset decides — `tfvars_is_null` tells an explicit null from a missing key), the payload patched in place, and the case logged to `export/terraform-version-fallbacks.log` plus a printed notice; any other per-workflow failure fails the run. `cmd_import` reads the org's execution preset (`sg_execution_preset`, `GET /orgs/{org}/` → `Settings.workflowDefaults`) and `preset_labels` (report.sh) turns it into the `preset (1.5.7)` / `preset (private:rg)` cells of the plan and the `SG_TF_FALLBACK_LABEL` used in messages. The trigger pass skips workflows that do not exist in SG, and `sg_api_post` returns 22 on 4xx so `sg_retry` (via `SG_NO_RETRY_RC`) does not retry definitive errors. `clean` removes local artifacts (`export/`, TF state, tool cache); `clean --all` also removes config. `completion bash|zsh` prints a completion script (`cmd_completion`; commands/options come from `SG_COMMANDS`/`SG_OPTIONS`, keep them in sync with the parser and the host flags); like `clean` it always runs natively. Output is concise by default — `apply` captures terraform's init/plan output and shows only progress + the final summary (or the full log on failure, `TF_IN_AUTOMATION=1`/`-no-color`); `-v`/`--verbose` (`SG_VERBOSE`) streams everything and un-gates the convert detail lines. Colored logging via `sg_step`/`sg_log`/`sg_success`/`sg_warn`/`sg_err`/`sg_row`; paths shown relative via `sg_rel` (all auto-off when not a TTY / `NO_COLOR`). Long steps show a live progress line (`sg_run_quiet` for terraform init/apply, `run_parallel