feat!: read raw-free medallion names and fail loudly when absent - #7
Merged
Conversation
Read the raw-free medallion SSM keys published by gen3-aws-data-pipeline >= v2.0.0 (buckets/silver|gold, glue/db/bronze|silver|gold) and replace the silent rc.get() reads with a _req_key guard that raises ConfigError naming the missing key and the likely cause (pipeline older than v2.0.0). Previously an absent key propagated as None: dbt-env dropped G3DT_DB_* export lines and emitted s3://None/dbt/ data dirs, and the release model search silently widened to an account-wide Glue catalog walk. BREAKING CHANGE: dbt-env variables renamed: G3DT_DB_RAW_BRONZE -> G3DT_DB_BRONZE, G3DT_DB_RAW_SILVER -> G3DT_DB_SILVER, G3DT_DB_RAW_GOLD -> G3DT_DB_GOLD, G3DT_DB_RAW_SILVER_CI -> G3DT_DB_SILVER_CI, G3DT_DB_RAW_GOLD_CI -> G3DT_DB_GOLD_CI. Toolkit >= 3 pairs only with a gen3-aws-data-pipeline deployment >= v2.0.0 (which publishes the raw-free SSM keys); older deployments must upgrade the pipeline or pin gen3-dataops-toolkit<3.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pairs with aws-gen3-pipeline v2.0.0 (raw-free SSM keys). Two changes:
glue/db/bronze|silver|goldandbuckets/silver|gold, and emitsG3DT_DB_BRONZE|SILVER|GOLD(+_CI).G3DT_S3_*_DATA_DIR[_CI]names unchanged. Release search scoping reads the new DB keys._req_key, which raisesConfigErrornaming the missing key and the likely cause (pipeline older than v2.0.0). Previously a missing key made vars vanish silently, emitteds3://None/dbt/, and sent release search on an account-wide catalog walk — a new test pins the loud behaviour and documents that history.BREAKING CHANGE (v3.0.0):
G3DT_DB_RAW_*vars are gone; toolkit ≥3 requires a pipeline deployment ≥ v2.0.0.348 tests pass.