fix(deploy): keep on-chain validator off master hosts #101
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,10 +1,12 @@ | ||
| # Master role — gateway enabled, published on VPC for validator peers. | ||
| # Used with: docker compose -f docker-compose.yml -f deploy/compose/role-master.yml --profile master | ||
| # | ||
| # On-chain weight submit must NOT run here (sole submitter = validator host). | ||
| # Co-located validator is local-e2e only — env-local.yml clears the profile. | ||
| services: | ||
| gateway: | ||
| profiles: ["master"] | ||
| ports: | ||
| - "8080:8080" | ||
| validator: | ||
| profiles: ["never"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -416,23 +416,35 @@ fi | |
| docker compose ${COMPOSE_FILES[*]} ${PROFILE_ARGS[*]} \$UP_PROFILE "\${UP_ARGS[@]}" | ||
| # Profile-disabled services are not started, but an older compose project may | ||
| # still be running them. On validator, force-remove master-only challenge | ||
| # surfaces so smoke health does not see stale unhealthy containers. On master, | ||
| # force-remove the validator so a prior dual-submitter cannot fight the | ||
| # validator-host wallet for WeightsSetRateLimit / CRV4 commits. | ||
| if [[ '$ROLE' == 'validator' ]]; then | ||
| docker compose ${COMPOSE_FILES[*]} rm -sf \ | ||
| prism-challenge design-challenge design-egress-proxy socket-proxy \ | ||
| >/dev/null 2>&1 || true | ||
|
Comment on lines
422
to
425
There was a problem hiding this comment. 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files | rg '(^|/)deploy/scripts/remote-deploy\.sh$|(^|/)deploy/.*/role-.*\.ya?ml$|(^|/)deploy/[^/]+\.ya?ml$|(^|/)docker-compose.*ya?ml$' || true
echo "== remote-deploy relevant =="
if [ -f deploy/scripts/remote-deploy.sh ]; then
wc -l deploy/scripts/remote-deploy.sh
sed -n '350,460p' deploy/scripts/remote-deploy.sh | cat -n
fi
echo "== compose service/profile snippets =="
rg -n "gateway|updater|validator|role-validator|role-master|profile:" deploy --glob '*.{yml,yaml,sh,tf,hcl}' | head -n 200
echo "== behavioral probe: docker compose rm semantics from compose docs? no runtime =="
echo "Inspect compose files for profiles explicitly to validate transition hypothesis:"
python3 - <<'PY'
from pathlib import Path
for p in sorted(Path('deploy').rglob('*')):
if p.is_file() and p.suffix in {'.yml','.yaml'}:
text=p.read_text(errors='ignore')
if 'gateway' in text or 'updater' in text or 'profile:' in text:
print(f'--- {p} ---')
for i,line in enumerate(text.splitlines(),1):
if any(s in line for s in ['gateway','updater','profile:','profiles:', 'profiles']):
print(f'{i}: {line}')
PYRepository: BaseIntelligence/base Length of output: 30485 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== role-validator.yml complete =="
cat -n deploy/compose/role-validator.yml
echo "== role-master.yml complete =="
cat -n deploy/compose/role-master.yml
echo "== base docker service profile definitions for gateway/updater =="
python3 - <<'PY'
from pathlib import Path
import re
for p in [Path('docker-compose.yml'), Path('docker-compose.e2e.yml')]:
if p.exists():
text=p.read_text(errors='ignore')
lines=text.splitlines()
in_svc=False
svc=None
for i,line in enumerate(lines,1):
if re.match(r'^[A-Za-z0-9_\-=/.]+\s*:$', line):
in_svc=True; svc=line.split(':')[0]
if svc in {'gateway','updater'}:
print(f'{p}:{i}: {line.strip()}')
while i < len(lines) and (not re.match(r'^[A-Za-z0-9_][A-Za-z0-9_./-]*:', lines[i]) or lines[i].startswith(' ') or lines[i].startswith('\t')):
print(f'{i+1}: {lines[i]}')
i+=1
break
PY
echo "== master profile arguments context =="
sed -n '60,85p;230,255p;396,440p' deploy/scripts/remote-deploy.sh | cat -n
echo "== read-only compose YAML service/profile model =="
python3 - <<'PY'
from pathlib import Path
import re
def parse_overrides(fname):
hits=[]
t=Path(fname).read_text(errors='ignore').splitlines()
for i,line in enumerate(t,1):
m=re.match(r'^\s+(gateway|updater|validator|prism-challenge|design-challenge|design-egress-proxy|socket-proxy):\s*(.*)?$', line)
if m:
svc=m[1]; tail=m[2]
profiles=[]
while i < len(t):
nxt=t[i]
if re.match(r'^\s{4}profiles:\s*\[([^\]]*)\]', nxt):
profiles=profiles + [x.strip().strip('"\'') for x in re.findall(r'[^,\]\s"]+', nxt)]
elif nxt.strip().startswith('profiles:'):
profiles=[]
elif re.match(r'^[A-Za-z0-9_][A-Za-z0-9_./-]*:', nxt):
break
i+=1
hits.append((svc, tail or '', profiles))
return hits
for f in ['deploy/compose/role-master.yml','deploy/compose/role-validator.yml']:
print(f'--- {f} ---')
for h in parse_overrides(f):
print(h)
PYRepository: BaseIntelligence/base Length of output: 8422 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== docker compose files =="
git ls-files | rg '(^docker-compose\.ya?ml$|(^|/)compose/.*\.ya?ml$)'
echo "== gateway/updater definitions in base compose =="
for f in docker-compose.yml docker-compose.e2e.yml; do
[ -f "$f" ] || continue
echo "--- $f ---"
rg -n "gateway|updater|profiles:" "$f" -A 8 -B 2 || true
done
echo "== all gateway/updater occurrences =="
rg -n "^\s*gateway:\s*$|^\s*updater:\s*$|profiles:" deploy docker-compose.yml docker-compose.e2e.yml -A 5 -B 2 || trueRepository: BaseIntelligence/base Length of output: 15895 Remove stale gateways during validator redeploy.
🤖 Prompt for AI AgentsSource: Coding guidelines |
||
| elif [[ '$ROLE' == 'master' ]]; then | ||
| docker compose ${COMPOSE_FILES[*]} ${PROFILE_ARGS[*]} rm -sf validator \ | ||
| >/dev/null 2>&1 || true | ||
|
Comment on lines
+427
to
+428
There was a problem hiding this comment. 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -u
echo "== locate file =="
git ls-files | rg '(^|/)remote-deploy\.sh$' || true
echo "== file stats =="
wc -l deploy/scripts/remote-deploy.sh
echo "== relevant lines 380-460 =="
sed -n '380,460p' deploy/scripts/remote-deploy.sh | cat -n -v | sed 's/^/L/'
echo "== search validator/profile/compose refs =="
rg -n "validator|base-validator-1|base-prod-validator|profile|PROFILE_ARGS|COMPOSE_FILES|docker compose ps|profile:|profiles:" deploy -SRepository: BaseIntelligence/base Length of output: 50377 🏁 Script executed: #!/bin/bash
set -u
echo "== compose files and profiles =="
fd -e yml -e yaml . deploy | sort | xargs -r rg -n "name:|services:|profiles:|profile:" -S || true
echo "== validator occurrences in repo =="
rg -n "\bvalidator\b|base-validator-1|base-prod-validator|ROLE|role-master" deploy -SRepository: BaseIntelligence/base Length of output: 50377 🏁 Script executed: #!/bin/bash
set -u
echo "== remote-deploy master profile and validator check =="
sed -n '236,260p;416,448p' deploy/scripts/remote-deploy.sh | cat -n
echo "== master role compose =="
cat -n deploy/compose/role-master.yml
echo "== env compose validator sections =="
for f in deploy/compose/env-prod.yml deploy/compose/env-staging.yml deploy/compose/env-local.yml; do
echo "--- $f ---"
cat -n "$f" | sed -n '1,70p'
done
echo "== script error handling/exit behavior around absent validator message =="
rg -n "set -[ef]|trap|main|function main|\\}# exit|exit 0|echo \"remote-deploy: done|echo \"master: validator absent" deploy/scripts/remote-deploy.shRepository: BaseIntelligence/base Length of output: 16582 🌐 Web query:
💡 Result: The docker compose ps command is not currently profile-aware, meaning it does not automatically filter its output to match the profiles specified in the command [1][2]. Even if you execute docker compose --profile ps, the command will typically list all containers defined in your project, regardless of whether those containers belong to an active profile or an inactive one [1][2]. This behavior is a known limitation of the docker compose ps command, which was designed to maintain backward compatibility by listing all containers associated with the project [2]. While you can use the --status flag to filter by the state of the containers (e.g., --status running), this does not implicitly filter out containers based on their configured profiles [3][4]. If you need to limit the output to specific containers, you may need to rely on alternative methods such as using standard CLI tools (e.g., piping to grep) or specifically targeting the service names directly in the command (e.g., docker compose ps <service_name>), as the project continues to prioritize compatibility over adding profile-aware filtering to the ps command [2][5]. Citations:
Fail closed on validator removal and inspection.
🤖 Prompt for AI AgentsSource: Coding guidelines |
||
| fi | ||
| docker compose ${COMPOSE_FILES[*]} ${PROFILE_ARGS[*]} \$UP_PROFILE ps | ||
| # Local health probes via published tunnels if present, else container exec. | ||
| sleep 5 | ||
| if [[ '$ROLE' == 'validator' ]]; then | ||
| if curl -fsS -m 5 http://127.0.0.1:18080/healthz >/dev/null 2>&1; then | ||
| echo "validator tunnel health: \$(curl -fsS -m 5 http://127.0.0.1:18080/healthz)" | ||
| elif docker compose ${COMPOSE_FILES[*]} ${PROFILE_ARGS[*]} exec -T validator curl -fsS -m 5 http://127.0.0.1:8080/healthz >/dev/null 2>&1; then | ||
| echo "validator health: ok (in-container)" | ||
| else | ||
| echo "validator health: probe deferred (container may still be starting)" | ||
| fi | ||
| fi | ||
| if [[ '$ROLE' == 'master' ]]; then | ||
| if docker compose ${COMPOSE_FILES[*]} ${PROFILE_ARGS[*]} ps --status running --services 2>/dev/null | grep -qx validator; then | ||
| echo "remote-deploy: ERROR: validator still running on master (dual submitter)" >&2 | ||
| exit 1 | ||
| fi | ||
| echo "master: validator absent (sole on-chain submitter is validator host)" | ||
| if docker compose ${COMPOSE_FILES[*]} ${PROFILE_ARGS[*]} exec -T gateway curl -fsS -m 5 http://127.0.0.1:8080/healthz >/dev/null 2>&1; then | ||
| echo "gateway health: ok" | ||
| else | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
Repository: BaseIntelligence/base
Length of output: 50377
🏁 Script executed:
Repository: BaseIntelligence/base
Length of output: 26702
🏁 Script executed:
Repository: BaseIntelligence/base
Length of output: 50380
🏁 Script executed:
Repository: BaseIntelligence/base
Length of output: 34677
🏁 Script executed:
Repository: BaseIntelligence/base
Length of output: 27022
Serialize D24 leaf projection with
award_round.run_emittercan submitNotAttemptedleaves beforeaward_roundpersists scores and submits scored leaves. Sinceemitted_epochonly guards the emitter andaward_rounddoes not recheck it,emit_leavescan expose a stale D24 projection to sealing. Use one shared async lock, or a store-backed transaction, held across score persistence and leaf submission inaward_round; acquire it before recheckingemitted_epochinemitter_tickas well.🤖 Prompt for AI Agents
Source: Coding guidelines