Repository navigation
fix: bound PACK inflation and push body size in parsePush (sfind-79af3f3c) - #47
Open
devin-ai-integration[bot] wants to merge 4 commits into
Open
devin-ai-integration[bot] wants to merge 4 commits into
devin-ai-integration[bot] wants to merge 4 commits into
Conversation
…79af3f3c) Inflate each PACK object with a single inflateSync call capped by MAX_INFLATED_OBJECT_SIZE / MAX_INFLATED_PACK_SIZE instead of one zlib write per input byte, stop after the declared entry count, log object metadata instead of full contents, and cap the buffered git pack request body at MAX_PACK_BODY_SIZE instead of 1gb.
Author
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
…t exact pack budget (sfind-79af3f3c)
| activeInflations++; | ||
| return; | ||
| } | ||
| await new Promise<void>((resolve) => inflationQueue.push(resolve)); |
Author
There was a problem hiding this comment.
🟥 Unbounded push queue exhausts proxy memory
When two pushes occupy inflation slots, inflationQueue retains every additional push's buffered PACK without a limit. Anonymous requests reach this queue before permission checks, so enough queued pushes can exhaust proxy memory.
Was this helpful? React with 👍 or 👎 to provide feedback.
Author
There was a problem hiding this comment.
I'm leaving this one without a code change, for these reasons:
- The queue adds no memory of its own. It stores only a
resolveclosure per waiting push. The compressed PACK it waits on isreq.bodyRaw, whichextractRawBodyhas already fully buffered (capped atMAX_PACK_BODY_SIZE) before the chain orparsePushruns. Each waiting push therefore holds exactly what it would hold onmain, or with no queue at all. The difference is that without the queue, all of those pushes would also be inflating at once, each up toMAX_INFLATED_PACK_SIZE. - The real gap is concurrent body buffering. How many ~100 MB bodies can be buffered at once is decided by the HTTP/route layer, before this parser is reached. Capping the queue length here would reject some requests sooner, but would not stop an attacker from holding N connections that each buffer a body.
- The fix belongs elsewhere. That needs a concurrency or rate limit on
isPackPostrequests inextractRawBody, or a body/connection limit in the reverse proxy in front of GitProxy. That is a separate change with its own trade-offs, such as rejecting legitimate pushes under load, so I've flagged it for the maintainers rather than adding it to this parser fix.
I'm leaving the thread open so a maintainer can decide whether they want a route-level limit, either in this PR or as a follow-up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Corrige o finding
sfind-79af3f3c(low/high): o parser de PACK emparsePushinflava objetos sem limite de saída e com uma escrita no zlib (e umaPromise) por byte de entrada. ComoparsePushroda antes de qualquer autenticação/checagem de repositório, um push anônimo podia travar ou derrubar o proxy por CPU/memória.O que mudou
decompressGitObjects(src/proxy/processors/push-action/parsePush.ts)zlib.inflateassíncrona (promisificada,info: true+maxOutputLength), executada no threadpool do libuv — não bloqueia o event loop. O fim de cada objeto é localizado viaengine.bytesWritten(sem mais escrita byte a byte).MAX_INFLATED_OBJECT_SIZE= 100 MiB por objeto eMAX_INFLATED_PACK_SIZE= 256 MiB no total do PACK. Objetos vazios ainda são aceitos quando o orçamento chega exatamente a zero.MAX_CONCURRENT_PACK_INFLATIONS= 2 PACKs são inflados ao mesmo tempo (fila FIFO; o slot é liberado emfinally), limitando a memória agregada de inflação entre pushes concorrentes.getContents(buffer, numEntries)).getContentspassa a logar só metadados (contentLength) em vez de serializar o conteúdo completo dos objetos.extractRawBody(src/proxy/routes/index.ts) usavagetRawBody(..., { limit: '1gb' }); agora usaMAX_PACK_BODY_SIZE('100000kb'), o mesmo valor já usado pelo Express emsrc/proxy/index.ts(ambos referenciam a constante emsrc/proxy/processors/constants.ts).Mudança de comportamento (sem mudança de contrato de API)
Pushes com body acima de ~100 MB, objeto inflado acima de 100 MiB ou PACK inflado acima de 256 MiB passam a ser rejeitados com erro de inflação. Nenhuma API pública/assinatura exportada mudou.
Fora de escopo
Limitar quantos bodies de push (~100 MB cada) podem estar bufferizados simultaneamente é decidido na camada HTTP/rota, antes do parser — exigiria um limite de concorrência/rate limit em
extractRawBodyou no reverse proxy à frente do GitProxy (ver thread de review).Medições locais (harness chamando
getContents, Node 22)mainInvalid string lengthRelated Issue
Code scan finding
sfind-79af3f3c070b42aa9b72f931bea858cd(scanscan-70082f73b42b43fda8ae0a7ecbe7fedf).Checklist
General
Documentation
Configuration
config.schema.json) was modified: (não modificado)Tests
npm test) — veja nota abaixonpm run lintandnpm run format:check) — jobLintingverde no CInpm run check-types) — passa no CIComo testar
test/parsePushLimits.test.ts(novo): limites por objeto/PACK (com constantes mockadas), objeto vazio no limite exato, yield ao event loop durante a inflação e no máximoMAX_CONCURRENT_PACK_INFLATIONSinflações simultâneas (falha se o limite for removido).test/testParsePush.test.ts: parada no número de entradas declarado e rejeição rápida (<5 s) de um payload real acima deMAX_INFLATED_OBJECT_SIZE.test/extractRawBody.test.ts:getRawBodychamado comMAX_PACK_BODY_SIZE.Localmente: 63/63 nesses 3 arquivos. No CI,
Linting,e2e (vitest),e2e (cypress)e CodeQL passam; nos jobsbuild-*, 60 arquivos de teste passam. As falhas restantes não vêm desta mudança e também aparecem em outros PRs abertos (ex.: #46):test/ConfigLoader.test.ts > loadRemoteConfigbuscaproxy.config.jsondo upstreamfinos/git-proxye falha comInvalid value for key "sink".MongoDB did not initialize within 20 secondsantes da instalação.Devin-Org: engineering
Link to Devin session: https://devin-gtm.devinenterprise.com/sessions/a53cfdf248f24a4ab811ed0da7f0bbdc
Open in Devin Desktop: https://devin-gtm.devinenterprise.com/desktop/session/a53cfdf248f24a4ab811ed0da7f0bbdc?variant=devin
Devin Review