Repository navigation
Add generation environment and source - #1
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| const worker = async () => { | ||
| while (next < items.length) await work(items[next++]); | ||
| }; | ||
| await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker)); |
There was a problem hiding this comment.
🔍 One failed job strands queued attempts
If generateAttempt throws, its worker stops and pool rejects. Queued attempts remain unprocessed, and the command exits without a final tally.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Intended. A thrown error here is a setup error (a missing key, an unwritable folder), so stopping is correct. Model and transport failures do not throw: they are recorded in completion.json, and the run continues. A rerun resumes and skips finished attempts.
| completion = JSON.parse(await readFile(path.join(dir, 'completion.json'), 'utf8')); | ||
| } catch (error) { | ||
| if (error.code === 'ENOENT') return null; | ||
| throw error; |
There was a problem hiding this comment.
🟡 Partial completion blocks all attempts
When a run stops during completion.json writing, savedAttempt throws on partial JSON. The rerun aborts instead of marking that attempt unfinished, blocking other attempts.
Learn more
A model call creates completion.json using a normal write after the provider replies. If the process stops mid-write, the directory and a truncated completion file remain. On rerun, parsing that file throws before the existing-directory check can mark the attempt unfinished. The worker pool then rejects and stops scheduling other attempts.
Example: A run stops after writing the first half of spiral-staircase/1/completion.json. The next generate command exits on a JSON parse error instead of reporting that attempt unfinished and continuing with the remaining tasks.
Recommended fix: Treat a JSON parse failure as an unfinished attempt by returning null from savedAttempt. The caller already checks that the attempt directory exists and refuses to resend its request.
| completion = JSON.parse(await readFile(path.join(dir, 'completion.json'), 'utf8')); | |
| } catch (error) { | |
| if (error.code === 'ENOENT') return null; | |
| throw error; | |
| completion = JSON.parse(await readFile(path.join(dir, 'completion.json'), 'utf8')); | |
| } catch (error) { | |
| if (error.code === 'ENOENT' || error instanceof SyntaxError) return null; | |
| throw error; |
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Fixed at the source. completion.json is now written to completion.json.partial and then renamed, so it is complete or absent. A run that stops before the rename leaves only the folder, and a rerun reports that attempt as unfinished and continues.
1 participant
Adds the code that generates attempts for the 100 tasks: one request and one reply per attempt, through OpenRouter or any OpenAI-compatible server. The README describes the request and how to run it. The grader comes in a later release.
node tools/3jsbench.js generate --model MODEL --out DIRsaves each reply verbatim asDIR/<task>/<n>/asset.ts, beside the request, every response and the parsed completion.node tools/3jsbench.js prompts --out DIRwrites each task's exact messages.npm testruns offline.🤖 Generated with Claude Code