I work on LLM inference, mostly on Apple silicon with MLX, Core ML and the Neural Engine. A lot of that is measuring runtimes: how well the cache gets reused, what happens under concurrency, whether the same request gives the same output, where the latency actually goes. When I find a bug in something I depend on, I send the fix upstream — including to MLX, oMLX, and Apple's coremltools.

meowcoder.com · Research notes · tc3oliver@gmail.com

Apple coremltools

• Open · #2876: MLModel.predict() held the GIL for the whole native Core ML call, so other Python threads in the process stalled behind it. The patch releases it around the native call, with a threading test.

MLX

• Merged · #4615: The distributed launcher could lose a rank's final output when the process exited before its pipes were drained. The fix reads both pipes to EOF without blocking.

oMLX

• Merged · #3685: The SDPA256 prefill path was picked by how much memory happened to be free, so the same request could give different temperature-0 output in two processes. Now the choice depends only on the input shape.
• Merged · #3840 + #3842: On hybrid attention/recurrent models the SpecPrefill draft cache never hit: a restored cache looked empty, and recurrent state wasn't saved at block boundaries.
• Merged · #3664: The Responses API dropped namespace tool groups, which is how Codex passes MCP tools, so the model never saw them.
• Merged · #4035: After a failed SpecPrefill request, stale state could hang the engine or make the retry prefill the wrong tokens.

_{Status syncs from GitHub weekly. The rest: All upstream pull requests.}

laya-apple: runs Laya on the MLX GPU and the Neural Engine concurrently, with every output checked against upstream Laya. Profiling showed GPU tail latency climbing whenever the Neural Engine ran; tracing it led to the coremltools patch above. laya-apple uses Core ML's async API instead of waiting for the patch, and falls back to its earlier path if it detects the slow state. Median GPU result return on an M4 Max went from 4–9 ms to about 0.04 ms, with no mismatches across 154 validation runs. Research notes · PyPI

llm-inference-systems: experiments on what inference optimizations cost later requests: a sparse prefill that stops the reusable prefix growing, speculative decoding where verify cost matters more than acceptance rate, and rebuilding the reusable state in the background. The oMLX maintainer closed my pull request (#3964) without merging, arguing recovery at best matches SpecPrefill off; I never ran a clean off to compare against. Case study · Engineering notes

Embed2ASR: a small reader trained on a frozen EmbeddingGemma 2 audio encoder transcribes Mandarin, English and code-switched speech. On one MI300X it ran 4.16× faster than Whisper large-v3, which wins on most test sets. Case study · Write-up

qwen3.8-27b-5070ti-eval: a pre-registered evaluation of a 27B model on one 16 GB RTX 5070 Ti, with raw data and graders. It scores 92.1 % on HumanEval+. The first competitor spilled out of VRAM mid-run, so I voided that run and round 1 makes no comparison. Under WSL2 a 128K context loaded without an error, then decoded one test request at 5.9 tok/s, apparently because some memory had landed in system RAM. Article · Evidence index

PiShip: lets a company ship Pi as its own coding agent without forking it. Pi keeps the agent loop and tools; PiShip adds OIDC login, short-lived credentials for an internal LLM gateway, policy, MCP rules and a sandbox. If a required sandbox can't start, the command doesn't run. Pre-release; every release archive is attested and built twice to check that the payloads match. TypeScript. Status · Enterprise integration

Smaller things: SignalForge, a self-hosted intelligence pipeline; version-aware-code-mcp, an MCP server that keeps a coding agent's code search on the right commit; deepseek-v4-flash-mi300x, a vLLM serving baseline on AMD MI300X; claude-team-kit, a Claude Code plugin that limits concurrent agent-team workers and shows them in a Mission Control pane (pre-release); Shouri; my coding-agent skills.

I write up research at study.meowcoder.com, in Traditional Chinese. Two good places to start: 當 prefill 變快，agent 反而變慢 and 償還 reusable state 的債.

20 pull requests to projects I don't maintain: 11 merged · 9 open.

Apple coremltools (1 open)

• ○ #2876 Release the GIL during native MLModel prediction

MLX (1 merged)

• ✓ #4615 Fix lost rank output in the distributed launcher

oMLX (6 merged · 7 open)

• ✓ #4035 fix(scheduler): clear SpecPrefill state after failures and cache rejects
• ✓ #3685 fix(attention): keep SDPA256 prefill on the bounded route
• ✓ #3842 feat(specprefill): preserve draft recurrent state at cache boundaries
• ✓ #3840 fix(specprefill): derive draft cache position from attention layers
• ✓ #3746 fix(ane): avoid impossible sequence-length guidance below the ANE minimum
• ✓ #3664 fix(responses): route namespace tool groups through to the model and back
• ○ #4500 perf(mtp): let greedy chains draft through the coarse lm_head (opt-in)
• ○ #4461 fix(specprefill): keep manual_rope output in the input dtype
• ○ #4215 fix(mtp): keep greedy Qwen3.5 verify independent of the draft depth
• ○ #3962 test: reset the image decode cache between tests
• ○ #3811 fix(specprefill): keep selected tokens at their original positions on mRoPE VLMs
• ○ #3762 feat(anthropic): accept per-request SpecPrefill overrides on /v1/messages
• ○ #3756 fix(specprefill): preserve the full static system/tool prefix

Other projects (4 merged · 1 open)

• ✓ NandhaKishorM/laya#260 docs: add laya-apple to Community Tools
• ✓ eiei114/pi-model-fallback#64 fix(error-status): parse "Error: <status>" with no colon after the status
• ✓ DrJuChunKoO/TransPal-gemini-transcriber#1 feat: add resume-from-interruption support for long audio transcriptions
• ✓ Baseflow/screenrecorder#9 fix: fix the black background
• ○ ray-project/llmperf#100 feat: Support reasoning_content in OpenAI chat completions streaming response