Skip to content

Upgrade llama.cpp Java binding to 4.2.0 and adapt adapters and loader to new API - #64

Merged
jsilvanus merged 1 commit into
mainfrom
codex/compile-and-run-inference-system
Sep 11, 2026
Merged

jsilvanus merged 1 commit into
mainfrom
codex/compile-and-run-inference-system

Conversation

@jsilvanus

Copy link
Copy Markdown
Owner

Motivation

  • Unify the JVM and Android engines on the current de.kherud:llama:4.2.0 binding to ensure consistent GGUF compatibility and a request-oriented streaming API.
  • Adapt the inference adapters to the newer llama binding API shapes so local inference and streaming work correctly with the updated native core.
  • Improve streaming behavior so tokens are emitted lazily as they are generated and make model identification clearer via modelVersion.
  • Avoid exposing a nullable artifact path on the public loader API by consolidating shared logic into a private implementation.

Description

  • Bumped the llama binding dependency to de.kherud:llama:4.2.0 in engine/modelruntime and engine/androidapp and added clarifying comments about binding selection in the Android Gradle file.
  • Updated LlamaCppAdapter to the new parameters and constructor shapes by using ModelParameters().setModel(...).setCtxSize(...).setThreads(...).setThreadsBatch(...).setBatchSize(...).setUbatchSize(...).setGpuLayers(...) and calling LlamaModel(params) instead of the old constructor, and updated modelVersion to java-llama.cpp-4.2.0.
  • Changed generation code in both LlamaCppAdapter and AndroidLlamaCppAdapter to use InferenceParameters(prompt) with setNPredict, setTemperature, setTopP, and setTopK, and iterate model.generate(parameters) to stream token deltas as they arrive.
  • Refactored ModelLoader to introduce a private loadInternal implementation that accepts a nullable artifactPath and exposed two public overloads for normal id resolution and exact artifact loading.
  • Adjusted tests in GgufRot13FixtureTest to construct ModelParameters and InferenceParameters using the new API shapes.

Testing

  • Ran ./gradlew :engine:modelruntime:test, which executed the JVM unit tests including GgufRot13FixtureTest, and the suite completed successfully (native inference checks are skipped when the platform lacks a native binary).
  • Ran a full build with ./gradlew build to verify compilation across modules, and the build completed successfully.

Codex Task

@jsilvanus
jsilvanus merged commit 459fb20 into main Sep 11, 2026
4 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant