AWS has documented two paths for serving models with llama.cpp on Amazon SageMaker AI: a standard AWS Deep Learning Container deployment for OpenAI-compatible chat completions, and an extended runtime using SemIf for structured decisions. The distinction matters for agent workflows. A chat endpoint generates an answer; the decision path returns probabilities over options declared by the application, without decoding an answer token. Both can run on CPU or GPU infrastructure, according to AWS’s deployment guide.
For cloud architects, the deployment question therefore starts with the required output. AWS’s published container provides the chat-serving route. Its structured-decision route adds SemIf and a child container image. OpenAI-compatible chat payloads and option-scoring requests serve different purposes, and the documented deployment paths reflect that difference.
What a decision model returns
The ggml-org article on Hugging Face describes a decision model as a model that scores supplied options rather than generating text. It reads the input once and returns a provided option with an attached probability. The article identifies request routing, content moderation, checking whether an agent’s step worked, and choosing its next action as typical uses.
That interface gives the application control over the available choices. For an agent, the useful architectural distinction is that a decision can be represented directly as a choice among declared options. The application still has to determine what those options mean and what action to take with the result. An attached probability is a model output; it should not, by itself, be read as independently established decision quality.
The ggml-org article documents decision-model support in the llama.cpp server through /v1/systemone. AWS describes SemIf, formerly OpenJev, as an extension that returns probabilities over declared options without decoding an answer token. These descriptions explain the decision-serving mechanism, but they do not establish that the standard SageMaker chat container exposes the same interface unchanged.
The standard SageMaker deployment path
AWS’s standard path packages the model as model.tar.gz in Amazon Simple Storage Service, or Amazon S3, and references the published Deep Learning Container image directly. The container finds the first .gguf file under /opt/ml/model.
Requests pass through SageMaker AI InvokeEndpoint. The container maps POST /invocations to llama-server at /v1/chat/completions and accepts the OpenAI Chat Completions schema. This is compatibility at the chat request interface, within the documented SageMaker invocation path.
AWS also documents the SM_LLAMA_CPP_* family of environment variables, which the container passes to llama.cpp arguments. These are the configuration bridge between the managed container and its model runtime. The supplied evidence identifies the variable family but does not provide individual settings or a complete deployment configuration, so it does not support a copy-and-run configuration example.
The model-discovery behavior deserves attention when preparing the artifact: AWS specifies the first .gguf file under the model directory. Architects should account for that behavior when deciding what to package, rather than assuming that the container selects among multiple files according to an undocumented rule.
The structured-decision path adds a child image
For structured decisions, AWS extends the runtime with SemIf. Its documented build path uses AWS CodeBuild to create a child image and stores that image in Amazon Elastic Container Registry, or Amazon ECR. The practical difference is an additional image-building step beyond referencing the published container directly.
| Deployment concern | Standard chat path | Structured-decision path |
|---|---|---|
| Runtime setup | Reference the published llama.cpp Deep Learning Container | Extend the runtime with SemIf |
| Image preparation | Use the published image directly | Build a child image with AWS CodeBuild and store it in Amazon ECR |
| Response behavior | OpenAI-compatible chat completions | Probabilities over declared options without decoding an answer token |
The implication for deployment planning is straightforward: choose the serving interface before treating the container setup as complete. The chat mapping to /v1/chat/completions explains the standard route; decision support through /v1/systemone is a separate documented capability. The available evidence does not supply the complete SageMaker routing configuration for structured-decision requests.
AWS also names other implementations: Jev, TypeSafe’s decision model and /v1/systemone API; Kev, a model server designed for TypeSafe API compatibility; and Strands Decider 2B, which AWS describes as purpose-built for agent workflows with calibrated outputs. These references establish alternatives in the decision-serving ecosystem, without providing comparative deployment or accuracy results for them.
Model selection involves language, images and licensing
The ggml-org article lists models with different sizes and input capabilities. A selection illustrates the trade-offs:
| Model | Parameters | Languages listed | Image support | License |
|---|---|---|---|---|
| Julia-1 | 144M | 50+ | No | Apache 2.0 |
| Laya | 421M | English | No | Apache 2.0 |
| Kev-4B | 4B | English | No | Apache 2.0 |
| OpenJev | 27B | English, German, French, Hindi, Chinese and Japanese | Yes | CC BY-NC 4.0 |
| Clef | 27B | English | Yes | Apache 2.0 |
OpenJev is described as able to read images such as documents or screenshots, with its vision projector downloading automatically. That is a model-specific capability, not a property established for every decision model. The license differences likewise remain relevant when selecting an artifact; the models should not be treated as interchangeable simply because they appear in the same runtime discussion.
What AWS’s CPU and GPU results show
AWS reports testing 231 public JevBench tasks with Qwen3.5-4B Q4_K_M. Its two endpoint configurations produced the following results:
| Configuration | Correct answers | Accuracy | p50 latency | p95 latency |
|---|---|---|---|---|
| <code>ml.c6i.8xlarge</code>, 32 threads; <code>llama-cpp:server-sagemaker-cpu-v1</code> | 179/231 | 77.5% | 1.715 seconds | 19.291 seconds |
| <code>ml.g6e.xlarge</code>, full GPU offload; <code>llama-cpp:server-sagemaker-cuda-v1</code> | 180/231 | 77.9% | 0.352 seconds | 0.599 seconds |
AWS describes the GPU endpoint as 4.9 times faster at p50 and 32.2 times faster at p95. These are AWS-reported results for the listed model and configurations, rather than an independently verified performance guarantee.
The accuracy difference amounts to one additional correct answer across the 231 tasks. In this comparison, the larger observed difference is latency, particularly p95, which describes the slower end of the measured response distribution. That makes the benchmark relevant to infrastructure selection, but it does not establish the same improvement for another workload.
The ggml-org article separately reports median answer times on one NVIDIA RTX PRO 6000, including 3 ms for Julia-1, 12 ms for Kev-4B and 43 ms for OpenJev. Those model timings and AWS’s SageMaker endpoint results have different stated measurement settings. They should not be combined into a single deployment ranking.
For an agent that must choose among declared actions, the structured-decision interface is the relevant starting point. AWS’s evidence supports a managed deployment route and a measured CPU–GPU latency comparison. Selecting an image, model and instance still requires keeping the output contract, model capabilities and benchmark conditions aligned with the application’s actual decision task.
