Models, Standard

Amazon SageMaker JumpStart Model Leaderboard: Compare Foundation Models on Quality, Speed and Cost

Symbolic scene of scattered model gauges converging into one glowing leaderboard platform with ranked model orbs

Choosing a foundation model usually starts with a simple question and ends in a long comparison. You read one model card, then another, and the benchmark scores you find were published by different teams, under different conditions, on different hardware. The SageMaker JumpStart Model Leaderboard is Amazon’s answer to that problem: a single view inside SageMaker Studio where open-weight JumpStart models are compared side by side on quality, speed and cost, measured the same way for every model.

Amazon SageMaker JumpStart gives you hundreds of open-weight foundation models you can deploy quickly. That choice is powerful, but it makes the starting question harder: which model should you start with? It does not remove the need to test a model yourself, but it gives you a consistent shortlist instead of incomparable numbers.

What the SageMaker JumpStart Model Leaderboard is

The leaderboard is a new view of the JumpStart base models page in SageMaker Studio, right next to the Categories and All models views. It uses the same search bar and filters you already know, and it opens on a leaderboard of large language models (LLMs). You see rankings without answering setup questions first. At launch, it covers 70+ models from providers including Meta, Google, Qwen, IBM and Arcee, with more added over time.

For each model, the leaderboard shows two kinds of information. First, quality scores on three public benchmarks that AWS runs itself, using the same evaluation setup for every model so the numbers are comparable. Second, performance and cost metrics measured by sending a standardized test workload to an endpoint deployed with each JumpStart deployment configuration, so you can see how fast a model responds and what it costs to run. Because both kinds of numbers come from the same place and the same method, you can make a like-for-like comparison instead of stitching together data from different websites.

What it measures: quality benchmarks

Three glowing model orbs in test chambers under one identical evaluation beam, with light bars of different heights

A benchmark is a standardized test: a fixed set of questions with known answers. Running every model through the same test gives you a score you can compare. At launch, the leaderboard reports three reasoning benchmarks for LLMs, and shows each one separately rather than blending them into a single number, so you can weigh the benchmarks that matter for your application. Higher is better on all three.

The three benchmarks are:

  • BIG-Bench Hard (BBH): a set of challenging tasks that require multi-step reasoning, such as logical deduction, tracking objects and multi-step arithmetic. JumpStart scores 21 of the 23 BBH tasks, covering 25 of the 27 subtasks, and excludes the movie_recommendation and causal_judgement tasks because of dataset licensing restrictions.
  • GPQA Diamond: graduate-level science questions that are difficult to answer by searching the web, a good signal of deep reasoning. It consists of 198 multiple-choice questions in biology, physics and chemistry.
  • MMLU-Pro: broad knowledge and reasoning across many subjects, such as science, law and engineering, using harder multiple-choice questions than the original MMLU.

The comparability comes from the method, not just the choice of tests. AWS runs each benchmark with the same evaluation harness and settings across all models on the leaderboard, following the original benchmark papers where available and using inspect_evals implementations as an additional reference. Responses that are missing, cannot be parsed or are cut off are scored as incorrect, and each question is attempted up to 30 times when transient API errors occur.

One detail matters when you compare model types. Reasoning and non-reasoning models are evaluated with different generation parameters. Reasoning models are run at temperature 0.6, top_p 0.95, top_k 20, min_p 0 and max_tokens 32768. Non-reasoning models are run at temperature 0, greedy decoding. GPQA Diamond scores for reasoning models are averaged over 10 independent passes. That difference is why comparisons between reasoning and non-reasoning models are approximate.

What it measures: speed and cost

A glowing model orb speeding through two control gates, one showing speed and one weighing time and cost

Quality scores alone do not tell you what a model will cost or how it will feel to use. The leaderboard therefore shows three performance and cost metrics for each model, measured on SageMaker under a standardized test workload:

  • Time to first token (TTFT): how long a user waits before the model starts responding, in seconds. Lower is better, and it matters most for chat.
  • Throughput per request (tokens/s): how many tokens the model generates per second for a single request. Higher is better, and it matters for batch and high-volume workloads. The leaderboard throughput column shows the number of concurrent requests used.
  • Instance cost ($/hr): the estimated cost per hour given a specified instance type. It is the on-demand price for the Studio Hosting instance type the model was benchmarked on, in the current Region. The instance type sets this cost, not the model, so two models benchmarked on the same instance type show the same instance cost.

Two menus above the leaderboard control which performance and cost numbers you see. The use case profile, selected from the View benchmarks for your use case menu, picks the profile closest to your workload; the default is Max Context. The profile changes only the performance and cost metrics. Quality benchmark scores are the same for every profile. The optimization goal, selected from the Optimized for menu, picks which deployment configuration’s metrics are shown, because a model can have several deployment configurations for each profile. The default is Balanced.

Each optimization goal picks a different configuration:

  • Balanced: the configuration with the lowest estimated cost per token, balancing what you pay per hour against how fast the instance generates tokens.
  • Cost optimized: the configuration with the lowest hourly instance price. Latency can be higher and throughput lower than with other goals.
  • Latency optimized: the configuration with the lowest time to first token.
  • Throughput optimized: the configuration with the highest throughput. These configurations can use speculative decoding.

For a chatbot your customers interact with, Latency optimized is a sensible start; for overnight document processing, Throughput optimized or Cost optimized may fit better. The performance and cost columns update when you change either menu, and quality scores stay the same.

Estimated cost reflects SageMaker instance cost only, based on the on-demand hourly rate for the configuration’s default instance type. It excludes data transfer, storage, inference component costs and discounts such as Reserved Instances or AWS Savings Plans; AWS points readers to the AWS Pricing Calculator for billing estimates.

The Instance cost column appears only if your Studio execution role has permission to query AWS Price List Service pricing. The AmazonSageMakerFullAccess policy does not include pricing permissions, so attach the AWSPriceListServiceFullAccess managed policy, or allow the pricing:GetProducts action. Without that permission, the other metrics still appear, but instance costs cannot be displayed.

How to use it: from browsing to a deployed model

To follow the walkthrough you need access to SageMaker Studio and permission to deploy JumpStart models.

Step 1: Open the leaderboard. In SageMaker Studio, open the Models page, then select the Leaderboard view. It has its own URL, so you can bookmark it or share a direct link with your team.

Step 2: Narrow the list with filters. Use the same search and filters as elsewhere in JumpStart to keep only models that fit your requirements. Common filters include license, provider, and input and output modality. Start with hard requirements — for example, a license that allows commercial use — before you look at scores. Requirements rule models in or out; scores only order what is left.

Step 3: Choose a use case profile and optimization goal. Select the profile closest to your workload (default: Max Context), then a goal from the Optimized for menu (default: Balanced). The performance and cost columns update, and quality scores stay the same.

Step 4: Read the heatmap and customize columns. Each column has a blue heatmap: the deeper the blue, the stronger the result in that column. Sort any benchmark column to rank models by the metric you care about most, use the column settings (gear icon) to hide metrics you do not need, and scroll horizontally to see more. The model name columns stay frozen as you scroll.

Step 5: Deploy or customize the model. Choose a model to open it, then deploy it or, for models that support serverless customization, customize it. If you deploy, choose the same use case and optimization goal you compared on, because each metric applies to the configuration it was measured on.

A worked example: picking a model for a chat assistant

Suppose you are building an internal help-desk assistant. Employees ask questions and expect fast, accurate answers that follow a set response format. You need a commercially licensed model and want to keep costs predictable. After filtering by license and choosing a chat-like use case profile with Latency optimized, your shortlist might look like this:

ModelBBHMMLU-ProTTFT (s)Throughput (tokens/s)Instance cost ($/hr)
Model A (large)95.987.22.3337$37.66
Model B (medium)94.284.90.9844$5.75
Model C (small)77.466.10.2761$2.61

Model A has the highest knowledge score but also the highest cost and the slowest first response. Model B scores almost as well, holds up on multi-step reasoning, and has a lower hourly instance cost. For a help desk, where reasoning through a question and responding quickly matter more than the last few points of general knowledge, Model B is the better starting point. As AWS puts it, a bigger model is not always the better choice, and the leaderboard makes that trade-off visible in seconds.

From there, you would deploy Model B, test it with real questions from your help desk, and keep Model C in mind as a lower-cost option if quality holds up. The gap between Model B and Model A on BBH is only 1.7 points, while the cost gap is more than sixfold. That is the kind of comparison the leaderboard is built to surface.

What the leaderboard does not do

The leaderboard’s limits are as important as its numbers. First, use it to build a shortlist, not to make the final call. Benchmarks measure general ability. Always evaluate your top two or three models on your own data and prompts before going to production.

Second, compare models within a benchmark. Each benchmark has its own difficulty, so a score on one benchmark is not directly comparable with a score on another. Treat close scores as ties: small differences can fall within normal run-to-run variation. Look for clear gaps.

Third, account for model type. Reasoning and non-reasoning models are evaluated with different generation parameters, and GPQA Diamond scores for reasoning models are averaged over 10 passes, so comparisons between the two types are approximate. Fourth, remember coverage is growing. Not every JumpStart model has benchmark scores yet, and launch benchmarks focus on LLMs. A missing score does not establish weakness.

Finally, leaderboard metrics can differ from those published by model providers and other external sources, because many benchmarks do not have a standard methodology. Public benchmark questions might also appear in a model’s training data, which can inflate scores, and scores can change with prompt format, few-shot examples and answer-extraction rules. Measured latency and throughput come from standardized test workloads, so your results can differ based on prompt length, output length, concurrency and Region. The leaderboard gives you a consistent comparison; it does not promise that your workload will reproduce its numbers exactly.

What’s next for the leaderboard

AWS says it is continuing to expand the leaderboard in three areas: private hub support, so organizations that use curated JumpStart private hubs can see a leaderboard for their approved models; more task leaderboards beyond LLM text generation, such as object detection, text embeddings and image understanding; and more models and benchmarks as new models are added to JumpStart. Next: open the Models page in SageMaker Studio, shortlist two or three models, and test them with your own prompts before choosing one for production.