Inside the catalog and the picker

This page is for people working on the runtime itself. What a grade is, how a deployment writes its catalog and pins, and how a job asks for a model are in the guide; this page holds the mechanics behind them. Tokenization lives in this package too, because counting is a property of the model.

The catalog: Models and ModelSpec

Models is the static catalog facade (mirrors Prompts) and PURE catalog - what exists, what it can do, what it costs; WHO gets a model is the ModelPicker's business: spec(id) (throws ModelNotFoundException on a miss), findSpec(id) (null on a miss, for the billing path), all(), variants(identity), pool(Grade) (the candidate rung - specs whose grade IS the argument; over-qualified is capable but not a candidate), embeddingsPool(), decisionPool(), and updateSpec(id, spec) - the sole channel for learned factual rate limits, which replaces the stored spec and refreshes its limiter. reload() adopts the deployment's catalog as it stands now - backend, resolution cache and learned overrides dropped, for the moment a discovery has just rewritten models.json under a running process; the freshly written file carries the account's limits fresher than any learned header.

A ModelSpec is one concrete (provider, model) deployment: an endpoint-suffixed id (claude-opus-5-bedrock), a shared logical identity (opus-5), providerKey, wireModelId, capability fields (context/output tokens, vision, documents, thinking mode/budget), and the endpoint's tpm/rpm and cache multipliers. equals/hashCode are by id, so a spec is a stable rate-limiter key even as its learned limits move. Provider subtypes add endpoint-specific fields (AnthropicModelSpec.cacheBreakpoints); StandardModelSpec covers the rest.

AnthropicNaming is how every surface that serves Claude names it the same way: isClaude(wireModelId) recognizes a Claude wire id in any surface's spelling (us.anthropic.claude-opus-4-7, claude-opus-4-5-20251101-v1:0), identityOf reads the identity (opus-4.7), and catalogId(identity, channel) names the entry (claude-opus-4-7-bedrock). It sits here, beside ModelLineage, because the providers that use it - the direct API, Bedrock's native surface, Mantle, Converse - live in different provider modules, and it is naming alone, with no client behind it.

Pricing

Pricing lives on the spec as optional inputPricePerMillion / outputPricePerMillion (nullable; null means no configured price) in the entry's currency, an ISO 4217 code an entry carries or inherits from its provider_defaults (the shipped fragments say USD). No provider reports cost on the wire, only tokens: a price is what a person typed from a list page or an invoice, in whatever currency that page bills in (Azure in local currency, Qwen and DeepSeek in yuan, Mistral in euros, Bedrock per region). A price with no currency is a load error. The runtime accounts a call's cost in the entry's currency and never converts: a rollup sums per currency, and a spend cap stated in one currency refuses a call priced in another rather than guessing a rate. Prices are public list prices per endpoint, so they sit per models.json entry rather than in provider_defaults (Bedrock and direct variants of one identity can differ, and same-provider variants like gpt-5 / gpt-5-mini differ). Cache prices default to the base input rate scaled by the cache multipliers (exact where the public cache price is that simple multiple, as with Anthropic); a model whose cache price diverges sets cacheReadPricePerMillion / cacheWritePricePerMillion explicitly (e.g. OpenAI's GPT-5 caches at 0.1x while its billing multiplier is 0.5). getEffectiveCacheReadMultiplier() / getEffectiveCacheWriteMultiplier() expose the resolved weight (the cache-to-input price ratio when an explicit price is set, the nominal multiplier otherwise); both billable-token accounting and cost read through them, so the two stay reconcilable. isPriced() says whether an entry carries the input and output prices and the currency a call's cost needs (an embeddings or decision entry needs no output price). Consumers reverse-look-up a recorded model id via Models.findSpec to price it.

Status

status says whether a spec may be served, and who said it may not: OPEN (the default, omitted in the file) is picked and pinged; DISABLED is the deployment's own decision, written by hand and never touched by the discovery; DEPRECATED is the vendor's word, written by the discovery from the provider's lifecycle flag; UNLISTED is the account's word, written by the discovery when its listing no longer names an open entry (a deleted Azure deployment, access withdrawn on Bedrock); UNREACHABLE is this deployment's reach, written by the discovery when the provider's endpoint is not served where the process points (a Bedrock surface absent from the configured region) or the entry's own ping fails for a reason no retry changes. Those two the discovery also clears, reopening and pinging the entry when a later listing names the model again, or the provider answers again. No closed status removes the entry, because the call record stores whatever id served each logged call and Models.findSpec reverse-looks-up that id to price it: dropping an entry would silently unprice every historical row that used it. So a closed spec stays fully resolvable (by catalog id and by wire-id alias, with its prices intact) while the resolution gate (ModelPickers.resolve) refuses to serve one to new traffic and the discovery never pings it. note is free text on an entry, never read by the runtime: the discovery writes one on every entry it adds (the date, the ancestor whose shape it inherited), a person writes why a model is disabled.

Where specs come from: the backend and the fragments

Specs come from a pluggable ModelsBackend (the immutable DEFAULTS layer under the facade's runtime override layer), selected via ModelSettings.backendClass. The default JsonModelsBackend discovers the catalog on the classpath the way logback finds its configuration. The deployment's own file is the catalog when it exists, found the way logback finds its configuration and the same way on every machine: the file named by -Dnucleo.models=<path>, else every models.json on the classpath, enumerated with the nearest to the application taken - launchers order the application's own classes before its dependencies - and the shadowed copies named in a warning. A named file that cannot be read is refused rather than passed over for the classpath. CatalogDiscovery writes the file --out names, else the one the property names (JsonModelsBackend.namedFile()); an application's catalog goes into its src/main/resources, where the next build carries it onto the classpath. It is complete and alone - the deployment decides what exists, what it costs and what it may carry, and nothing shipped in a jar is consulted beside it. Only when no such file exists does the runtime fall back to the fragments: every META-INF/nucleo/seed_models.json on the classpath, one per provider artifact, carrying a bootstrap kernel rather than the provider's whole catalog - the current model of each shipped family per channel, with its public facts (identity, wire id, ceilings, vision, thinking mode, list prices, grade) and the provider's published entry-tier limits. The kernel is deliberately small because a shipped catalog freezes at release age and its models retire; its job is to make one credential sufficient - serve the grades before any discovery runs, seat the discovery's classifier, and anchor grading, id conventions and family lineage (including facts a classifier cannot know, like requires_lax) - and the discovery authors everything else into the deployment's file. A fresh checkout with one credential runs before anyone authors anything, under a boot warning that names the file to author; a classpath with neither a file nor a fragment refuses to load, naming the places it looked. Two entries with one id are a load error, within a file and across fragments alike, and a fragment may not declare pins.

Within a file, provider_defaults (per provider key) sits under each models entry, so an entry carries only what differs from its provider's defaults; the explicit spec_type selects the concrete spec subtype; every entry is validated fail-fast (a misspelled limit key throws at load rather than silently defaulting to zero).

Provider links

A catalog is written for the providers of the classpath that wrote it, and a process whose classpath carries others reads the same file: a native image built without the Anthropic SDK reads the catalog a JVM build wrote with it. So a written catalog records its providers, "providers": {"<key>": {"platform": "<platform>", "addressing": "<addressing>"}}, and each entry its own platform, both written by ProviderLinks.stamp (the discovery and the demo's first-run seed call it). A provider's addressing (ClientProvider.addressing) is the endpoint family its wire ids are spelled for: its platform by default, and its own where its endpoint names models its own way (Mantle's bare Claude ids, which bedrock-runtime refuses, are bedrock-mantle). The record names every provider the writer carries, and every provider an entry names that the writer does not, when the entry or the catalog's earlier record states its platform. At load, an entry whose provider this classpath does not carry is linked to one here that addresses its model the same way: ProviderLinks.resolve takes, among the providers of that addressing (the record's word for the entry's provider, else its platform) and of the entry's kind (ModelKind.ofProviderKey), the first in key order that claims the model (ClientProvider.claims, a provider written for that family: the Anthropic ones claim Claude), else the first that serves it (ClientProvider.serves). The linked entry keeps its id, wire id and every field it states; its own spec_type is dropped so its new provider's defaults shape it. An entry nothing here can serve is left out of the loaded catalog, so no picker and no selector offers it, and a pin naming it is dropped so its seat falls to the picker: one no provider here of its addressing fits (a Mantle entry in a build without the Anthropic add-on), and one with no platform on record whose spec type no artifact here declares (a present provider's entry naming an undeclared spec type is still a load error). All of this is in memory: the raw views the discovery reads (rawEntries, rawPins, rawProviders) keep the file as written, the runtime's own load warns once per source, saying what differs and what was done, and once more for any pins it dropped (a view a tool or a status page opens on the same file stays quiet), and only the discovery rewrites the file. An entry with no platform on record and a spec type this classpath builds loads as written; no picker lands on it while its provider is absent (ProviderLinksTest, CatalogLinkingTest).

An entry has a seat in the runtime (ModelSpec.hasSeat()) when it carries a grade (an LLM entry, on a rung) or rides an -embeddings or -decision key; a seat is what pickers, pools and the benchmark land on. An entry may instead state a modality the runtime has no client for: input_modalities and output_modalities, the provider's own words as the discovery recorded them from the account's listing (TEXT, IMAGE, VIDEO, SPEECH, EMBEDDING, ...). An entry whose stated modalities are not text in and text out, and not an embedding out - an image generator, a speech model, a transcriber whose input states no text - is a model the account offers, written so the catalog states what the account has: it carries no grade (the loader refuses one), needs none of the limits and ceilings a call would need (none is ever made), and is reachable by id alone (SeatlessEntryTest). Hand-written LLM and embeddings entries state no modalities and are held to what a call needs.

pins is the deployment's model policy in the same file - {"SMALL": ["<id>", "<id>"], ..., "embeddings": "<id>", "decision": "<id>"} - read by DefaultModelPicker, so a deployment states which entries serve each grade, in its order of preference with the first the grade's default (an array of ids, each a graded LLM entry of that grade or above and named once; an empty array declares none), which embeddings entry the corpus is tied to and which decision entry answers decisions, without writing a picker class; Models.pins() exposes it as a CatalogPins, and is null for a catalog that declares no pins at all.

Kinds

Every entry belongs to one of three client families, ModelKind (ModelSpec.kind()), and the kind follows the provider key: providers registered under *-embeddings keys serve the EMBEDDINGS family, providers under *-decision keys the DECISION family, and every other provider the LLM family. A seat asks for a kind and the resolution gate refuses an entry of another kind whichever picker answered, on the picked and the pinned path alike (DecisionResolutionTest). An LLM entry carries a grade; an embeddings entry may pin embedding_dimensions; a decision entry carries no grade, since it is not on the ladder, has no output ceiling the runtime reads (it generates nothing, so the loader leaves max_output_tokens alone and the shipped fragment writes 0), must state max_context_tokens, the state it reads, and states exactly one bound: tpm for a hosted endpoint bounded by its quota window (rpm beside it where the endpoint publishes one; without it the request budget derives as tpm / 1000), or max_concurrent for a model served from a machine the deployment owns and bounded by how many requests its server takes at once. Both and neither are refused at load, as is max_concurrent on any other kind and a grade pin naming a decision entry (DecisionCatalogTest). Which account admission takes for a call follows the bound: a token reservation on the entry's bucket for a window, one permit on the entry's ModelGate for a concurrency (DecisionDemandTest).

Building the deployment's file

Building the deployment's file is one development-time action, CatalogDiscovery, in the discovery/ subpackage: it walks the providers on the classpath, lists what each account can reach, pings every open entry, and merges the result into the catalog under the rules that doc states. Each LLM entry carries its identity's grade (MICRO < SMALL < MEDIUM < LARGE < XL < MEGA), identity-uniform across endpoint variants - the same curation judgment as status. Billing records the spec id and spec.getProviderKey() on every call.

Per-entry translations

A seat speaks the framework's vocabulary - a Grade (which model), a Depth (how much effort), an OutputSize rung (how much answer) - and the catalog entry is where those words become numbers, because the translation depends on the model and even on the endpoint. Each translation has a framework default that works without any declaration, and an optional per-entry override for the engineer who knows better; a partial override is refused at load, so every rung of a translated entry comes from one source:

thinking_mode says what the thinking number does: EXTENDED sends it as budget_tokens; ADAPTIVE lets the provider pick from the effort word and keeps the number as headroom and reservation; REASONING_EFFORT (the OpenAI reasoning generation, direct or as gpt-oss on Converse) reasons inside the ordinary completion ceiling on every call, IMMEDIATE included, so the number is headroom above the answer and the client sends reasoning_effort from the depth; NONE attaches nothing. The two ThinkingMode predicates keep the consumers apart: reasoningReserved (is headroom booked - true for REASONING_EFFORT at every depth) feeds the reservation and the wire ceiling, thinkingActive (does a native thinking block come back and supersede the prose reasoning field - Anthropic only) feeds the request config and the schema strip.

Resolution: grades, the picker, the gate

Call sites never resolve models. A job declares its need in getRequirements() - requireModel(Grade, Depth, ...) / requireEmbeddings(...) - and each returned ModelBinding (a write-once deferred binding cell, wired two-way to its conversation when one exists) is resolved by the dispatcher at the requirements-resolution step, per attempt, BEFORE any resource is acquired: the deployment's ModelPicker (configured via ModelSettings.pickerClass, consulted only through the ModelPickers.resolve gate) turns the seat + situation into a spec; the gate enforces kind (embeddings vs LLM), the ComplianceEnvelope sealed into the dispatcher (an envelope refusal of an unpinned seat is not terminal: the seat is served from the best rung the envelope permits - a CEILING seat walks downward from the picker's ceiling, since it declared no floor, and an explicit-grade seat walks upward from its floor; a pin the envelope refuses stays refused, and a walk refused whole raises ModelResolutionError, an Error - a deployment whose policy permits no model for a seat is broken, not correctable), the inputs (an input the conversation carries, ConversationContext.carriedInputs, that the binding did not declare through ModelBinding.setSends is refused, and so is a declared input the entry does not accept: Input.IMAGES needs supports_vision, Input.DOCUMENTS needs supports_documents), deprecation, and last, on the picked path alone since a pin declares none, the grade floor (under-qualification refused, over-qualification legal; the floor is the picker's ceiling at most, so a seat asking for a grade above everything the deployment serves is lowered to the ceiling, once-warned, instead of refused); the binding then prices itself (wired: conversation counted under the resolved tokenizer plus the conversation's output reserve; prompt form: text counted plus the declared output and the depth's thinking; flat: the declared input count plus the same) and the reservation meters against the resolved spec's limiter. ConversationContext.getModel() is delegation to the binding - a pre-resolution read throws. Transparent-retry re-entry re-runs resolution with the accumulated attempt history, which is what makes failover a pure picker policy. The shipped default picker (DefaultModelPicker) serves a seat from the first entry, in this order, that is eligible - open, a provider artifact on the classpath declaring its provider, its data-share posture provisioned where it needs one, its provider's credential in the store (a local check), permitted by the envelope - and accepts every input the binding declared: the grade's entries in the order the catalog's pins give them (CatalogPins.grades()), one that cannot be called skipped and logged once with the reason, one that does not accept a declared input passed over silently; then the grade's other entries, cheapest first by list price per million (input plus output; an unpriced entry after every priced one, in catalog order), logged once at WARN naming the entry and the pins that would make the choice deliberate; then the same two steps at the next rung up, logged once; and only a grade with nothing at it or above raises ModelResolutionError, naming per provider what to provide (a seat above everything configured never gets that far: the gate lowers it to the picker's ceiling first). The payload never chooses: the default picker reads what the binding declared and never what the conversation carries, and it caches nothing, so its choice follows the credentials and the order as they change (DefaultModelPickerTest). Which model serves which grade stays a deployment decision, stated in models.json or in a picker class - the only legal homes for model ids. A picker class is declared as an instance, once at startup and before the first resolution, through ModelPickers.use(picker); ModelSettings.pickerClass names a no-arg class for a deployment that prefers configuration. A second declaration, or one after a resolution, throws.

Embeddings are frozen configuration, not policy. Stored vectors are only comparable to vectors from the model that produced them, so the embeddings model must be specified precisely and can never change ad-hoc: it does not flow through provide() at all. The deployment declares it once - ModelPicker.embeddingsSpec() - and ModelPickers.resolveEmbeddings gates that declaration (kind, envelope, deprecation) and FREEZES the first resolved id for the process lifetime; a declaration that diverges mid-process refuses instead of silently corrupting every similarity search. The availability turnoff never touches it (embeddings seats never reach pick()), and changing the model is a deliberate corpus migration, never a resolution outcome.

Decision models are deployment configuration, not policy. Decision models are not on the capability ladder: they are ranked by a different trade, and a deployment runs one of them. So a decision seat (requireDecision) never reaches provide() either: the deployment declares its entry - ModelPicker.decisionSpec(), the decision pin under DefaultModelPicker (unless no provider on this classpath serves it), else the first callable decision entry whose provider serves its wire id (a decision provider's endpoint is the credential's host and one host serves one family, so a catalog carrying both the Jev and the Kev entry on the same key serves the one the endpoint lists) with a warning naming the pin that would make it deliberate - and ModelPickers.resolveDecision gates that declaration (kind, envelope, deprecation). Unlike embeddings the declaration is not frozen, since nothing stored depends on which decision model answered; a picker declaring none refuses by name (DecisionResolutionTest).

Exact pins. A job may name its exact spec - requireModel(ModelSpec, Depth, ...) / requireEmbeddings(ModelSpec, tokens) - discouraged (a grade seat lets the picker keep the system resilient) but first-class. The declaration FORM sets the kind and validates the spec's kind at mint; the pin skips ONLY the picker consult - ModelPickers.resolvePinned still runs the envelope, kind, inputs, and deprecation walls (no grade floor: a pin declares none) - and a retry re-mints the same spec, so pins have no failover, structurally. The string-literal rule is untouched: pins take ModelSpec objects, and ids still live only in models.json and the picker class.

Ceiling. ModelPicker.ceiling() is the strongest grade the deployment serves, the dual of a seat's floor. It is derived, never pinned: DefaultModelPicker takes the highest rung with an entry it can call, so it follows the credentials as they change, and a ceiling or CEILING key in pins is refused at load. A seat that wants the best model available (every chat) declares Grade.CEILING, the last constant of the ladder and not a rung: Grade.rungs() omits it, atLeast refuses it, and no catalog entry or pool may carry it. ModelPickers.resolveWith translates a CEILING seat to the picker's ceiling before the picker sees it, so one chat class rides MEGA where the deployment can call a MEGA entry and XL where its strongest callable entry is XL, with no app subclass naming a rung; a picker whose ceiling() is null or names CEILING itself is refused there, and a picker's own switch treats CEILING as unreachable.

Availability state. Besides ceiling(), the ModelPicker interface carries two default methods any implementation may ignore: recordProbe(ProbeOutcome) (an availability observation, returning the displaced one for drift) and describePins() (the picker's standing grade pins, for reporting). AbstractModelPicker is the stateful base: subclasses implement pick(Seat, Situation), and the final provide template refuses a spec whose probes say it is down - availability-shaped failures only (connection/5xx/not-found; a 429 is busy, an auth failure is a config incident, an envelope exclusion is neither - none of those ever turn a model off), with no success inside the 24h window, and never on missing data. A deployment's availability probe is the only writer, through ModelPickers.recordProbe; the runtime ships the ping recipe (CatalogPing) and no scheduled probe. Pins bypass the turnoff template deliberately - that is how a probe observes recovery, and it means an explicit pin can ride a turned-off model; catalog deprecation is the gate's and refuses pins too.

529 failover. Two halves, both requiring an undeprecated, envelope-permitted, not probe-dead candidate of the SAME identity (identical weights, different pipe) that passes the deployment's route policy - eligibleForFailover(candidate, situation), ABSTRACT on AbstractModelPicker because which endpoints may substitute is a deployment decision the framework cannot answer (return false to disable substitution entirely). Named answers ship as intermediate bases: BedrockOnlyModelPicker (substitution stays on Bedrock, never a direct endpoint), ZdrOnlyModelPicker (Bedrock AND not requiresLax - and via the vetPick family hook it refuses its own subclass's PICKS too, so a ZDR promise cannot drift), and OpenModelPicker (no route policy; the envelope is the only fence). The retry half: when the route a pick would repeat just answered overloaded, the retry serves a sibling route, excluding every route already overloaded in this execution. The proportional half: a drowning route needs NEW traffic taken off it too, or nothing alleviates the pressure - the adaptive limiter already integrates the provider's signals (529 weighted 3x, successes decay it), so its slowdown factor drives the diversion of fresh picks to the healthiest sibling: none below 2x, 1 - 1/x between (half at 2x, two thirds at 3x), everything at 10x, with the candidate required to be strictly less throttled than the pick. The share decays to zero as the route recovers - no state beyond what the throttle already keeps. No candidate means the pick stands and the dispatcher's backoff continues.

Tokenization

TokenizerFactory.get().forModel(spec) hands out the TokenCounter for a spec. The shipped DefaultTokenizerFactory resolves by provider key rather than model name (a Bedrock-served Claude's wire id would defeat a name-prefix check): Anthropic and Bedrock-Converse models count through the Claude counter, Gemini models through a Gemini-calibrated one, Cohere models through a Cohere-calibrated one, OpenAI models through tiktoken; register overrides a specific model. Counters are memoized per spec; the per-family calibration numbers (multipliers, image base tokens, PDF tokens per KB) are Cl100kTokenCounter's own measured constants. Counting rides with the catalog because it is a property of the model: the clients' pre-flight estimate, the binding's pricing and the context-window management all count through the same counter.

Related packages

• Discovering an account's models - CatalogDiscovery and the listing/ping machinery that authors a deployment's models.json
• Model clients and rate limits - the clients a spec resolves to: provider abstraction, rate limiting, prompt caching, embeddings
• Conversations: what every call carries - the conversation the binding wires to, output declarations, outputReserve