Open-Weight Models for Our AI Router

Quick Summary

Key takeaways from the article
  • Open-weight models let you choose the model and its host separately. Where requests are processed depends on who runs inference, not where the model was developed.
  • In Zoolatech’s bounded four-feature evaluation, routing reduced model charges from $86.95 to $42.79—about 51%. About three-quarters of the router’s work went to a lower-cost model; this was not a hosting comparison.
  • Token rates tell only part of the story. Handovers between models accounted for 41% of the router bill while representing less than 3% of the text.
  • Independent hosting and self-hosting offer different controls and costs. Check processing location, request recipients, and retention; self-hosting also adds GPU capacity, maintenance, and operational costs.
  • Compare deployments on the same workflow using total charges divided by accepted results, alongside verification quality and response times. Both sides passed the evaluation’s checks, but premium-only sessions produced about four times as many automated tests.

“Developed in China” is not the same as “runs in China.”

That one sentence changes the economics of open-weight models. Our AI Router evaluation shows how, in dollars.

In September 2026, the router sent an API-contract task to GLM-5.2 instead of Claude Opus 4.8. GLM-5.2 wrote the contract and found a live bug: an invalid API key returned an empty string, so the documented 401 never fired.

A capable model did real work for a fraction of the price. The open question is where that model should run.

Open-Weight Models

What Open Weights Change

Three layers, three separate decisions:

  • The assistant: the tool a developer talks to, such as a coding agent or our router.
  • The model: the engine that generates responses.
  • The host: the infrastructure that runs the model and processes each request.

When a developer publishes its weights, anyone with the hardware and license can run the model. That gives a business options beyond the developer’s own API.

It does not make inference free, and licenses still apply. GLM-5.2 is MIT; Kimi K2.6 uses a modified MIT license with its own conditions.

Where Your Data Actually Goes

Where a request goes depends on the service and deployment you use, not on where the weights were trained. Three routes exist:

  1. Developer’s API: the model author runs inference. Check its processing location and data policy.
  2. Independent provider: a hosting company runs its own copy of the weights. Verify it is hosted inference, not a gateway forwarding requests upstream.
  3. Your own deployment: you control location, network and logging.

Providers do put this in writing. Together states that models on its own infrastructure do not call out to the model author. DeepInfra states its hosted models run in US data centers.

Those are provider statements, not audits. Whichever route you pick, three checks remain:

  • Where inference runs
  • Who receives request content, including fallbacks and tools
  • What is stored, for how long, and whether it trains anything

One Model, Four Hosts: Kimi K2.6

Kimi K2.6 is the worked example from our July hosting research. It is not a model we have tested in the router.

Published rates, checked November 9, 2026, USD per million tokens:

ProviderUncached inputCached inputOutputStatus
DeepInfra0.750.15$3.50Serverless; deployment labeled FP4
Novita AI0.800.16$3.40Serverless; quantization not stated
Fireworks AIBilled by GPU timeServerless retired Sept 25, 2026; dedicated deployment only
Together AINot verifiedJuly rate ($1.20 / $4.50) no longer in current listings

Two token-priced options remain. DeepInfra has the lower input rate; Novita has the lower output rate. Which wins depends on the workload.

What the Bill Looks Like

Model charges = uncached input + cached input + output + any separately billed usage.

What Drives AI Cost

Illustrative totals at the rates above, not measured traffic:

Token mixDeepInfraNovita AI
10M uncached input + 1M output11.0011.40
1M uncached input + 1M output4.254.20
1M uncached + 9M cached input + 1M output5.605.64

Input-heavy work favors DeepInfra. Output-heavy work favors Novita. The third row shows how much cache pricing matters when 90% of input is a reread.

Our own evaluation shows why that row deserves attention:

  • Rereading conversation history was 72% of the premium-only bill
  • Handovers between models were 41% of the router bill, under 3% of the text
  • Caches are scoped per model and account and can expire

Switching providers mid-session is not free, even when per-token rates look identical.

Dedicated Deployment and Self-Hosting

Fireworks answers a different question: a dedicated deployment instead of a shared endpoint. It is a researched option, not part of our router.

Fireworks documentsLimit
GLM-5.2 and Kimi K2.6 on dedicated GPUsAvailability and quota must match the account
Regional deployments; Enterprise data-residency controlsEnterprise feature, not default
No storage or logging of inference without opt-inBatch and training workflows persist differently

Residency has a price. Together’s US-only endpoint charges 1.25 times its standard rate, and it supports GLM-5.3 and Kimi K3, not the models above.

Self-hosting replaces token prices with capacity costs:

  • Provisioned GPUs, busy or idle
  • Storage and networking
  • Upgrades and maintenance
  • Monitoring and on-call

Moonshot’s deployment guide gives a single-node example of eight H200 GPUs. One configuration, not a minimum.

Same Model, Different Results

Two deployments of the same model are not guaranteed to behave the same way. DeepInfra labels its Kimi K2.6 FP4; Moonshot documents native INT4. Neither label proves degraded or identical quality.

Benchmarks help shortlist. Z.ai’s GLM-5.2 model card reports:

BenchmarkGLM-5.2Claude Opus 4.8
Terminal Bench 2.1, Terminus-281.085.0
NL2Repo48.969.7

Developer-reported, not our tests. The gap is small on terminal work and wide on repository-level generation.

Our four-feature run adds what benchmarks cannot. Both sides passed every test and check, but the premium-only sessions left about four times as many automated tests behind. Same outcome, different depth of verification.

Test One Workflow Before You Route to It

Before any model and host enters a routing configuration, measure five things on a representative workflow:

  1. Accepted results against agreed requirements.
  2. Total model charges, including every attempt and reread.
  3. Verification performed and human corrections needed.
  4. Response times and failures under expected load.
  5. Approved processing destinations, including fallbacks.

The metric that ties them together: total charges divided by accepted results.

Our evaluation is that assessment at small scale. Four features, $86.95 versus $42.79. The saving came from routing and shorter conversations, not from any host or self-hosting.

Open weights let you judge the model and the service separately. Judge them on the same workflow.