RootInference

Smart Routing

The best model and provider for every request — by price, latency and quality

Define routing policies once and RootInference picks the best model and provider per request. Optimise for cost, latency or quality, set fallback chains, pin rules per API key, and A/B new models against production traffic — all without touching application code.

Design a routing policyNo routing surcharge — pay model cost only
Tracerootinference route --policy latency-first

  1. Chat completion received under latency-first policy

  2. 6 candidate providers serve the requested model class

  3. Selected provider-2: p50 189 ms, $0.9/M input

  4. Fallback chain armed: provider-4, then provider-7

  5. Completed in 1.4 s for 2.1 credits, quality check passed

  6. Estimated saving vs pinned frontier model: 63%

Capabilities

What it does

Cut inference spend and latency without sacrificing quality — routing decisions become policy, not code.

  • Price, latency and quality routing policies with per-request overrides
  • Model fallback chains — degrade gracefully from frontier to mid-tier
  • Per-key routing rules for teams, environments and features
  • A/B model comparison on live traffic with side-by-side metrics
  • Provider selection by real-time latency and error-rate telemetry
  • Cost ceilings and budget-aware routing per key or team
  • Route economy models by default, escalate to frontier on demand

The routing decision

Every request, routed in nine steps

The whole decision happens in milliseconds of routing overhead — and every step is driven by live telemetry, not static provider lists.

  1. Receive the request on the OpenAI-compatible endpoint

  2. Resolve the requested model or model class from the catalog

  3. Load the routing policy for the API key (price, latency or quality)

  4. Score candidate providers on live latency, error-rate and price telemetry

  5. Apply per-request overrides and cost ceilings

  6. Select the winning provider and arm the fallback chain

  7. Stream the response back in the unified schema

  8. Meter tokens and price the request in credits

  9. Feed the outcome back into provider scores for the next request

See Smart Routing on your own traffic

Point your existing OpenAI SDK at our base URL and try it in minutes — or book a demo to plan a production rollout.

Get an API keyView pricing