[{"data":1,"prerenderedAt":823},["ShallowReactive",2],{"article/model-agnostic-ai-layer-fallbacks":3},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":8,"description":9,"featured":6,"author":10,"categories":11,"slug":12,"image":13,"imageAlt":23,"published":24,"draft":6,"createdAt":25,"updatedAt":25,"faqs":26,"body":42,"_type":817,"_id":818,"_source":819,"_file":820,"_stem":821,"_extension":822,"isInteractive":6,"interactiveConfig":-1},"/articles/model-agnostic-ai-layer-fallbacks","articles",false,"","Build an LLM Fallback Layer Before Your Model Vanishes","When Anthropic disabled Fable 5 and Mythos 5 for every customer, hardcoded model IDs broke. Build a model-agnostic provider layer with typed fallbacks.","Thomas Findlay","AI, Typescript, Javascript, React","model-agnostic-ai-layer-fallbacks",[14,15,16,17,18,19,20,21,22],"/images/articles/model-agnostic-ai-layer-fallbacks/model-agnostic-ai-layer-fallbacks-640w.avif","/images/articles/model-agnostic-ai-layer-fallbacks/model-agnostic-ai-layer-fallbacks-1024w.avif","/images/articles/model-agnostic-ai-layer-fallbacks/model-agnostic-ai-layer-fallbacks-1920w.avif","/images/articles/model-agnostic-ai-layer-fallbacks/model-agnostic-ai-layer-fallbacks-640w.webp","/images/articles/model-agnostic-ai-layer-fallbacks/model-agnostic-ai-layer-fallbacks-1024w.webp","/images/articles/model-agnostic-ai-layer-fallbacks/model-agnostic-ai-layer-fallbacks-1920w.webp","/images/articles/model-agnostic-ai-layer-fallbacks/model-agnostic-ai-layer-fallbacks-640w.png","/images/articles/model-agnostic-ai-layer-fallbacks/model-agnostic-ai-layer-fallbacks-1024w.png","/images/articles/model-agnostic-ai-layer-fallbacks/model-agnostic-ai-layer-fallbacks-1920w.png","One request routed through a single provider boundary to three swappable model backends, one unplugged and grayed out, two live.",true,"2026-06-13T00:00:00",[27,30,33,36,39],{"question":28,"answer":29},"What happened to Fable 5 and Mythos 5?","On 12 June 2026, the US government issued a national-security export-control directive ordering Anthropic to suspend all access to Fable 5 and Mythos 5 for any foreign national, including foreign nationals inside the US and Anthropic's own non-citizen employees. To comply, Anthropic disabled both models for every customer within hours. Access to all other Anthropic models, including Opus 4.8, Sonnet 4.6, and Haiku 4.5, was unaffected, and Anthropic said it hopes to reinstate access as soon as possible.",{"question":31,"answer":32},"What is an LLM fallback layer?","An LLM fallback layer is a single boundary in your code that every model call goes through, with an ordered list of models to try if the primary one fails. The application asks the boundary for a completion and never names a model directly. If the first model returns an availability error, the layer transparently retries the next model in the chain, so one provider decision cannot take your feature down.",{"question":34,"answer":35},"Should I build my own provider abstraction or use a gateway like LiteLLM or OpenRouter?","Build a thin abstraction yourself when you call one or two providers and want zero new infrastructure, full control over the fallback logic, and no extra network hop. Adopt a gateway like LiteLLM, OpenRouter, or Vercel AI Gateway when you call several providers, want a unified API and centralized observability, and accept a managed dependency in exchange for not maintaining the routing code. Most teams start with a thin in-process layer and move to a gateway once the provider count and the observability needs grow.",{"question":37,"answer":38},"Can I swap from Claude to GPT-5.5 without rewriting my prompts?","You can swap the call without rewriting your code, but not always without revisiting your prompts. The wire format differs between providers, so a cross-provider fallback needs an adapter or a gateway that normalizes the request and response. Even after that, prompts tuned for one model can produce different output on another, so a cross-provider fallback should be validated with the same evals you run on the primary.",{"question":40,"answer":41},"Does an LLM fallback layer add latency?","The boundary itself adds almost nothing, a function call and an object lookup. The latency cost appears only on failover: when the primary model errors, you pay its time-to-failure plus the full latency of the fallback call. An in-process layer adds no network hop, while a hosted gateway adds one round trip to the gateway on every request, which is usually a few milliseconds against an LLM response measured in seconds.",{"type":43,"children":44,"toc":801},"root",[45,53,83,90,95,101,117,137,142,148,161,170,182,195,208,214,219,226,231,239,248,293,314,320,325,333,342,363,368,374,379,387,396,415,420,427,436,441,446,452,481,486,672,677,690,696,701,706,711,716,722,727,772,785,791,796],{"type":46,"tag":47,"props":48,"children":49},"element","p",{},[50],{"type":51,"value":52},"text","If one provider decision can take your product down, route every model call through a single boundary and treat the model ID as runtime configuration, not a hardcoded constant. Behind that boundary, keep an ordered list of models and fail over to the next when the active one returns an availability error. That pattern is an LLM fallback layer, and it is what separates a feature that survives a sudden model shutdown from one that does not.",{"type":46,"tag":47,"props":54,"children":55},{},[56,58,65,67,73,75,81],{"type":51,"value":57},"This targets the Anthropic TypeScript SDK and any OpenAI-compatible gateway. Code examples use current Claude model ids as of June 2026: ",{"type":46,"tag":59,"props":60,"children":62},"code",{"className":61},[],[63],{"type":51,"value":64},"claude-opus-4-8",{"type":51,"value":66},", ",{"type":46,"tag":59,"props":68,"children":70},{"className":69},[],[71],{"type":51,"value":72},"claude-sonnet-4-6",{"type":51,"value":74},", and ",{"type":46,"tag":59,"props":76,"children":78},{"className":77},[],[79],{"type":51,"value":80},"claude-haiku-4-5-20251001",{"type":51,"value":82},".",{"type":46,"tag":84,"props":85,"children":87},"h2",{"id":86},"tldr-route-every-model-call-through-one-boundary",[88],{"type":51,"value":89},"TL;DR: route every model call through one boundary",{"type":46,"tag":47,"props":91,"children":92},{},[93],{"type":51,"value":94},"Put one provider boundary between your app and any model. The application asks that boundary for a completion and passes intent (the prompt, the token budget), never a model ID. Model selection lives in configuration: a primary model and an ordered fallback chain you can change without touching feature code. The boundary fails over only on availability errors (the model is gone, overloaded, or rate-limited), and never retries a request the provider rejected as malformed, because retrying a bad request burns the whole chain for nothing. Build this as a thin in-process layer when you call one or two providers; reach for a gateway like LiteLLM, OpenRouter, or Vercel AI Gateway once you call several. The model ID is config, not a constant.",{"type":46,"tag":84,"props":96,"children":98},{"id":97},"what-concretely-broke-when-fable-5-and-mythos-5-went-dark",[99],{"type":51,"value":100},"What concretely broke when Fable 5 and Mythos 5 went dark",{"type":46,"tag":47,"props":102,"children":103},{},[104,106,115],{"type":51,"value":105},"On 12 June 2026, Anthropic received a US government export-control directive at 5:21pm ET ordering it to suspend all access to Fable 5 and Mythos 5 for any foreign national, including foreign nationals inside the US and its own non-citizen employees. The only way to comply within hours was to disable both models for every customer. Anthropic described the trigger as a narrow, non-universal jailbreak and said that if the same standard were applied across the industry, it \"would essentially halt all new model deployments for all frontier model providers.\" You can read ",{"type":46,"tag":107,"props":108,"children":112},"a",{"href":109,"rel":110},"https://www.anthropic.com/news/fable-mythos-access",[111],"nofollow",[113],{"type":51,"value":114},"Anthropic's statement",{"type":51,"value":116}," in full.",{"type":46,"tag":47,"props":118,"children":119},{},[120,122,128,130,135],{"type":51,"value":121},"Here is the part that matters for how you build. Every other Anthropic model stayed up. A team whose code said ",{"type":46,"tag":59,"props":123,"children":125},{"className":124},[],[126],{"type":51,"value":127},"model: 'claude-fable-5'",{"type":51,"value":129}," in a route handler returned errors to users until somebody shipped a code change, ran CI, and deployed. A team that read the model ID from configuration changed one value and pointed the same call at ",{"type":46,"tag":59,"props":131,"children":133},{"className":132},[],[134],{"type":51,"value":64},{"type":51,"value":136},". Same outage, same provider, two completely different incident timelines.",{"type":46,"tag":47,"props":138,"children":139},{},[140],{"type":51,"value":141},"The lesson is not \"Anthropic is risky.\" It is that model availability is a production dependency you do not control, and you should architect for it the way you already architect for a payment API that can 503. A hardcoded model ID is the single point of failure.",{"type":46,"tag":84,"props":143,"children":145},{"id":144},"why-a-hardcoded-model-id-is-a-single-point-of-failure",[146],{"type":51,"value":147},"Why a hardcoded model ID is a single point of failure",{"type":46,"tag":47,"props":149,"children":150},{},[151,153,159],{"type":51,"value":152},"The fastest way to ship an AI feature is to call the SDK directly in the handler that needs it. Its model lives right there in the call, the response shape is whatever that provider returns, and it works on the first try. That is exactly the shape an AI assistant generates when you ask it to \"summarize the ticket with Claude,\" the same demo-grade default catalogued in ",{"type":46,"tag":107,"props":154,"children":156},{"href":155},"/blog/vibe-coding-vs-production-coding-react",[157],{"type":51,"value":158},"AI-generated React code, 9 patterns that fail in production",{"type":51,"value":160},", and it is the shape that broke on 12 June.",{"type":46,"tag":47,"props":162,"children":163},{},[164],{"type":46,"tag":165,"props":166,"children":167},"strong",{},[168],{"type":51,"value":169},"src/features/tickets/summarize.ts",{"type":46,"tag":171,"props":172,"children":177},"pre",{"className":173,"code":175,"language":176,"meta":7},[174],"language-ts","import Anthropic from '@anthropic-ai/sdk'\n\nconst anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY })\n\nexport async function summarizeTicket(ticket: string): Promise\u003Cstring> {\n  const response = await anthropic.messages.create({\n    model: 'claude-fable-5',\n    max_tokens: 512,\n    messages: [{ role: 'user', content: `Summarize this support ticket:\\n\\n${ticket}` }],\n  })\n\n  const block = response.content[0]\n  return block.type === 'text' ? block.text : ''\n}\n","ts",[178],{"type":46,"tag":59,"props":179,"children":180},{"__ignoreMap":7},[181],{"type":51,"value":175},{"type":46,"tag":47,"props":183,"children":184},{},[185,187,193],{"type":51,"value":186},"Two design decisions are fused into one function here, and that is the problem. The model ID ",{"type":46,"tag":59,"props":188,"children":190},{"className":189},[],[191],{"type":51,"value":192},"claude-fable-5",{"type":51,"value":194}," is welded to the business logic of summarizing a ticket, so the only way to change the model is to edit, review, and redeploy this file. Worse, the provider client is instantiated inline, so the same coupling repeats in every feature that calls a model. When Fable 5 went dark, this function did not degrade. It threw, and it kept throwing until a human intervened.",{"type":46,"tag":47,"props":196,"children":197},{},[198,200,206],{"type":51,"value":199},"The deeper issue is that the response handling is provider-specific too. That ",{"type":46,"tag":59,"props":201,"children":203},{"className":202},[],[204],{"type":51,"value":205},"response.content[0].text",{"type":51,"value":207}," access assumes the Anthropic message shape. Swap to another provider and the parsing breaks alongside the model name. The naive integration couples three things that should move independently: which model runs, which provider client makes the call, and how your app reads the result. Decouple them and a shutdown becomes a config change.",{"type":46,"tag":84,"props":209,"children":211},{"id":210},"a-model-agnostic-provider-interface-in-typescript",[212],{"type":51,"value":213},"A model-agnostic provider interface in TypeScript",{"type":46,"tag":47,"props":215,"children":216},{},[217],{"type":51,"value":218},"The goal is one call signature your whole app uses, with the provider details hidden behind it. Your features ask for a completion in terms they own (a prompt, a token budget) and get back a plain string plus the metadata they care about. Which model produced it, and through which SDK, is the boundary's concern.",{"type":46,"tag":220,"props":221,"children":223},"h3",{"id":222},"one-call-signature-many-backends",[224],{"type":51,"value":225},"One call signature, many backends",{"type":46,"tag":47,"props":227,"children":228},{},[229],{"type":51,"value":230},"Start with the contract. Every provider in the system implements the same interface, so the rest of the app depends on the interface and never on a concrete SDK.",{"type":46,"tag":47,"props":232,"children":233},{},[234],{"type":46,"tag":165,"props":235,"children":236},{},[237],{"type":51,"value":238},"src/ai/provider.ts",{"type":46,"tag":171,"props":240,"children":243},{"className":241,"code":242,"language":176,"meta":7},[174],"import Anthropic from '@anthropic-ai/sdk'\n\nexport interface CompletionRequest {\n  prompt: string\n  maxTokens: number\n  system?: string\n}\n\nexport interface CompletionResult {\n  text: string\n  model: string\n}\n\nexport class ModelUnavailableError extends Error {\n  constructor(public readonly model: string, cause?: unknown) {\n    super(`Model ${model} is unavailable`)\n    this.name = 'ModelUnavailableError'\n    this.cause = cause\n  }\n}\n\nexport class BadRequestError extends Error {\n  constructor(message: string, cause?: unknown) {\n    super(message)\n    this.name = 'BadRequestError'\n    this.cause = cause\n  }\n}\n\nexport interface ModelProvider {\n  complete(model: string, request: CompletionRequest): Promise\u003CCompletionResult>\n}\n\nexport class AnthropicProvider implements ModelProvider {\n  private readonly client: Anthropic\n\n  constructor(apiKey: string) {\n    this.client = new Anthropic({ apiKey })\n  }\n\n  async complete(model: string, request: CompletionRequest): Promise\u003CCompletionResult> {\n    try {\n      const response = await this.client.messages.create({\n        model,\n        max_tokens: request.maxTokens,\n        system: request.system,\n        messages: [{ role: 'user', content: request.prompt }],\n      })\n      const block = response.content[0]\n      const text = block && block.type === 'text' ? block.text : ''\n      return { text, model }\n    } catch (error) {\n      if (error instanceof Anthropic.APIError) {\n        if (error.status === 400) {\n          throw new BadRequestError(error.message, error)\n        }\n        if (error.status === 404 || error.status === 429 || error.status >= 500) {\n          throw new ModelUnavailableError(model, error)\n        }\n      }\n      throw error\n    }\n  }\n}\n",[244],{"type":46,"tag":59,"props":245,"children":246},{"__ignoreMap":7},[247],{"type":51,"value":242},{"type":46,"tag":47,"props":249,"children":250},{},[251,253,259,261,267,269,275,277,283,285,291],{"type":51,"value":252},"The interface is the entire point. ",{"type":46,"tag":59,"props":254,"children":256},{"className":255},[],[257],{"type":51,"value":258},"complete",{"type":51,"value":260}," takes a model ID as an argument rather than baking it in, so a single provider instance can drive any model the provider serves. The Anthropic-specific knowledge, the ",{"type":46,"tag":59,"props":262,"children":264},{"className":263},[],[265],{"type":51,"value":266},"messages.create",{"type":51,"value":268}," call shape and the ",{"type":46,"tag":59,"props":270,"children":272},{"className":271},[],[273],{"type":51,"value":274},"content[0].text",{"type":51,"value":276}," parsing, lives in exactly one class. Everything downstream works with ",{"type":46,"tag":59,"props":278,"children":280},{"className":279},[],[281],{"type":51,"value":282},"CompletionRequest",{"type":51,"value":284}," and ",{"type":46,"tag":59,"props":286,"children":288},{"className":287},[],[289],{"type":51,"value":290},"CompletionResult",{"type":51,"value":292},", which carry no provider vocabulary at all.",{"type":46,"tag":47,"props":294,"children":295},{},[296,298,304,306,312],{"type":51,"value":297},"The translation of provider errors into two of our own error types is the load-bearing decision. A 404 (model not found, which is what a disabled model returns), a 429 (rate limited), or any 5xx becomes a ",{"type":46,"tag":59,"props":299,"children":301},{"className":300},[],[302],{"type":51,"value":303},"ModelUnavailableError",{"type":51,"value":305},", the signal that says \"try the next model.\" A 400 becomes a ",{"type":46,"tag":59,"props":307,"children":309},{"className":308},[],[310],{"type":51,"value":311},"BadRequestError",{"type":51,"value":313},", the signal that says \"stop, this request is broken and the next model will reject it too.\" We will lean on that distinction in the fallback layer. Hold onto it.",{"type":46,"tag":220,"props":315,"children":317},{"id":316},"model-selection-as-configuration-not-code",[318],{"type":51,"value":319},"Model selection as configuration, not code",{"type":46,"tag":47,"props":321,"children":322},{},[323],{"type":51,"value":324},"With the interface in place, model choice becomes data. The chain that broke on 12 June should be editable without opening a feature file, ideally from environment configuration so you can change it per deploy or even at runtime.",{"type":46,"tag":47,"props":326,"children":327},{},[328],{"type":46,"tag":165,"props":329,"children":330},{},[331],{"type":51,"value":332},"src/ai/config.ts",{"type":46,"tag":171,"props":334,"children":337},{"className":335,"code":336,"language":176,"meta":7},[174],"export interface ModelChain {\n  primary: string\n  fallbacks: string[]\n}\n\nfunction parseChain(value: string | undefined, fallbackDefault: ModelChain): ModelChain {\n  if (!value) return fallbackDefault\n  const ids = value.split(',').map(id => id.trim()).filter(Boolean)\n  if (ids.length === 0) return fallbackDefault\n  const [primary, ...fallbacks] = ids\n  return { primary, fallbacks }\n}\n\nexport const ticketSummaryChain: ModelChain = parseChain(process.env.TICKET_SUMMARY_MODELS, {\n  primary: 'claude-opus-4-8',\n  fallbacks: ['claude-sonnet-4-6', 'claude-haiku-4-5-20251001'],\n})\n",[338],{"type":46,"tag":59,"props":339,"children":340},{"__ignoreMap":7},[341],{"type":51,"value":336},{"type":46,"tag":47,"props":343,"children":344},{},[345,347,353,355,361],{"type":51,"value":346},"The chain reads from ",{"type":46,"tag":59,"props":348,"children":350},{"className":349},[],[351],{"type":51,"value":352},"TICKET_SUMMARY_MODELS",{"type":51,"value":354}," as a comma-separated list and falls back to a sane default if the variable is unset. On 12 June, a team running this would have set ",{"type":46,"tag":59,"props":356,"children":358},{"className":357},[],[359],{"type":51,"value":360},"TICKET_SUMMARY_MODELS=claude-opus-4-8,claude-sonnet-4-6,claude-haiku-4-5-20251001",{"type":51,"value":362}," and shipped nothing. The model that vanished is simply absent from the list.",{"type":46,"tag":47,"props":364,"children":365},{},[366],{"type":51,"value":367},"Notice the chain is named per use case rather than global. Ticket summaries can run on a cheaper model than, say, code generation, so each feature owns its own ordered preference. The configuration is where product decisions about cost and quality live, separated from the mechanism that executes them. That separation is what lets a non-engineer change the model in an incident with a single environment variable.",{"type":46,"tag":84,"props":369,"children":371},{"id":370},"a-typed-fallback-chain-that-fails-over-on-availability-not-just-rate-limits",[372],{"type":51,"value":373},"A typed fallback chain that fails over on availability, not just rate limits",{"type":46,"tag":47,"props":375,"children":376},{},[377],{"type":51,"value":378},"Most fallback code people write only catches rate limits. That is the common case, so it is the one tutorials show. The 12 June shutdown was not a rate limit. It was a model that returned a 404 because it no longer existed, and a fallback layer that only retries on 429 would have sailed straight past the one error that mattered. The layer has to treat \"the model is gone\" and \"the model is throttling you\" as the same class of recoverable failure, and treat \"your request is malformed\" as a hard stop.",{"type":46,"tag":47,"props":380,"children":381},{},[382],{"type":46,"tag":165,"props":383,"children":384},{},[385],{"type":51,"value":386},"src/ai/client.ts",{"type":46,"tag":171,"props":388,"children":391},{"className":389,"code":390,"language":176,"meta":7},[174],"import {\n  type ModelProvider,\n  type CompletionRequest,\n  type CompletionResult,\n  type ModelChain,\n  ModelUnavailableError,\n  BadRequestError,\n} from './provider'\n\nexport class AiClient {\n  constructor(private readonly provider: ModelProvider) {}\n\n  async complete(chain: ModelChain, request: CompletionRequest): Promise\u003CCompletionResult> {\n    const models = [chain.primary, ...chain.fallbacks]\n    let lastError: unknown\n\n    for (const model of models) {\n      try {\n        return await this.provider.complete(model, request)\n      } catch (error) {\n        if (error instanceof BadRequestError) {\n          throw error\n        }\n        if (error instanceof ModelUnavailableError) {\n          lastError = error\n          continue\n        }\n        throw error\n      }\n    }\n\n    throw new ModelUnavailableError(\n      chain.primary,\n      new Error(`All models in the chain were unavailable. Last error: ${String(lastError)}`),\n    )\n  }\n}\n",[392],{"type":46,"tag":59,"props":393,"children":394},{"__ignoreMap":7},[395],{"type":51,"value":390},{"type":46,"tag":47,"props":397,"children":398},{},[399,401,406,408,413],{"type":51,"value":400},"The loop walks the chain in order and returns the first success. A ",{"type":46,"tag":59,"props":402,"children":404},{"className":403},[],[405],{"type":51,"value":303},{"type":51,"value":407}," is recorded and the loop moves to the next model, which is the failover. A ",{"type":46,"tag":59,"props":409,"children":411},{"className":410},[],[412],{"type":51,"value":311},{"type":51,"value":414}," rethrows immediately, because a request the provider considered malformed will be just as malformed for every other model, and retrying it down the whole chain wastes time, tokens, and money while the user waits. Any error type we did not classify also rethrows, because silently failing over on an error you do not understand hides bugs.",{"type":46,"tag":47,"props":416,"children":417},{},[418],{"type":51,"value":419},"Wiring it into the feature replaces the hardcoded call with a request the boundary fulfills.",{"type":46,"tag":47,"props":421,"children":422},{},[423],{"type":46,"tag":165,"props":424,"children":425},{},[426],{"type":51,"value":169},{"type":46,"tag":171,"props":428,"children":431},{"className":429,"code":430,"language":176,"meta":7},[174],"import { AiClient } from '@/ai/client'\nimport { AnthropicProvider } from '@/ai/provider'\nimport { ticketSummaryChain } from '@/ai/config'\n\nconst client = new AiClient(new AnthropicProvider(process.env.ANTHROPIC_API_KEY ?? ''))\n\nexport async function summarizeTicket(ticket: string): Promise\u003Cstring> {\n  const result = await client.complete(ticketSummaryChain, {\n    prompt: `Summarize this support ticket:\\n\\n${ticket}`,\n    maxTokens: 512,\n  })\n  return result.text\n}\n",[432],{"type":46,"tag":59,"props":433,"children":434},{"__ignoreMap":7},[435],{"type":51,"value":430},{"type":46,"tag":47,"props":437,"children":438},{},[439],{"type":51,"value":440},"The feature no longer knows what a model ID is. It knows it wants a summary, it knows its token budget, and it trusts the boundary to pick a live model. Compare this to the version at the top of the article: the business logic is identical, but the failure mode changed from \"throws until a human deploys\" to \"transparently degrades to the next model in the chain.\" That is the whole return on the abstraction.",{"type":46,"tag":47,"props":442,"children":443},{},[444],{"type":51,"value":445},"One caveat worth stating plainly. This chain is Anthropic-to-Anthropic, which is the realistic fast fix on shutdown day because every model shares the same SDK and the same response shape, so failover is free of translation. The moment a fallback target is a different provider, you need an adapter that normalizes the request and response, which is the next section's trade-off. Failover within one provider is cheap; failover across providers is a project.",{"type":46,"tag":84,"props":447,"children":449},{"id":448},"build-it-yourself-or-adopt-a-gateway",[450],{"type":51,"value":451},"Build it yourself or adopt a gateway?",{"type":46,"tag":47,"props":453,"children":454},{},[455,457,464,465,472,473,480],{"type":51,"value":456},"Once you accept that model selection belongs behind a boundary, the next question is who owns that boundary. You can keep the thin in-process layer from the previous sections, or you can route through a managed AI gateway that gives you a unified API across providers and a fallback chain as configuration. By mid-2026 most enterprise teams call several model providers, so a vendor-neutral abstraction has become the default rather than an exotic choice. The three gateways teams reach for are ",{"type":46,"tag":107,"props":458,"children":461},{"href":459,"rel":460},"https://docs.litellm.ai/docs/routing",[111],[462],{"type":51,"value":463},"LiteLLM",{"type":51,"value":66},{"type":46,"tag":107,"props":466,"children":469},{"href":467,"rel":468},"https://openrouter.ai/docs/features/model-routing",[111],[470],{"type":51,"value":471},"OpenRouter",{"type":51,"value":74},{"type":46,"tag":107,"props":474,"children":477},{"href":475,"rel":476},"https://vercel.com/docs/ai-gateway",[111],[478],{"type":51,"value":479},"Vercel AI Gateway",{"type":51,"value":82},{"type":46,"tag":47,"props":482,"children":483},{},[484],{"type":51,"value":485},"Here is how the options compare on the dimensions that decide it.",{"type":46,"tag":487,"props":488,"children":489},"table",{},[490,521],{"type":46,"tag":491,"props":492,"children":493},"thead",{},[494],{"type":46,"tag":495,"props":496,"children":497},"tr",{},[498,504,509,513,517],{"type":46,"tag":499,"props":500,"children":501},"th",{},[502],{"type":51,"value":503},"Dimension",{"type":46,"tag":499,"props":505,"children":506},{},[507],{"type":51,"value":508},"DIY layer",{"type":46,"tag":499,"props":510,"children":511},{},[512],{"type":51,"value":463},{"type":46,"tag":499,"props":514,"children":515},{},[516],{"type":51,"value":471},{"type":46,"tag":499,"props":518,"children":519},{},[520],{"type":51,"value":479},{"type":46,"tag":522,"props":523,"children":524},"tbody",{},[525,554,582,617,645],{"type":46,"tag":495,"props":526,"children":527},{},[528,534,539,544,549],{"type":46,"tag":529,"props":530,"children":531},"td",{},[532],{"type":51,"value":533},"Setup cost",{"type":46,"tag":529,"props":535,"children":536},{},[537],{"type":51,"value":538},"A few files, no infra",{"type":46,"tag":529,"props":540,"children":541},{},[542],{"type":51,"value":543},"Self-host the proxy or run as a library",{"type":46,"tag":529,"props":545,"children":546},{},[547],{"type":51,"value":548},"API key, no infra",{"type":46,"tag":529,"props":550,"children":551},{},[552],{"type":51,"value":553},"API key, tightest on Vercel",{"type":46,"tag":495,"props":555,"children":556},{},[557,562,567,572,577],{"type":46,"tag":529,"props":558,"children":559},{},[560],{"type":51,"value":561},"Provider coverage",{"type":46,"tag":529,"props":563,"children":564},{},[565],{"type":51,"value":566},"What you adapt by hand",{"type":46,"tag":529,"props":568,"children":569},{},[570],{"type":51,"value":571},"Broad, many providers normalized",{"type":46,"tag":529,"props":573,"children":574},{},[575],{"type":51,"value":576},"Broad catalog behind one API",{"type":46,"tag":529,"props":578,"children":579},{},[580],{"type":51,"value":581},"Broad, managed routing",{"type":46,"tag":495,"props":583,"children":584},{},[585,590,595,607,612],{"type":46,"tag":529,"props":586,"children":587},{},[588],{"type":51,"value":589},"Fallback control",{"type":46,"tag":529,"props":591,"children":592},{},[593],{"type":51,"value":594},"Total, you write the loop",{"type":46,"tag":529,"props":596,"children":597},{},[598,600],{"type":51,"value":599},"Configurable routing and ",{"type":46,"tag":107,"props":601,"children":604},{"href":602,"rel":603},"https://docs.litellm.ai/docs/proxy/reliability",[111],[605],{"type":51,"value":606},"retries/fallbacks",{"type":46,"tag":529,"props":608,"children":609},{},[610],{"type":51,"value":611},"Automatic model routing and fallback",{"type":46,"tag":529,"props":613,"children":614},{},[615],{"type":51,"value":616},"Managed failover policies",{"type":46,"tag":495,"props":618,"children":619},{},[620,625,630,635,640],{"type":46,"tag":529,"props":621,"children":622},{},[623],{"type":51,"value":624},"Lock-in",{"type":46,"tag":529,"props":626,"children":627},{},[628],{"type":51,"value":629},"None, it is your code",{"type":46,"tag":529,"props":631,"children":632},{},[633],{"type":51,"value":634},"Low, open source you can self-host",{"type":46,"tag":529,"props":636,"children":637},{},[638],{"type":51,"value":639},"Medium, routed through their API",{"type":46,"tag":529,"props":641,"children":642},{},[643],{"type":51,"value":644},"Higher, strongest inside Vercel",{"type":46,"tag":495,"props":646,"children":647},{},[648,653,658,663,668],{"type":46,"tag":529,"props":649,"children":650},{},[651],{"type":51,"value":652},"Observability",{"type":46,"tag":529,"props":654,"children":655},{},[656],{"type":51,"value":657},"You build it",{"type":46,"tag":529,"props":659,"children":660},{},[661],{"type":51,"value":662},"Built-in logging and spend tracking",{"type":46,"tag":529,"props":664,"children":665},{},[666],{"type":51,"value":667},"Dashboard and usage analytics",{"type":46,"tag":529,"props":669,"children":670},{},[671],{"type":51,"value":667},{"type":46,"tag":47,"props":673,"children":674},{},[675],{"type":51,"value":676},"The DIY column is the layer you already saw. You own every line, there is no new network hop, and there is no third party in your request path. The cost is that cross-provider normalization and observability are now your job, and that job grows with every provider you add.",{"type":46,"tag":47,"props":678,"children":679},{},[680,682,688],{"type":51,"value":681},"When to pick each comes down to provider count and who you want maintaining the routing. Build it yourself when you call one or two providers, want zero added latency and zero new dependency, and value full control over exactly when failover fires. Reach for LiteLLM when you want broad provider coverage with the option to self-host and keep lock-in low, which suits teams that are cost-sensitive and infrastructure-comfortable. Reach for OpenRouter when you want the widest model catalog behind a single key with the least setup and are happy to route through a hosted API. Reach for Vercel AI Gateway when you are already on Vercel and want managed failover and observability with the least wiring. The honest default for a small team is to start with the in-process layer and graduate to a gateway when the provider count crosses two or three and you find yourself rebuilding spend tracking and unified logging by hand. That wrapping-a-flaky-dependency instinct is the same one behind ",{"type":46,"tag":107,"props":683,"children":685},{"href":684},"/blog/how-to-use-rtk-query-the-right-scalable-way",[686],{"type":51,"value":687},"using RTK Query the right, scalable way",{"type":51,"value":689},": centralize the awkward integration once so the rest of the app stays clean.",{"type":46,"tag":84,"props":691,"children":693},{"id":692},"what-a-fallback-layer-cannot-save-you-from",[694],{"type":51,"value":695},"What a fallback layer cannot save you from",{"type":46,"tag":47,"props":697,"children":698},{},[699],{"type":51,"value":700},"A fallback layer keeps your feature responding when a model disappears. It does nothing about whether the response is any good. This is the part teams skip, and it is where the quiet failures live.",{"type":46,"tag":47,"props":702,"children":703},{},[704],{"type":51,"value":705},"The first gap is prompt and output drift between models. A prompt tuned against Opus 4.8 can return differently shaped output on Sonnet 4.6 or Haiku 4.5, and a cross-provider fallback to GPT-5.5 or Gemini 3.1 Pro widens that gap further. If your code parses the model's output as JSON, or depends on a specific tone or structure, a silent failover can hand downstream code a shape it was not built for. The boundary stayed up; the contract quietly changed.",{"type":46,"tag":47,"props":707,"children":708},{},[709],{"type":51,"value":710},"The second gap is evaluation. Knowing whether a fallback model is acceptable means running the same evals against every model in the chain, not just the primary, before you ship. Without that, your fallback is a guess that you will only test for the first time during an outage, which is the worst possible moment to discover that Haiku 4.5 mangles the format your invoice parser expects.",{"type":46,"tag":47,"props":712,"children":713},{},[714],{"type":51,"value":715},"The third gap is cost and capability. Models in a chain rarely cost the same per token, and they rarely have the same context window or tool-use behavior. A failover from a small model to a large one can multiply your bill, and a failover from a large-context model to a smaller one can truncate a prompt that fit a moment ago. Capability gaps cut both ways, and the fallback layer is blind to all of it. A fallback layer is a resilience tool, not a quality or cost guarantee.",{"type":46,"tag":84,"props":717,"children":719},{"id":718},"a-resilience-checklist-for-your-ai-layer",[720],{"type":51,"value":721},"A resilience checklist for your AI layer",{"type":46,"tag":47,"props":723,"children":724},{},[725],{"type":51,"value":726},"Run this against any feature that calls a model before it ships to a paying user.",{"type":46,"tag":728,"props":729,"children":730},"ul",{},[731,737,742,747,752,757,762,767],{"type":46,"tag":732,"props":733,"children":734},"li",{},[735],{"type":51,"value":736},"Route every model call through one boundary. No feature file names a model ID directly.",{"type":46,"tag":732,"props":738,"children":739},{},[740],{"type":51,"value":741},"Make model selection configuration. The primary and the fallback chain are editable without a code change, ideally from environment variables you can flip per deploy.",{"type":46,"tag":732,"props":743,"children":744},{},[745],{"type":51,"value":746},"Classify provider errors into two buckets: availability errors (404, 429, 5xx) that trigger failover, and bad-request errors (400) that must stop the chain immediately.",{"type":46,"tag":732,"props":748,"children":749},{},[750],{"type":51,"value":751},"Order the chain by cost and quality per use case, not globally. Summaries and code generation deserve different chains.",{"type":46,"tag":732,"props":753,"children":754},{},[755],{"type":51,"value":756},"Run your evals against every model in the chain, not just the primary, so a failover does not silently degrade output quality.",{"type":46,"tag":732,"props":758,"children":759},{},[760],{"type":51,"value":761},"Log which model actually served each request. When you fail over, you want the metric and the breadcrumb, both to alert on and to debug with.",{"type":46,"tag":732,"props":763,"children":764},{},[765],{"type":51,"value":766},"Decide a cross-provider strategy before you need it: an adapter you maintain, or a gateway that normalizes for you. Anthropic-to-Anthropic failover is free; cross-provider failover needs real translation and its own evals.",{"type":46,"tag":732,"props":768,"children":769},{},[770],{"type":51,"value":771},"Set a sensible per-model timeout so a hung model spends its budget and yields to the next one rather than freezing the request.",{"type":46,"tag":47,"props":773,"children":774},{},[775,777,783],{"type":51,"value":776},"The shape of this list is the same hardening instinct from ",{"type":46,"tag":107,"props":778,"children":780},{"href":779},"/blog/hardening-ai-generated-react-app-for-production",[781],{"type":51,"value":782},"hardening an AI-generated React app for production",{"type":51,"value":784},", pointed at the runtime model dependency instead of the code the model writes.",{"type":46,"tag":84,"props":786,"children":788},{"id":787},"conclusion",[789],{"type":51,"value":790},"Conclusion",{"type":46,"tag":47,"props":792,"children":793},{},[794],{"type":51,"value":795},"The 12 June shutdown was not a one-off. It is a preview of every way a model can leave your dependency graph without warning: a regulatory order, a deprecation notice, a pricing change that makes a model uneconomic overnight. The teams that shrugged it off were not lucky. They had already decided that a model ID is configuration and that every model call goes through one boundary that can fail over.",{"type":46,"tag":47,"props":797,"children":798},{},[799],{"type":51,"value":800},"If you ship one thing from this article, make it the boundary. Start with the thin in-process layer, because it is a few files and it carries no new dependency, and move to a gateway when the provider count justifies the managed routing and the unified observability. Then go further than availability: put your fallback models through the same evals as your primary, because the next outage will test them whether you did or not. The model will change. Your call site should not have to.",{"title":7,"searchDepth":802,"depth":802,"links":803},2,[804,805,806,807,812,813,814,815,816],{"id":86,"depth":802,"text":89},{"id":97,"depth":802,"text":100},{"id":144,"depth":802,"text":147},{"id":210,"depth":802,"text":213,"children":808},[809,811],{"id":222,"depth":810,"text":225},3,{"id":316,"depth":810,"text":319},{"id":370,"depth":802,"text":373},{"id":448,"depth":802,"text":451},{"id":692,"depth":802,"text":695},{"id":718,"depth":802,"text":721},{"id":787,"depth":802,"text":790},"markdown","content:articles:model-agnostic-ai-layer-fallbacks.md","content","articles/model-agnostic-ai-layer-fallbacks.md","articles/model-agnostic-ai-layer-fallbacks","md",1781356919063]