ModelRail
ModelRail is a production AI gateway: one API for routing, failover, usage limits, and billing across multiple language models — launching first for developers in Nigeria.
Overview
Teams calling multiple LLM providers directly end up duplicating routing, retry, and billing logic per integration. ModelRail collapses that into a single API surface, with a prepaid credit wallet built to remove payment friction for the initial Nigeria launch market.
Problem
The routing layer had to pick the right model for a request accurately — without adding the latency and cost of a separate LLM call just to decide which model to use.
Role & ownership
Designed and built ModelRail's routing engine, provider integrations, and usage-control layer — the workload-alias system, the two-tier router, rate limiting and spend caps, and the prepaid wallet for the Nigeria launch.
Constraints
- Routing decisions could not themselves cost an extra model call — the router had to stay static-mapping-plus-heuristics, not a second LLM in the request path.
- Usage needed hard limits: rate limits and monthly spend caps with clear error responses, to prevent runaway usage and surprise bills.
Architecture
Calling applications target a stable workload alias — modelrail-auto, modelrail-chat, modelrail-reasoning, modelrail-code, or modelrail-tools — instead of a specific provider model.
Each alias resolves through a two-tier router: a static mapping table handles the common case, with lightweight heuristics deciding when a request needs a different tier. Nothing in that path is itself a model call.
Requests then go through provider routing with failover, so a provider outage or rate limit doesn't surface as a failure to the caller. NestJS (on Fastify) handles the request layer; Redis and BullMQ back rate limiting and queued work; MongoDB persists usage, wallet, and billing state.
Technical decisions
Decision
Static mapping + heuristics over an LLM-based router
A static mapping table covers the common routing case, with heuristics handling the rest — accurate enough without a second model round-trip on every request.
Decision
Workload aliases instead of exposing raw model names
Aliases like modelrail-chat or modelrail-reasoning describe what a request needs, not which vendor model serves it, so calling code stays stable when the underlying model or provider changes.
Decision
Rate limits and spend caps enforced at the gateway
Usage limits and monthly spend caps live in the gateway layer, with clear error responses, so a single client can't create a runaway bill or crowd out other traffic.
Decision
Prepaid wallet for the Nigeria launch
A prepaid credit wallet removes payment friction for developers in the initial Nigeria launch market, rather than requiring a card-based billing flow.
Trade-offs
Trade-off
Static mapping + heuristics instead of a learned/LLM router
Heuristic routing needs manual upkeep as new models and workload types are added. A learned router would generalize better, but reintroduces the latency and cost this design exists to avoid.
Trade-off
Provider failover handled at the gateway layer
Centralizing failover in the gateway makes it a critical-path component — its own availability now bounds every provider's effective availability to callers.
Challenges
- Keeping heuristic routing accurate without a model-based fallback meant tuning the static mapping and heuristics against real workload patterns, rather than leaning on a router that could reason its way through edge cases.
Not published
A fuller account of specific incidents and debugging stories from the ModelRail build isn't published yet.
Outcome
- Shipped the workload-alias system (modelrail-auto, modelrail-chat, modelrail-reasoning, modelrail-code, modelrail-tools) so applications call a stable alias instead of a specific model.
- Rate limits and monthly spend caps live in production, with clear error responses to prevent runaway usage and surprise bills.
- A prepaid credit wallet live for the Nigeria launch, removing payment friction for developers there.