ConceptGame logoConceptGame
LIVELive Product

LLMCascade — Free-Tier LLM Dispatcher

Standalone free-tier LLM dispatcher that selects an eligible free model from live rate-limit budgets, calls the provider’s native HTTP API, and falls back on failure. Framework-independent RouterClient plus optional FastAPI layer (self-hosted).

Sliding-window RPD/RPM/RPS/TPM budgets with round-robin, least_used, and priority_first selection across 13+ providers, plus dashboard, admin key UI, and optional MongoDB stats.

  • YAML model registry with Pydantic validation and live free-tier budgets
  • RouterClient library + optional FastAPI /v1/complete API
  • Status dashboard, admin provider keys, historical stats
  • Retry + multi-model fallback across Groq, Gemini, OpenRouter, and more
PythonFastAPIPydanticMongoDBGroqGeminiOpenRouter
LLMCascade status dashboard with configured models and health panel

System Architecture

Diagram focused on system boundaries, data flow, and integration responsibilities.

LLMCascade Architecture Diagram

Client / App

HTTP or Python SDK

  • - POST /v1/complete
  • - RouterClient library
  • - Optional API-key auth

Dispatcher Core

Python, FastAPI, Pydantic

  • - YAML model registry
  • - RPD / RPM / RPS / TPM budgets
  • - round-robin / least_used / priority_first
  • - Retry + multi-model fallback
  • - Async queue + worker pool

LLM Providers

Native HTTP APIs

  • - Groq
  • - Gemini
  • - OpenRouter
  • - Together, Cerebras, Mistral, + more
Complete Request
Provider Calls

Dashboard & Admin

  • - /dashboard health + budgets
  • - /admin/providers encrypted keys
  • - Event / error logs

Stats Store

MongoDB

Historical stats when MONGODB_URI is set; /stats UI for source grouping.