FreeLLMAPI

One OpenAI-compatible endpoint. Every free LLM provider. Real-time dashboard.

Aggregate the free tiers from Google, Groq, Cerebras, NVIDIA, Mistral, OpenRouter, GitHub Models, Cohere, Cloudflare, HuggingFace, Z.ai, Ollama, Kilo, Pollinations, LLM7, OVH AI Endpoints, plus any custom OpenAI-compatible endpoint — behind a single /v1/chat/completions drop-in. A smart router picks the best available key for each request, fails over transparently when a provider rate-limits you, queues requests instead of dropping them, routes outbound traffic through an HTTP proxy pool of your choice, and tracks everything in a live WebSocket dashboard.

---

What's new in v1.3

• 🛰️ HTTP proxy pool — pool many proxies, choose a strategy (round-robin, least-latency, random, or single), per-key assignment that rotates hourly, dashboard with stats, geolocation, health-checker, bulk select / enable / disable / test / delete, and a France-only find-proxies scraper. A single proxy URL set in Settings is also tracked in the dashboard.
• 📊 Live dashboard v2 — in-flight requests now show the proxy in use, input tokens and a growing output token counter that ticks up as the stream comes back. New session totals (Tokens IN / Tokens OUT).
• 🔑 Keys page v2 — model discovery now syncs the catalog: new models from the provider are added, stale ones are removed. Bulk auto-discover for custom endpoints, search bar, inline label/RPM/TPM editing.
• 🌐 19 providers — Groq, Cerebras, Mistral, OpenRouter, NVIDIA NIM, GitHub Models, Cohere, Cloudflare, Google, Z.ai, HuggingFace, Ollama Cloud, Kilo Gateway, Pollinations, LLM7, OVH AI Endpoints, Amazon Bedrock (EU, paid — your own AWS capacity), Anthropic (native format), plus any custom OpenAI-compatible URL.
• 🛠 Operational fixes — fixed fd-leak crash on test-all, dispatcher cache, SOCKS test endpoint, model-stats scoping per user, pool-strategy persistence across restarts.
• 🧪 728 server tests passing in ~27s. Forgejo Actions workflow on every push and every v* tag.

---

Contents

• What's new in v1.3
• Features
• Supported providers
• Quick start
• Install script
• Docker
• Node.js + PM2
• Migrating from an older install
• Using the API
• HTTP proxy pool
• Dashboard
• Configuration
• How routing works
• Project structure
• Continuous integration
• Contributing
• Disclaimer

---

Features

Gateway

• Single OpenAI-compatible endpoint — /v1/chat/completions, /v1/models. Any OpenAI client library works unchanged.
• Smart failover router — tries provider keys in scored order (success rate, latency, rate-limit headroom). Falls over to the next key/provider silently.
• Request queueing — when all keys for a model are rate-limited, requests are held and retried instead of immediately returning 429. Configurable timeout.
• Per-user isolation — each user has their own provider keys, gateway keys, proxies, and request history. Zero cross-tenant data leakage.
• Gateway API keys — mint scoped keys (freellmapi-…) for your apps with per-key RPM/TPD limits. Enable/disable without deleting.
• Outbound rate limits — set per-provider-key RPM/TPM caps enforced between the router and the upstream API.
• Context handoff — injects a compact system message when a session switches model mid-stream so the new model knows where things left off.
• HTTP proxy pool — distribute outbound calls across many HTTP proxies (round-robin, least-latency, random), or pin to a single proxy. Per-key hourly assignment.
• AES-256-GCM encryption — all provider API keys are encrypted at rest. The plaintext never leaves your server.

Dashboard

• Live WebSocket dashboard — in-flight request counter, per-request routing events (which provider/model/proxy is being tried), per-key RPM gauges, overview counters — all pushed in real time without polling.
• In-flight proxy + tokens — active rows show the proxy name, input tokens, and a growing output token counter that ticks up live as the model streams.
• Session token totals — Tokens IN and Tokens OUT aggregated over the dashboard session.
• Per-user live counts — counts and events are scoped to the logged-in user.
• Keys page — masked key display, inline label/RPM/TPM editing, health status, one-click model discovery, custom endpoint with bulk auto-discover.
• Per-key deep stats — click any key to expand: latency percentiles (p50/p95/p99), 24-hour hourly bar chart, per-model breakdown, error breakdown, recent 25 requests, active cooldowns.
• Live status dots — each key's indicator updates in real time when the server detects a 401/403/429. No manual refresh needed.
• Playground — interactive chat with model selector, system prompt editor, and a live routing indicator showing which provider/model/proxy is handling the current request.
• Proxy pool page — bulk select, enable / disable / delete / test, per-proxy stats panel, geolocation, health checker.
• Analytics — request volume, error rates, cost estimates, per-provider breakdown, exportable history.

Operations

• Structured JSON logger — every request logs timestamp, level, req_id, user, key prefix, proxy, provider, and model. Secrets masked. Daily rotation with configurable retention.
• Universal installer — install.sh auto-detects Docker vs Node.js+PM2, generates keys, writes .env, builds, and starts the service. Prompts to migrate from an older install.
• Migration script — migrate-from-old.sh finds an older FreeLLMAPI install, verifies the encryption key, and imports provider keys and request history non-destructively.
• find-proxies script — scans a list of candidate free-proxy URLs, geo-IP-verifies the country, tests connectivity, and writes the working set to working-proxies.txt for one-click pool import.
• Docker — multi-stage Dockerfile, docker-compose.yml, named volume for the database.
• PM2 — ecosystem.config.cjs with autorestart, memory cap, and structured log rotation.
• Kubernetes / Podman — k8s/ manifests for podman play kube or kubectl apply.
• Forgejo Actions — .gitea/workflows/ci.yml and release.yml run install + build + test on every push and every v* tag.

---

Supported providers

Provider	Models	Auth
Google Gemini	Gemini 2.5 Flash, 2.0 Flash, 2.5 Pro preview	API key
Groq	Llama 3.3/4, Qwen3, Gemma, compound-beta	API key
Cerebras	Qwen3 235B, Llama 3.3 70B	API key
Mistral	Large 3, Medium 3.5, Codestral, Devstral	API key
OpenRouter	20+ free-tier models via :free routes	API key
GitHub Models	GPT-4.1, GPT-4o, Phi-4, Llama 3.3	GitHub PAT
Cloudflare Workers AI	Kimi K2, GLM-4.7, Llama, Granite	Account ID + token
Cohere	Command R+, Command-A (trial key)	API key
NVIDIA NIM	40 RPM free tier (eval-only ToS)	API key
HuggingFace	Inference router → DeepSeek, Kimi, Qwen3	API key
Z.ai (Zhipu)	GLM-4.5, GLM-4.7 Flash	API key
Ollama Cloud	GLM-4.7, Kimi K2, Qwen3	API key
Kilo Gateway	:free routes, no key required	Keyless
Pollinations	GPT-OSS 20B, no key required	Keyless
LLM7	GPT-OSS, Llama 3.1, GLM, no key required	Keyless
OVH AI Endpoints	Qwen3.5 397B, Llama 3.3, no key required	Keyless
Amazon Bedrock (EU)	Your AWS account's Bedrock models, EU regions only — paid, not a free tier	Bedrock API key (bearer)
Anthropic	Claude Sonnet 4, Opus 4, Haiku 3.5, 3.5 Sonnet, 3 Opus	API key
Custom endpoint	Any OpenAI-compatible URL — llama.cpp, LM Studio, vLLM, Ollama, Novita, etc.	API key (or keyless)

Keyless providers don't require an API key. The Keys page stores a sentinel row for them so routing treats the platform as configured.

Amazon Bedrock (EU) is the one paid provider in the list — it exists so you can route your own AWS capacity through the same endpoint, and it is off unless you add a key. It uses the AWS-recommended bedrock-runtime OpenAI-compatible endpoint with an Amazon Bedrock API key as a plain bearer token — no SigV4 signing and no AWS SDK. Pick the region with BEDROCK_REGION (EU allowlist, default eu-central-1); a non-EU value is refused so inference never silently leaves the EU. Because Bedrock model access is granted per AWS account, no models ship in the catalog: add the key, then Keys → Fetch all models imports exactly what your account is entitled to.

---

Quick start

git clone https://git.pandem.fr/outage.sh/FreeLLMapi freellmapi
cd freellmapi

# Recommended — universal installer
bash install.sh

# Or manually
cp .env.example .env
# Edit .env — set ENCRYPTION_KEY, ADMIN_USERNAME, ADMIN_PASSWORD
npm install && npm run build
node server/dist/index.js

Open http://localhost:3001 and log in with your admin credentials.

---

Install script

bash install.sh             # auto-detects Docker vs Node.js+PM2
bash install.sh --docker    # force Docker Compose
bash install.sh --node      # force Node.js + PM2
bash install.sh --port 8080 # custom port
bash install.sh --lan       # bind to 0.0.0.0 (LAN access)
bash install.sh --yes       # non-interactive / CI

The script detects Docker or Node.js, generates ENCRYPTION_KEY, prompts for admin credentials, writes .env, builds, starts the service, waits for the health check, and prints the access URL and management commands. After .env is written it asks if you want to migrate keys from an older install.

---

Docker

cp .env.example .env
# Set ENCRYPTION_KEY, ADMIN_USERNAME, ADMIN_PASSWORD in .env

docker compose up -d

# Verify
curl http://localhost:3001/api/ping

# Logs
docker compose logs -f

# Update
git pull && docker compose build && docker compose up -d

The database is stored in a named Docker volume (freellmapi-data) and survives container restarts and image rebuilds.

LAN access: set HOST_BIND=0.0.0.0 in .env to expose the port to your local network.

---

Node.js + PM2

Requires Node.js 20+.

cp .env.example .env   # edit before proceeding

npm install
npm run build

pm2 start ecosystem.config.cjs
pm2 save
pm2 startup            # enable autostart on reboot

# Logs
pm2 logs api-gateway

# Update
git pull && npm run build && pm2 restart api-gateway

The database lives at server/data/freeapi.db. Back it up before updates.

---

Migrating from an older install

If you ran a previous version of FreeLLMAPI and want to carry your provider keys forward:

# Auto-search the machine for an old install
bash migrate-from-old.sh

# Or point directly at the old repo directory
bash migrate-from-old.sh /path/to/old/freellmapi

The script:

1. Locates the old freeapi.db and reads its ENCRYPTION_KEY
2. Verifies decryption works before touching anything
3. Handles ENCRYPTION_KEY mismatches — adopt the old key (recommended if the new DB is empty) or re-encrypt everything on the fly with the new key
4. Imports all api_keys rows, skipping duplicates. Optionally imports request history.
5. Never deletes from either database — fully non-destructive

After running, restart the server and go to Keys → Check all to confirm the imported keys are still valid with each provider.

---

Using the API

FreeLLMAPI speaks standard OpenAI API. Point any client at your server:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3001/v1",
    api_key="freellmapi-your-gateway-key-here",
)

response = client.chat.completions.create(
    model="auto",   # router picks best available, or name a specific model
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

curl http://localhost:3001/v1/chat/completions \
  -H "Authorization: Bearer freellmapi-your-gateway-key-here" \
  -H "Content-Type: application/json" \
  -d '{"model":"auto","messages":[{"role":"user","content":"Hello!"}]}'

Listing models

curl http://localhost:3001/v1/models \
  -H "Authorization: Bearer freellmapi-your-gateway-key-here"

Returns only models the gateway can actually route. A model is listed when it is switched on on the Models tab and you hold an enabled key for its provider. A catalogue entry is a promise: discovery clients list whatever they are handed, so advertising a model you turned off — or one with no key behind it — just puts a choice in your model picker that can only ever fail.

Add ?available=0 to see the full catalogue instead, each entry tagged available with an unavailable_reason of disabled (switched off on the Models tab) or no_key (no enabled key for that provider) — useful for checking what adding a given provider key would unlock.

Use auto to let the router choose, or request a specific model by name. Ids are provider-qualified (groq/openai-gpt-oss-120b), so the same model served by several providers stays distinguishable; the bare name also resolves.

Model prices

Every model carries a price — free routes included — and it is the paid equivalent: what those tokens would cost on a paid API. A free route's own list price is $0, which is true and tells you nothing; the number worth seeing on a free-tier gateway is what you didn't pay.

{
  "id": "openrouter/inclusionai/ling-3.0-flash-vl:free",
  "pricing": {
    "prompt": "0.00000006",     // per token, OpenRouter's wire convention
    "completion": "0.00000018",
    "input_per_million": 0.06,
    "output_per_million": 0.18,
    "source": "openrouter",     // openrouter | models.dev | litellm | fallback
    "estimated": false          // true = no feed knew it; flat fallback rate
  },
  "list_price": { "input_per_million": 0, "output_per_million": 0 }  // what this route charges
}

Prices are resolved per model from OpenRouter, models.dev and LiteLLM, so a provider that publishes no prices of its own (NVIDIA NIM, Groq, Cerebras…) still gets one from whichever feed knows the model. Ids are matched across naming conventions — @cf/ prefixes, model:120b size tags, _ separators, :free / -free suffixes, qwen-3 vs qwen3 — because the benchmark only exists under the paid spelling. A model no feed has heard of (stealth releases, first-party routes like groq/compound) gets the flat fallback rate with estimated: true, so a client can say "about $x" rather than quote a number it cannot stand behind.

Per-request cost

Every completion reports what it cost — chat (streaming and non-streaming) — in the usage.cost field OpenAI-compatible clients read as the authoritative charge:

"usage": { "prompt_tokens": 120000, "completion_tokens": 30000, "cost": 0.066 }

This is what lets a harness show a running total for traffic routed through the gateway. Coding agents (omp, Cline, …) park models discovered from a gateway at zero catalog cost, precisely because a gateway's prices are its own — so usage.cost is not just the preferred source, it is the only one.

It is the same number the Analytics tab's savings figure is built from: one resolver prices the request, the response reports it, and the ledger stores it. The API and the dashboard cannot quote different figures.

Showing prices in a coding harness

usage.cost drives a harness's session cost total. Its model picker is a separate matter: harnesses refuse to price models discovered from a gateway using that gateway's plain /v1/models. OMP says so outright — "discovered proxy and gateway models … discovery keeps them at a local-unknown zero cost" — and that is a fair default, because an arbitrary OpenAI-compatible endpoint has no agreed way to state a price, so trusting one would be guesswork. The effect is that every model reads free in the picker no matter what /v1/models says.

LiteLLM's management endpoints are that agreed way, and the gateway serves them. You do not need a second provider entry or a second copy of your key — add discovery.type: litellm to the provider you already have:

providers:
  my-freellmapi:              # whatever you already called it
    baseUrl: http://172.17.1.2:3001/v1
    apiKey: FREELLMAPI_KEY    # the key you already configured
    api: openai-completions
    discovery:
      type: litellm           # ← the only line you add

Discovery strips the trailing /v1 and probes /model_group/info, /v2/model/info, /model/info and /v1/model/info at the host root; runtime calls keep using your /v1 base URL unchanged. All four are served, at the root and under /v1, so either convention works. The client parses input_cost_per_token / output_cost_per_token, max_input_tokens, max_output_tokens, supports_vision, supports_reasoning and supported_openai_params:

{
  "model_group": "openrouter/inclusionai/ling-3.0-flash-vl:free",
  "input_cost_per_token": 6e-8,      // → $0.06/M in the picker
  "output_cost_per_token": 1.8e-7,   // → $0.18/M
  "max_input_tokens": 262144,
  "max_output_tokens": 128000,
  "supports_reasoning": true,
  "mode": "chat"
}

Nothing is pinned locally, so a refresh picks up new models, rotated rosters and re-priced routes on its own. Model ids are the same provider-qualified ones /v1/chat/completions accepts, so a model chosen from the picker routes without translation.

If your harness has no LiteLLM discovery path, there is a static fallback — a generated models.yml fragment. It needs re-running whenever the catalogue changes, which is why it is the second choice:

curl -H "Authorization: Bearer freellmapi-your-gateway-key-here" \
  "http://localhost:3001/v1/models/omp-overrides.yaml?provider=freellmapi" \
  > ~/.config/omp/models.yml

---

OpenAI models

The default setup works out of the box — just add an OpenAI-compatible API key on the Keys page and you're ready:

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
)

curl http://localhost:3001/v1/chat/completions \
  -H "Authorization: Bearer freellmapi-your-gateway-key-here" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"Hello!"}]}'

---

Anthropic models

Anthropic models use the same gateway key — the router transparently translates your OpenAI-format request to Anthropic's native API format. No extra setup needed beyond adding an Anthropic API key on the Keys page:

response = client.chat.completions.create(
    model="claude-sonnet-4",
    messages=[{"role": "user", "content": "Hello!"}],
)

curl http://localhost:3001/v1/chat/completions \
  -H "Authorization: Bearer freellmapi-your-gateway-key-here" \
  -H "Content-Type: application/json" \
  -d '{"model":"claude-sonnet-4","messages":[{"role":"user","content":"Hello!"}]}'

Anthropic also supports native tool calling, thinking/reasoning, and streaming through the same gateway key — the router handles all protocol translation automatically.

Thinking / reasoning effort

Pass reasoning_effort to control how much a reasoning model thinks. Accepted values: none | minimal | low | medium | high | xhigh.

curl http://localhost:3001/v1/chat/completions \
  -H "Authorization: Bearer freellmapi-your-gateway-key-here" \
  -H "Content-Type: application/json" \
  -d '{"model":"nvidia/z-ai/glm-5.2","reasoning_effort":"none","messages":[{"role":"user","content":"2+2"}]}'

The gateway translates the level into whatever shape each upstream actually honors, so you never send provider-specific fields yourself: Gemini gets thinkingConfig.thinkingBudget, Nemotron 3.5 gets chat_template_kwargs.enable_thinking + thinking_token_budget, Zhipu gets thinking.type, Mistral gets its 2-value none|high enum. /v1/models advertises compat.supportsReasoningEffort per model so a client can show the control only where it works.

GLM-5.x caveat (genuine model limitation, not a gateway one). GLM's chat template resolves the effort to max unless the value is literally high — the template logic is 'high' if reasoning_effort == 'high' else 'max'. So the gateway clamps low/medium/high all onto high, and xhigh onto max. low and medium therefore think the same amount as high — the graduation between them cannot be expressed on GLM. If you want GLM to think less, use none or minimal, which turn thinking off outright via enable_thinking: false. Anything else reasons at high or above.

Image generation

POST /v1/images/generations is OpenAI-compatible. The first run after upgrading seeds a Pollinations FLUX (keyless) model on the imagegen platform that uses Pollinations.ai's anonymous legacy image endpoint — no card, no key, no rate limit to speak of. The row appears in Models / Fallback Chain with a rose Image badge and an auto-seeded sentinel key, so image generation works out of the box with zero configuration.

curl -X POST http://localhost:3001/v1/images/generations \
  -H "Authorization: Bearer freellmapi-your-gateway-key-here" \
  -H "Content-Type: application/json" \
  -d '{"prompt": "a red apple on a white background, studio lighting", "n": 1, "size": "1024x1024"}'

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3001/v1",
    api_key="freellmapi-your-gateway-key-here",
)

img = client.images.generate(
    model="imagegen/pollinations-flux",
    prompt="a red apple on a white background, studio lighting",
    n=1,
    size="1024x1024",
)
# img.data[0].b64_json is a base64-encoded image

Request fields:

Field	Type	Default	Notes
prompt	string (1-4000)	required	The text prompt.
model	string	auto	The image-gen model id, e.g. imagegen/pollinations-flux.
n	integer (1-4)	1	Number of images. The Pollinations endpoint serves one per request, so n>1 issues sequential calls.
size	"WxH"	1024x1024	Width x height in pixels, both clamped to 64-2048.
response_format	url | b64_json	b64_json	b64_json is the only format actually served (the Pollinations endpoint returns raw image bytes); url is accepted for OpenAI client compatibility.

Response shape (200 OK):

{
  "created": 1753819200,
  "data": [
    { "b64_json": "iVBORw0KGgo...", "revised_prompt": "a red apple on a white background, studio lighting" }
  ]
}

Errors:

• 401 invalid_api_key - the bearer token is missing or wrong.
• 422 no_image_gen_model - no image-gen model is enabled in the Fallback Chain. Enable imagegen/pollinations-flux (or another image-gen row) and retry.
• 422 invalid_request - the body is malformed or prompt is missing.
• 502 image_generation_failed - the upstream returned a 5xx; the message includes the upstream status text.

Add another image-gen backend: the imagegen platform is a normal provider slot. To add e.g. a Hugging Face-backed image-gen model, register a second provider on the same /v1/images/generations route, seed rows with supports_image_generation = 1 via a follow-up migration, and the route will pick whichever row is at the top of the fallback chain.

Gateway API keys

Create keys in the dashboard under Keys → Gateway keys. The raw key (freellmapi-…) is shown once at creation — only the hash is stored server-side, so you should copy it right away.

Each gateway key is provider-agnostic — the same key works for OpenAI models, Anthropic models, Google models, or any other configured provider. The router picks the best provider based on your fallback chain. You never need a different gateway key per provider.

How to add provider credentials (e.g., Anthropic, OpenAI, etc.):

1. Go to Keys page in the dashboard
2. Select the provider platform (e.g., Anthropic)
3. Paste your provider API key (e.g., Anthropic's sk-ant-… key, OpenAI's sk-… key)
4. Click Fetch all models to import that provider's catalog
5. Start making requests — the gateway key (freellmapi-…) routes to the right provider automatically

Example: Anthropic setup

1. Get an API key from console.anthropic.com
2. On the Keys page, select Anthropic and paste the key
3. Click Fetch all models
4. Use your gateway key with model="claude-sonnet-4" — the router handles the translation to Anthropic's native API format transparently

---

HTTP proxy pool

v1.3 ships a full HTTP proxy pool. Configure it from the Proxies page or via env / .env:

Env var	Default	Description
PROXY_URL	—	Single proxy URL. Wins over DB. e.g. http://user:pass@host:port
PROXY_POOL_STRATEGY	none	none (off), random, round-robin, least-latency
PROXY_BYPASS	—	Comma-separated platform names that skip the proxy (e.g. groq,google)

Single proxy — set PROXY_URL once. The dashboard tracks request count, success rate, latency, and bytes.

Pool — add rows in the Proxies page (label, URL, optional country), pick a strategy, and the router distributes each request across the pool. Per-key assignment rotates every hour to spread load and avoid burning any single egress.

Bypass list — platforms you don't want proxied (e.g. low-latency providers like Groq that you want to reach directly).

SOCKS proxies — supported via socks5:// and socks4:// URLs.

find-proxies script — scans a candidate URL list, geo-IP-verifies the country (France by default), probes connectivity, and writes the working set to working-proxies.txt for bulk import:

# 1. Find a batch of working proxies
node scripts/find-proxies.mjs --out working-proxies.txt

# 2. Import in the dashboard: Proxies → Bulk import → paste file

---

Dashboard

Live dashboard (v1.3 highlights)

• In-flight proxy name — every active row shows which proxy the request is using
• Growing token counts — output_tokens ticks up live as the model streams
• Session totals — Tokens IN and Tokens OUT aggregated over the dashboard session
• Per-user counts — events scoped to the logged-in user
• Real-time routing events — which provider/model was tried, latency, outcome, all pushed over WebSocket

Keys page

Add provider credentials for any supported service. Features per key:

• Live status dot — updates in real time: green (healthy), amber (rate-limited), red (error/disabled/401)
• Inline editing — pencil icon to edit label, RPM cap, TPM cap without re-entering the key
• Stats panel — click the dot or ⌄ chevron to expand deep stats:
  • Total requests, success rate, error count, rate-limit hits
  • Latency: min / p50 / avg / p95 / p99 / max
  • 24-hour bar chart, colour-coded by error rate
  • Per-model breakdown with success rate and token counts
  • Error breakdown with occurrence counts
  • Last 25 requests with model, latency, tokens, and timestamp
  • Active cooldowns with reset times
• Model discovery — one click calls the provider's /v1/models endpoint, adds new models to the catalog, and removes stale ones. Removal is decided per provider, not per key: free tiers hand out different rosters per account, so a model is retired only once no key on that provider lists it any more. The summary line counts what actually changed, so a refresh that changed nothing reads all zeros. Custom endpoint mode imports all advertised models in one shot.
• Click the summary line to see what changed, not just how much: every model added and retired, and every thinking level, context window, and price that moved, each with its value on both sides of the change (33K → 131K, in $1/M / out $2/M → in $4/M / out $9/M). One entry per counted change, so the number and the list can never disagree.
• Your keys outrank the catalog. The signed catalog also retires models it used to ship and no longer lists — but never one your own key still serves. The catalog is a snapshot of what is free somewhere; your key listing is what your account is actually given, and when they disagree your account wins. Without that rule the two fought: discovery added a model your key serves, then the next catalog apply (every boot, and twice a day) deleted it again, so it never appeared no matter how often you refreshed.

Custom endpoints: use the "Custom" provider to connect any OpenAI-compatible URL. Toggle "Auto-discover" to bulk-register all models the endpoint advertises.

Proxies

The proxy pool page lets you:

• Add HTTP / HTTPS / SOCKS proxies with labels and (optional) country
• Pick a pool strategy: none, random, round-robin, least-latency
• Bulk select with checkboxes and apply enable / disable / delete / test across many rows
• Click a row to expand its stats panel (request count, success rate, latency, bytes)
• Health-checker pings every enabled proxy on a schedule and marks failing ones

Playground

Interactive chat with:

• Full model selector (providers, auto)
• System prompt editor
• Live routing indicator — shows the provider, model, and proxy the router is currently trying while a request is in flight

Analytics

Request history with per-provider breakdown, token usage, latency trends, and cost estimates.

Three things the numbers mean, because each is easy to read wrong:

• Avg latency is time to first token — how fast the gateway starts answering. Avg duration is the whole request: queue wait, any failover attempts, and the full response stream. A long duration is normal for an agentic turn with a big context and says nothing bad about the provider; latency is the figure that makes providers comparable. Latency shows — when nothing in the window recorded a first-token time (it is measured on the streaming path).
• Est. savings is what the tokens you actually sent would have cost on paid APIs, priced per request at the moment it was served and stored on the row (requests.paid_cost_usd). It is a measurement of the selected range, not a projection, so it only changes when traffic does — a pricing refresh or a retired model never rewrites what last week cost. Errors and cancellations cost nothing and are excluded.
• Requests counts everything, including requests cancelled or abandoned before routing resolved. Those belong to no provider and no model, so the per-provider and per-model panels leave them out — which is why those panels can total less than the Requests card. The card's tooltip reports the gap so the two reconcile.

Range windows are aligned to whole UTC hours (24h) or whole UTC days (7d, 30d), and the charts cover exactly the window the cards count.

---

Configuration

Variable	Default	Description
ENCRYPTION_KEY	(required)	64-char hex string for AES-256-GCM. Generate: node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
PORT	3001	HTTP listen port
HOST_BIND	127.0.0.1	Bind interface. Set 0.0.0.0 for LAN access.
ADMIN_USERNAME	—	Bootstrap admin username (first run only)
ADMIN_PASSWORD	—	Bootstrap admin password, min 8 chars (first run only)
PROXY_RATE_LIMIT_RPM	120	Max /v1 requests per minute per client IP. 0 = disabled.
REQUEST_ANALYTICS_RETENTION_DAYS	90	Days to keep request history
REQUEST_ANALYTICS_MAX_ROWS	100000	Maximum request history rows
REQUEST_QUEUE_TIMEOUT_SECONDS	60	Seconds to hold a queued request before returning 429
MODEL_REFRESH_INTERVAL_MINUTES	—	Periodic live model list refresh. Unset = boot-time only.
FREELLMAPI_CONTEXT_HANDOFF	—	Set on_model_switch to inject context messages on model switch
BEDROCK_REGION	eu-central-1	Amazon Bedrock EU region. EU-only allowlist; non-EU values are refused.
PROXY_URL	—	Single proxy URL. e.g. http://user:pass@host:port
PROXY_POOL_STRATEGY	none	none / random / round-robin / least-latency
PROXY_BYPASS	—	Comma-separated platform names to skip the proxy
DASHBOARD_ORIGINS	—	Extra CORS origins for the dashboard (comma-separated)
LOG_LEVEL	INFO	DEBUG | INFO | WARN | ERROR | FATAL
GATEWAY_INSTANCES	1	PM2: number of gateway worker processes

---

How routing works

When a request arrives at /v1/chat/completions:

1. Resolve the model — auto uses the whole fallback chain; anything else looks up provider keys for that model ID.
2. Follow your order — candidates are tried in the fallback-chain order you set on the Models page (drag to reorder), with rate-limited models demoted by a temporary penalty.
3. Pick a proxy — for the resolved key, look up its current pool assignment (or fall back to the single PROXY_URL); the assignment rotates every hour so each key gets a fair share.
4. Rotate keys fairly — for the chosen provider, keys are tried least-recently-used first. Every key gets served once before any key repeats, and rotation state lives in the database (api_keys.use_seq), so it survives restarts. A key you just added has never been used, so it goes to the front of the queue and is picked on the very next request for that provider. Within that order the router takes the first key that isn't on cooldown, hasn't hit its outbound RPM/TPM cap, and has a healthy proxy.
5. On failure — a rate-limit response (429) puts the key on a cooldown; an auth failure (401/403) marks it invalid and pushes a real-time status update to the dashboard.
6. Queue if exhausted — if all candidates are on cooldown the request parks in an in-memory queue and retries when the earliest cooldown expires, up to REQUEST_QUEUE_TIMEOUT_SECONDS.
7. Context handoff — if the winning model differs from the previous turn in the same session, a compact system message is prepended so the new model has context.

Every routing decision is logged (provider, model, key prefix, proxy, latency) and streamed to the live dashboard via WebSocket.

When a model gets auto-disabled

If a provider answers 404/410 for a model, the gateway can disable its catalog row so the router stops paying a round-trip to a dead endpoint on every request. That verdict requires proof the model is gone, not that one key couldn't reach it:

• the provider says so in words — retired, end of life, no longer available, no endpoints found — which is authoritative on its own, or
• a bare 404/410 reproduces on 2+ different keys within 15 minutes.

A bare 404 on a single key is not enough. On gateways that grant model access per account (NVIDIA NIM, Bedrock, OpenRouter tiers) a 404 usually just means that key isn't entitled, and it also comes from a dead proxy's error page or a transient routing fault. Such a request fails over normally and the model stays enabled.

Auto-disabled rows are marked auto_disabled and stay visible on the Models page so you can re-enable them. Keys → Fetch all models also restores them automatically when the provider lists the model again. Requesting one returns a 400 that says the disable was automatic and how to undo it — rather than implying you turned it off.

---

Project structure

freellmapi/
├── server/src/
│   ├── routes/        REST endpoints — /api/*, /v1/*
│   │                    (proxy, keys, gateway-keys, admin, fallback, responses, settings…)
│   ├── services/      Router, queue, health checker, scoring, WebSocket push, auth, context handoff, proxy-health
│   ├── providers/     Per-provider OpenAI-compatible adapters (google, openai-compat, cohere, cloudflare)
│   ├── db/            SQLite (better-sqlite3), migration runner, model catalog migrations
│   └── lib/           Crypto, proxy pool, logger, error handling
├── client/src/
│   ├── pages/         Keys, Proxies, Gateway keys, Playground, Analytics, Live dashboard, Fallback, Premium, Admin…
│   └── components/    Shared UI components (shadcn/ui)
├── shared/            TypeScript types shared between server and client
├── desktop/           Electron desktop wrapper
├── scripts/           find-proxies.mjs and other ops helpers
├── k8s/               Kubernetes / Podman play kube manifests
├── .gitea/workflows/  Forgejo Actions (ci.yml, release.yml)
├── .github/workflows/ GitHub Actions (ci.yml, docker.yml)
├── ecosystem.config.cjs  PM2 process definition
├── Dockerfile         Multi-stage Docker build
├── docker-compose.yml
├── install.sh         Universal installer
├── migrate-from-old.sh  Import keys from an older FreeLLMAPI install
└── find-proxies.mjs   Free-proxy scraper / verifier

---

Continuous integration

Two Forgejo Actions workflows ship with the repo and are picked up automatically by your Forgejo runner:

Workflow	Trigger	What it does
.gitea/workflows/ci.yml	push to main, pull request	npm ci → build server → build client → run 728 server tests
.gitea/workflows/release.yml	push of any v* tag (e.g. v1.3)	same as CI, then posts a green-build summary to the run page

A push of the v1.3 tag will start a release build on the runner and report success or failure on the release page. Run the same suite locally with:

npm install
npm run build
npm test -w server

---

Contributing

git clone https://git.pandem.fr/outage.sh/FreeLLMapi freellmapi
cd freellmapi
npm install
npm run dev        # server on :3001, Vite dev server on :5173

• npm run build — full production build (server + client)
• npm test -w server — server test suite (728 tests)
• npm run build:server — server only
• npm run desktop:dev — Electron desktop wrapper in dev mode
• npm run desktop:dist — build a distributable Electron binary

PRs welcome. Keep changes focused; include tests for new server behaviour.

---

Disclaimer

FreeLLMAPI routes requests to third-party AI provider APIs under their respective free-tier terms. It does not circumvent rate limits — it distributes load across multiple API keys you legitimately own. Review each provider's Terms of Service before use. NVIDIA's free NIM tier is for evaluation only. The authors are not responsible for ToS violations or API key misuse.

---

MIT License · Based on FreeLLMAPI