Mixture of Experts

Mixture of Experts (MoE) lets one POST /v1/api/chat/completions request run a prompt through several models in parallel and return one combined OpenAI-compatible answer. You attach an optional b3iq.mixture object to the request you already send; a plain request with a single model and no b3iq.mixture is completely unchanged.

The b3iq block is consumed by B3IQ routing and stripped before any node sees the request, so nodes serve each expert exactly like a normal single-model call.

How It Works

When a request carries b3iq.mixture, B3IQ plans one sub-call per expert (plus one for the aggregator, if the strategy uses one) and dispatches the experts in parallel. Each sub-call is routed independently through the same selection policy as a single-model request — supply is picked by health, readiness, and policy, and privacy class and your key's model policy apply to every expert and the aggregator alike.

Every sub-call stays inside the machines on your account, the same own-nodes-strict scope any gateway key already dispatches to — ownership is verified server-side on every sub-call, so the fan-out can never leave your fleet. Each expert model (and the aggregator model, for synthesize) must be installed and serving on at least one machine you own, and the request must be unpinned: a mixture combined with a node pin returns a 400, because a mixture always spreads across the fleet by policy. A failed mixture is never silently retried elsewhere — there is no fallback outside the machines you own, and a quorum failure is returned as-is. Mixtures are not available on the encrypted private-jobs path or direct node dispatch.

Once the experts return, a quorum check decides whether the mixture succeeded, and the strategy combines the surviving answers into the final response. The response is a standard chat.completion whose usage aggregates every sub-call, plus a metadata-only b3iq.mixture trace.

Strategies

Configuration

All fields live under b3iq.mixture on the chat-completion request body.

Send A Mixture

A synthesize mixture: two experts answer in parallel, then an aggregator model writes the final response. Model ids come from GET /v1/api/models — called with your key, that list is already scoped to the models the machines on your account serve right now.

bash
: "${B3IQ_GATEWAY_BASE_URL:?set B3IQ_GATEWAY_BASE_URL}": "${B3IQ_GATEWAY_KEY_FILE:?set B3IQ_GATEWAY_KEY_FILE}"cfg="$(mktemp)"chmod 600 "$cfg"trap 'rm -f "$cfg"' EXIT{  printf 'url = "%s/chat/completions"\n' "${B3IQ_GATEWAY_BASE_URL%/}"  printf 'request = "POST"\n'  printf 'header = "Authorization: Bearer %s"\n' "$(sed -n '1p' "$B3IQ_GATEWAY_KEY_FILE")"  printf 'header = "Content-Type: application/json"\n'} > "$cfg"curl --fail --silent --show-error --config "$cfg" --data-binary @- <<'JSON'{  "model": "qwen3-8b",  "messages": [{ "role": "user", "content": "Prove that sqrt(2) is irrational." }],  "b3iq": {    "mixture": {      "strategy": "synthesize",      "experts": [        { "model": "qwen3-8b", "weight": 2 },        { "model": "llama3.1-8b" }      ],      "aggregator": { "model": "qwen3-8b" }    }  }}JSON

javascript
import { readFileSync } from "node:fs";const baseURL = process.env.B3IQ_GATEWAY_BASE_URL;const token = readFileSync(process.env.B3IQ_GATEWAY_KEY_FILE, "utf8").trim();const response = await fetch(`${baseURL}/chat/completions`, {  method: "POST",  headers: {    authorization: `Bearer ${token}`,    "content-type": "application/json"  },  body: JSON.stringify({    model: "qwen3-8b",    messages: [{ role: "user", content: "Prove that sqrt(2) is irrational." }],    b3iq: {      mixture: {        strategy: "synthesize",        experts: [          { model: "qwen3-8b", weight: 2 },          { model: "llama3.1-8b" }        ],        aggregator: { model: "qwen3-8b" }      }    }  })});console.log(await response.json());

A vote mixture needs no aggregator — matching answers pool their weights and the winner is returned verbatim:

json
{  "model": "qwen3-8b",  "messages": [{ "role": "user", "content": "What is 17 * 24? Answer with the number only." }],  "b3iq": {    "mixture": {      "strategy": "vote",      "experts": [        { "model": "qwen3-8b" },        { "model": "llama3.1-8b" },        { "model": "mistral-small" }      ],      "min_experts": 2    }  }}

Response And The Mixture Trace

The response is a standard chat.completion. usage aggregates tokens across every sub-call (experts plus aggregator), and b3iq.mixture carries a trace of what ran. For synthesize, the response model is the aggregator model; otherwise it is your advisory top-level model.

json
{  "id": "chatcmpl-…",  "object": "chat.completion",  "model": "qwen3-8b",  "choices": [    { "index": 0, "message": { "role": "assistant", "content": "…" }, "finish_reason": "stop" }  ],  "usage": { "prompt_tokens": 96, "completion_tokens": 210, "total_tokens": 306 },  "b3iq": {    "request_id": "…",    "mixture": {      "strategy": "synthesize",      "degraded": false,      "experts": [        { "role": "expert", "model": "qwen3-8b", "status": "ok", "node_id": "b3iq_…", "usage": { "total_tokens": 84 } },        { "role": "expert", "model": "llama3.1-8b", "status": "ok", "node_id": "b3iq_…", "usage": { "total_tokens": 91 } },        { "role": "aggregator", "model": "qwen3-8b", "status": "ok", "node_id": "b3iq_…", "usage": { "total_tokens": 131 } }      ]    }  }}

The trace is metadata only — strategy, per-sub-call role, model, node id, status, and token counts. It never contains expert or aggregator text, and intermediate answers are never persisted; they exist only in memory while the mixture combines.

degraded: true means the mixture completed with a partial failure: one or more sub-calls did not succeed, or the synthesize aggregator failed and the answer fell back to the top-weighted surviving expert. The request still returns 200 — check the trace's per-sub-call status fields for what happened.

Streaming

stream: true is supported. The fan-out is fully buffered and settled first, then the final combined answer is delivered as SSE, with usage and the b3iq.mixture trace on the terminal chunk. Because every expert must finish before the stream opens, time-to-first-byte for a streamed mixture equals the full fan-out latency — a mixture is inherently slower than a single-model call.

Cost And Budgets

A mixture is free, like every request served by the machines on your account — no per-sub-call charge and no aggregate quote to afford. What is budgeted is capacity: the estimated token total across every sub-call is checked against the mixture token budget up front, each sub-call is metered like a normal request, and everything counts toward your account's daily token quota (default uncapped). See Usage & Quotas.

Errors And Limits

Mixture rejections use the same envelopes as other hosted errors — see Errors. The common cases:

Deployment-configurable limits apply per request: expert count (default 4), an aggregate token budget across all sub-calls, and an aggregate wall-clock deadline for the whole mixture. Keep max_tokens and the expert count modest — every expert multiplies the token and time budget the request consumes.

Related