Runtimes
Which local inference runtimes B3IQ supports, which one a given machine should use, and how one-click install verifies each build.
A runtime is the local engine that actually runs a model on your machine. B3IQ routes work to your node, but the runtime is what loads the weights and produces tokens. Picking the right one is mostly about your hardware, and the setup wizard does the picking for you in most cases.
You do not need to memorize any of this. The wizard detects your hardware, pre-selects a sensible default, and labels every option in plain language. This page exists so you understand what it chose and why.
Supported Runtimes
| Runtime | Best for | Runs on |
|---|---|---|
| Ollama | The simplest start. Pick a model from a curated list and it just runs. | Works on any computer |
| llama.cpp | The recommended default. OpenAI-compatible, squeezes the most out of a gaming or workstation GPU, and downloads models on demand. | Works on any computer (Metal-accelerated on Apple Silicon) |
| MLX | Fastest on Macs. Apple-Silicon-native, tuned for unified memory. | Apple Silicon Macs only (needs Python 3.10+) |
| LocalAI | Image, audio, embedding, and rerank models from the LocalAI hub. | Linux and Apple Silicon macOS |
| OpenAI-compatible endpoint | Advanced users running their own server (vLLM, SGLang, Xinference). | Whatever your server runs |
If you are not sure, take the wizard's pre-selected default. Ollama is the easiest first run; llama.cpp is the recommended default for getting the most out of a GPU. Both run on any computer.
What The Wizard Shows You
Rather than asking you to reason about drivers and architectures, the wizard labels each runtime with one of four plain-language eligibility states based on the hardware it detected:
| Label | What it means |
|---|---|
| Works on any computer | Runs on CPU, and uses a GPU if you have one. Safe choice anywhere. |
| Needs an NVIDIA graphics card | Only offered when an NVIDIA GPU is present on this machine. |
| Apple Silicon Macs only | Offered on M-series Macs; hidden on Intel Macs and other platforms. |
| Not supported on this machine | This runtime cannot run on your hardware, so the wizard will not let you start a doomed install. |
One-Click Install
The setup wizard can install a supported runtime for you. Each runtime in the catalog is pinned to a known build, so you get a tested version rather than whatever happens to be latest.
Before B3IQ installs a pinned runtime build it verifies the download against a
known checksum, and stops and reports the failure rather than running an
unverified binary. This applies to the host installers and to one-click binary
runtime installs. MLX provisions through pip over TLS.
On Apple Silicon, MLX is one-click: B3IQ provisions it for you in a managed
Python virtual environment (you can also install it yourself with
pip install mlx-lm). Where a one-click install genuinely isn't available for a
runtime on your machine, the wizard shows a copy-paste command and a docs link
instead of an Install button.
Picking A Runtime By Platform
On Linux, llama.cpp is the recommended default and works on any computer. If you have an NVIDIA GPU, the GPU-accelerated build is offered automatically and gets the most out of the card. Ollama is the easiest alternative if you would rather pick from a curated model list. Advanced operators can point B3IQ at a self-hosted vLLM, SGLang, or Xinference endpoint.
On Apple Silicon Macs, B3IQ prefers MLX (fastest, Apple-native) and falls back to a Metal-accelerated llama.cpp build. See the note below for exactly how the Mac auto-provision decides. MLX is provisioned one-click in a managed Python environment (Python 3.10 or newer). On Intel Macs, llama.cpp is the default.
On Windows, llama.cpp is the recommended default and works on any computer. A GPU-accelerated build is offered when an NVIDIA card is detected. Ollama is the simplest alternative. The Windows host installer is checksum-verified the same way as the runtime installs.
On Apple Silicon Macs, B3IQ provisions the fastest option your machine can support. It installs MLX when possible, and falls back to a Metal-accelerated llama.cpp build, so your Apple Silicon hardware is used for inference.
Hardware Detection
B3IQ reads your machine's hardware so it can offer the right runtimes and the right accelerated builds. It detects:
| Hardware | How B3IQ uses it |
|---|---|
| NVIDIA GPU | Detected; unlocks GPU-accelerated builds and NVIDIA-only runtimes. |
| AMD GPU | Detected; used by GPU-accelerated builds where supported. |
| Apple Silicon | Detected; unlocks MLX and Metal-accelerated llama.cpp. |
| NPU | Detected and reported for visibility only. Not yet used for serving (future work). |
B3IQ can see a neural processing unit if your machine has one, and reports it for visibility. NPU acceleration is not yet wired into serving. Do not pick a node on the assumption that its NPU is doing inference today; serving still runs on CPU, GPU, or Apple Silicon.
Self-Healing vs Self-Hosted
Not every runtime gets the same crash recovery. B3IQ-installed runtimes are watched by the host agent and restarted automatically if they crash. A self-hosted OpenAI-compatible endpoint (vLLM, SGLang, Xinference) is a different story — B3IQ shows its endpoint on the machine's Runtimes tab, but it cannot restart a process it doesn't manage, so that card carries a won't self-heal badge. If a self-hosted engine goes down, restarting it is on you.
Listing Models For A Runtime
Once a runtime is installed, list the models actually available on your node rather than guessing at artifact names. The host agent exposes an OpenAI-compatible list:
bashcurl --fail --silent --show-error http://127.0.0.1:8831/v1/models
Use the ids returned here when configuring a model profile. The available set changes as you install or remove models, so always list rather than hard-code.
The same installed set is visible remotely, without SSHing into the box. The b3iq CLI can list it —
bashb3iq models list --machine <machine>b3iq runtimes list --machine <machine>
— and the machine's Models and Runtimes tabs in the account dashboard show the same statuses.