Export GPU metrics
Scrape the GPU metrics of your organization's machines into Prometheus, Grafana Cloud or Datadog with an API key.
GET https://control-plane.b3iq.org/api/dev/metrics returns the GPU metrics of the machines your
organization is using, in the Prometheus text format. The metric names, help text and units are the ones
NVIDIA's dcgm-exporter uses, so a dashboard built for dcgm-exporter reads your B3IQ machines as they are.
Your scraper runs in your own infrastructure and pulls the metrics over HTTPS. Nothing is installed on B3IQ machines, and B3IQ does not push anything to you.
Create a key
The export is available to organizations. In the dashboard, switch to your organization with the organization switcher (not "Personal account"), open the API keys page, and create a key with Metrics, read-only switched on. Send it as a bearer token:
bashcurl -H "Authorization: Bearer $B3IQ_API_KEY" https://control-plane.b3iq.org/api/dev/metrics
A key created in "Personal account" cannot have Metrics access and gets a 403. A dashboard session without an organization gets a 403 with "Metrics export is available to organizations. Create the API key inside an organization." A key limited to some machines reads only those. If your organization is using no machines, the response is empty, with status 200.
Which machines you see
You see a machine's cards while your organization is the one using them.
- A machine your organization holds is in the export.
- A machine your organization rents is in the export from the start of the lease until it ends.
- On a machine that runs VMs, the renting organization sees only its own VM's cards. The owning organization stops seeing those cards from the start of the lease until the VM is destroyed, and keeps seeing the rest of the machine.
- A machine on a bare-metal lease leaves its owner's export for the length of the lease and comes back when the lease ends.
- A machine in Earn mode shows as up, with its Xid counts but no card readings.
On a machine that runs VMs, the readings for a rented VM's cards come from the B3IQ guest agent inside that
VM. If the renter stops the agent, those cards' readings stop too. The machine's b3iq_node_up and
b3iq_node_info series carry on.
Machines report every 60 seconds and the endpoint caches its answer for 15 seconds, so a value is at most about 75 seconds old.
Prometheus
Run Prometheus where you like, and add a scrape job:
yamlscrape_configs: - job_name: b3iq scheme: https metrics_path: /api/dev/metrics scrape_interval: 60s honor_labels: true authorization: type: Bearer credentials: b3iq_key_... static_configs: - targets: ["control-plane.b3iq.org"]
honor_labels: true keeps the instance label the endpoint sets, which is the machine id, so dashboard
12239 lists each machine. Without it, Prometheus sets instance to the scrape target and keeps the machine id
as exported_instance. The node_id label has the machine id either way. To keep the key out of the
config file, use credentials_file in place of credentials.
Grafana Cloud
Grafana Cloud's Metrics Endpoint integration scrapes the endpoint for you, with no Prometheus of your own.
- In Grafana Cloud, add the Metrics Endpoint integration and create a scrape job.
- Set the URL to
https://control-plane.b3iq.org/api/dev/metrics, pick Bearer authentication with your API key, and keep the 1 minute interval. - Import dashboard 12239, the NVIDIA DCGM Exporter Dashboard.
The temperature, power, utilization and memory panels fill in. The SM clock and tensor core panels stay empty, because the endpoint does not export those fields.
A scrape job has no honor_labels setting, so instance is the scrape target and the machine id arrives as
exported_instance and node_id. Pick machines by node_id instead. Variable and panel names can differ between dashboard revisions, so adjust to what you see: in the dashboard's settings, change the Instance variable's query to
label_values(DCGM_FI_DEV_GPU_TEMP, node_id) and each panel's instance=~"${instance}" to
node_id=~"${instance}".
Datadog
The Datadog Agent's OpenMetrics check can scrape the endpoint. The Agent runs in your own infrastructure,
on any host that can reach control-plane.b3iq.org. Add conf.d/openmetrics.d/conf.yaml:
yamlinit_config:instances: - openmetrics_endpoint: https://control-plane.b3iq.org/api/dev/metrics headers: Authorization: Bearer b3iq_key_... namespace: b3iq min_collection_interval: 60 metrics: - DCGM_FI_DEV_GPU_UTIL - DCGM_FI_DEV_FB_USED - DCGM_FI_DEV_FB_FREE - DCGM_FI_DEV_GPU_TEMP - DCGM_FI_DEV_POWER_USAGE - b3iq_node_up - b3iq_node_last_heartbeat_timestamp_seconds - b3iq_node_info - b3iq_gpu_xid_errors
Datadog bills what this check collects as custom metrics. Name the metrics you want, as above, rather than a
wildcard: a wildcard would also bill any metric the endpoint adds later. The Xid counter is listed without
its _total suffix, which is how the check names counters, and Datadog reports it with a .count suffix. With namespace: b3iq, the metrics arrive as
b3iq.DCGM_FI_DEV_GPU_UTIL, b3iq.b3iq_node_up and b3iq.b3iq_gpu_xid_errors.count.
The check stops at 2,000 metrics per instance unless you set max_returned_metrics. An eight-GPU machine
sends 43 series, so raise the limit if your machines together send more than 2,000.
Metrics
Each NVIDIA card sends these, labelled gpu (the card's index on the machine), UUID, device
(nvidia and the index), modelName, Hostname (the machine's name, or its id when it has none),
node_id and instance (both the machine id):
| Metric | Type | Unit |
|---|---|---|
DCGM_FI_DEV_GPU_UTIL | gauge | % |
DCGM_FI_DEV_FB_USED | gauge | MiB, rounded down |
DCGM_FI_DEV_FB_FREE | gauge | MiB, rounded down |
DCGM_FI_DEV_GPU_TEMP | gauge | °C |
DCGM_FI_DEV_POWER_USAGE | gauge | W |
b3iq_gpu_xid_errors_total | counter | Xid events, with an xid label for the Xid code |
Each machine sends these, labelled node_id and instance:
| Metric | Type | Value |
|---|---|---|
b3iq_node_up | gauge | 1, or 0 once the machine has missed two heartbeats |
b3iq_node_last_heartbeat_timestamp_seconds | gauge | Unix time of the last heartbeat |
b3iq_node_info | gauge | Always 1, with Hostname and driver_version labels |
- A value the machine did not measure has no series. It never shows as 0.
- While a machine is down, only its
b3iq_node_up,b3iq_node_last_heartbeat_timestamp_secondsandb3iq_node_infoseries are sent, so a stale reading does not draw a flat line. - Cards from other vendors send no card series. Up to eight cards per machine are reported; cards nine and up send no card series and no Xid series.
- A machine in Earn mode sends no card readings, the same as its charts on the dashboard. Its Xid count still shows.
- The Xid count covers the 90 days of events B3IQ keeps. On a bare-metal lease it starts at your lease. Cards in a VM lease report no Xid counts. When an event passes 90 days the count goes down, and Prometheus reads that as a counter reset.
A machine sends 3 series, plus 5 per NVIDIA card, plus 1 for each card and Xid code it has seen. An eight-GPU machine with no Xid errors sends 43. That is the per-machine maximum before Xid series, because only eight cards are reported.
Limits and errors
Each key may make 120 requests a minute. A scraper on a 60 second interval makes one.
| Status | When |
|---|---|
| 401 | The key is wrong, revoked or expired. |
| 403 | The key does not have Metrics access, was created outside an organization, or the X-Org-ID header names an organization the key is not bound to. Personal API keys always get 403. |
| 429 | The key made more than 120 requests in a minute. The response carries Retry-After: 60. |
| 502 | B3IQ could not check your permissions in the organization. The next scrape tries again. |
| 503 | B3IQ could not read the metrics or the organization's members. The next scrape tries again. |

