> ## Documentation Index
> Fetch the complete documentation index at: https://docs.b3iq.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Export GPU metrics

Scrape the GPU metrics of your organization's machines into Prometheus, Grafana Cloud or Datadog with an API key.

`GET https://control-plane.b3iq.org/api/dev/metrics` returns the GPU metrics of the machines your
organization is using, in the Prometheus text format. The metric names, help text and units are the ones
NVIDIA's dcgm-exporter uses, so a dashboard built for dcgm-exporter reads your B3IQ machines as they are.

Your scraper runs in your own infrastructure and pulls the metrics over HTTPS. Nothing is installed on
B3IQ machines, and B3IQ does not push anything to you.

## Create a key

The export is available to organizations. In the dashboard, switch to your organization with the
organization switcher (not "Personal account"), open the [API keys](/developers/api-keys) page, and create a
key with **Metrics, read-only** switched on. Send it as a bearer token:

```bash
curl -H "Authorization: Bearer $B3IQ_API_KEY" https://control-plane.b3iq.org/api/dev/metrics
```

A key created in "Personal account" cannot have Metrics access and gets a 403. A dashboard session
without an organization gets a 403 with "Metrics export is available to organizations. Create the
API key inside an organization." A key limited to some machines reads only those. If your organization is using no machines,
the response is empty, with status 200.

## Which machines you see

You see a machine's cards while your organization is the one using them.

- A machine your organization holds is in the export.
- A machine your organization rents is in the export from the start of the lease until it ends.
- On a machine that runs VMs, the renting organization sees only its own VM's cards. The owning
  organization stops seeing those cards from the start of the lease until the VM is destroyed, and keeps
  seeing the rest of the machine.
- A machine on a bare-metal lease leaves its owner's export for the length of the lease and comes back when
  the lease ends.
- A machine in Earn mode shows as up, with its Xid counts but no card readings.

On a machine that runs VMs, the readings for a rented VM's cards come from the B3IQ guest agent inside that
VM. If the renter stops the agent, those cards' readings stop too. The machine's `b3iq_node_up` and
`b3iq_node_info` series carry on.

Machines report every 60 seconds and the endpoint caches its answer for 15 seconds, so a value is at most
about 75 seconds old.

## Prometheus

Run Prometheus where you like, and add a scrape job:

```yaml
scrape_configs:
  - job_name: b3iq
    scheme: https
    metrics_path: /api/dev/metrics
    scrape_interval: 60s
    honor_labels: true
    authorization:
      type: Bearer
      credentials: b3iq_key_...
    static_configs:
      - targets: ["control-plane.b3iq.org"]
```

`honor_labels: true` keeps the `instance` label the endpoint sets, which is the machine id, so dashboard
12239 lists each machine. Without it, Prometheus sets `instance` to the scrape target and keeps the machine id
as `exported_instance`. The `node_id` label has the machine id either way. To keep the key out of the
config file, use `credentials_file` in place of `credentials`.

## Grafana Cloud

Grafana Cloud's Metrics Endpoint integration scrapes the endpoint for you, with no Prometheus of your own.

1. In Grafana Cloud, add the **Metrics Endpoint** integration and create a scrape job.
2. Set the URL to `https://control-plane.b3iq.org/api/dev/metrics`, pick Bearer authentication with your API
   key, and keep the 1 minute interval.
3. Import dashboard [12239](https://grafana.com/grafana/dashboards/12239), the NVIDIA DCGM Exporter Dashboard.

The temperature, power, utilization and memory panels fill in. The SM clock and tensor core panels stay
empty, because the endpoint does not export those fields.

A scrape job has no `honor_labels` setting, so `instance` is the scrape target and the machine id arrives as
`exported_instance` and `node_id`. Pick machines by `node_id` instead. Variable and panel names can differ between dashboard revisions, so adjust to what you see: in the dashboard's settings, change the Instance variable's query to
`label_values(DCGM_FI_DEV_GPU_TEMP, node_id)` and each panel's `instance=~"${instance}"` to
`node_id=~"${instance}"`.

## Datadog

The Datadog Agent's OpenMetrics check can scrape the endpoint. The Agent runs in your own infrastructure,
on any host that can reach `control-plane.b3iq.org`. Add `conf.d/openmetrics.d/conf.yaml`:

```yaml
init_config:

instances:
  - openmetrics_endpoint: https://control-plane.b3iq.org/api/dev/metrics
    headers:
      Authorization: Bearer b3iq_key_...
    namespace: b3iq
    min_collection_interval: 60
    metrics:
      - DCGM_FI_DEV_GPU_UTIL
      - DCGM_FI_DEV_FB_USED
      - DCGM_FI_DEV_FB_FREE
      - DCGM_FI_DEV_GPU_TEMP
      - DCGM_FI_DEV_POWER_USAGE
      - b3iq_node_up
      - b3iq_node_last_heartbeat_timestamp_seconds
      - b3iq_node_info
      - b3iq_gpu_xid_errors
```

Datadog bills what this check collects as custom metrics. Name the metrics you want, as above, rather than a
wildcard: a wildcard would also bill any metric the endpoint adds later. The Xid counter is listed without
its `_total` suffix, which is how the check names counters, and Datadog reports it with a `.count` suffix. With `namespace: b3iq`, the metrics arrive as
`b3iq.DCGM_FI_DEV_GPU_UTIL`, `b3iq.b3iq_node_up` and `b3iq.b3iq_gpu_xid_errors.count`.

The check stops at 2,000 metrics per instance unless you set `max_returned_metrics`. An eight-GPU machine
sends 43 series, so raise the limit if your machines together send more than 2,000.

## Metrics

Each NVIDIA card sends these, labelled `gpu` (the card's index on the machine), `UUID`, `device`
(`nvidia` and the index), `modelName`, `Hostname` (the machine's name, or its id when it has none),
`node_id` and `instance` (both the machine id):

| Metric | Type | Unit |
| --- | --- | --- |
| `DCGM_FI_DEV_GPU_UTIL` | gauge | % |
| `DCGM_FI_DEV_FB_USED` | gauge | MiB, rounded down |
| `DCGM_FI_DEV_FB_FREE` | gauge | MiB, rounded down |
| `DCGM_FI_DEV_GPU_TEMP` | gauge | °C |
| `DCGM_FI_DEV_POWER_USAGE` | gauge | W |
| `b3iq_gpu_xid_errors_total` | counter | Xid events, with an `xid` label for the Xid code |

Each machine sends these, labelled `node_id` and `instance`:

| Metric | Type | Value |
| --- | --- | --- |
| `b3iq_node_up` | gauge | 1, or 0 once the machine has missed two heartbeats |
| `b3iq_node_last_heartbeat_timestamp_seconds` | gauge | Unix time of the last heartbeat |
| `b3iq_node_info` | gauge | Always 1, with `Hostname` and `driver_version` labels |

- A value the machine did not measure has no series. It never shows as 0.
- While a machine is down, only its `b3iq_node_up`, `b3iq_node_last_heartbeat_timestamp_seconds` and
  `b3iq_node_info` series are sent, so a stale reading does not draw a flat line.
- Cards from other vendors send no card series. Up to eight cards per machine are reported; cards nine and up send no card series and no Xid series.
- A machine in Earn mode sends no card readings, the same as its charts on the dashboard. Its Xid count
  still shows.
- The Xid count covers the 90 days of events B3IQ keeps. On a bare-metal lease it starts at your lease.
  Cards in a VM lease report no Xid counts.
  When an event passes 90 days the count goes down, and Prometheus reads that as a counter reset.

A machine sends 3 series, plus 5 per NVIDIA card, plus 1 for each card and Xid code it has seen. An
eight-GPU machine with no Xid errors sends 43. That is the per-machine maximum before Xid series, because only eight cards are reported.

## Limits and errors

Each key may make 120 requests a minute. A scraper on a 60 second interval makes one.

| Status | When |
| --- | --- |
| 401 | The key is wrong, revoked or expired. |
| 403 | The key does not have Metrics access, was created outside an organization, or the `X-Org-ID` header names an organization the key is not bound to. Personal API keys always get 403. |
| 429 | The key made more than 120 requests in a minute. The response carries `Retry-After: 60`. |
| 502 | B3IQ could not check your permissions in the organization. The next scrape tries again. |
| 503 | B3IQ could not read the metrics or the organization's members. The next scrape tries again. |