---
title: 'Grafana Cloud'
description: 'LLM Observability with Grafana Cloud and OpenLIT'
---

<Frame>
  <img src="/images/grafana-cloud-dashboard-1.png" />
  <img src="/images/grafana-cloud-dashboard-2.png" />
</Frame>

To directly send OpenTelemetry metrics and traces generated by OpenLIT SDK from your AI Application to Grafana Cloud, Follow the below steps.
<Steps>
  <Step title="Get your Grafana Cloud Credentials">
    <Steps>
      1. Sign in to Grafana Cloud Portal and select your Grafana Cloud Stack.
      1. Click **Configure** in the OpenTelemetry section.
        ![](https://grafana.com/media/docs/grafana-cloud/application-observability/opentelemetry-tile.png)
      1. In the **Password / API Token** section, click on **Generate now** to create a new API token:
         - Give the API token a name, for example `openlit`
         - Click on **Create token**
         - Click on **Close** without copying the token
         - Copy and Save the value for `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_EXPORTER_OTLP_HEADERS`
         <Note>Replace the space after `Basic` with `%20`: OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic%20[base64 instanceID:token]"</Note>
         ![](https://grafana.com/media/docs/grafana-cloud/application-observability/opentelemetry-env-vars.png)
    </Steps>
  </Step>
  <Step title="Add the following two lines to your application code:">
    <Tabs>
      <Tab title="Setup using function arguments">
      ```python
      import openlit

      openlit.init(
        otlp_endpoint="YOUR_GRAFANA_OTEL_GATEWAY_URL", 
        otlp_headers="YOUR_GRAFANA_OTEL_GATEWAY_AUTH"
      )
      ```

      Replace:
      1. `YOUR_GRAFANA_OTEL_GATEWAY_URL` with the `OTEL_EXPORTER_OTLP_ENDPOINT` value you copied in Step 1. 
        * Example - `https://otlp-gateway-<zone>.grafana.net/otlp`
      1. `YOUR_GRAFANA_OTEL_GATEWAY_AUTH` with the `OTEL_EXPORTER_OTLP_HEADERS` value you copied in Step 1. 
        * Example - `Authorization=Basic%20<base64 encoded Instance ID and API Token>`
      </Tab>
      <Tab title="Setup using Environment Variables">
      ```python
      import openlit

      openlit.init()
      ```

      Run the following command to configure the OTEL endpoint and headers to send metrics and traces to Grafana Cloud:
      ```shell
      export OTEL_EXPORTER_OTLP_ENDPOINT = "YOUR_GRAFANA_OTEL_GATEWAY_URL"
      export OTEL_EXPORTER_OTLP_HEADERS = "YOUR_GRAFANA_OTEL_GATEWAY_AUTH"
      ```

      Replace:
      1. `YOUR_GRAFANA_OTEL_GATEWAY_URL` with the `OTEL_EXPORTER_OTLP_ENDPOINT` value you copied in Step 1. 
         * Example - `https://otlp-gateway-<zone>.grafana.net/otlp`
      1. `YOUR_GRAFANA_OTEL_GATEWAY_AUTH` with the `OTEL_EXPORTER_OTLP_HEADERS` value you copied in Step 1. 
         * Example - `Authorization=Basic%20<base64 encoded Instance ID and API Token>`
      </Tab>
    </Tabs>
  Refer to the OpenLIT [Python SDK repository](https://github.com/openlit/openlit/tree/main/sdk/python) for more advanced configurations and use cases.
  </Step>
  <Step title="Import the pre-built Dashboard">
  1. Log into your Grafana Cloud Instance.
  1. Click **Dashboards** in the primary menu.
  1. Click **New** and select **Import** in the drop-down menu.
  1. Copy the dashboard JSON provided in the accordion named [`Dashboard`](#dashboard) below.
     1. To get the GPU Monitoring dashboard, Copy the dashboard JSON provided in the accordion named [`GPU Dashboard`](#gpu-dashboard)
  1. Paste the dashboard JSON text directly into the text area.
  1. Click **Import**.
  1. Save the dashboard.
  <Accordion title="Dashboard">
      ```json
      {
      "annotations": {
        "list": [
          {
            "builtIn": 1,
            "datasource": {
              "type": "grafana",
              "uid": "-- Grafana --"
            },
            "enable": true,
            "hide": true,
            "iconColor": "rgba(0, 211, 255, 1)",
            "name": "Annotations & Alerts",
            "type": "dashboard"
          }
        ]
      },
      "editable": true,
      "fiscalYearStartMonth": 0,
      "graphTooltip": 1,
      "id": 14,
      "links": [
        {
          "asDropdown": false,
          "icon": "bolt",
          "includeVars": false,
          "keepTime": false,
          "tags": [],
          "targetBlank": true,
          "title": "OpenLIT Github",
          "tooltip": "Github",
          "type": "link",
          "url": "https://github.com/openlit/openlit"
        },
        {
          "asDropdown": false,
          "icon": "doc",
          "includeVars": false,
          "keepTime": false,
          "tags": ["GPU", "AI"],
          "targetBlank": true,
          "title": "OpenLIT Docs",
          "tooltip": "Documentation",
          "type": "link",
          "url": "https://docs.openlit.io/"
        }
      ],
      "panels": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "gridPos": {
            "h": 5,
            "w": 12,
            "x": 0,
            "y": 0
          },
          "id": 21,
          "options": {
            "code": {
              "language": "plaintext",
              "showLineNumbers": false,
              "showMiniMap": false
            },
            "content": "---\n# GenAI Observability\n\nThis dashboard displays the usage stats of LLMs, Vector Databases and GPUs, tracking OpenTelemetry Traces and Metrics sent using [OpenLIT](https://github.com/openlit/openlit).\n\n---",
            "mode": "markdown"
          },
          "pluginVersion": "11.1.0-71516",
          "transparent": true,
          "type": "text"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "description": "This panel displays the LLM Request Rate ",
          "fieldConfig": {
            "defaults": {
              "color": {
                "fixedColor": "blue",
                "mode": "palette-classic-by-name"
              },
              "mappings": [],
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "red",
                    "value": null
                  },
                  {
                    "color": "#EAB839",
                    "value": 10
                  },
                  {
                    "color": "#6ED0E0",
                    "value": 100
                  }
                ]
              },
              "unit": "reqps"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 2,
            "w": 12,
            "x": 12,
            "y": 0
          },
          "id": 22,
          "options": {
            "colorMode": "background",
            "graphMode": "area",
            "justifyMode": "auto",
            "orientation": "auto",
            "percentChangeColorMode": "standard",
            "reduceOptions": {
              "calcs": [
                "lastNotNull"
              ],
              "fields": "",
              "values": false
            },
            "showPercentChange": true,
            "textMode": "value_and_name",
            "wideLayout": true
          },
          "pluginVersion": "11.1.0-71516",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "builder",
              "expr": "sum(rate(gen_ai_requests_total{telemetry_sdk_name=\"openlit\"}[$__rate_interval]))",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "LLM Request Rate",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "transparent": true,
          "type": "stat"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "description": "This panel displays the VectorDB Request Rate",
          "fieldConfig": {
            "defaults": {
              "color": {
                "fixedColor": "blue",
                "mode": "palette-classic-by-name"
              },
              "mappings": [],
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "red",
                    "value": null
                  },
                  {
                    "color": "#EAB839",
                    "value": 10
                  },
                  {
                    "color": "#6ED0E0",
                    "value": 100
                  }
                ]
              },
              "unit": "reqps"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 2,
            "w": 12,
            "x": 12,
            "y": 2
          },
          "id": 23,
          "options": {
            "colorMode": "background",
            "graphMode": "area",
            "justifyMode": "auto",
            "orientation": "auto",
            "percentChangeColorMode": "standard",
            "reduceOptions": {
              "calcs": [
                "lastNotNull"
              ],
              "fields": "",
              "values": false
            },
            "showPercentChange": true,
            "textMode": "value_and_name",
            "wideLayout": true
          },
          "pluginVersion": "11.1.0-71516",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "builder",
              "expr": "sum(rate(db_requests_total{telemetry_sdk_name=\"openlit\"}[$__rate_interval]))",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "VectorDB Request Rate",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "transparent": true,
          "type": "stat"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "description": "This panel displays the total cost incurred from using GenAI models. It reflects the financial impact of operational activities, offering insights into budgetary allocation and efficiency. Tracking this helps in effective cost management and financial planning for GenAI usage.",
          "fieldConfig": {
            "defaults": {
              "color": {
                "fixedColor": "blue",
                "mode": "shades"
              },
              "mappings": [],
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "green",
                    "value": null
                  },
                  {
                    "color": "red",
                    "value": 80
                  }
                ]
              },
              "unit": "currencyUSD"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 3,
            "w": 12,
            "x": 12,
            "y": 4
          },
          "id": 2,
          "options": {
            "colorMode": "background",
            "graphMode": "area",
            "justifyMode": "auto",
            "orientation": "auto",
            "percentChangeColorMode": "standard",
            "reduceOptions": {
              "calcs": [
                "lastNotNull"
              ],
              "fields": "",
              "values": false
            },
            "showPercentChange": true,
            "textMode": "auto",
            "wideLayout": true
          },
          "pluginVersion": "11.1.0-71516",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "sum(gen_ai_usage_cost_USD_sum{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "Total Usage Cost",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "transparent": true,
          "type": "stat"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "description": "This panel displays the total number of successful requests made to the GenAI system. A successful request is one that is completed without errors, indicating seamless operation and effective utilization of the GenAI service. Tracking this helps in understanding the reliability and performance of the GenAI system.",
          "fieldConfig": {
            "defaults": {
              "color": {
                "mode": "palette-classic-by-name"
              },
              "mappings": [],
              "min": 0,
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "green",
                    "value": null
                  },
                  {
                    "color": "red",
                    "value": 80
                  }
                ]
              },
              "unit": "none"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 5,
            "w": 6,
            "x": 0,
            "y": 5
          },
          "id": 1,
          "options": {
            "colorMode": "background",
            "graphMode": "area",
            "justifyMode": "auto",
            "orientation": "auto",
            "percentChangeColorMode": "standard",
            "reduceOptions": {
              "calcs": [
                "lastNotNull"
              ],
              "fields": "",
              "values": false
            },
            "showPercentChange": true,
            "textMode": "auto",
            "wideLayout": true
          },
          "pluginVersion": "11.1.0-71516",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "sum by() (gen_ai_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "Total Successful GenAI Requests",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "transparent": true,
          "type": "stat"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "description": "This panel displays the total count of successful VectorDB requests. It indicates requests that have been processed completely without any errors, showcasing VectorDB's effectiveness and reliability. Watching this helps gauge how well VectorDB is performing and its success in handling operations.",
          "fieldConfig": {
            "defaults": {
              "color": {
                "mode": "thresholds"
              },
              "mappings": [],
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "green",
                    "value": null
                  },
                  {
                    "color": "red",
                    "value": 80
                  }
                ]
              },
              "unit": "none"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 5,
            "w": 6,
            "x": 6,
            "y": 5
          },
          "id": 7,
          "options": {
            "colorMode": "background",
            "graphMode": "area",
            "justifyMode": "auto",
            "orientation": "auto",
            "percentChangeColorMode": "standard",
            "reduceOptions": {
              "calcs": [
                "lastNotNull"
              ],
              "fields": "",
              "values": false
            },
            "showPercentChange": true,
            "textMode": "auto",
            "wideLayout": true
          },
          "pluginVersion": "11.1.0-71516",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "sum(db_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "Total Successful VectorDB Requests",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "transparent": true,
          "type": "stat"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "description": "This panel displays the total number of tokens consumed by GenAI requests, providing a direct measure of usage. Monitoring this helps in assessing the demand on GenAI services and guiding resource allocation or optimization strategies.",
          "fieldConfig": {
            "defaults": {
              "color": {
                "fixedColor": "purple",
                "mode": "shades"
              },
              "mappings": [],
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "green",
                    "value": null
                  },
                  {
                    "color": "red",
                    "value": 80
                  }
                ]
              },
              "unit": "none"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 3,
            "w": 6,
            "x": 12,
            "y": 7
          },
          "id": 3,
          "options": {
            "colorMode": "background",
            "graphMode": "area",
            "justifyMode": "auto",
            "orientation": "auto",
            "percentChangeColorMode": "standard",
            "reduceOptions": {
              "calcs": [
                "lastNotNull"
              ],
              "fields": "",
              "values": false
            },
            "showPercentChange": true,
            "textMode": "auto",
            "wideLayout": true
          },
          "pluginVersion": "11.1.0-71516",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "sum(gen_ai_usage_tokens_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "Total Usage Tokens",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "transparent": true,
          "type": "stat"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "description": "This panel displays the average cost per use of the GenAI models and related services. It provides insights into the cost-effectiveness of interactions with GenAI, helping to identify trends in expense per operation. Monitoring this assists in optimizing budget allocation and improving cost efficiency in GenAI utilization.",
          "fieldConfig": {
            "defaults": {
              "color": {
                "fixedColor": "blue",
                "mode": "shades"
              },
              "mappings": [],
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "green",
                    "value": null
                  },
                  {
                    "color": "#EAB839",
                    "value": 0.5
                  },
                  {
                    "color": "red",
                    "value": 1
                  }
                ]
              },
              "unit": "currencyUSD"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 3,
            "w": 6,
            "x": 18,
            "y": 7
          },
          "id": 5,
          "options": {
            "colorMode": "background",
            "graphMode": "none",
            "justifyMode": "auto",
            "orientation": "auto",
            "percentChangeColorMode": "standard",
            "reduceOptions": {
              "calcs": [
                "lastNotNull"
              ],
              "fields": "",
              "values": false
            },
            "showPercentChange": true,
            "textMode": "auto",
            "wideLayout": true
          },
          "pluginVersion": "11.1.0-71516",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "avg by() (gen_ai_usage_cost_USD_sum{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "Avg Usage Cost",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "transparent": true,
          "type": "stat"
        },
        {
          "datasource": {
            "type": "tempo",
            "uid": "${traceDatasource}"
          },
          "description": "This panel displays the distribution of request durations for both GenAI and VectorDB services. It highlights how long requests take to complete, from the shortest to the longest durations, offering insights into system performance and efficiency. Understanding this distribution helps in identifying bottlenecks and optimizing response times for both services.",
          "fieldConfig": {
            "defaults": {
              "color": {
                "fixedColor": "blue",
                "mode": "palette-classic"
              },
              "custom": {
                "fillOpacity": 81,
                "gradientMode": "opacity",
                "hideFrom": {
                  "legend": false,
                  "tooltip": false,
                  "viz": false
                },
                "lineWidth": 3,
                "stacking": {
                  "group": "A",
                  "mode": "none"
                }
              },
              "mappings": [],
              "min": 0,
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "green",
                    "value": null
                  },
                  {
                    "color": "red",
                    "value": 10
                  }
                ]
              },
              "unit": "s"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 7,
            "w": 24,
            "x": 0,
            "y": 10
          },
          "id": 4,
          "options": {
            "legend": {
              "calcs": [],
              "displayMode": "list",
              "placement": "bottom",
              "showLegend": false
            }
          },
          "pluginVersion": "11.1.0-69372",
          "targets": [
            {
              "datasource": {
                "type": "tempo",
                "uid": "${traceDatasource}"
              },
              "filters": [
                {
                  "id": "status",
                  "operator": "=",
                  "scope": "intrinsic",
                  "tag": "status",
                  "value": "ok",
                  "valueType": "keyword"
                },
                {
                  "id": "f755ab99",
                  "operator": "=",
                  "scope": "span",
                  "tag": "telemetry.sdk.name",
                  "value": [
                    "openlit"
                  ],
                  "valueType": "string"
                }
              ],
              "limit": 20,
              "query": "{status=ok && span.telemetry.sdk.name=\"openlit\" && span.gen_ai.application_name=~\"$application\" && span.gen_ai.environment=~\"$environment\"}",
              "queryType": "traceql",
              "refId": "A",
              "tableType": "traces"
            }
          ],
          "title": "Request Duration Distribution",
          "type": "histogram"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "description": "This panel displays the ranking of GenAI models based on their usage frequency. It identifies which models are most popular or in-demand, providing insights into user preferences and operational trends. Analyzing this helps in resource allocation, optimizing model availability, and understanding which GenAI features are driving usage.",
          "fieldConfig": {
            "defaults": {
              "color": {
                "mode": "continuous-BlYlRd"
              },
              "mappings": [],
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "green",
                    "value": null
                  },
                  {
                    "color": "red",
                    "value": 80
                  }
                ]
              },
              "unit": "none"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 10,
            "w": 14,
            "x": 0,
            "y": 17
          },
          "id": 17,
          "options": {
            "displayMode": "gradient",
            "maxVizHeight": 300,
            "minVizHeight": 16,
            "minVizWidth": 8,
            "namePlacement": "auto",
            "orientation": "horizontal",
            "reduceOptions": {
              "calcs": [
                "lastNotNull"
              ],
              "fields": "",
              "values": false
            },
            "showUnfilled": true,
            "sizing": "auto",
            "valueMode": "text"
          },
          "pluginVersion": "11.1.0-71516",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "topk(5, sum by(gen_ai_request_model) (gen_ai_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}))",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "__auto",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "title": "Top GenAI Models by Usage",
          "transparent": true,
          "type": "bargauge"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "description": "This panel displays the distribution of GenAI requests based on the environment, such as production, development, testing, etc. It shows where the requests are being utilized, providing a clear picture of operational focus and deployment strategies. Tracking this helps in aligning resources appropriately and optimizing the performance across different environments.",
          "fieldConfig": {
            "defaults": {
              "color": {
                "mode": "palette-classic"
              },
              "custom": {
                "hideFrom": {
                  "legend": false,
                  "tooltip": false,
                  "viz": false
                }
              },
              "mappings": [],
              "min": 0,
              "unit": "none"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 5,
            "w": 5,
            "x": 14,
            "y": 17
          },
          "id": 15,
          "options": {
            "legend": {
              "displayMode": "list",
              "placement": "bottom",
              "showLegend": false
            },
            "pieType": "pie",
            "reduceOptions": {
              "calcs": [
                "lastNotNull"
              ],
              "fields": "",
              "values": false
            },
            "tooltip": {
              "mode": "single",
              "sort": "none"
            }
          },
          "pluginVersion": "11.1.0-69372",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "sum by(gen_ai_environment) (gen_ai_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "__auto",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "title": "GenAI Requests by Environment",
          "type": "piechart"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "description": "This panel displays the distribution of GenAI requests based on the application, such as production, development, testing, etc. It shows where the requests are being utilized, providing a clear picture of operational focus and deployment strategies. Tracking this helps in aligning resources appropriately and optimizing the performance across different environments.",
          "fieldConfig": {
            "defaults": {
              "color": {
                "mode": "palette-classic"
              },
              "custom": {
                "hideFrom": {
                  "legend": false,
                  "tooltip": false,
                  "viz": false
                }
              },
              "mappings": [],
              "min": 0,
              "unit": "none"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 5,
            "w": 5,
            "x": 19,
            "y": 17
          },
          "id": 27,
          "options": {
            "legend": {
              "displayMode": "list",
              "placement": "bottom",
              "showLegend": false
            },
            "pieType": "pie",
            "reduceOptions": {
              "calcs": [
                "lastNotNull"
              ],
              "fields": "",
              "values": false
            },
            "tooltip": {
              "mode": "single",
              "sort": "none"
            }
          },
          "pluginVersion": "11.1.0-69372",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "sum by(gen_ai_application) (gen_ai_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "__auto",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "title": "GenAI Requests by Application",
          "type": "piechart"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "description": "This panel displays the distribution of GenAI requests across different platforms such as OpenAI, Cohere, Anthropic, etc. It shows where requests are being sent, giving insights into platform popularity and usage patterns. Monitoring this helps in understanding platform preferences, and it can guide strategic decisions for integration or diversification of GenAI services.",
          "fieldConfig": {
            "defaults": {
              "color": {
                "mode": "palette-classic"
              },
              "custom": {
                "hideFrom": {
                  "legend": false,
                  "tooltip": false,
                  "viz": false
                }
              },
              "mappings": [],
              "min": 0,
              "unit": "none"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 5,
            "w": 5,
            "x": 14,
            "y": 22
          },
          "id": 16,
          "options": {
            "legend": {
              "displayMode": "list",
              "placement": "bottom",
              "showLegend": false
            },
            "pieType": "pie",
            "reduceOptions": {
              "calcs": [
                "lastNotNull"
              ],
              "fields": "",
              "values": false
            },
            "tooltip": {
              "mode": "single",
              "sort": "none"
            }
          },
          "pluginVersion": "11.1.0-69372",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "sum by(gen_ai_system) (gen_ai_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "__auto",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "title": "GenAI Requests by Platform",
          "type": "piechart"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "description": "This panel displays the breakdown of GenAI requests by type, such as Chat, Embedding, Image, Audio, and Fine Tuning. It reveals the diversity in usage, highlighting which types of requests are most common. Understanding this distribution assists in optimizing resources for the most demanded services and planning for future needs based on usage trends.",
          "fieldConfig": {
            "defaults": {
              "color": {
                "mode": "palette-classic"
              },
              "custom": {
                "hideFrom": {
                  "legend": false,
                  "tooltip": false,
                  "viz": false
                }
              },
              "mappings": [],
              "min": 0,
              "unit": "none"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 5,
            "w": 5,
            "x": 19,
            "y": 22
          },
          "id": 13,
          "options": {
            "legend": {
              "displayMode": "list",
              "placement": "bottom",
              "showLegend": false
            },
            "pieType": "pie",
            "reduceOptions": {
              "calcs": [
                "lastNotNull"
              ],
              "fields": "",
              "values": false
            },
            "tooltip": {
              "mode": "single",
              "sort": "none"
            }
          },
          "pluginVersion": "11.1.0-69372",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "sum by(gen_ai_type) (gen_ai_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "__auto",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "title": "GenAI Requests by Type",
          "type": "piechart"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "description": "This panel displays a comparative graph showing the average number of tokens consumed for completions and prompts against the average usage cost. It provides a visual representation of the relationship between the volume of data processed (in tokens) and the financial implications of using GenAI services. Analyzing this comparison helps in assessing cost-effectiveness and guiding strategic decisions for efficient resource utilization.",
          "fieldConfig": {
            "defaults": {
              "color": {
                "mode": "palette-classic"
              },
              "custom": {
                "axisBorderShow": false,
                "axisCenteredZero": false,
                "axisColorMode": "text",
                "axisLabel": "",
                "axisPlacement": "auto",
                "barAlignment": 1,
                "drawStyle": "line",
                "fillOpacity": 30,
                "gradientMode": "opacity",
                "hideFrom": {
                  "legend": false,
                  "tooltip": false,
                  "viz": false
                },
                "insertNulls": false,
                "lineInterpolation": "smooth",
                "lineStyle": {
                  "fill": "solid"
                },
                "lineWidth": 2,
                "pointSize": 5,
                "scaleDistribution": {
                  "type": "linear"
                },
                "showPoints": "always",
                "spanNulls": true,
                "stacking": {
                  "group": "A",
                  "mode": "none"
                },
                "thresholdsStyle": {
                  "mode": "off"
                }
              },
              "mappings": [],
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "green",
                    "value": null
                  },
                  {
                    "color": "red",
                    "value": 80
                  }
                ]
              },
              "unit": "none"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 8,
            "w": 24,
            "x": 0,
            "y": 27
          },
          "id": 6,
          "options": {
            "legend": {
              "calcs": [],
              "displayMode": "list",
              "placement": "bottom",
              "showLegend": true
            },
            "tooltip": {
              "mode": "single",
              "sort": "none"
            }
          },
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "avg(gen_ai_usage_input_tokens_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "Prompt Tokens",
              "range": true,
              "refId": "A",
              "useBackend": false
            },
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "builder",
              "expr": "avg(gen_ai_usage_completion_tokens_total{telemetry_sdk_name=\"openlit\"})",
              "fullMetaSearch": false,
              "hide": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "Completion Tokens",
              "range": true,
              "refId": "B",
              "useBackend": false
            },
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "builder",
              "expr": "avg(gen_ai_usage_cost_USD_bucket{telemetry_sdk_name=\"openlit\"})",
              "fullMetaSearch": false,
              "hide": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "Usage Cost",
              "range": true,
              "refId": "C",
              "useBackend": false
            }
          ],
          "title": "Average Token Consumption vs. Average Usage Cost Comparison",
          "type": "timeseries"
        },
        {
          "collapsed": true,
          "gridPos": {
            "h": 1,
            "w": 24,
            "x": 0,
            "y": 35
          },
          "id": 10,
          "panels": [
            {
              "datasource": {
                "type": "tempo",
                "uid": "${traceDatasource}"
              },
              "description": "This panel displays a detailed table of GenAI request traces, providing comprehensive insights into each request's timing, source, type, and outcome. It allows for the in-depth analysis of individual requests, facilitating troubleshooting, performance monitoring, and understanding user interactions with GenAI services. Tracking this helps in identifying patterns, potential issues, and opportunities for optimization.",
              "fieldConfig": {
                "defaults": {
                  "color": {
                    "mode": "continuous-BlPu"
                  },
                  "custom": {
                    "align": "auto",
                    "cellOptions": {
                      "type": "color-background"
                    },
                    "inspect": false
                  },
                  "mappings": [],
                  "thresholds": {
                    "mode": "absolute",
                    "steps": [
                      {
                        "color": "green"
                      }
                    ]
                  },
                  "unit": "s"
                },
                "overrides": []
              },
              "gridPos": {
                "h": 16,
                "w": 24,
                "x": 0,
                "y": 36
              },
              "id": 8,
              "options": {
                "cellHeight": "sm",
                "footer": {
                  "countRows": false,
                  "fields": "",
                  "reducer": [
                    "sum"
                  ],
                  "show": false
                },
                "showHeader": true
              },
              "pluginVersion": "11.1.0-71516",
              "targets": [
                {
                  "datasource": {
                    "type": "tempo",
                    "uid": "${traceDatasource}"
                  },
                  "filters": [
                    {
                      "id": "e7e29fde",
                      "operator": "!=",
                      "scope": "span",
                      "tag": "gen_ai.operation.name",
                      "value": [
                        "vectordb"
                      ],
                      "valueType": "string"
                    },
                    {
                      "id": "status",
                      "operator": "=",
                      "scope": "intrinsic",
                      "tag": "status",
                      "value": "ok",
                      "valueType": "keyword"
                    }
                  ],
                  "limit": 20,
                  "query": "{span.gen_ai.operation.name!=\"vectordb\" && status=ok && span.gen_ai.application_name=~\"$application\" && span.gen_ai.environment=~\"$environment\"} | select(span.gen_ai.prompt, span.gen_ai.completion, span.gen_ai.usage.cost)",
                  "queryType": "traceql",
                  "refId": "A",
                  "tableType": "traces"
                }
              ],
              "transparent": true,
              "type": "table"
            },
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "description": "",
              "gridPos": {
                "h": 2,
                "w": 24,
                "x": 0,
                "y": 52
              },
              "id": 25,
              "options": {
                "code": {
                  "language": "plaintext",
                  "showLineNumbers": false,
                  "showMiniMap": false
                },
                "content": "### VectorDB\n\n---",
                "mode": "markdown"
              },
              "pluginVersion": "11.1.0-71516",
              "transparent": true,
              "type": "text"
            },
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "description": "This panel displays the breakdown of database requests based on the type of operation, such as add, updates, deletes, and queries. It provides insights into the operational dynamics of database interactions, highlighting the most common actions performed. Understanding this distribution helps in optimizing database performance and planning for capacity based on operational needs.",
              "fieldConfig": {
                "defaults": {
                  "color": {
                    "mode": "continuous-BlYlRd"
                  },
                  "mappings": [],
                  "min": 0,
                  "thresholds": {
                    "mode": "absolute",
                    "steps": [
                      {
                        "color": "green"
                      },
                      {
                        "color": "red",
                        "value": 80
                      }
                    ]
                  },
                  "unit": "none"
                },
                "overrides": []
              },
              "gridPos": {
                "h": 9,
                "w": 14,
                "x": 0,
                "y": 54
              },
              "id": 20,
              "options": {
                "displayMode": "gradient",
                "maxVizHeight": 300,
                "minVizHeight": 16,
                "minVizWidth": 8,
                "namePlacement": "top",
                "orientation": "horizontal",
                "reduceOptions": {
                  "calcs": [
                    "lastNotNull"
                  ],
                  "fields": "",
                  "values": false
                },
                "showUnfilled": true,
                "sizing": "auto",
                "valueMode": "text"
              },
              "pluginVersion": "11.1.0-71516",
              "targets": [
                {
                  "datasource": {
                    "type": "prometheus",
                    "uid": "${metricsDatasource}"
                  },
                  "disableTextWrap": false,
                  "editorMode": "code",
                  "expr": "sum by(db_operation) (db_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
                  "fullMetaSearch": false,
                  "includeNullMetadata": true,
                  "instant": false,
                  "legendFormat": "__auto",
                  "range": true,
                  "refId": "A",
                  "useBackend": false
                }
              ],
              "title": "DB Requests by Operation",
              "transparent": true,
              "type": "bargauge"
            },
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "description": "This panel displays the distribution of requests across different VectorDB systems, such as Chroma, Pinecone, etc. It highlights which VectorDB services are most frequently used, providing insights into system preference and usage patterns. Monitoring this helps in evaluating the performance and scalability of each VectorDB system, guiding strategic decisions for technology adoption and integration.",
              "fieldConfig": {
                "defaults": {
                  "color": {
                    "mode": "palette-classic"
                  },
                  "custom": {
                    "hideFrom": {
                      "legend": false,
                      "tooltip": false,
                      "viz": false
                    }
                  },
                  "mappings": []
                },
                "overrides": []
              },
              "gridPos": {
                "h": 5,
                "w": 10,
                "x": 14,
                "y": 54
              },
              "id": 18,
              "options": {
                "legend": {
                  "displayMode": "list",
                  "placement": "bottom",
                  "showLegend": true
                },
                "pieType": "pie",
                "reduceOptions": {
                  "calcs": [
                    "lastNotNull"
                  ],
                  "fields": "",
                  "values": false
                },
                "tooltip": {
                  "mode": "single",
                  "sort": "none"
                }
              },
              "pluginVersion": "11.1.0-69950",
              "targets": [
                {
                  "datasource": {
                    "type": "prometheus",
                    "uid": "${metricsDatasource}"
                  },
                  "disableTextWrap": false,
                  "editorMode": "code",
                  "expr": "sum by(db_system) (db_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
                  "fullMetaSearch": false,
                  "includeNullMetadata": true,
                  "instant": false,
                  "legendFormat": "__auto",
                  "range": true,
                  "refId": "A",
                  "useBackend": false
                }
              ],
              "title": "DB Requests by System",
              "type": "piechart"
            },
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "description": "This panel displays the breakdown of DB requests by application name, offering insights into which applications are leveraging VectorDB services the most. This identification helps in understanding application-specific demand, guiding resource allocation, and supporting targeted optimization efforts for enhanced efficiency in application performance.\"",
              "fieldConfig": {
                "defaults": {
                  "color": {
                    "mode": "palette-classic"
                  },
                  "custom": {
                    "hideFrom": {
                      "legend": false,
                      "tooltip": false,
                      "viz": false
                    }
                  },
                  "mappings": [],
                  "min": 0,
                  "unit": "none"
                },
                "overrides": []
              },
              "gridPos": {
                "h": 4,
                "w": 5,
                "x": 14,
                "y": 59
              },
              "id": 19,
              "options": {
                "legend": {
                  "displayMode": "list",
                  "placement": "bottom",
                  "showLegend": false
                },
                "pieType": "pie",
                "reduceOptions": {
                  "calcs": [
                    "lastNotNull"
                  ],
                  "fields": "",
                  "values": false
                },
                "tooltip": {
                  "mode": "single",
                  "sort": "none"
                }
              },
              "pluginVersion": "11.1.0-69372",
              "targets": [
                {
                  "datasource": {
                    "type": "prometheus",
                    "uid": "${metricsDatasource}"
                  },
                  "disableTextWrap": false,
                  "editorMode": "builder",
                  "expr": "sum by(gen_ai_application_name) (db_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
                  "fullMetaSearch": false,
                  "includeNullMetadata": true,
                  "instant": false,
                  "legendFormat": "__auto",
                  "range": true,
                  "refId": "A",
                  "useBackend": false
                }
              ],
              "title": "DB Requests by Application",
              "type": "piechart"
            },
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "description": "This panel displays the breakdown of DB requests by application name, offering insights into which applications are leveraging VectorDB services the most. This identification helps in understanding application-specific demand, guiding resource allocation, and supporting targeted optimization efforts for enhanced efficiency in application performance.\"",
              "fieldConfig": {
                "defaults": {
                  "color": {
                    "mode": "palette-classic"
                  },
                  "custom": {
                    "hideFrom": {
                      "legend": false,
                      "tooltip": false,
                      "viz": false
                    }
                  },
                  "mappings": [],
                  "min": 0,
                  "unit": "none"
                },
                "overrides": []
              },
              "gridPos": {
                "h": 4,
                "w": 5,
                "x": 19,
                "y": 59
              },
              "id": 26,
              "options": {
                "legend": {
                  "displayMode": "list",
                  "placement": "bottom",
                  "showLegend": false
                },
                "pieType": "pie",
                "reduceOptions": {
                  "calcs": [
                    "lastNotNull"
                  ],
                  "fields": "",
                  "values": false
                },
                "tooltip": {
                  "mode": "single",
                  "sort": "none"
                }
              },
              "pluginVersion": "11.1.0-69372",
              "targets": [
                {
                  "datasource": {
                    "type": "prometheus",
                    "uid": "${metricsDatasource}"
                  },
                  "disableTextWrap": false,
                  "editorMode": "builder",
                  "expr": "sum by(gen_ai_application_name) (db_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
                  "fullMetaSearch": false,
                  "includeNullMetadata": true,
                  "instant": false,
                  "legendFormat": "__auto",
                  "range": true,
                  "refId": "A",
                  "useBackend": false
                }
              ],
              "title": "DB Requests by Application",
              "type": "piechart"
            }
          ],
          "title": "GenAI Requests",
          "type": "row"
        },
        {
          "collapsed": true,
          "gridPos": {
            "h": 1,
            "w": 24,
            "x": 0,
            "y": 36
          },
          "id": 12,
          "panels": [
            {
              "datasource": {
                "type": "tempo",
                "uid": "${traceDatasource}"
              },
              "description": "This panel displays a table with detailed traces of VectorDB requests, offering insights into specifics like request timing, source, operation type, and results. It's designed for thorough examination of VectorDB interactions, enabling precise troubleshooting, performance evaluation, and user behavior understanding. By monitoring these traces, users can spot trends, diagnose problems, and improve operational efficiency in handling VectorDB requests.",
              "fieldConfig": {
                "defaults": {
                  "color": {
                    "mode": "palette-classic-by-name"
                  },
                  "custom": {
                    "align": "auto",
                    "cellOptions": {
                      "type": "color-background"
                    },
                    "inspect": false
                  },
                  "mappings": [],
                  "min": 0,
                  "thresholds": {
                    "mode": "absolute",
                    "steps": [
                      {
                        "color": "green"
                      },
                      {
                        "color": "red",
                        "value": 80
                      }
                    ]
                  },
                  "unit": "s"
                },
                "overrides": []
              },
              "gridPos": {
                "h": 15,
                "w": 24,
                "x": 0,
                "y": 37
              },
              "id": 9,
              "options": {
                "cellHeight": "sm",
                "footer": {
                  "countRows": false,
                  "fields": "",
                  "reducer": [
                    "sum"
                  ],
                  "show": false
                },
                "showHeader": true
              },
              "pluginVersion": "11.1.0-71516",
              "targets": [
                {
                  "datasource": {
                    "type": "tempo",
                    "uid": "${traceDatasource}"
                  },
                  "filters": [
                    {
                      "id": "e7e29fde",
                      "operator": "=",
                      "scope": "span",
                      "tag": "gen_ai.operation.name",
                      "value": [
                        "vectordb"
                      ],
                      "valueType": "string"
                    },
                    {
                      "id": "status",
                      "operator": "=",
                      "scope": "intrinsic",
                      "tag": "status",
                      "value": "ok",
                      "valueType": "keyword"
                    }
                  ],
                  "limit": 20,
                  "query": "{span.gen_ai.operation.name=\"vectordb\" && status=ok && span.gen_ai.application_name=~\"$application\" && span.gen_ai.environment=~\"$environment\"}",
                  "queryType": "traceql",
                  "refId": "A",
                  "tableType": "traces"
                }
              ],
              "transparent": true,
              "type": "table"
            }
          ],
          "title": "VectorDB Requests",
          "type": "row"
        },
        {
          "collapsed": false,
          "gridPos": {
            "h": 1,
            "w": 24,
            "x": 0,
            "y": 37
          },
          "id": 28,
          "panels": [],
          "title": "GPU",
          "type": "row"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "fieldConfig": {
            "defaults": {
              "color": {
                "mode": "continuous-BlPu"
              },
              "custom": {
                "align": "auto",
                "cellOptions": {
                  "applyToRow": true,
                  "mode": "gradient",
                  "type": "color-background"
                },
                "filterable": false,
                "inspect": true
              },
              "displayName": "GPU",
              "mappings": [],
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "green",
                    "value": null
                  },
                  {
                    "color": "red",
                    "value": 80
                  }
                ]
              },
              "unit": "string"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 2,
            "w": 24,
            "x": 0,
            "y": 38
          },
          "id": 43,
          "options": {
            "cellHeight": "lg",
            "footer": {
              "countRows": false,
              "enablePagination": false,
              "fields": "",
              "reducer": [
                "sum"
              ],
              "show": false
            },
            "showHeader": false
          },
          "pluginVersion": "11.1.0-71516",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "gpu_fan_speed{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}",
              "format": "table",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "__auto",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "transformations": [
            {
              "id": "organize",
              "options": {
                "excludeByName": {
                  "Time": true,
                  "Value": true,
                  "__name__": true,
                  "gen_ai_application_name": true,
                  "gen_ai_environment": true,
                  "gpu_index": true,
                  "gpu_uuid": true,
                  "job": true,
                  "telemetry_sdk_name": true
                },
                "includeByName": {},
                "indexByName": {},
                "renameByName": {
                  "gpu_name": "Name",
                  "gpu_uuid": "UUID"
                }
              }
            }
          ],
          "transparent": true,
          "type": "table"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "description": "This panel displays the LLM Request Rate ",
          "fieldConfig": {
            "defaults": {
              "color": {
                "fixedColor": "blue",
                "mode": "continuous-GrYlRd"
              },
              "mappings": [],
              "min": 0,
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "red",
                    "value": null
                  }
                ]
              },
              "unit": "percent"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 3,
            "w": 6,
            "x": 0,
            "y": 40
          },
          "id": 34,
          "options": {
            "colorMode": "background",
            "graphMode": "area",
            "justifyMode": "auto",
            "orientation": "auto",
            "percentChangeColorMode": "inverted",
            "reduceOptions": {
              "calcs": [
                "lastNotNull"
              ],
              "fields": "",
              "values": false
            },
            "showPercentChange": true,
            "textMode": "value_and_name",
            "wideLayout": true
          },
          "pluginVersion": "11.1.0-71516",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "avg(gpu_utilization{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "Average GPU Utilization",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "transparent": true,
          "type": "stat"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "description": "",
          "fieldConfig": {
            "defaults": {
              "color": {
                "fixedColor": "blue",
                "mode": "continuous-GrYlRd"
              },
              "mappings": [],
              "min": 0,
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "red",
                    "value": null
                  }
                ]
              },
              "unit": "celsius"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 3,
            "w": 6,
            "x": 6,
            "y": 40
          },
          "id": 36,
          "options": {
            "colorMode": "background",
            "graphMode": "area",
            "justifyMode": "auto",
            "orientation": "auto",
            "percentChangeColorMode": "inverted",
            "reduceOptions": {
              "calcs": [
                "lastNotNull"
              ],
              "fields": "",
              "values": false
            },
            "showPercentChange": true,
            "textMode": "value_and_name",
            "wideLayout": true
          },
          "pluginVersion": "11.1.0-71516",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "avg(gpu_temperature{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "Average GPU Temperature",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "transparent": true,
          "type": "stat"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "description": "This panel displays the LLM Request Rate ",
          "fieldConfig": {
            "defaults": {
              "color": {
                "fixedColor": "blue",
                "mode": "continuous-GrYlRd"
              },
              "mappings": [],
              "min": 0,
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "red",
                    "value": null
                  }
                ]
              },
              "unit": "watt"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 3,
            "w": 6,
            "x": 12,
            "y": 40
          },
          "id": 35,
          "options": {
            "colorMode": "background",
            "graphMode": "area",
            "justifyMode": "auto",
            "orientation": "auto",
            "percentChangeColorMode": "standard",
            "reduceOptions": {
              "calcs": [
                "lastNotNull"
              ],
              "fields": "",
              "values": false
            },
            "showPercentChange": true,
            "textMode": "value_and_name",
            "wideLayout": true
          },
          "pluginVersion": "11.1.0-71516",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "avg(gpu_power_draw{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "Average GPU Power Draw",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "transparent": true,
          "type": "stat"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "description": "This panel displays the LLM Request Rate ",
          "fieldConfig": {
            "defaults": {
              "color": {
                "fixedColor": "blue",
                "mode": "continuous-GrYlRd"
              },
              "mappings": [],
              "min": 0,
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "red",
                    "value": null
                  }
                ]
              },
              "unit": "percent"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 3,
            "w": 6,
            "x": 18,
            "y": 40
          },
          "id": 37,
          "options": {
            "colorMode": "background",
            "graphMode": "area",
            "justifyMode": "auto",
            "orientation": "auto",
            "percentChangeColorMode": "inverted",
            "reduceOptions": {
              "calcs": [
                "lastNotNull"
              ],
              "fields": "",
              "values": false
            },
            "showPercentChange": true,
            "textMode": "value_and_name",
            "wideLayout": true
          },
          "pluginVersion": "11.1.0-71516",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "avg(gpu_memory_used{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}/gpu_memory_available{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "Average GPU Memory Usage",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "transparent": true,
          "type": "stat"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "fieldConfig": {
            "defaults": {
              "color": {
                "mode": "palette-classic-by-name"
              },
              "custom": {
                "axisBorderShow": true,
                "axisCenteredZero": false,
                "axisColorMode": "text",
                "axisLabel": "Percentage",
                "axisPlacement": "auto",
                "barAlignment": 0,
                "drawStyle": "line",
                "fillOpacity": 1,
                "gradientMode": "hue",
                "hideFrom": {
                  "legend": false,
                  "tooltip": false,
                  "viz": false
                },
                "insertNulls": false,
                "lineInterpolation": "smooth",
                "lineStyle": {
                  "fill": "solid"
                },
                "lineWidth": 1,
                "pointSize": 5,
                "scaleDistribution": {
                  "type": "linear"
                },
                "showPoints": "auto",
                "spanNulls": 3600000,
                "stacking": {
                  "group": "A",
                  "mode": "none"
                },
                "thresholdsStyle": {
                  "mode": "off"
                }
              },
              "mappings": [],
              "max": 100,
              "min": 0,
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "green",
                    "value": null
                  },
                  {
                    "color": "red",
                    "value": 80
                  }
                ]
              },
              "unit": "percent"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 9,
            "w": 12,
            "x": 0,
            "y": 43
          },
          "id": 38,
          "options": {
            "legend": {
              "calcs": [
                "lastNotNull",
                "mean",
                "max"
              ],
              "displayMode": "table",
              "placement": "bottom",
              "showLegend": true
            },
            "tooltip": {
              "mode": "single",
              "sort": "none"
            }
          },
          "pluginVersion": "11.1.0-71516",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "gpu_utilization{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "{{gpu_uuid}}-{{gpu_name}}",
              "range": true,
              "refId": "A",
              "useBackend": false
            },
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "builder",
              "expr": "gpu_dec_utilization{telemetry_sdk_name=\"openlit\"}",
              "fullMetaSearch": false,
              "hide": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "{{gpu_uuid}}-{{gpu_name}}",
              "range": true,
              "refId": "B",
              "useBackend": false
            },
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "builder",
              "expr": "gpu_enc_utilization{telemetry_sdk_name=\"openlit\"}",
              "fullMetaSearch": false,
              "hide": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "{{gpu_uuid}}-{{gpu_name}}",
              "range": true,
              "refId": "C",
              "useBackend": false
            }
          ],
          "title": "GPU Utilization",
          "type": "timeseries"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "fieldConfig": {
            "defaults": {
              "color": {
                "mode": "palette-classic-by-name"
              },
              "custom": {
                "axisBorderShow": true,
                "axisCenteredZero": false,
                "axisColorMode": "text",
                "axisLabel": "Temperature (°C)",
                "axisPlacement": "auto",
                "barAlignment": 0,
                "drawStyle": "line",
                "fillOpacity": 1,
                "gradientMode": "opacity",
                "hideFrom": {
                  "legend": false,
                  "tooltip": false,
                  "viz": false
                },
                "insertNulls": false,
                "lineInterpolation": "smooth",
                "lineStyle": {
                  "fill": "solid"
                },
                "lineWidth": 1,
                "pointSize": 5,
                "scaleDistribution": {
                  "type": "linear"
                },
                "showPoints": "auto",
                "spanNulls": true,
                "stacking": {
                  "group": "A",
                  "mode": "none"
                },
                "thresholdsStyle": {
                  "mode": "off"
                }
              },
              "mappings": [],
              "max": 100,
              "min": 0,
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "green",
                    "value": null
                  },
                  {
                    "color": "red",
                    "value": 80
                  }
                ]
              },
              "unit": "celsius"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 9,
            "w": 12,
            "x": 12,
            "y": 43
          },
          "id": 39,
          "options": {
            "legend": {
              "calcs": [
                "lastNotNull",
                "mean",
                "max"
              ],
              "displayMode": "table",
              "placement": "bottom",
              "showLegend": true
            },
            "tooltip": {
              "mode": "single",
              "sort": "none"
            }
          },
          "pluginVersion": "11.1.0-71516",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "gpu_temperature{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "{{gpu_uuid}}-{{gpu_name}}",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "title": "GPU Temperature",
          "type": "timeseries"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "fieldConfig": {
            "defaults": {
              "color": {
                "mode": "palette-classic-by-name"
              },
              "custom": {
                "axisBorderShow": true,
                "axisCenteredZero": false,
                "axisColorMode": "text",
                "axisLabel": "",
                "axisPlacement": "auto",
                "barAlignment": 0,
                "drawStyle": "bars",
                "fillOpacity": 50,
                "gradientMode": "hue",
                "hideFrom": {
                  "legend": false,
                  "tooltip": false,
                  "viz": false
                },
                "insertNulls": false,
                "lineInterpolation": "linear",
                "lineStyle": {
                  "fill": "solid"
                },
                "lineWidth": 1,
                "pointSize": 3,
                "scaleDistribution": {
                  "type": "linear"
                },
                "showPoints": "always",
                "spanNulls": 3600000,
                "stacking": {
                  "group": "A",
                  "mode": "percent"
                },
                "thresholdsStyle": {
                  "mode": "off"
                }
              },
              "mappings": [],
              "min": 0,
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "green",
                    "value": null
                  },
                  {
                    "color": "red",
                    "value": 80
                  }
                ]
              },
              "unit": "decmbytes"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 9,
            "w": 12,
            "x": 0,
            "y": 52
          },
          "id": 40,
          "options": {
            "legend": {
              "calcs": [
                "lastNotNull",
                "mean",
                "max"
              ],
              "displayMode": "table",
              "placement": "bottom",
              "showLegend": true
            },
            "tooltip": {
              "mode": "single",
              "sort": "none"
            }
          },
          "pluginVersion": "11.1.0-71516",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "gpu_memory_used{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "Memory Used",
              "range": true,
              "refId": "A",
              "useBackend": false
            },
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "builder",
              "expr": "gpu_memory_free{telemetry_sdk_name=\"openlit\"}",
              "fullMetaSearch": false,
              "hide": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "Memory Free",
              "range": true,
              "refId": "C",
              "useBackend": false
            },
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "builder",
              "expr": "gpu_memory_total{telemetry_sdk_name=\"openlit\"}",
              "fullMetaSearch": false,
              "hide": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "Memory Total",
              "range": true,
              "refId": "B",
              "useBackend": false
            }
          ],
          "title": "GPU Memory Usage",
          "type": "timeseries"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "fieldConfig": {
            "defaults": {
              "color": {
                "mode": "palette-classic-by-name"
              },
              "custom": {
                "axisBorderShow": true,
                "axisCenteredZero": false,
                "axisColorMode": "text",
                "axisLabel": "Fan Speed (0-100)",
                "axisPlacement": "auto",
                "barAlignment": 0,
                "drawStyle": "line",
                "fillOpacity": 1,
                "gradientMode": "opacity",
                "hideFrom": {
                  "legend": false,
                  "tooltip": false,
                  "viz": false
                },
                "insertNulls": false,
                "lineInterpolation": "smooth",
                "lineStyle": {
                  "fill": "solid"
                },
                "lineWidth": 1,
                "pointSize": 5,
                "scaleDistribution": {
                  "type": "linear"
                },
                "showPoints": "auto",
                "spanNulls": true,
                "stacking": {
                  "group": "A",
                  "mode": "none"
                },
                "thresholdsStyle": {
                  "mode": "off"
                }
              },
              "mappings": [],
              "max": 100,
              "min": 0,
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "green",
                    "value": null
                  },
                  {
                    "color": "red",
                    "value": 80
                  }
                ]
              },
              "unit": "none"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 9,
            "w": 12,
            "x": 12,
            "y": 52
          },
          "id": 41,
          "options": {
            "legend": {
              "calcs": [
                "lastNotNull",
                "mean",
                "max"
              ],
              "displayMode": "table",
              "placement": "bottom",
              "showLegend": true
            },
            "tooltip": {
              "mode": "single",
              "sort": "none"
            }
          },
          "pluginVersion": "11.1.0-71516",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "gpu_fan_speed{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "{{gpu_uuid}}-{{gpu_name}}",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "title": "GPU Fan Speed",
          "type": "timeseries"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${metricsDatasource}"
          },
          "fieldConfig": {
            "defaults": {
              "color": {
                "mode": "palette-classic-by-name"
              },
              "custom": {
                "axisBorderShow": true,
                "axisCenteredZero": false,
                "axisColorMode": "text",
                "axisLabel": "",
                "axisPlacement": "auto",
                "barAlignment": 0,
                "drawStyle": "bars",
                "fillOpacity": 50,
                "gradientMode": "hue",
                "hideFrom": {
                  "legend": false,
                  "tooltip": false,
                  "viz": false
                },
                "insertNulls": false,
                "lineInterpolation": "linear",
                "lineStyle": {
                  "fill": "solid"
                },
                "lineWidth": 1,
                "pointSize": 3,
                "scaleDistribution": {
                  "type": "linear"
                },
                "showPoints": "always",
                "spanNulls": 3600000,
                "stacking": {
                  "group": "A",
                  "mode": "percent"
                },
                "thresholdsStyle": {
                  "mode": "off"
                }
              },
              "mappings": [],
              "min": 0,
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "green",
                    "value": null
                  },
                  {
                    "color": "red",
                    "value": 80
                  }
                ]
              },
              "unit": "watt"
            },
            "overrides": []
          },
          "gridPos": {
            "h": 7,
            "w": 24,
            "x": 0,
            "y": 61
          },
          "id": 42,
          "options": {
            "legend": {
              "calcs": [
                "lastNotNull",
                "mean",
                "max"
              ],
              "displayMode": "table",
              "placement": "bottom",
              "showLegend": true
            },
            "tooltip": {
              "mode": "single",
              "sort": "none"
            }
          },
          "pluginVersion": "11.1.0-71516",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "code",
              "expr": "gpu_power_draw{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "Power Draw",
              "range": true,
              "refId": "A",
              "useBackend": false
            },
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${metricsDatasource}"
              },
              "disableTextWrap": false,
              "editorMode": "builder",
              "expr": "gpu_power_limit{telemetry_sdk_name=\"openlit\"}",
              "fullMetaSearch": false,
              "hide": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "Power Limit",
              "range": true,
              "refId": "C",
              "useBackend": false
            }
          ],
          "title": "GPU Memory Usage",
          "type": "timeseries"
        }
      ],
      "refresh": "",
      "schemaVersion": 39,
      "tags": [
        "LLM",
        "VectorDB",
        "GenAI",
        "GPU"
      ],
      "templating": {
        "list": [
          {
            "hide": 0,
            "includeAll": false,
            "label": "Trace Datasource",
            "multi": false,
            "name": "traceDatasource",
            "options": [],
            "query": "tempo",
            "queryValue": "",
            "refresh": 1,
            "regex": "",
            "skipUrlSync": false,
            "type": "datasource"
          },
          {
            "hide": 0,
            "includeAll": false,
            "label": "Metrics Datasource",
            "multi": false,
            "name": "metricsDatasource",
            "options": [],
            "query": "prometheus",
            "queryValue": "",
            "refresh": 1,
            "regex": "",
            "skipUrlSync": false,
            "type": "datasource"
          },
          {
            "current": {
              "selected": true,
              "text": [
                "All"
              ],
              "value": [
                "$__all"
              ]
            },
            "datasource": {
              "type": "prometheus",
              "uid": "${metricsDatasource}"
            },
            "definition": "label_values(gen_ai_application_name)",
            "hide": 0,
            "includeAll": true,
            "label": "Application",
            "multi": true,
            "name": "application",
            "options": [],
            "query": {
              "qryType": 1,
              "query": "label_values(gen_ai_application_name)",
              "refId": "PrometheusVariableQueryEditor-VariableQuery"
            },
            "refresh": 1,
            "regex": "",
            "skipUrlSync": false,
            "sort": 0,
            "type": "query"
          },
          {
            "current": {
              "selected": true,
              "text": [
                "All"
              ],
              "value": [
                "$__all"
              ]
            },
            "datasource": {
              "type": "prometheus",
              "uid": "${metricsDatasource}"
            },
            "definition": "label_values(gen_ai_environment)",
            "hide": 0,
            "includeAll": true,
            "label": "Environment",
            "multi": true,
            "name": "environment",
            "options": [],
            "query": {
              "qryType": 1,
              "query": "label_values(gen_ai_environment)",
              "refId": "PrometheusVariableQueryEditor-VariableQuery"
            },
            "refresh": 1,
            "regex": "",
            "skipUrlSync": false,
            "sort": 1,
            "type": "query"
          }
        ]
      },
      "time": {
        "from": "now-1h",
        "to": "now"
      },
      "timepicker": {},
      "timezone": "browser",
      "title": "GenAI Observability",
      "uid": "cdiz9piuoa3gge",
      "version": 2,
      "weekStart": ""
    }
      ```
  </Accordion>
  
  
  <Accordion title="GPU Dashboard">
      ```json
      {
        "annotations": {
          "list": [
            {
              "builtIn": 1,
              "datasource": {
                "type": "grafana",
                "uid": "-- Grafana --"
              },
              "enable": true,
              "hide": true,
              "iconColor": "rgba(0, 211, 255, 1)",
              "name": "Annotations & Alerts",
              "type": "dashboard"
            }
          ]
        },
        "editable": true,
        "fiscalYearStartMonth": 0,
        "graphTooltip": 0,
        "id": 13,
        "links": [],
        "panels": [
          {
            "fieldConfig": {
              "defaults": {},
              "overrides": []
            },
            "gridPos": {
              "h": 5,
              "w": 24,
              "x": 0,
              "y": 0
            },
            "id": 3,
            "options": {
              "code": {
                "language": "plaintext",
                "showLineNumbers": false,
                "showMiniMap": false
              },
              "content": "---\n# GPU Monitoring\n\nThis dashboard displays the usage stats of NVIDIA GPUs like temperature, utilization, memory usage and more generated using [OpenLIT](https://github.com/openlit/openlit).\n\n---",
              "mode": "markdown"
            },
            "pluginVersion": "11.2.0-73830",
            "title": "",
            "transparent": true,
            "type": "text"
          },
          {
            "datasource": {
              "type": "prometheus",
              "uid": "${metricsDatasource}"
            },
            "fieldConfig": {
              "defaults": {
                "color": {
                  "mode": "continuous-BlPu"
                },
                "custom": {
                  "align": "auto",
                  "cellOptions": {
                    "applyToRow": true,
                    "mode": "gradient",
                    "type": "color-background"
                  },
                  "filterable": false,
                  "inspect": true
                },
                "displayName": "GPU",
                "mappings": [],
                "thresholds": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "green",
                      "value": null
                    },
                    {
                      "color": "red",
                      "value": 80
                    }
                  ]
                },
                "unit": "string"
              },
              "overrides": []
            },
            "gridPos": {
              "h": 2,
              "w": 24,
              "x": 0,
              "y": 4
            },
            "id": 2,
            "options": {
              "cellHeight": "lg",
              "footer": {
                "countRows": false,
                "enablePagination": false,
                "fields": "",
                "reducer": [
                  "sum"
                ],
                "show": false
              },
              "showHeader": false
            },
            "pluginVersion": "11.2.0-73830",
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${metricsDatasource}"
                },
                "disableTextWrap": false,
                "editorMode": "code",
                "expr": "gpu_fan_speed{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}",
                "format": "table",
                "fullMetaSearch": false,
                "includeNullMetadata": true,
                "instant": false,
                "legendFormat": "__auto",
                "range": true,
                "refId": "A",
                "useBackend": false
              }
            ],
            "title": "",
            "transformations": [
              {
                "id": "organize",
                "options": {
                  "excludeByName": {
                    "Time": true,
                    "Value": true,
                    "__name__": true,
                    "gen_ai_application_name": true,
                    "gen_ai_environment": true,
                    "gpu_index": true,
                    "gpu_uuid": true,
                    "job": true,
                    "telemetry_sdk_name": true
                  },
                  "includeByName": {},
                  "indexByName": {},
                  "renameByName": {
                    "gpu_name": "Name",
                    "gpu_uuid": "UUID"
                  }
                }
              }
            ],
            "transparent": true,
            "type": "table"
          },
          {
            "datasource": {
              "type": "prometheus",
              "uid": "${metricsDatasource}"
            },
            "description": "This panel displays the LLM Request Rate ",
            "fieldConfig": {
              "defaults": {
                "color": {
                  "fixedColor": "blue",
                  "mode": "continuous-GrYlRd"
                },
                "mappings": [],
                "min": 0,
                "thresholds": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "red",
                      "value": null
                    }
                  ]
                },
                "unit": "percent"
              },
              "overrides": []
            },
            "gridPos": {
              "h": 3,
              "w": 6,
              "x": 0,
              "y": 6
            },
            "id": 4,
            "options": {
              "colorMode": "background",
              "graphMode": "area",
              "justifyMode": "auto",
              "orientation": "auto",
              "percentChangeColorMode": "inverted",
              "reduceOptions": {
                "calcs": [
                  "lastNotNull"
                ],
                "fields": "",
                "values": false
              },
              "showPercentChange": true,
              "textMode": "value_and_name",
              "wideLayout": true
            },
            "pluginVersion": "11.2.0-73830",
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${metricsDatasource}"
                },
                "disableTextWrap": false,
                "editorMode": "code",
                "expr": "avg(gpu_utilization{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
                "fullMetaSearch": false,
                "includeNullMetadata": true,
                "instant": false,
                "legendFormat": "Average GPU Utilization",
                "range": true,
                "refId": "A",
                "useBackend": false
              }
            ],
            "title": "",
            "transparent": true,
            "type": "stat"
          },
          {
            "datasource": {
              "type": "prometheus",
              "uid": "${metricsDatasource}"
            },
            "description": "",
            "fieldConfig": {
              "defaults": {
                "color": {
                  "fixedColor": "blue",
                  "mode": "continuous-GrYlRd"
                },
                "mappings": [],
                "min": 0,
                "thresholds": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "red",
                      "value": null
                    }
                  ]
                },
                "unit": "celsius"
              },
              "overrides": []
            },
            "gridPos": {
              "h": 3,
              "w": 6,
              "x": 6,
              "y": 6
            },
            "id": 5,
            "options": {
              "colorMode": "background",
              "graphMode": "area",
              "justifyMode": "auto",
              "orientation": "auto",
              "percentChangeColorMode": "inverted",
              "reduceOptions": {
                "calcs": [
                  "lastNotNull"
                ],
                "fields": "",
                "values": false
              },
              "showPercentChange": true,
              "textMode": "value_and_name",
              "wideLayout": true
            },
            "pluginVersion": "11.2.0-73830",
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${metricsDatasource}"
                },
                "disableTextWrap": false,
                "editorMode": "code",
                "expr": "avg(gpu_temperature{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
                "fullMetaSearch": false,
                "includeNullMetadata": true,
                "instant": false,
                "legendFormat": "Average GPU Temperature",
                "range": true,
                "refId": "A",
                "useBackend": false
              }
            ],
            "title": "",
            "transparent": true,
            "type": "stat"
          },
          {
            "datasource": {
              "type": "prometheus",
              "uid": "${metricsDatasource}"
            },
            "description": "This panel displays the LLM Request Rate ",
            "fieldConfig": {
              "defaults": {
                "color": {
                  "fixedColor": "blue",
                  "mode": "continuous-GrYlRd"
                },
                "mappings": [],
                "min": 0,
                "thresholds": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "red",
                      "value": null
                    }
                  ]
                },
                "unit": "watt"
              },
              "overrides": []
            },
            "gridPos": {
              "h": 3,
              "w": 6,
              "x": 12,
              "y": 6
            },
            "id": 6,
            "options": {
              "colorMode": "background",
              "graphMode": "area",
              "justifyMode": "auto",
              "orientation": "auto",
              "percentChangeColorMode": "standard",
              "reduceOptions": {
                "calcs": [
                  "lastNotNull"
                ],
                "fields": "",
                "values": false
              },
              "showPercentChange": true,
              "textMode": "value_and_name",
              "wideLayout": true
            },
            "pluginVersion": "11.2.0-73830",
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${metricsDatasource}"
                },
                "disableTextWrap": false,
                "editorMode": "code",
                "expr": "avg(gpu_power_draw{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
                "fullMetaSearch": false,
                "includeNullMetadata": true,
                "instant": false,
                "legendFormat": "Average GPU Power Draw",
                "range": true,
                "refId": "A",
                "useBackend": false
              }
            ],
            "title": "",
            "transparent": true,
            "type": "stat"
          },
          {
            "datasource": {
              "type": "prometheus",
              "uid": "${metricsDatasource}"
            },
            "description": "This panel displays the LLM Request Rate ",
            "fieldConfig": {
              "defaults": {
                "color": {
                  "fixedColor": "blue",
                  "mode": "continuous-GrYlRd"
                },
                "mappings": [],
                "min": 0,
                "thresholds": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "red",
                      "value": null
                    }
                  ]
                },
                "unit": "percent"
              },
              "overrides": []
            },
            "gridPos": {
              "h": 3,
              "w": 6,
              "x": 18,
              "y": 6
            },
            "id": 7,
            "options": {
              "colorMode": "background",
              "graphMode": "area",
              "justifyMode": "auto",
              "orientation": "auto",
              "percentChangeColorMode": "inverted",
              "reduceOptions": {
                "calcs": [
                  "lastNotNull"
                ],
                "fields": "",
                "values": false
              },
              "showPercentChange": true,
              "textMode": "value_and_name",
              "wideLayout": true
            },
            "pluginVersion": "11.2.0-73830",
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${metricsDatasource}"
                },
                "disableTextWrap": false,
                "editorMode": "code",
                "expr": "avg(gpu_memory_used{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}/gpu_memory_available{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",
                "fullMetaSearch": false,
                "includeNullMetadata": true,
                "instant": false,
                "legendFormat": "Average GPU Memory Usage",
                "range": true,
                "refId": "A",
                "useBackend": false
              }
            ],
            "title": "",
            "transparent": true,
            "type": "stat"
          },
          {
            "datasource": {
              "type": "prometheus",
              "uid": "${metricsDatasource}"
            },
            "fieldConfig": {
              "defaults": {
                "color": {
                  "mode": "palette-classic-by-name"
                },
                "custom": {
                  "axisBorderShow": true,
                  "axisCenteredZero": false,
                  "axisColorMode": "text",
                  "axisLabel": "Percentage",
                  "axisPlacement": "auto",
                  "barAlignment": 0,
                  "barWidthFactor": 0.6,
                  "drawStyle": "line",
                  "fillOpacity": 1,
                  "gradientMode": "hue",
                  "hideFrom": {
                    "legend": false,
                    "tooltip": false,
                    "viz": false
                  },
                  "insertNulls": false,
                  "lineInterpolation": "smooth",
                  "lineStyle": {
                    "fill": "solid"
                  },
                  "lineWidth": 1,
                  "pointSize": 5,
                  "scaleDistribution": {
                    "type": "linear"
                  },
                  "showPoints": "auto",
                  "spanNulls": 3600000,
                  "stacking": {
                    "group": "A",
                    "mode": "none"
                  },
                  "thresholdsStyle": {
                    "mode": "off"
                  }
                },
                "mappings": [],
                "max": 100,
                "min": 0,
                "thresholds": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "green",
                      "value": null
                    },
                    {
                      "color": "red",
                      "value": 80
                    }
                  ]
                },
                "unit": "percent"
              },
              "overrides": []
            },
            "gridPos": {
              "h": 9,
              "w": 12,
              "x": 0,
              "y": 9
            },
            "id": 8,
            "options": {
              "legend": {
                "calcs": [
                  "lastNotNull",
                  "mean",
                  "max"
                ],
                "displayMode": "table",
                "placement": "bottom",
                "showLegend": true
              },
              "tooltip": {
                "mode": "single",
                "sort": "none"
              }
            },
            "pluginVersion": "11.2.0-73830",
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${metricsDatasource}"
                },
                "disableTextWrap": false,
                "editorMode": "code",
                "expr": "gpu_utilization{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}",
                "fullMetaSearch": false,
                "includeNullMetadata": true,
                "instant": false,
                "legendFormat": "{{gpu_uuid}}-{{gpu_name}}",
                "range": true,
                "refId": "A",
                "useBackend": false
              },
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${metricsDatasource}"
                },
                "disableTextWrap": false,
                "editorMode": "builder",
                "expr": "gpu_dec_utilization{telemetry_sdk_name=\"openlit\"}",
                "fullMetaSearch": false,
                "hide": false,
                "includeNullMetadata": true,
                "instant": false,
                "legendFormat": "{{gpu_uuid}}-{{gpu_name}}",
                "range": true,
                "refId": "B",
                "useBackend": false
              },
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${metricsDatasource}"
                },
                "disableTextWrap": false,
                "editorMode": "builder",
                "expr": "gpu_enc_utilization{telemetry_sdk_name=\"openlit\"}",
                "fullMetaSearch": false,
                "hide": false,
                "includeNullMetadata": true,
                "instant": false,
                "legendFormat": "{{gpu_uuid}}-{{gpu_name}}",
                "range": true,
                "refId": "C",
                "useBackend": false
              }
            ],
            "title": "GPU Utilization",
            "type": "timeseries"
          },
          {
            "datasource": {
              "type": "prometheus",
              "uid": "${metricsDatasource}"
            },
            "fieldConfig": {
              "defaults": {
                "color": {
                  "mode": "palette-classic-by-name"
                },
                "custom": {
                  "axisBorderShow": true,
                  "axisCenteredZero": false,
                  "axisColorMode": "text",
                  "axisLabel": "Temperature (°C)",
                  "axisPlacement": "auto",
                  "barAlignment": 0,
                  "barWidthFactor": 0.6,
                  "drawStyle": "line",
                  "fillOpacity": 1,
                  "gradientMode": "opacity",
                  "hideFrom": {
                    "legend": false,
                    "tooltip": false,
                    "viz": false
                  },
                  "insertNulls": false,
                  "lineInterpolation": "smooth",
                  "lineStyle": {
                    "fill": "solid"
                  },
                  "lineWidth": 1,
                  "pointSize": 5,
                  "scaleDistribution": {
                    "type": "linear"
                  },
                  "showPoints": "auto",
                  "spanNulls": true,
                  "stacking": {
                    "group": "A",
                    "mode": "none"
                  },
                  "thresholdsStyle": {
                    "mode": "off"
                  }
                },
                "mappings": [],
                "max": 100,
                "min": 0,
                "thresholds": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "green",
                      "value": null
                    },
                    {
                      "color": "red",
                      "value": 80
                    }
                  ]
                },
                "unit": "celsius"
              },
              "overrides": []
            },
            "gridPos": {
              "h": 9,
              "w": 12,
              "x": 12,
              "y": 9
            },
            "id": 9,
            "options": {
              "legend": {
                "calcs": [
                  "lastNotNull",
                  "mean",
                  "max"
                ],
                "displayMode": "table",
                "placement": "bottom",
                "showLegend": true
              },
              "tooltip": {
                "mode": "single",
                "sort": "none"
              }
            },
            "pluginVersion": "11.2.0-73830",
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${metricsDatasource}"
                },
                "disableTextWrap": false,
                "editorMode": "code",
                "expr": "gpu_temperature{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}",
                "fullMetaSearch": false,
                "includeNullMetadata": true,
                "instant": false,
                "legendFormat": "{{gpu_uuid}}-{{gpu_name}}",
                "range": true,
                "refId": "A",
                "useBackend": false
              }
            ],
            "title": "GPU Temperature",
            "type": "timeseries"
          },
          {
            "datasource": {
              "type": "prometheus",
              "uid": "${metricsDatasource}"
            },
            "fieldConfig": {
              "defaults": {
                "color": {
                  "mode": "palette-classic-by-name"
                },
                "custom": {
                  "axisBorderShow": true,
                  "axisCenteredZero": false,
                  "axisColorMode": "text",
                  "axisLabel": "",
                  "axisPlacement": "auto",
                  "barAlignment": 0,
                  "barWidthFactor": 0.6,
                  "drawStyle": "bars",
                  "fillOpacity": 50,
                  "gradientMode": "hue",
                  "hideFrom": {
                    "legend": false,
                    "tooltip": false,
                    "viz": false
                  },
                  "insertNulls": false,
                  "lineInterpolation": "linear",
                  "lineStyle": {
                    "fill": "solid"
                  },
                  "lineWidth": 1,
                  "pointSize": 3,
                  "scaleDistribution": {
                    "type": "linear"
                  },
                  "showPoints": "always",
                  "spanNulls": 3600000,
                  "stacking": {
                    "group": "A",
                    "mode": "percent"
                  },
                  "thresholdsStyle": {
                    "mode": "off"
                  }
                },
                "mappings": [],
                "min": 0,
                "thresholds": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "green",
                      "value": null
                    },
                    {
                      "color": "red",
                      "value": 80
                    }
                  ]
                },
                "unit": "decmbytes"
              },
              "overrides": []
            },
            "gridPos": {
              "h": 9,
              "w": 12,
              "x": 0,
              "y": 17
            },
            "id": 10,
            "options": {
              "legend": {
                "calcs": [
                  "lastNotNull",
                  "mean",
                  "max"
                ],
                "displayMode": "table",
                "placement": "bottom",
                "showLegend": true
              },
              "tooltip": {
                "mode": "single",
                "sort": "none"
              }
            },
            "pluginVersion": "11.2.0-73830",
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${metricsDatasource}"
                },
                "disableTextWrap": false,
                "editorMode": "code",
                "expr": "gpu_memory_used{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}",
                "fullMetaSearch": false,
                "includeNullMetadata": true,
                "instant": false,
                "legendFormat": "Memory Used",
                "range": true,
                "refId": "A",
                "useBackend": false
              },
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${metricsDatasource}"
                },
                "disableTextWrap": false,
                "editorMode": "builder",
                "expr": "gpu_memory_free{telemetry_sdk_name=\"openlit\"}",
                "fullMetaSearch": false,
                "hide": false,
                "includeNullMetadata": true,
                "instant": false,
                "legendFormat": "Memory Free",
                "range": true,
                "refId": "C",
                "useBackend": false
              },
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${metricsDatasource}"
                },
                "disableTextWrap": false,
                "editorMode": "builder",
                "expr": "gpu_memory_total{telemetry_sdk_name=\"openlit\"}",
                "fullMetaSearch": false,
                "hide": false,
                "includeNullMetadata": true,
                "instant": false,
                "legendFormat": "Memory Total",
                "range": true,
                "refId": "B",
                "useBackend": false
              }
            ],
            "title": "GPU Memory Usage",
            "type": "timeseries"
          },
          {
            "datasource": {
              "type": "prometheus",
              "uid": "${metricsDatasource}"
            },
            "fieldConfig": {
              "defaults": {
                "color": {
                  "mode": "palette-classic-by-name"
                },
                "custom": {
                  "axisBorderShow": true,
                  "axisCenteredZero": false,
                  "axisColorMode": "text",
                  "axisLabel": "Fan Speed (0-100)",
                  "axisPlacement": "auto",
                  "barAlignment": 0,
                  "barWidthFactor": 0.6,
                  "drawStyle": "line",
                  "fillOpacity": 1,
                  "gradientMode": "opacity",
                  "hideFrom": {
                    "legend": false,
                    "tooltip": false,
                    "viz": false
                  },
                  "insertNulls": false,
                  "lineInterpolation": "smooth",
                  "lineStyle": {
                    "fill": "solid"
                  },
                  "lineWidth": 1,
                  "pointSize": 5,
                  "scaleDistribution": {
                    "type": "linear"
                  },
                  "showPoints": "auto",
                  "spanNulls": true,
                  "stacking": {
                    "group": "A",
                    "mode": "none"
                  },
                  "thresholdsStyle": {
                    "mode": "off"
                  }
                },
                "mappings": [],
                "max": 100,
                "min": 0,
                "thresholds": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "green",
                      "value": null
                    },
                    {
                      "color": "red",
                      "value": 80
                    }
                  ]
                },
                "unit": "none"
              },
              "overrides": []
            },
            "gridPos": {
              "h": 9,
              "w": 12,
              "x": 12,
              "y": 17
            },
            "id": 11,
            "options": {
              "legend": {
                "calcs": [
                  "lastNotNull",
                  "mean",
                  "max"
                ],
                "displayMode": "table",
                "placement": "bottom",
                "showLegend": true
              },
              "tooltip": {
                "mode": "single",
                "sort": "none"
              }
            },
            "pluginVersion": "11.2.0-73830",
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${metricsDatasource}"
                },
                "disableTextWrap": false,
                "editorMode": "code",
                "expr": "gpu_fan_speed{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}",
                "fullMetaSearch": false,
                "includeNullMetadata": true,
                "instant": false,
                "legendFormat": "{{gpu_uuid}}-{{gpu_name}}",
                "range": true,
                "refId": "A",
                "useBackend": false
              }
            ],
            "title": "GPU Fan Speed",
            "type": "timeseries"
          },
          {
            "datasource": {
              "type": "prometheus",
              "uid": "${metricsDatasource}"
            },
            "fieldConfig": {
              "defaults": {
                "color": {
                  "mode": "palette-classic-by-name"
                },
                "custom": {
                  "axisBorderShow": true,
                  "axisCenteredZero": false,
                  "axisColorMode": "text",
                  "axisLabel": "",
                  "axisPlacement": "auto",
                  "barAlignment": 0,
                  "barWidthFactor": 0.6,
                  "drawStyle": "bars",
                  "fillOpacity": 50,
                  "gradientMode": "hue",
                  "hideFrom": {
                    "legend": false,
                    "tooltip": false,
                    "viz": false
                  },
                  "insertNulls": false,
                  "lineInterpolation": "linear",
                  "lineStyle": {
                    "fill": "solid"
                  },
                  "lineWidth": 1,
                  "pointSize": 3,
                  "scaleDistribution": {
                    "type": "linear"
                  },
                  "showPoints": "always",
                  "spanNulls": 3600000,
                  "stacking": {
                    "group": "A",
                    "mode": "percent"
                  },
                  "thresholdsStyle": {
                    "mode": "off"
                  }
                },
                "mappings": [],
                "min": 0,
                "thresholds": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "green",
                      "value": null
                    },
                    {
                      "color": "red",
                      "value": 80
                    }
                  ]
                },
                "unit": "watt"
              },
              "overrides": []
            },
            "gridPos": {
              "h": 7,
              "w": 24,
              "x": 0,
              "y": 25
            },
            "id": 12,
            "options": {
              "legend": {
                "calcs": [
                  "lastNotNull",
                  "mean",
                  "max"
                ],
                "displayMode": "table",
                "placement": "bottom",
                "showLegend": true
              },
              "tooltip": {
                "mode": "single",
                "sort": "none"
              }
            },
            "pluginVersion": "11.2.0-73830",
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${metricsDatasource}"
                },
                "disableTextWrap": false,
                "editorMode": "code",
                "expr": "gpu_power_draw{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}",
                "fullMetaSearch": false,
                "includeNullMetadata": true,
                "instant": false,
                "legendFormat": "Power Draw",
                "range": true,
                "refId": "A",
                "useBackend": false
              },
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${metricsDatasource}"
                },
                "disableTextWrap": false,
                "editorMode": "builder",
                "expr": "gpu_power_limit{telemetry_sdk_name=\"openlit\"}",
                "fullMetaSearch": false,
                "hide": false,
                "includeNullMetadata": true,
                "instant": false,
                "legendFormat": "Power Limit",
                "range": true,
                "refId": "C",
                "useBackend": false
              }
            ],
            "title": "GPU Memory Usage",
            "type": "timeseries"
          }
        ],
        "preload": false,
        "schemaVersion": 39,
        "tags": [],
        "templating": {
          "list": [
            {
              "description": "Prometheus datasource holding GPU Monitoring data",
              "label": "Metrics Datasource",
              "name": "metricsDatasource",
              "options": [],
              "query": "prometheus",
              "refresh": 1,
              "regex": "",
              "type": "datasource"
            },
            {
              "current": {
                "text": [
                  "chatbot"
                ],
                "value": [
                  "chatbot"
                ]
              },
              "definition": "label_values(gen_ai_application_name)",
              "includeAll": true,
              "label": "Application",
              "multi": true,
              "name": "application",
              "options": [],
              "query": {
                "qryType": 1,
                "query": "label_values(gen_ai_application_name)",
                "refId": "PrometheusVariableQueryEditor-VariableQuery"
              },
              "refresh": 1,
              "regex": "",
              "type": "query"
            },
            {
              "current": {
                "text": [
                  "staging"
                ],
                "value": [
                  "staging"
                ]
              },
              "definition": "label_values(gen_ai_environment)",
              "includeAll": true,
              "label": "Environment",
              "multi": true,
              "name": "environment",
              "options": [],
              "query": {
                "qryType": 1,
                "query": "label_values(gen_ai_environment)",
                "refId": "PrometheusVariableQueryEditor-VariableQuery"
              },
              "refresh": 1,
              "regex": "",
              "type": "query"
            }
          ]
        },
        "time": {
          "from": "2024-08-07T05:20:57.710Z",
          "to": "2024-08-07T05:23:27.562Z"
        },
        "timepicker": {},
        "timezone": "browser",
        "title": "GPU Monitoring",
        "uid": "adu2uyzl5ih34e",
        "version": 6,
        "weekStart": ""
      }
      ```
  </Accordion>

  </Step>
</Steps>


---

<CardGroup cols={2}>
<Card title="Integrations" href="/latest/integrations/introduction" icon='circle-nodes'>
Start Monitoring your LLM Application 
</Card>
</CardGroup>
