The Problem
You’ve got 3 environments: dev, staging, prod. Each one needs the same dashboard. So you duplicate the dashboard 3 times. Then you add a metric to one. You remember to update the other two. Mostly.
Six months later, the prod dashboard is out of sync with dev. You don’t notice until something breaks.
Grafana variables solve this. One dashboard. Multiple environments. One source of truth.
What Are Dashboard Variables?
Variables are placeholders in your dashboard that change based on user input or data source values. Think of them as dashboard-level parameters.
Instead of hardcoding job="api" in every query, you use $job. User selects “api”, “worker”, or “scheduler” from a dropdown. The dashboard updates everywhere.
Setting Up Variables
Go to Dashboard Settings → Variables → New Variable.
Here’s a practical example: environment selector.
Name: envType: ConstantValue: prodWait, that’s boring. Let’s do it better:
Name: envType: CustomCustom Options: dev, staging, prodSelected Value: prodRefresh: NeverNow your dashboard has a dropdown with 3 options. When you pick one, $env updates everywhere.
Real Example: Multi-Environment Prometheus Dashboard
You’re monitoring an API across dev/staging/prod. Your Prometheus jobs are labeled:
job="api-dev"job="api-staging"job="api-prod"
Step 1: Create Environment Variable
Name: environmentType: CustomOptions: dev, staging, prodDefault: prodMulti-select: OffStep 2: Create Job Variable (Conditional)
This one pulls data from Prometheus based on the environment:
Name: jobType: QueryDatasource: PrometheusQuery: label_values(up{environment="$environment"}, job)Refresh: On Dashboard LoadThis query says: “Give me all distinct job names where environment matches the selected value.”
Step 3: Use in Panels
In a panel’s Prometheus query:
rate(http_requests_total{job="$job"}[5m])Or more sophisticated:
rate(http_requests_total{environment="$environment", job=~"$job"}[5m])Now when a user picks environment=staging and job=api, the dashboard shows staging API metrics. Pick environment=prod, and it shows prod. Same dashboard.
Advanced: Regex Filters
Variables support regex patterns. Suppose you have hosts: host-1, host-2, api-1, api-2. Create a variable:
Name: host_filterType: CustomOptions: host-*, api-*, *Default: host-*Then in your query:
up{instance=~"$host_filter"}Pick api-* to see only API hosts. Pick * to see everything.
Multi-Select Variables
Allow users to pick multiple values:
Name: status_codesType: CustomOptions: 200, 404, 500, 503Multi-select: OnSelected Value: [200, 500, 503]In your Prometheus query, use =~ (regex match):
rate(http_requests_total{status=~"$status_codes"}[5m])Grafana converts [200, 500, 503] into 200|500|503 (regex OR). Clean.
Templating in Titles
Variables aren’t just for queries. Use them in panel titles:
HTTP Requests - $environment ($job)When you switch environments, the title updates. Helps prevent confusion.
JSON Variable Example
Here’s what a full dashboard JSON looks like with variables:
{ "dashboard": { "title": "API Monitoring", "templating": { "list": [ { "name": "environment", "type": "custom", "options": { "values": ["dev", "staging", "prod"] }, "current": { "value": "prod" } }, { "name": "job", "type": "query", "datasource": "Prometheus", "query": "label_values(up{environment=\"$environment\"}, job)", "refresh": 1, "current": { "value": "api" } } ] } }}If you’re version-controlling dashboards (you should be), commit this JSON to git. When you export/import, variables come along.
Pro Tips
Populate Variables from Loki Labels
If you’re using Loki for logs:
Name: serviceType: QueryDatasource: LokiQuery: label_values(job)Now your log filtering matches your metrics. Consistency across observability.
Use ad-hoc Filters
Toggle Ad-hoc filters on. Users can click labels in graphs to filter. Grafana adds the filters as query parameters in the URL.
https://grafana.local/d/xyz?var-env=prod&var-job=apiShareable links. Reproducible views.
Chain Variables
A variable’s query can reference another variable:
Name: podQuery: label_values(kube_pod_labels{namespace="$namespace"}, pod)User picks namespace, pod list updates. Powerful.
Common Gotchas
Variable not updating? Check the query. If it has a syntax error, it silently fails. Check the Prometheus console directly.
Multi-select not working in PromQL? Use =~ for regex match, not =. Grafana converts [a, b] into a|b (regex).
Want to hide a variable from users? Use a Constant type. It won’t show in the UI but still updates queries.
The Payoff
One dashboard. Any environment. Any subset of services. Users can troubleshoot without bugging you for dashboard links.
It’s 30 minutes of setup. Saves hours of maintenance.
Do it.
Chaining Variables
Variables can depend on each other. Select a datacenter and the host list updates automatically:
- Create
$datacentervariable (query:label_values(up, datacenter)) - Create
$hostvariable (query:label_values(up{datacenter="$datacenter"}, instance))
Now $host only shows hosts in the selected $datacenter. No more filtering through 200 hosts when you only care about one region.
This is the feature that makes dashboards actually useful instead of overwhelming.