Monitoring VyOS with Prometheus and Grafana

Grafana allows you to query, visualize, alert on and understand your metrics no matter where they are stored. Prometheus is an open-source monitoring system for which Grafana provides out-of-the-box support. This topic walks you through the steps to create a dashboard in Grafana to display system metrics for a server monitored by Prometheus.

On VyOS:

Using node_exporter (with some scripts/text collector to also monitor container and ip addresses) and blackbox_exporter to export metrics.

The scripts need to be put in /config/container/node_exporter/textfile_collector (please create the directory for scripts)

set system task-scheduler task ip-export interval 1h
set system task-scheduler task ip-export executable path '/config/container/node_exporter/textfile_collector:
/export-ip-metrics.py'
set system task-scheduler task podman-export executable path '/config/container/node_exporter/textfile_collector:
/export-podman-metrics.py'
set system task-scheduler task podman-export interval '1m'

To set up node_exporter container:

Run this op-command to pull the new image:

add container image quay.io/prometheus/node-exporter:v1.8.1

Set these commands to configure container attributes:

set container name node-exporter image quay.io/prometheus/node-exporter:v1.8.1
set container name node-exporter argument '--path.procfs /host/proc --path.sysfs /host/sys --path.rootfs /host/rootfs --collector.textfile.directory=/var/lib/node_exporter/textfile_collector --collector.filesystem.ignored-mount-points ^(sys|proc|dev|etc|host)'
set container name node-exporter restart on-failure
set container name node-exporter memory 256
set container name node-exporter shared-memory 64
set container name node-exporter allow-host-networks
set container name node-exporter volume osrelease source '/etc/os-release'
set container name node-exporter volume osrelease destination '/etc/os-release'
set container name node-exporter volume osrelease mode ro
set container name node-exporter volume procfs source '/proc'
set container name node-exporter volume procfs destination '/host/proc'
set container name node-exporter volume procfs mode ro
set container name node-exporter volume rootfs source '/'
set container name node-exporter volume rootfs destination '/host/rootfs'
set container name node-exporter volume rootfs mode ro
set container name node-exporter volume sysfs source '/sys'
set container name node-exporter volume sysfs destination '/host/sys'
set container name node-exporter volume sysfs mode ro
set container name node-exporter volume txtfile source '/config/container/node_exporter/textfile_collector'
set container name node-exporter volume txtfile destination '/var/lib/node_exporter/textfile_collector'
set container name node-exporter volume txtfile mode ro

Blackbox exporter needs a simple configuration to be put in /config/container/blackbox_exporter/config:

To set up blackbox_exporter container, first pull the image and then add the parameters:

run add container image quay.io/prometheus/blackbox-exporter:v0.25.0
set container name blackbox-exporter image quay.io/prometheus/blackbox-exporter:v0.25.0
set container name blackbox-exporter argument '--config.file=/config/blackbox.yml'
set container name blackbox-exporter restart on-failure
set container name blackbox-exporter allow-host-networks
set container name blackbox-exporter memory 128
set container name blackbox-exporter shared-memory 32
set container name blackbox-exporter capability net-raw
set container name blackbox-exporter volume config source '/config/container/blackbox_exporter/config'
set container name blackbox-exporter volume config destination '/config'

On Prometheus

Configure scraping:

- job_name: vyos-ne
  static_configs:
    - targets: [ 'vyos.example.local:9100' ]
  relabel_configs:
    - source_labels: [__address__]
      target_label: instance
      regex: '([^:]+)(:[0-9]+)?'
      replacement: '${1}'


- job_name: vyos-be
  metrics_path: /probe
  scrape_interval: 60s
  #params:
  #  module: [dns_ipv4, dns_ipv6]
  static_configs:
    - targets:
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
  relabel_configs:
    - source_labels: [__address__]
      regex: (.*)\@(.*)
      target_label: __param_module
      replacement: ${1}
    - source_labels: [__address__]
      regex: (.*)\@(.*)
      target_label: __param_target
      replacement: ${2}
    - source_labels: [__address__]
      regex: (.*)\@(.*)
      target_label: module
      replacement: ${1}
    - source_labels: [__address__]
      regex: (.*)\@(.*)
      target_label: probe
      replacement: ${2}
    - target_label: __address__
      replacement: vyos.example.local:9115
    - target_label: instance
      replacement: 'vyos.example.local'

(ns.example.net 1 being your upstream dns server of choice, ns4/6 being your own dns server like pihole, example.net some host you use to check general internet connectivity)

On Grafana

With everything in place, we just need to put in the dashboard.

You will need to tweak a few things like the dashboard variables to identity wan/lan interfaces, blackbox exporter labels etc.

Dashboard: