Monitoring — DevOps Notes

Prometheus scrapes metrics from your servers and stores them as time series. Grafana visualizes them. Together they give you a proper observability stack without paying for a SaaS product.

Installing node_exporter

node_exporter exposes system metrics — CPU, memory, disk, network — over HTTP so Prometheus can scrape them. Install it on every server you want to monitor:

wget https://github.com/prometheus/node_exporter/releases/download/v1.8.1/node_exporter-1.8.1.linux-amd64.tar.gz
tar xf node_exporter-1.8.1.linux-amd64.tar.gz
mv node_exporter-1.8.1.linux-amd64/node_exporter /usr/local/bin/

Create a systemd service:

[Unit]
Description=node_exporter

[Service]
ExecStart=/usr/local/bin/node_exporter
Restart=always

[Install]
WantedBy=multi-user.target

Enable and start: systemctl enable --now node_exporter. Metrics are available at http://localhost:9100/metrics.

Prometheus scrape config

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'servers'
    static_configs:
      - targets:
          - 'server1:9100'
          - 'server2:9100'
          - 'server3:9100'

First Grafana dashboard

After connecting Grafana to Prometheus as a data source, import dashboard ID 1860 — the Node Exporter Full dashboard. It covers everything: CPU usage, memory, disk I/O, network traffic, load average. No manual panel creation needed.

Run Prometheus and Grafana in Docker to avoid dependency conflicts. A simple compose file gets both running in under five minutes.

Prometheus evaluates alert rules and sends firing alerts to Alertmanager. Alertmanager decides where to send them. Getting alerts to Telegram takes about 15 minutes to set up.

Create a Telegram bot

Message @BotFather on Telegram, run /newbot, and save the token it gives you. Then message your new bot and get your chat ID from https://api.telegram.org/bot<TOKEN>/getUpdates.

Alertmanager config

route:
  receiver: telegram

receivers:
  - name: telegram
    telegram_configs:
      - bot_token: 'YOUR_BOT_TOKEN'
        chat_id: YOUR_CHAT_ID
        message: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'

A basic alert rule

groups:
  - name: servers
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 2m
        annotations:
          summary: "Server {{ $labels.instance }} is down"

When a server stops responding to scrapes for 2 minutes, you get a Telegram message. Simple and reliable.

Monitoring & alerting