> This is a troubleshooting log template. Facts first, evidence second, conclusions last. ## TL;DR - **Symptoms**: The DNS service becomes unresponsive; the whole machine stutters/freezes; sometimes it looks like storage I/O issues or a “disk drop”. - **Pattern**: Random, usually after **1+ day** of continuous running. - **Recovery**: **Power off and let it cool down** for a while, then it works again. - **Current conclusion (probabilistic)**: \_\_\_\_ (e.g., thermals / power / storage-controller reset). ## Impact - **Direct impact**: \_\_\_\_ (e.g., LAN-wide DNS timeouts). - **Indirect impact**: \_\_\_\_ (services that depend on name resolution). - **Time window**: \_\_\_\_ (start time / duration / auto-recovery?). ## Environment (reproducible) ### Hardware - **CPU**: Intel® N100 (4C/4T, 6MB cache, up to 3.4GHz) - **GPU**: Intel® UHD Graphics - **RAM**: 8GB LPDDR5 (single channel, onboard, 4800MHz) - **Storage**: UFS 2.1 256G - **Wireless**: Intel AX200/201 (Wi‑Fi 6 / Bluetooth 5.2) - **Ethernet**: 2.5G RJ45 ×1 (PoE IEEE 802.3at) - **Ports**: USB 3.2 Gen2 Type‑A ×2; USB 3.2 Gen2 Type‑C ×1 (Alt DP/PD); HDMI ×1 - **Power**: 65W USB‑C Power Delivery adapter ### Software - **Actual OS**: Linux (current kernel: `6.8.12-5-pve`; factory OS: Windows 11 Home) - **Kernel (if Linux)**: `6.8.12-5-pve` - **Bootloader (GRUB)**: `grub2 2.06-13+pmx2` - **DNS stack**: \_\_\_\_ (dnsmasq / unbound / AdGuard Home / Pi‑hole / other) - **Deployment**: \_\_\_\_ (systemd / Docker / other) - **Other services**: \_\_\_\_ - **Logging policy**: \_\_\_\_ (write frequency / rotation / same disk?). ## Problem statement (facts only) - **First seen**: \_\_\_\_ - **Frequency**: about every \_\_\_\_ hours/days (random). - **Observable symptoms during failure**: - **DNS**: \_\_\_\_ (timeouts / refusal / high latency) - **System**: \_\_\_\_ (SSH/RDP reachable? CPU/memory spike?) - **Storage**: \_\_\_\_ (mount disappears / read-only remount / I/O error / device reset) - **Network**: \_\_\_\_ (packet loss? NIC reset?) - **Recovery**: \_\_\_\_ (power off for \_\_\_\_ minutes; does reboot also work: \_\_\_\_) - **Load correlation**: \_\_\_\_ (QPS / logging / other tasks). ## Immediate mitigation (timeline) - **T+0**: \_\_\_\_ (e.g., switch upstream DNS) - **T+5m**: \_\_\_\_ (stop DNS or high-IO workloads) - **T+10m**: \_\_\_\_ (collect logs and metrics snapshots) - **Result**: \_\_\_\_ ## Evidence model (align everything by timestamp) - **One timeline**: failure start time, temperature peak, I/O errors, service timeouts. - **One core question**: is DNS dead, or is the storage/system dead and DNS is just the first visible symptom? ## Hypotheses (prioritized) - **H1: Thermals / overheating → throttling or protection → instability** - Evidence: \_\_\_\_ - **H2: Power instability (PD/PoE/adapter) → controller reset** - Evidence: \_\_\_\_ - **H3: Storage media/controller issue → I/O stall or device drop** - Evidence: \_\_\_\_ - **H4: Software resource exhaustion (FD/memory/log amplification/IO saturation)** - Evidence: \_\_\_\_ - **H5: Kernel/driver bug (power management, device PM policy)** - Evidence: \_\_\_\_ ## Data to capture (during failure) - **Temps/fans**: \_\_\_\_ (`sensors`, `/sys/class/thermal/`, Windows tools) - **System logs**: \_\_\_\_ (`journalctl`, Windows Event Viewer) - **Storage logs**: \_\_\_\_ (I/O error, timeout, reset, read-only remount) - **Resource metrics**: \_\_\_\_ (CPU/load/iowait/memory/disk throughput) - **DNS metrics**: \_\_\_\_ (QPS, cache hit ratio, upstream latency, error rate) - **Client checks**: \_\_\_\_ (`dig/nslookup` outputs) ## Key evidence (3–5 snippets) ```text [time] ____ ``` ```text [time] ____ ``` ```text [time] ____ ``` ## Investigation (Attempt → Observation → Conclusion) ### Attempt A: \_\_\_\_ - Change: \_\_\_\_ - Observation: \_\_\_\_ - Conclusion: \_\_\_\_ ### Attempt B: \_\_\_\_ - Change: \_\_\_\_ - Observation: \_\_\_\_ - Conclusion: \_\_\_\_ ### Attempt C: \_\_\_\_ - Change: \_\_\_\_ - Observation: \_\_\_\_ - Conclusion: \_\_\_\_ ## Final conclusion (probability, not absolute claims) - **Most likely root cause**: \_\_\_\_ - **Secondary factors**: \_\_\_\_ - **Ruled out**: \_\_\_\_ (with evidence) ## Fix plan (short-term vs long-term) ### Short-term mitigation - \_\_\_\_ (e.g., improve cooling) - \_\_\_\_ (e.g., cap power / disable aggressive power saving) - \_\_\_\_ (e.g., reduce disk writes / configure log rotation) ### Long-term fix - \_\_\_\_ (e.g., replace power adapter / replace storage / BIOS update) - \_\_\_\_ (e.g., airflow redesign / thermal pads) - \_\_\_\_ (e.g., kernel/OS change) ### Validation - Uptime: \_\_\_\_ days without recurrence - Temperature: CPU ≤ \_\_\_\_°C; storage ≤ \_\_\_\_°C - Logs: no I/O errors / resets / read-only remount ## Retrospective - Biggest learning: \_\_\_\_ - What I will do earlier next time: \_\_\_\_ (monitoring, alerting, auto log capture) - Open questions: \_\_\_\_ ## Attempt plan: Monitor 192.168.22.18 (Minisforum) via a monitoring LXC (192.168.22.19) on PVE 192.168.22.12 > Goal: turn “it feels like overheating / disk drop / DNS died” into an evidence-aligned timeline: **temps + kernel/storage errors + real DNS query success/latency + host reachability**. > > Note: **never put passwords into configs or the repo**. Any credentials should live in root-only local files on the monitoring host. ### Extra pitfall: port `22` is reachable, but `ssh` still hangs / drops (more common across subnets) This one is misleading: `nc` says `22/tcp open`, so people assume “network is fine”. But `ssh` can still stall during handshake and eventually time out (or die with `Broken pipe`). #### Topology (what happened in this incident) - **Client subnet**: `192.168.11.0/24` - **Target host**: `192.168.22.12` (PVE) - **Same-subnet control point**: `192.168.22.18` (SSH from here to `.12` is stable) - **Correlated signal**: DNS server (e.g. `192.168.22.53:53`) experienced timeouts — reverse lookups can amplify SSH handshake latency. #### Symptoms (typical) - `nc -vz -G 2 192.168.22.12 22` reports **succeeded** (TCP handshake is OK) - `ssh root@192.168.22.12` hangs and later shows: - `ssh_dispatch_run_fatal: Connection to 192.168.22.12 port 22: Operation timed out` - or `Read from remote host 192.168.22.12: Operation timed out` / `client_loop: send disconnect: Broken pipe` #### Quick triage (split “port is open” into smaller truths) 1. **Check whether the SSH banner is returned immediately** (protocol only, no auth): ```bash nc -v 192.168.22.12 22 ``` You should quickly see `SSH-2.0-OpenSSH_...`. If the TCP connect succeeds but the banner is **slow/missing**, the issue is likely in **sshd/system path** (DNS reverse lookup stalls, high load, policy/ACL causing unstable data path), not “port blocked”. 2. **Capture where SSH stalls** (useful evidence): ```bash ssh -vvv -o ConnectTimeout=5 -o ServerAliveInterval=5 -o ServerAliveCountMax=1 root@192.168.22.12 ``` #### Minimal fix (eliminate DNS/GSSAPI special cases first) If you can SSH from a same-subnet host (e.g. `.22.18`) into `.22.12`, prefer the **server-side** fix: 1. On `.22.12`, edit `/etc/ssh/sshd_config` and make sure you have: - `UseDNS no` - `GSSAPIAuthentication no` 2. Validate and restart: ```bash sshd -t && systemctl restart ssh systemctl status ssh --no-pager ``` > Practical rationale: when DNS/reverse lookup is flaky, `UseDNS yes` can block parts of the handshake/audit path. Disabling it removes an entire class of “it depends on DNS today” failures. #### If it is still flaky: continue with the “cross-subnet triad” - **Return path routing** (the target must know how to route back to `192.168.11.0/24`): ```bash ip route get 192.168.11.12 ``` - **Firewall/ACL/state tracking**: allow not only `22/tcp` inbound but also the correct return direction and conntrack behavior (especially with gateways/NAT/policy routing). - **MTU/fragmentation**: tunnels/PPPoE/VLAN paths can yield “SYN works but data is unstable”; validate with DF-sized pings on Linux and temporarily lower MSS/MTU to confirm. ### Architecture (minimal, but sufficient) - **192.168.22.19 (monitoring LXC on PVE 192.168.22.12)**: Prometheus + Grafana + blackbox_exporter (optional Pushgateway) - **192.168.22.18 (target host)**: node_exporter (host metrics) + (optional) smartctl_exporter + custom textfile metrics (kernel/I/O error counter) Data model: - **Time series (Prometheus)**: scrape every 15s, keep at least 48h around failures. - **Active probing (blackbox_exporter)**: probe `.18/.54` from `.19` to separate “host down” from “service down”. - **Alerting**: pin the incident timestamp immediately. --- ### Attempt D: Bring up the monitoring stack (Prometheus + Grafana + blackbox_exporter) #### D0. Pre-checks (network & ports) From `192.168.22.19`, verify reachability to `192.168.22.18`: ```bash ping 192.168.22.18 Test-NetConnection 192.168.22.18 -Port 22 ``` Ports used (defaults): - `.18:9100` node_exporter - `.18:9633` smartctl_exporter (optional) - `.19:9090` Prometheus - `.19:3000` Grafana - `.19:9115` blackbox_exporter - `.19:9091` Pushgateway (optional) #### D1. Deploy node_exporter on 192.168.22.18 (systemd, stable) Run on `.18`: ```bash export VER="1.7.0" curl -fsSL -o /tmp/node_exporter.tar.gz "https://github.com/prometheus/node_exporter/releases/download/v${VER}/node_exporter-${VER}.linux-amd64.tar.gz" tar -C /tmp -xzf /tmp/node_exporter.tar.gz install -m 0755 /tmp/node_exporter-${VER}.linux-amd64/node_exporter /usr/local/bin/node_exporter useradd --system --no-create-home --shell /usr/sbin/nologin nodeexp || true mkdir -p /var/lib/node_exporter/textfile_collector chown -R nodeexp:nodeexp /var/lib/node_exporter cat >/etc/systemd/system/node_exporter.service <<'EOF' [Unit] Description=Prometheus Node Exporter After=network-online.target Wants=network-online.target [Service] User=nodeexp Group=nodeexp ExecStart=/usr/local/bin/node_exporter \ --web.listen-address=:9100 \ --collector.textfile.directory=/var/lib/node_exporter/textfile_collector Restart=on-failure RestartSec=2 [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable --now node_exporter systemctl status node_exporter --no-pager ``` Verify (from `.12` or any LAN host): ```bash curl -fsS http://192.168.22.18:9100/metrics | head ``` #### D2. (Recommended) Add a kernel/storage error counter as a custom metric The keywords that matter: `I/O error|timeout|reset|remount read-only|EXT4-fs error|BTRFS error|blk_update_request` Use node_exporter **textfile collector** and a timer: ```bash cat >/usr/local/bin/collect_kernel_error_metrics.sh <<'EOF' #!/usr/bin/env bash set -euo pipefail OUT="/var/lib/node_exporter/textfile_collector/kernel_errors.prom" TMP="$(mktemp)" COUNT="$(journalctl -k --since "10 min ago" --no-pager 2>/dev/null | \ grep -Eic 'I/O error|timeout|reset|remount read-only|EXT4-fs error|BTRFS error|blk_update_request|nvme|ufs' || true)" NOW="$(date +%s)" { echo "# HELP kernel_error_events_10m Number of kernel error-like events in last 10 minutes" echo "# TYPE kernel_error_events_10m gauge" echo "kernel_error_events_10m ${COUNT}" echo "# HELP kernel_error_scrape_time_seconds Last collect time" echo "# TYPE kernel_error_scrape_time_seconds gauge" echo "kernel_error_scrape_time_seconds ${NOW}" } >"${TMP}" mv "${TMP}" "${OUT}" EOF chmod +x /usr/local/bin/collect_kernel_error_metrics.sh cat >/etc/systemd/system/kernel-error-metrics.service <<'EOF' [Unit] Description=Collect kernel error metrics for node_exporter textfile collector [Service] Type=oneshot ExecStart=/usr/local/bin/collect_kernel_error_metrics.sh EOF cat >/etc/systemd/system/kernel-error-metrics.timer <<'EOF' [Unit] Description=Run kernel error metrics collector every 60s [Timer] OnBootSec=30s OnUnitActiveSec=60s AccuracySec=1s [Install] WantedBy=timers.target EOF systemctl daemon-reload systemctl enable --now kernel-error-metrics.timer ``` Verify: ```bash curl -fsS http://192.168.22.18:9100/metrics | grep -E '^kernel_error_events_10m|^kernel_error_scrape_time_seconds' || true ``` #### D3. (Optional) smartctl_exporter for disk health If the storage device does not expose SMART/health, this may not work. Still worth trying. ```bash apt-get update apt-get install -y smartmontools export VER="0.12.0" curl -fsSL -o /tmp/smartctl_exporter.tar.gz "https://github.com/prometheus-community/smartctl_exporter/releases/download/v${VER}/smartctl_exporter-${VER}.linux-amd64.tar.gz" tar -C /tmp -xzf /tmp/smartctl_exporter.tar.gz install -m 0755 /tmp/smartctl_exporter-${VER}.linux-amd64/smartctl_exporter /usr/local/bin/smartctl_exporter useradd --system --no-create-home --shell /usr/sbin/nologin smartctl-exp || true cat >/etc/systemd/system/smartctl_exporter.service <<'EOF' [Unit] Description=Prometheus Smartctl Exporter After=network-online.target Wants=network-online.target [Service] User=smartctl-exp Group=smartctl-exp ExecStart=/usr/local/bin/smartctl_exporter --web.listen-address=:9633 Restart=on-failure RestartSec=2 [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable --now smartctl_exporter ``` #### D4. Deploy Prometheus + Grafana + blackbox_exporter on 192.168.22.12 (PVE) via LXC (recommended) Since `192.168.22.12` is also a PVE host, avoid piling Docker onto the host OS. The simplest operationally is: **one Debian LXC container running the monitoring stack with systemd**. ##### D4.1 Create a Debian LXC container on PVE (.12) You can do it in the PVE Web UI, or via CLI like this (replace ``, ``; LXC IP is pinned to `192.168.22.19`): ```bash # Run on the PVE host 192.168.22.12 pct create local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \ --hostname monitoring \ --cores 2 --memory 2048 --swap 512 \ --rootfs local-lvm:8 \ --net0 name=eth0,bridge=vmbr0,ip=192.168.22.19/24,gw= \ --unprivileged 1 pct set -features keyctl=1,nesting=1 pct start ``` ##### D4.2 Install Prometheus in the container (systemd) Enter the container: ```bash pct exec -- bash ``` Inside the container: ```bash apt-get update apt-get install -y ca-certificates curl tar useradd --system --no-create-home --shell /usr/sbin/nologin prometheus || true mkdir -p /etc/prometheus /var/lib/prometheus chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus export VER="2.54.1" curl -fsSL -o /tmp/prometheus.tar.gz "https://github.com/prometheus/prometheus/releases/download/v${VER}/prometheus-${VER}.linux-amd64.tar.gz" tar -C /tmp -xzf /tmp/prometheus.tar.gz install -m 0755 /tmp/prometheus-${VER}.linux-amd64/prometheus /usr/local/bin/prometheus install -m 0755 /tmp/prometheus-${VER}.linux-amd64/promtool /usr/local/bin/promtool cp -r /tmp/prometheus-${VER}.linux-amd64/consoles /etc/prometheus/ cp -r /tmp/prometheus-${VER}.linux-amd64/console_libraries /etc/prometheus/ chown -R prometheus:prometheus /etc/prometheus ``` Create `/etc/prometheus/prometheus.yml` (scrape `.18` + active probes; DNS is the AdGuardDNS LXC at `.54`): ```yaml global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: "minisforum-node" static_configs: - targets: ["192.168.22.18:9100"] - job_name: "minisforum-smartctl" static_configs: - targets: ["192.168.22.18:9633"] - job_name: "blackbox-icmp" metrics_path: /probe params: module: [icmp] static_configs: - targets: - 192.168.22.18 - 192.168.22.54 relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: 127.0.0.1:9115 - job_name: "blackbox-tcp" metrics_path: /probe params: module: [tcp_connect] static_configs: - targets: - 192.168.22.18:22 - 192.168.22.18:9100 - 192.168.22.54:53 - 192.168.22.54:3000 relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: 127.0.0.1:9115 ``` Create the systemd service: ```bash cat >/etc/systemd/system/prometheus.service <<'EOF' [Unit] Description=Prometheus After=network-online.target Wants=network-online.target [Service] User=prometheus Group=prometheus ExecStart=/usr/local/bin/prometheus \ --config.file=/etc/prometheus/prometheus.yml \ --storage.tsdb.path=/var/lib/prometheus \ --storage.tsdb.retention.time=30d Restart=on-failure RestartSec=2 [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable --now prometheus ``` ##### D4.3 Install blackbox_exporter in the container (systemd) ```bash useradd --system --no-create-home --shell /usr/sbin/nologin blackbox || true mkdir -p /etc/blackbox_exporter chown -R blackbox:blackbox /etc/blackbox_exporter export VER="0.25.0" curl -fsSL -o /tmp/blackbox.tar.gz "https://github.com/prometheus/blackbox_exporter/releases/download/v${VER}/blackbox_exporter-${VER}.linux-amd64.tar.gz" tar -C /tmp -xzf /tmp/blackbox.tar.gz install -m 0755 /tmp/blackbox_exporter-${VER}.linux-amd64/blackbox_exporter /usr/local/bin/blackbox_exporter cat >/etc/blackbox_exporter/blackbox.yml <<'EOF' modules: icmp: prober: icmp timeout: 5s tcp_connect: prober: tcp timeout: 5s EOF chown -R blackbox:blackbox /etc/blackbox_exporter cat >/etc/systemd/system/blackbox_exporter.service <<'EOF' [Unit] Description=Prometheus Blackbox Exporter After=network-online.target Wants=network-online.target [Service] User=blackbox Group=blackbox ExecStart=/usr/local/bin/blackbox_exporter \ --config.file=/etc/blackbox_exporter/blackbox.yml \ --web.listen-address=:9115 Restart=on-failure RestartSec=2 [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable --now blackbox_exporter ``` ##### D4.4 Install Grafana in the container (systemd) Do not commit any Grafana admin password. Install it, then set the password manually. ```bash apt-get install -y apt-transport-https software-properties-common wget gpg mkdir -p /etc/apt/keyrings wget -qO- https://apt.grafana.com/gpg.key | gpg --dearmor >/etc/apt/keyrings/grafana.gpg echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" >/etc/apt/sources.list.d/grafana.list apt-get update apt-get install -y grafana systemctl enable --now grafana-server ``` Reset admin password (run in the container): ```bash grafana-cli admin reset-admin-password ``` --- ### Attempt E: Deploy AdGuardDNS on 192.168.22.18 via LXC (fixed IP: 192.168.22.54) + real DNS query probing Goal: isolate DNS service from the host and **pin the DNS IP to `192.168.22.54`** so monitoring targets never drift. #### E0. Create the AdGuardDNS LXC container on PVE (.18) with IP 192.168.22.54 Example (replace ``, ``): ```bash # Run on the PVE host 192.168.22.18 pct create local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \ --hostname adguarddns \ --cores 1 --memory 1024 --swap 256 \ --rootfs local-lvm:4 \ --net0 name=eth0,bridge=vmbr0,ip=192.168.22.54/24,gw= \ --unprivileged 1 pct set -features keyctl=1,nesting=1 pct start ``` #### E1. Install AdGuard Home inside the container (as AdGuardDNS) Enter the container: ```bash pct exec -- bash ``` Inside the container: ```bash apt-get update apt-get install -y ca-certificates curl tar export VER="0.107.57" curl -fsSL -o /tmp/adguardhome.tar.gz "https://github.com/AdguardTeam/AdGuardHome/releases/download/v${VER}/AdGuardHome_linux_amd64.tar.gz" tar -C /opt -xzf /tmp/adguardhome.tar.gz /opt/AdGuardHome/AdGuardHome -s install systemctl status AdGuardHome --no-pager ``` First-time setup: - Admin UI: `http://192.168.22.54:3000` - DNS: `192.168.22.54:53` (UDP/TCP) #### E1. Real DNS query probe using `dig` + Pushgateway (credential-free) Use `.12` to run `dig` periodically and push two metrics: success and latency. 1. Optionally install Pushgateway on the monitoring side (the `.12` monitoring LXC) via systemd: ```bash apt-get update apt-get install -y ca-certificates curl tar useradd --system --no-create-home --shell /usr/sbin/nologin pushgateway || true mkdir -p /etc/pushgateway /var/lib/pushgateway chown -R pushgateway:pushgateway /etc/pushgateway /var/lib/pushgateway export VER="1.10.0" curl -fsSL -o /tmp/pushgateway.tar.gz "https://github.com/prometheus/pushgateway/releases/download/v${VER}/pushgateway-${VER}.linux-amd64.tar.gz" tar -C /tmp -xzf /tmp/pushgateway.tar.gz install -m 0755 /tmp/pushgateway-${VER}.linux-amd64/pushgateway /usr/local/bin/pushgateway cat >/etc/systemd/system/pushgateway.service <<'EOF' [Unit] Description=Prometheus Pushgateway After=network-online.target Wants=network-online.target [Service] User=pushgateway Group=pushgateway ExecStart=/usr/local/bin/pushgateway --web.listen-address=:9091 Restart=on-failure RestartSec=2 [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable --now pushgateway systemctl status pushgateway --no-pager ``` 2. Add a Prometheus scrape job (use `127.0.0.1` since Pushgateway runs on the same container): ```yaml - job_name: "pushgateway" honor_labels: true static_configs: - targets: ["127.0.0.1:9091"] ``` 3. Create a probe script (DNS IP is pinned to `192.168.22.54`): ```bash cat >/opt/monitoring/dns_probe_push.sh <<'EOF' #!/usr/bin/env bash set -euo pipefail DNS_IP="192.168.22.54" NAME="www.baidu.com" PUSH_URL="http://127.0.0.1:9091/metrics/job/dns_probe/instance/${DNS_IP}" OUT="$(dig +tries=1 +time=2 +stats @"${DNS_IP}" "${NAME}" A 2>/dev/null || true)" if echo "${OUT}" | grep -qE '^;; ANSWER SECTION:'; then SUCCESS=1 else SUCCESS=0 fi LAT_MS="$(echo "${OUT}" | awk -F': ' '/^;; Query time:/{print $2}' | awk '{print $1}' || true)" LAT_MS="${LAT_MS:-0}" cat </dev/null # TYPE dns_probe_success gauge dns_probe_success ${SUCCESS} # TYPE dns_probe_latency_ms gauge dns_probe_latency_ms ${LAT_MS} METRICS EOF chmod +x /opt/monitoring/dns_probe_push.sh ``` 4. Schedule it (during investigation, 15s is useful): ```bash crontab -e */1 * * * * /opt/monitoring/dns_probe_push.sh */1 * * * * sleep 15; /opt/monitoring/dns_probe_push.sh */1 * * * * sleep 30; /opt/monitoring/dns_probe_push.sh */1 * * * * sleep 45; /opt/monitoring/dns_probe_push.sh ``` #### E2. (Optional) AdGuard Home internal stats (requires admin API creds; never commit them) If you need AdGuard-specific stats (blocked/allowed/qps/upstream latency), run an exporter on `.12` that talks to AdGuard’s admin API. Store credentials in a root-only local file (e.g. `/opt/monitoring/secrets/adguard.env`, `chmod 600`) and keep it out of Git.