By Manny Fernandez

November 18, 2026

Prove ISP Packet Loss With a Raspberry Pi and SmokePing

Prove ISP Packet Loss With a Raspberry Pi and SmokePing

No fluff. Just the config that works.

Your firewall keeps logging WAN link-monitor failures, a health check on a server times out a few times a day, and every call to the ISP ends with “reboot the gateway.” This guide builds the thing that ends that conversation: an independent probe that records ISP packet loss at one-minute resolution, hop by hop, with a second ISP running as a control. When you open the ticket, the evidence points at their network, not yours.

Item Details
Objective Measure loss and latency on the primary ISP from outside the firewall, layered so the first failing hop is obvious, with a second ISP as a control path.
Audience Network and security engineers running a dual-WAN FortiGate in a lab, home office or small site.
Result A Raspberry Pi 5 running SmokePing and timed mtr snapshots, plus a tuned FortiGate link monitor.

Why the probe has to sit outside the firewall

If you measure through the firewall, two things work against you. First, failover hides the outage: with update-static-route and update-policy-route enabled on the link monitor, the FortiGate moves traffic to the backup WAN, so a probe behind it sees a latency bump instead of loss. Second, the ISP can always say the firewall caused it. A probe plugged straight into a spare LAN port on the ISP gateway, using a spare address from your static block, removes both problems.

The probe also needs a second network interface. The built-in port lives on the ISP gateway with a public address and nothing in front of it, so it must accept nothing inbound. A USB Ethernet dongle on the internal LAN carries management and the control probes.

Prerequisites and architecture

  • Raspberry Pi 5 (4 or 8 GB is plenty), official 27 W USB-C supply, active cooling.
  • A high-endurance microSD card. A probe writes all day, every day.
  • A USB 3 Gigabit Ethernet dongle for the management interface.
  • One unused address from the primary ISP’s static block and a free LAN port on its gateway.
  • A UPS outlet. A power blip on the probe looks exactly like an ISP outage in the graphs.
  • A FortiGate with two WANs on FortiOS 7.x, and Docker on the Pi.
Component Lab address Role
ISP A gateway (CPE) 198.18.10.30 First layer 3 hop for the static block
FortiGate port1 198.18.10.1/27 Primary WAN on ISP A
FortiGate port2 DHCP, gateway 198.19.20.1 Backup WAN on ISP B
FortiGate LAN 10.0.10.1/24 Internal gateway
Probe eth0 198.18.10.29/27 ISP A probes, default route, nothing inbound
Probe eth1 (dongle) 10.0.10.5/24 SSH, web UI, ISP B control probes, no default gateway
ISP A first hop 198.18.100.1 First router past the CPE that answers ping
ISP A backbone 198.18.101.1 A core hop further upstream

Figure 1: Probe topology: ISP A probes leave eth0 directly on the CPE, control probes ride ISP B through the FortiGate

Raspberry Pi probe on the ISP A gateway beside a FortiGate, with a second path through ISP B as a control

Step 1: Fix the link monitor before you blame the ISP

Goal: Stop false failovers so you are chasing real loss.

Action: A link monitor pinging one public resolver is fragile. Many resolvers rate-limit or deprioritize ICMP, so a healthy circuit can lose a burst of pings to one target. With update-static-route enabled (the default), the FortiGate then removes the default route for a few seconds and resets sessions. Probe three independent providers and give it more tolerance.

config system link-monitor
    edit "wan1-sla"
        set srcintf "port1"
        set gateway-ip 198.18.10.30
        set server "1.1.1.1" "8.8.8.8" "9.9.9.9"
        set protocol ping
        set interval 500
        set failtime 10
        set recoverytime 5
        set update-static-route enable
    next
end

interval is in milliseconds and failtime counts missed probes, so 500 ms with a failtime of 10 means about five seconds of loss before the route is pulled. Defaults are documented in the FortiOS link-monitor CLI reference.

GUI Verification: Log and Report, System Events shows link-monitor up and down events with timestamps. Keep these; they become corroborating evidence later.

diagnose sys link-monitor status
Note: If the alerts continue after this change and all three targets fail together, the circuit really is dropping. That is what the probe proves.

Step 2: Flash and harden the Pi

Goal: A headless, key-only Linux host.

Action: In Raspberry Pi Imager, choose Raspberry Pi OS (other), then Raspberry Pi OS Lite (64-bit). In the settings screen set the hostname, your user, SSH with public-key only, and your time zone. Boot it on the internal LAN first, then update and install Docker.

sudo apt update && sudo apt full-upgrade -y
timedatectl                  # System clock synchronized: yes
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
sudo reboot

Verification: docker run --rm hello-world succeeds after the reboot. Accurate time matters more than anything else on this box; every piece of evidence is a timestamp.

Step 3: Two interfaces, two jobs

Goal: eth0 owns the default route onto ISP A. eth1 carries management with no default route, and anything sourced from its address returns through the FortiGate.

Action: NetworkManager handles both, including a source-based routing rule into a separate table. Run this while the Pi is still on the LAN; it writes the profiles without touching the live connection, so your SSH session survives.

#!/usr/bin/env bash
set -euo pipefail
LAN_GW=10.0.10.1

# Keep existing DHCP profiles from returning after reboot
nmcli -g NAME,TYPE con show | while IFS=: read -r n t; do
  [ "$t" = "802-3-ethernet" ] || continue
  case "$n" in isp-a|lan-mgmt) continue ;; esac
  nmcli con mod "$n" connection.autoconnect no
done

# eth0: ISP A, public static, default route
nmcli con add type ethernet ifname eth0 con-name isp-a \
  ipv4.method manual ipv4.addresses 198.18.10.29/27 \
  ipv4.gateway 198.18.10.30 ipv4.dns "1.1.1.1 8.8.8.8" \
  ipv6.method disabled

# eth1: LAN, no default route in main, own table 200
nmcli con add type ethernet ifname eth1 con-name lan-mgmt \
  ipv4.method manual ipv4.addresses 10.0.10.5/24 \
  ipv4.never-default yes \
  ipv4.routes "10.0.10.0/24 table=200, 0.0.0.0/0 $LAN_GW table=200" \
  ipv4.routing-rules "priority 100 from 10.0.10.5 table 200" \
  ipv6.method disabled

The routing rule is the trick: packets sourced from 10.0.10.5 (SSH replies, the web UI, the control probes) use table 200 and leave eth1 toward the FortiGate. Everything else follows the main table out eth0.

Verification: after the cable move in Step 7, ip rule shows from 10.0.10.5 lookup 200 and ip route show table 200 shows the default via 10.0.10.1.

Step 4: Lock down the public interface with nftables

Goal: eth0 answers nothing. Management only on eth1.

Action: Manage a single table so Docker’s own rules are left alone on reload. Install it now, enable it for boot, and do not load it until the cables move, or you will cut your own SSH session.

#!/usr/sbin/nft -f
table inet probe
delete table inet probe

table inet probe {
  chain input {
    type filter hook input priority filter; policy drop;
    iif "lo" accept
    ct state established,related accept
    ct state invalid drop
    iifname "eth1" tcp dport { 22, 80 } accept
    iifname "eth1" icmp type echo-request accept
    iifname "docker0" accept
  }
}
sudo install -m 0644 nftables.conf /etc/nftables.conf
sudo systemctl enable nftables

Verification: sudo nft list table inet probe after boot, and a scan of 198.18.10.29 from outside your network (a phone on cellular is enough) gets no answer, not even ping.

Step 5: Pin the control path to ISP B on the FortiGate

Goal: Probes sourced from 10.0.10.5 always leave through ISP B, so you have a clean comparison for the same minutes.

Action: An IP pool only changes what the source is translated to; it does not choose the egress interface. Routing does. Add a policy route, then a firewall policy that allows the traffic out port2 with NAT.

config router policy
    edit 0
        set input-device "lan"
        set src "10.0.10.5/255.255.255.255"
        set dst "0.0.0.0/0.0.0.0"
        set gateway 198.19.20.1
        set output-device "port2"
        set comments "Probe control path via ISP B"
    next
end

Policy routes are evaluated before the routing table, top down, and a route that names both a gateway and an interface only matches when that gateway is reachable through the interface. See FortiOS policy routes. Set the gateway explicitly: if ISP B’s default route has a higher distance, it is not in the active routing table while ISP A is up, so a lookup by interface alone can fail.

diagnose firewall proute list
diagnose sniffer packet port2 'host 10.0.10.5 and icmp' 4 10

GUI Verification: Network, Policy Routes lists the entry, and Policy and Objects shows hits on the LAN to port2 policy.

Step 6: SmokePing in Docker with layered targets

Goal: One-minute loss and latency for each layer of the ISP A path, plus the same internet targets through ISP B.

Action: Use the LinuxServer image with host networking so the probes use the Pi’s real interfaces and routing rule. The image keeps its configuration in /config and graphs in /data, and serves the UI under /smokeping/ (LinuxServer SmokePing docs).

services:
  smokeping:
    image: lscr.io/linuxserver/smokeping:latest
    container_name: smokeping
    network_mode: host
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/New_York
    volumes:
      - ./config:/config
      - ./data:/data
    restart: unless-stopped

Put these three files in ./config before the first start. The FPing probe’s sourceaddress is a probe-level variable (it maps to fping’s -S option per the SmokePing FPing probe manual), so the control path gets its own subprobe.

Probes

*** Probes ***
+ FPing
binary = /usr/sbin/fping

++ FPingISPA

++ FPingISPB
sourceaddress = 10.0.10.5

Database

*** Database ***
step     = 60
pings    = 20

# consfn mrhb steps total
AVERAGE  0.5   1  20160
AVERAGE  0.5  15   8640
AVERAGE  0.5  60   8760
    MIN  0.5  15   8640
    MAX  0.5  15   8640

That is 20 pings every 60 seconds: 14 days at one-minute resolution, 90 days at 15 minutes, a year at one hour.

Targets

*** Targets ***
probe = FPingISPA
menu = Top
title = ISP A vs ISP B packet loss

+ ISPA
menu = ISP A (direct on CPE)
title = ISP A path from 198.18.10.29
++ cpe
menu = 1 CPE gateway
title = CPE 198.18.10.30
host = 198.18.10.30
++ access
menu = 2 ISP access
title = First answering hop 198.18.100.1
host = 198.18.100.1
++ backbone
menu = 3 ISP backbone
title = Backbone 198.18.101.1
host = 198.18.101.1
++ cloudflare
menu = 4 Cloudflare
title = 1.1.1.1
host = 1.1.1.1
++ google
menu = 4 Google
title = 8.8.8.8
host = 8.8.8.8

+ ISPB
menu = ISP B (control)
title = Control path via FortiGate
probe = FPingISPB
++ fortigate
menu = 1 FortiGate LAN
title = 10.0.10.1
host = 10.0.10.1
++ cloudflare
menu = 2 Cloudflare
title = 1.1.1.1 via ISP B
host = 1.1.1.1
++ google
menu = 2 Google
title = 8.8.8.8 via ISP B
host = 8.8.8.8

Find your own layer targets with traceroute -n 1.1.1.1 from the primary path. The first hop is the CPE. The next hop often ignores ping, which is normal; use the first hop past it that answers.

Step 7: Move the cables and start collecting

Goal: The probe in its permanent spot, everything starting on its own.

sudo shutdown -h now
# eth0 to a spare CPE LAN port, dongle to the LAN switch, Pi on the UPS
cd ~/probe/smokeping && docker compose up -d
docker logs -f smokeping

Note the time of the move. That gap in the graphs is you, not the ISP.

Step 8: Hop-by-hop snapshots every 15 minutes

Goal: An mtr report near every event, so you can show the ISP the exact hop where loss started.

#!/usr/bin/env bash
OUT="$HOME/probe/mtr/$(date +%F)"
mkdir -p "$OUT"
{
  echo "# $(date '+%F %T %Z') source 198.18.10.29 (eth0)"
  mtr -rwzbc 100 -I eth0 1.1.1.1
} > "$OUT/$(date +%H%M).txt" 2>&1
find "$HOME/probe/mtr" -mindepth 1 -maxdepth 1 \
  -type d -mtime +60 -exec rm -rf {} +
(crontab -l 2>/dev/null; echo "*/15 * * * * $HOME/probe/mtr-capture.sh") | crontab -

Verification and validation

Both paths must leave where you think they do. Check from the Pi:

curl -4 -s ifconfig.me; echo
curl -4 -s --interface 10.0.10.5 ifconfig.me; echo
ip rule; ip route show table 200
docker ps --format '{{.Names}} {{.Status}}'

Expected success output:

198.18.10.29
198.19.20.10
0:      from all lookup local
100:    from 10.0.10.5 lookup 200
...
default via 10.0.10.1 dev eth1
smokeping Up 10 minutes

The first address is the probe’s own static on ISP A. The second is ISP B’s public address. If the second one shows an ISP A address instead, the control traffic is matching your normal LAN policy; see the gotchas below. Open http://10.0.10.5/smokeping/ after an hour and drill into a single target for the most detail.

Reading the graphs: whose problem is it?

Loss starts at Points to What to ask for
CPE gateway Cable, port or the gateway itself Gateway swap or a line test
ISP first hop The ISP access network Line test, dispatch, port check at the node
ISP backbone Upstream in the ISP core Escalation to network operations
Internet targets only Peering or the destination Usually not the access circuit
ISP B too, same minutes Your side: power, LAN, the probe Fix your house before calling anyone

The ISP packet loss evidence package

  • A list of events: date, time, duration and loss percentage.
  • SmokePing graphs showing the layer where loss starts.
  • The ISP B graphs for the same windows, showing a clean control.
  • The mtr snapshot closest to each event.
  • The CPE’s own event log entries and the FortiGate link-monitor events at the same timestamps.

A week of this, ideally covering a few link-monitor alerts, is a ticket front-line support cannot close with a reboot script.

Troubleshooting and gotchas

SSH key rejected on first boot

If ~/.ssh/authorized_keys is missing on the Pi, the key never got applied during imaging. Enabling password login with a file named 99-temp.conf can still fail, because sshd keeps the first value it reads for each setting. Check what is in effect, then name the override so it sorts first:

sudo sshd -T | grep -i passwordauthentication
echo "PasswordAuthentication yes" | \
  sudo tee /etc/ssh/sshd_config.d/00-temp.conf
sudo systemctl restart ssh
# from your workstation: ssh-copy-id user@10.0.10.5
sudo rm /etc/ssh/sshd_config.d/00-temp.conf
sudo systemctl restart ssh

Control probes still leave through ISP A

The traffic from 10.0.10.5 is hitting the normal LAN to WAN1 policy and its IP pool. Confirm the policy route’s input-device is the interface the LAN actually arrives on (a VLAN subinterface if you use one), and that it sits above other policy routes. Then watch the decision:

diagnose debug flow filter addr 10.0.10.5
diagnose debug flow show function-name enable
diagnose debug flow trace start 20
diagnose debug enable

The dongle is not called eth1

Some builds name USB adapters by MAC address. Check before running the network script, and edit ifname and the nftables rules to match.

ip -br link

Sources

Is SCEP Viable Long Term? SCEP Alternatives, RFCs and a Migration Plan

Is SCEP Viable Long Term? SCEP Alternatives, RFCs and a Migration Plan

SCEP alternatives explained: RFC 8894 limits, EST, ACME device attestation and CMPv2, plus a step-by-step plan to retire SCEP safely.

Read More
FortiGate 60F “Firmware Failed Signature Validation” Warning: What It Means and How to Get to the Latest Code

FortiGate 60F “Firmware Failed Signature Validation” Warning: What It Means and How to Get to the Latest Code

Seeing firmware failed signature validation on a FortiGate 60F? Learn why it is a false alarm, verify the image, and upgrade to FortiOS 7.6.8.

Read More
VRF Explained: Virtual Routing and Forwarding on FortiGate, Cisco and Juniper

VRF Explained: Virtual Routing and Forwarding on FortiGate, Cisco and Juniper

Learn virtual routing and forwarding (VRF), then build VRFs and route leaking on FortiGate, Cisco IOS XE and Juniper Junos with tested CLI.

Read More