Troubleshooting IPSec VPNs on Fortigate Firewalls

By Manny Fernandez

November 12, 2019

Troubleshooting IPSec VPNs on Fortigate Firewalls

Lets start with a little primer on IPSec. I am going to describe some concepts of IPSec VPNs.

IPSec Primer

Authentication Header or AH – The AH protocol provides authentication service only. AH provides data integrity, data origin authentication, and an optional replay protection service. AH authenticates IP headers and their payloads, with the exception of certain header fields that can be legitimately changed in transit, such as the Time To Live (TTL) field.

Encapsulating Security Payload or ESP – The ESP protocol provides data confidentiality by using encryption and authentication (data integrity, data origin authentication, and replay protection). ESP can be used with confidentiality only, authentication only, or both confidentiality and authentication. When ESP provides authentication functions, it uses the same algorithms as AH, but the coverage is different. AH-style authentication authenticates the entire IP packet, including the outer IP header, while the ESP authentication mechanism authenticates only the IP datagram portion of the IP packet.

Transport Mode – Transport Mode provides a secure connection between two endpoints as it encapsulates IP’s payload.

Tunnel Mode – Tunnel Mode encapsulates the entire IP packet to provide a virtual “secure hop” between two gateways.

Main Mode – Main mode requires six packets back and forth, but affords complete security during the establishment of an IPsec connection.

Aggressive mode – The fallacy is that this is better since it is “aggressive” however, Aggressive mode uses half the exchanges providing a bit less security because some information is transmitted in cleartext. If you get audited, they WILL ding you on this. Remote access IPSec VPNs use aggressive mode.

Internet Key Exchange or IKE – Is the mechanism by which the two devices exchange the keys.

Phase I – The purpose of phase 1 is to establish a secure channel for control plane traffic. The result of a successful phase 1 operation is the establishment of an ISAKMP SA which is then used to encrypt and verify all further IKE communications. Phase 1 can operate in two modes: main and aggressive.

Phase II – IKE phase 2 establishes IPSec SAs (one in each direction) for the VPN connection, and is referred to as Quick Mode. At the conclusion of phase 2 each peer will be ready to pass data plane traffic through the VPN. Quick mode consists of 3 messages sent between peers (with an optional 4th message). All messages in phase 2 are secured using the ISAKMP SA established in phase 1.

Fortigate Debug Command

Diag Commands

To filter out VPNs so that you focus on the one VPN you are trying to troubleshoot

FW-01 # diagnose vpn ike log-filter 
list Display the current filter.
clear Erase the current filter.
name Phase1 name to filter by.
src-addr4 IPv4 source address range to filter by.
msrc-addr4 multiple IPv4 source address to filter by.
dst-addr4 IPv4 destination address range to filter by.
mdst-addr4 multiple IPv4 destination address to filter by.
src-addr6 IPv6 source address range to filter by.
msrc-addr6 multiple IPv6 source address to filter by.
dst-addr6 IPv6 destination address range to filter by.
mdst-addr6 multiple IPv6 destination addresses to filter by.
src-port Source port range to filter by.
dst-port Destination port range to filter by.
vd Index of virtual domain. -1 matches all.
interface Interface that IKE connection is negotiated over.
negate Negate the specified filter parameter.

Above you can see the different filtering criteria. This allows you to filter a VPN to a destination of 2.2.2.2 as an example:

diagnose vpn ike log-filter dst-addr4 2.2.2.2

Now you can run the following commands

diag debug app ike -1 
diag debug enable

Clearing Established Connections

diagnose vpn ike restart 
diagnose vpn ike gateway clear

Lets get started

I have created a VPN in my lab and I will break it at different points and identify it on the output of the debug commands.

Pre-Shared Key Mismatch

The first example, we are going to look at non-matching pre-shared keys

I will break down the sections:

Here we can see the first ISKMP proposal the firewall received.
This section shows it is receiving AES 128 with a Hash of SHA 256
Shows that we matched a particular VPN we have configured and it matches what I created GW1-to-GW2
Here we can see the platform connecting to/from.
Here we see the cause of the problem possible pre-shared secret mismatch.

Phase I – No Proposal Chosen

In this example, I left ONLY AES-128 SHA256 while the remote firewall had the AES-128 SHA256 removed causing a mismatch.

Phase 1 v1.png

Here we see the incoming proposal.
We can see AES-128 and SHA-256 as stated above.
This section shows my proposal and show us iterating through our proposals we have configured.
Another my proposal
Another my proposal
Another my proposal

Next screen shot for more output

Phase 1 v2.png

Finally, we see the error negotiation failure
Now we have the final outcome of no SA proposal chosen

PFS Mismatch

In this section, I removed PFS on one side of the VPN.

In this output, we can see:

Pre-shared Key authentication is successful
Authentication OK. This shows us Phase I is up.
Here we can see that Quick-Mode has failed.

In this output, we do not see a specific PFS error, but normally in Phase II these are the following situations you will find:

Proposal mismatch. In this scenario, you could have AES-256 SHA-256 but it not be configured on the other side.
Phase II Selectors not matching (you will see this next). Essentially, you would see 10.x.x.x/24 on one side but the other configured as 192.168.0.0/24 as an example.
PFS or Perfect Forward Secrecy.
NAT-T or NAT Traversal mismatch on either side.
And finally, Some remote firewalls such as Cisco, do not like Fortinet/Palo/Checkpoint etc groups on Phase II Selectors. Cisco would make you create separate Phase II selectors.

Phase II Selector Mismatch

In route-based VPNs we normally use 0.0.0.0/0 as the Phase II selectors. Because of this, you would not see this error. However if not:

Here we can see:

The error saying that the Phase II selector was the issue.
Give you what the local Phase II selector you have configured (where you are capturing the the debug from)
This give you the remote Phase II Selectors. This is useful because if it is a third party vendor, you can tell them what they are sending and what you are expecting.
Finally the error telling you no matching Phase II found.

Other Things to Look At

Policies – Ensure you have the corresponding policies to allow the interesting traffic to flow through the firewall. Remember that although the VPN may be using the WAN1 or WAN2 interface to get to the remote side, the policies need to reference the VPN interface NOT the WAN interfaces.
Network Address Translation (NAT) – Ensure that you have the correct NAT configuration you are expecting. Depending on Policy NAT or Central NAT, the configuration may change. You can take a look at the Central NAT Article I did a while back.
Routing – Weather you are running route-based VPN or policy-based VPN, you will need to have either a static route or a routing protocol. Again, ensure that the destination interface is the VPN interface and NOT the WAN interface.
Key Lifetime – Also, if the tunnel comes up and drops, you may need to check the key lifetime in Phase I and Phase II.
Interoperability – From an interoperability perspective, although the Fortigate can do address groups in the PhaseII selectors, other vendors such as Cisco does not like it and will require you to create separate Phase II selectors to match their crypto-map ACLs.

diag vpn ipsec tunnel details

Hope this helps.

Troubleshooting IPSec VPNs on Fortigate Firewalls

IPSec Primer

Fortigate Debug Command

Clearing Established Connections

Pre-Shared Key Mismatch

Phase I – No Proposal Chosen

PFS Mismatch

Phase II Selector Mismatch

Other Things to Look At

Recent posts