CYBER RUNBOOK

Palo Alto IPSec VPN Troubleshooting Runbook

Palo Alto IPSec VPN Troubleshooting Runbook

Site-to-site VPN troubleshooting gets easier when you separate the problem into three questions:

  1. Can both peers establish IKE Phase 1?
  2. Can they establish the IPSec Phase 2 / Child SA?
  3. Once the tunnel is up, can the intended application traffic pass in both directions?

Do not start by changing proposals, PSKs, or routing. First collect evidence and identify the first stage that is failing.

Scope and safety: Use this runbook for authorised support only. Never paste PSKs, private keys, customer data, public NAT mappings, or internal addressing into tickets, chat, or public documentation. Coordinate disruptive tests with the remote partner and follow your change-control process.

1. Record the incident facts

Capture the following before making a change:

Item Record
Gateway and tunnel name Exact configured names
Local / peer public IP Confirm the currently active addresses
IKE version IKEv1 or IKEv2
Impact One flow, one subnet, or all traffic
First observed Time, timezone, and change reference
Last known good Approximate date/time and recent changes
Interesting traffic Source, destination, protocol, and port
Partner contact Name and agreed testing window

Also verify whether the VPN is route-based or policy-based, and whether the gateway is expected to initiate, respond, or both.

2. Fast decision tree

Can the firewall reach the peer public IP?
        |
        +-- No --> Check default route, upstream ACL/NAT, ISP path, and peer availability.
        |
        +-- Yes --> Is IKE Phase 1 established?
                       |
                       +-- No --> Check IKE version, IDs, PSK/certificates, proposals, UDP 500/4500.
                       |
                       +-- Yes --> Is IPSec Phase 2 established?
                                      |
                                      +-- No --> Check proxy IDs/selectors, PFS, proposals, lifetimes.
                                      |
                                      +-- Yes --> Check route, security policy, NAT exemption, return path, and sessions.

3. Check IKE and IPSec state

Run these read-only commands on the active firewall. In an HA pair, verify which member owns the active dataplane before drawing conclusions.

show vpn ike-sa gateway <gateway-name>
show vpn ipsec-sa tunnel <tunnel-name>

Interpretation:

Observation Meaning Next action
No IKE SA Phase 1 has not formed Review reachability and IKE negotiation
INIT sent / repeated retransmits Local side is initiating but not receiving a valid response Confirm peer address, path, UDP 500/4500, and partner configuration
IKE SA established, no IPSec SA Phase 1 works but Phase 2 fails Compare selectors, proposals, PFS, and lifetimes
IKE and IPSec SA established Control plane is healthy Troubleshoot traffic forwarding

For an authorised, coordinated negotiation test, use the gateway test operation from the firewall UI or CLI. Do this only after confirming the partner is ready, because it can create unnecessary log noise or interfere with an in-progress negotiation.

4. Phase 1: IKE does not establish

4.1 Confirm the outbound route to the peer

The firewall must send IKE traffic out of the intended internet-facing interface and source address.

test routing fib-lookup virtual-router <virtual-router> ip <peer-public-ip>

Confirm the result shows the correct next hop, interface, and expected source IP. A wrong route is a common cause after cloud route-table or default-route changes.

4.2 Review VPN system logs

In Monitor > Logs > System, filter on the gateway name and VPN-related messages. Useful findings include:

  • IKEv2 SA negotiation started as initiator - local firewall has started negotiation.
  • Retransmits - no usable response was received.
  • ID_R does not match peers id - peer identity does not match the configured Local/Peer ID.
  • Authentication failure - incorrect PSK, certificate issue, or certificate trust failure.
  • unknown ikev2 peer - incoming negotiation did not match a configured gateway.

Use the incident time window and save relevant log lines with sensitive values removed.

4.3 Compare both ends, field by field

Ask the partner to confirm these exact items; do not exchange the PSK in plain text.

Setting Both ends must agree on
IKE version IKEv1 or IKEv2
Peer address Current public IP/FQDN and correct NAT mapping
IKE crypto profile Encryption, integrity/hash, DH group, lifetime
Authentication Matching PSK or mutually trusted certificates
IDs Local and peer IDs, including FQDN vs IP type
NAT traversal Enabled where either side is NATed
Initiation Initiator/responder expectation and liveness behaviour

4.4 Network-path checks

  • Permit UDP/500 and UDP/4500 between the two public endpoints.
  • If NAT is in use, validate NAT traversal and ensure no upstream device is rewriting or blocking UDP/4500.
  • Do not rely on ICMP alone: a peer may block ping while IKE works, or allow ping while IKE is blocked.
  • If the peer address is an FQDN, verify the current resolution and whether it changed recently.

5. Phase 2: IKE is up but IPSec is down

An established IKE SA proves that the peers can authenticate each other. It does not prove the encryption domain or Child SA is correct.

Compare these values on both sides:

Setting What to validate
Proxy IDs / traffic selectors Local and remote CIDRs, correct direction, no overlapping selectors
IPSec crypto profile Encryption, authentication, lifetime
PFS Enabled/disabled and DH group match
Encapsulation ESP/NAT-T expectation
Rekey behaviour Timers and any recently changed lifetime settings

Common patterns:

  • Phase 1 up, Phase 2 absent: proxy ID/selector mismatch is the first suspect.
  • Tunnel works only after manual initiation: check routing, interesting traffic, tunnel monitor design, and whether the peer accepts on-demand initiation.
  • Tunnel drops at regular intervals: compare Phase 2 lifetime and PFS settings, then correlate both sides’ rekey logs.

6. Tunnel is up but traffic does not pass

When both SAs are established, move from VPN negotiation to normal forwarding checks.

6.1 Verify route selection

Confirm the remote protected subnet points to the tunnel interface:

show routing route destination <remote-subnet>

The selected route must be in the correct virtual router and use the expected tunnel interface. Check route preference, static route changes, ECMP, BGP advertisements, and more-specific routes.

6.2 Validate policy and NAT

In the firewall rulebase, verify:

  • Correct source zone, destination zone, source CIDR, and destination CIDR.
  • The security rule is allowing the intended application, service, and user criteria.
  • Rule hit counters increase during the test.
  • A dedicated no-NAT rule appears above broad internet SNAT rules for protected-to-protected traffic.
  • The partner has a return route for the local subnet and does not NAT the return traffic unexpectedly.

6.3 Inspect sessions and traffic logs

Filter traffic logs by the test source/destination and compare the session end reason, action, rule, NAT source/destination, ingress/egress zones, and bytes in each direction.

A useful read-only CLI check is:

show session all filter source <test-source-ip> destination <test-destination-ip>

Look for these symptoms:

Symptom Likely area
No session is created Endpoint, route, or upstream reachability
Session denied Security policy or zone mismatch
Session matches SNAT Missing/incorrect NAT exemption
Bytes sent but no bytes received Partner return route, remote policy, host firewall, or asymmetric path
Both directions show bytes, application fails Host/service, application ACL, MTU/MSS, or DNS

7. MTU, MSS, and asymmetric routing

Do not overlook transport issues after the tunnel comes up.

  • Test a small TCP connection first, then a larger transfer.
  • If small traffic works but larger HTTPS or file transfers stall, investigate MTU/MSS clamping and PMTUD.
  • Confirm the return path is symmetric. A reply leaving a different firewall, internet breakout, or VPN often causes stateful inspection to drop it.
  • In cloud environments, check the VPC/VNet route tables, transit routing, security groups/NSGs, and load-balancer path as well as the firewall.

8. Tunnel monitoring and HA checks

Tunnel monitoring can improve failover, but only if its target is reachable through the intended protected path.

  • Use a stable remote monitoring destination agreed with the partner.
  • Confirm a route exists to the monitoring destination through the tunnel.
  • Avoid monitoring a remote device that intentionally blocks ICMP unless you have designed an alternative method.
  • In HA, validate the active device is negotiating and forwarding; inspect the passive device only for configuration/state comparison.
  • After any HA failover, confirm the peer accepts the new source path and that cloud NAT/public-IP associations are correct.

9. Evidence to attach to the incident

Capture only what is necessary and sanitise the rest:

  • Timestamped IKE/IPSec state from both peers.
  • Relevant VPN system-log messages.
  • FIB lookup for the peer public IP and remote protected subnet.
  • Security-policy and NAT-rule names/hit counts.
  • Traffic-log result for one controlled test flow.
  • Partner confirmation of matching parameters.
  • Recent-change references, if any.

10. Closure checklist

  • IKE SA is established.
  • IPSec SA is established for the intended selectors.
  • Test traffic passes in both directions.
  • Correct security and no-NAT rules receive hits.
  • Return path and remote policy are verified.
  • Tunnel monitoring/HA behaviour is confirmed where applicable.
  • Temporary debug filters or test rules have been removed.
  • Evidence, root cause, remediation, and follow-up actions are documented.

Author

Masud Rana is a Network & Security Team Lead focused on cloud security, network infrastructure, Palo Alto firewalls, Cloudflare, IPSec connectivity, and operational resilience. Connect on LinkedIn.