Lessons Learned: Palo Alto in VWire mode

I recently had the opportunity of deploying a PaloAlto PA-2020 in inline mode within a pre-exisiting network. PaloAlto (PA) refer to inline mode as VWIre –or Virtual Wire-. It worked fantastically well but I hit a snag when trying to access some internal servers.

In a nutshell, and greatly simplified, imagine a network setup as follows:

  • A vlan aware switch (no L3 routing capabilites)
  • Inter vlan routing is handled by a stateful firewall
  • The PA-2020 is set inline with no blocking rules, and allowing all VLAN traffic. Below is a diagram to help visualise the problem

palo alto vwire

Basically the internal user 1.1.1.1 was unable to access the internal server 2.2.2.2. Troubleshooting determined:

a. The internal user 1.1.1.1 was sending SYN packets but not receiving any response

b. The internal server 2.2.2.2 was receiving SYN packets from 1.1.1.1 and answering with SYN/ACK packets, but the final ACK packet required to complete the TCP 3 way handshake was not being received

c. The firewall was relaying both the first SYN packet and the second SYN/ACK packet

Everything pointed towards the PA-2020 having an issue with this three way handshake. The clue was in the PA-2020 logs which showed the same TCP connection as coming from both eth1/2 (correct) and eth1/1 (incorrect)

Once displayed in a diagram as above it becomes easy to visualise what is happening:

step 1. Internal User sends a packet like so:

SRC IP: 1.1.1.1 DST IP: 2.2.2.2

The PA sees this packet as coming in on it’s eth1/2 interface, logs the connection and as shown in the table on the top left of the diagram, it logs IP 1.1.1.1 as being reachable on interface eth1/2

step 2. Firewall routes packet. Since the firewall is acting as a router, it receives the packet from 1.1.1.1 and forwards it out of the same physical interface to it’s destination, from subinterface VLAN1 to subinterface VLAN2

So the palo alto sees the same packet as it saw in step one, but this time it arrives on interface eth1/1, so it updates its cache and notes IP 1.1.1.1 as being reachable on interface eth1/1

step 3. The server answers the SYN packet and sends it’s reply to the firewall:

SRC IP: 2.2.2.2 DST IP: 1.1.1.1

The palo alto dutifully notes that IP 2.2.2.2 is reachable on eth1/2

step 4. The firewall routes the server’s reply to the client, using the inverse of step 1, that is, from subinterface vlan2 to subinterface vlan1

This is where the PA gets confused. The last mapping it had shows that:

1.1.1.1 is reachable on eth1/1

(see step 2) , so it dutifully sends the packet out of that interface —- the wrong one. So the client never gets the reply, and the connection is never established

The solution in this case was to introduce source NAT, or hide NAT. On step 2, the firewall changes the source IP to the IP of one of its interfaces, say 2.2.2.1. In this way, the PA-2020 never sees the same IP on two different interfaces and everything works as it should