I recently had the opportunity of deploying a PaloAlto PA-2020 in inline mode within a pre-exisiting network. PaloAlto (PA) refer to inline mode as VWIre –or Virtual Wire-. It worked fantastically well but I hit a snag when trying to access some internal servers.
In a nutshell, and greatly simplified, imagine a network setup as follows:
– A vlan aware switch (no L3 routing capabilites)
– Inter vlan routing is handled by a stateful firewall
– The PA-2020 is set inline with no blocking rules, and allowing all VLAN traffic. Below is a diagram to help visualise the problem
Basically the internal user 22.214.171.124 was unable to access the internal server 126.96.36.199. Troubleshooting determined:
a. The internal user 188.8.131.52 was sending SYN packets but not receiving any response
b. The internal server 184.108.40.206 was receiving SYN packets from 220.127.116.11 and answering with SYN/ACK packets, but the final ACK packet required to complete the TCP 3 way handshake was not being received
c. The firewall was relaying both the first SYN packet and the second SYN/ACK packet
Everything pointed towards the PA-2020 having an issue with this three way handshake. The clue was in the PA-2020 logs which showed the same TCP connection as coming from both eth1/2 (correct) and eth1/1 (incorrect)
Once displayed in a diagram as above it becomes easy to visualise what is happening:
step 1. Internal User sends a packet like so:
|SRC IP: 18.104.22.168||DST IP: 22.214.171.124|
The PA sees this packet as coming in on it’s eth1/2 interface, logs the connection and as shown in the table on the top left of the diagram, it logs IP 126.96.36.199 as being reachable on interface eth1/2
step 2. Firewall routes packet. Since the firewall is acting as a router, it receives the packet from 188.8.131.52 and forwards it out of the same physical interface to it’s destination, from subinterface VLAN1 to subinterface VLAN2
So the palo alto sees the same packet as it saw in step one, but this time it arrives on interface eth1/1, so it updates its cache and notes IP 184.108.40.206 as being reachable on interface eth1/1
step 3. The server answers the SYN packet and sends it’s reply to the firewall:
|SRC IP: 220.127.116.11||DST IP: 18.104.22.168|
The palo alto dutifully notes that IP 22.214.171.124 is reachable on eth1/2
step 4. The firewall routes the server’s reply to the client, using the inverse of step 1, that is, from subinterface vlan2 to subinterface vlan1
This is where the PA gets confused. The last mapping it had shows that:
126.96.36.199 is reachable on eth1/1
(see step 2) , so it dutifully sends the packet out of that interface —- the wrong one. So the client never gets the reply, and the connection is never established
The solution in this case was to introduce source NAT, or hide NAT. On step 2, the firewall changes the source IP to the IP of one of its interfaces, say 188.8.131.52. In this way, the PA-2020 never sees the same IP on two different interfaces and everything works as it should
5 thoughts on “Lessons Learned: Palo Alto in VWire mode”
thanks for the info! we’re also facing these kind of issue in Vwire mode.
I spoke to Palo Alto about this and they said it only occurs when both interfaces are in the same zone… which is odd, because you cannot have a vwire config with two interfaces in the same zone (at least currently, not sure about in 2013).
You need to do run the below command and check the connectivity.
PA_FW# set deviceconfig setting session tcp-reject-non-syn no
And of course do a commit (either in CLI or in GUI) afterwards….
This isnt right. Ive done some testing and found I have a different outcome. all traffic ( ping and http ) able to traverse the firewall with no issue. yes, from the logs, I see the traffic traverse because they need to “tromboning” the wire. of course I assigned different zones to the interface pair because that is the requirement of vwire. Let me post the test document and I will share.