Policy Based Forwarding on a Palo Alto with different Virtual Routers

This guide is a little bit different to my other Policy Based Forwarding blog post because it uses different virtual routers for both ISP connections. This is quite common to have a distinct default route for both providers. So, in order to route certain traffic, e.g., http/https, to another ISP connection, policy based forwarding is used.

There are two documents from Palo Alto that give advises how to configure PBF.

I am using a PA-200 with PAN-OS 7.0.1. My lab is the following:

Palo Alto PBF with different VRs

(Note that, unlike Juniper ScreenOS, a zone is not tied to a virtual router. You actually can merge interfaces on different vrouters into the same zone. However, I prefer to configure an extra zone for each ISP to keep my security policies clearly separated.)

These are the configuration steps. See the descriptions under the screenshots for details:

Done.

Featured image “Kondensstreifen” by Rüdiger Stehn is licensed under CC BY-SA 2.0.

28 thoughts on “Policy Based Forwarding on a Palo Alto with different Virtual Routers

  1. Very good explaination, thanks. You use static routes between the two virtual routers. I would like to use automatic route redistribution, but not to the internet. What do you think?
    Kind regards
    Michael

      1. Thanks, for your answer. OSPF isn´t straight forward I found out so far.

        1. Hello Michael,
          what do you mean with “isn’t straight forward”? The implementation of OSPFv2? Or the routing between different VRs when using OSPF?
          (Ok, I still have not tried OSPFv2 with different VRs, but in general the OSPF implementation works quite good from my point of view.)

      2. This is a little old, but thought I’d chime in. You can’t run OSPF between VRs on the Palo unless you physically connect the two VRs together. What I ended up doing was to assign a loopback address on each VR and then use those IPs as peers for iBGP. You just set up static routes in each VR that point to the other VR’s loopback address via the next VR special interface. I can then share routes between the VRs and redistribute or filter into OSPF as needed. This really helps since I have each VR connecting to my core stitch and I can advertise the routes correctly from the Palo.

  2. Good morning, I need to implement in my PA 3020 three ISPs, two of which will be exited for the internet and one for my web services. How do I make this configuration so that I have a routing between the two Internet outlets with my ISP services?

    1. Hi Marcio. Sorry, but I cannot answer this within 1-2 sentences. ;) It requires a more profound design with a discussion about the pros and cons, etc. I am not offering this kind of consulting within the comments section in my blog. Sorry. ;)
      Please ask your local IT security provider or some security consultants for that.

      (Just a short hint: I would use own virtual routers for all three ISPs.)

    2. Did you find an answer to this? I’m deploying 5 ISPs! Every article is limited to 2.

    3. Marco, your asking for a “design” here, not a quick simple config. If you need help with what you ask, it indicates that you are brand new to simple networking, so you should definitely hire some one with PA/Network experience to do it for you, or start reading about basic networking first.

  3. Johannes, why create a new separate VR when you could have simply created another default with a higher metric/admin distance.

    Are there any pros/cons to either approach ?

    1. Hey Phill,

      when you are only using *outgoing* connections, than you are right: You don’t need a second virtual router. Have a look at my other blogpost linked in the very first sentence.

      BUT: If you have *incoming* connections on both ISPs, for example for NATed servers behind the firewall or for site-to-site VPN sessions, it is much better to have to distinct virtual routers, one for each ISP, with their default routes pointing to the ISPs. Otherwise you’ll run into trouble to have the correct returning route to the correspondent ISP to NOT have asymmetric routes.

      Cheers, Johannes

  4. I have three issues and hopefully you can point me in the right direction.

    1) Both of my ISPs have a DHCP address. When configuring the PBF to get the traffic out the primary ISP, I have to statically put in the next hop IP. Unfortunately, if the public IP changes (it doesn’t happen very often, but does happen), then the PBF rule no longer works. I tried leaving the next hop blank (it assumes 0.0.0.0), but the PBF goes down. For site-site VPN with the PBF, you can simply set the egress to the tunnel interface without a next hop, and that works.

    2) I currently use OSPF to exchange routes from the Palo Alto to my core switch and this works great when there was one ISP off the Palo. The issue I’m seeing now is if the connection to the ISP hanging off the same VR that’s connected to my core goes down, then the default route no longer is sent to the core via OSPF. Am I stuck putting in a static default route to the Palo to get around this? I’m trying to avoid this since there could be other default routes from the core besides to the Palo (i.e. other data center) and I want the core to learn those via regular routing protocols.

    3) I’m connecting a site-site VPN to a Cisco ASA using IKEv2 and both tunnels come up correctly. I have the PBF configured to monitor via the primary ISP VPN tunnel and this works great. The issue I’m having is that on the remote ASA, I’ve configured a dynamic VPN connection so that when my IP address changes on the ISP, I don’t lose connectivity via the VPN. I’m currently getting two VPN tunnels via this dynamic policy, but depending upon which tunnel comes up first, the ASA doesn’t route the traffic back the correct tunnel and so traffic gets dropped. If the primary ISP tunnel comes up first then things are ok. If the connection fails over to the secondary ISP, then things work OK as long as the primary tunnel goes down on the ASA. When the primary ISP comes back up, then the primary tunnel comes back up, but the ASA still sends traffic via the secondary tunnel causing an asymetric route. Is it possible to configure the Palo to leave the secondary VPN tunnel down until the primary goes offline and then bring up that tunnel once the primary tunnel goes down? Likewise, when the primary tunnel comes back up, the secondary should go back down. I think that would solve the asymetric routing from the ASA. I think I read that IKEv2 on the ASA doesn’t support multiple peers, but I didn’t want to switch to IKEv1 and since I’m uisng dymanic VPN, I don’t have any peers configured (traffic has to originate from behind the Palo in order to bring up the tunnel to the ASA).

    1. i resolved number 2 by using IBGP between the two virtual routers. After setting up a loopback address in each VR with a static route for each loopback using a next hop of the other VR, I was able to establish IBGP peering between the two VRs. I can then redistribute the default route from my primary ISP in VR1 with VR2 and then I can redistribute that route via OSPF to my core switch. This way, I can bring down the connection to the secondary ISP in VR2 and when the default route goes away, my core still has one being advertised by VR1 via BGP to VR2 and then via OSP to the core. PBF still is utilized to steer traffic to the appropriate ISP regardless of the default routes being advertised.

      1. Stephen,
        Are you able to post or send me the configuration you made for this, as I’m trying to do the exact same setup. I have OSPF routes coming from the inside LAN to the default VR and I want to redistribute these to the other VR which has a site to site VPN connected to it.

    2. Issue number 3 seems to have resolved itself. After some time, due to the PBF rules I have to monitor the remote network via the tunnel via the primary ISP, eventually, the tunnels via the secondary ISP go down and so there is only one connection established. It only seems to be an issue when both tunnels are up and running. I’ll tweak the idle timeout values to get the secondary tunnels to come down a little sooner to avoid the asymetrical routing.

      1. I spoke too soon for issue #3. When the primary ISP fails and the VPN is established on just the secondary, everything works. When the primary comes back up and the VPN is established through the primary ISP, then return routing from the ASA is hit-and-miss. The only way to resolve the issue is to restart the IPSEC tunnel on the Palo Alto for the secondary connections, and then the return routing from the ASA works. After some time, the secondary tunnels idle out and disconnect. I’m going to have to play around with the ASA configuration to see if I can get the routing working correctly. I have a feeling that it’s related to reverse route injection on the crypto entry on the ASA. I may have to put in two static routes, but not sure how to do that since the next hop is the outside interface on the ASA in both cases.

    3. Hey Stephen. Looks like you don’t have a standard or simple setup at all. ;) I am sorry, but I cannot solve such bigger problems during my daily business in my free time. Please ask your IT security partner for that. If you want support from me, feel free to send me an email.

      At least I can say that having ISP connections with DHCP is a problem at many setups. Especially when you have it on both sides! Please figure out whether you can upgrade to static IPs. I am not sure whether PBF works at all with DHCP addresses un the uplinks.
      Same for using the ASA on the other site of the VPNs. ASA only implements policy-based VPNs which aren’t that flexible. Consider using another firewall that is easier to manage when it comes to different VPN scenarios.
      (Of course I know that both hints are not helping for your current problems. ;) Sorry for that. But remember: When you’re creating complex solutions, you’ll create complex problems as well… Sometimes it’s better to create easier setups, such as with static IPs, than creating huge workarounds.)

      Cheers, Johannes

      1. I was able to resolve this a couple of ways. Unfortunately, getting static IPs isn’t an option from my provider unless I want to double the price per month and get a 1/10th of the connection speed. I agree that this would be much easier if I had Palos on both sides since there’s the built in tunnel monitoring mechanism that takes care of failovers automatically. I’m working on trying to replace the ASAs with Palos.

        Since I’m using OSPF between the Palo and my Cisco core switch, I was able to manipulate the routes to get this working.

        1) I used IP SLA on my core switch to monitor the primary ISP. I have a static default route pointing to the primary ISP VR on the the Palo. When the primary ISP goes down, the static default route is removed from the forwarding table on the core. To make sure that I still had a default route advertised, I added another interface on the backup ISP VR on the Palo to form a second OSPF neighbor with the core.

        2) To get the remote networks on the other side of the VPN tunnel advertised correctly, I set up path monitoring on the primary ISP VR on the Palo where I have static routes for the VPN traffic through the tunnels. When the primary ISP goes down, the static routes are removed from the primary ISP VR routing table. I also have static routes for the same VPN networks on the backup ISP VR on the Palo and are redistributed into OSPF, but with a metric of 250. This way, when both the primary and backup VR on the Palo are both advertising the same routes, the one from the primary takes precedence.

        3) To get the ASAs to behave better, I used a tracked static host route to my primary ISP IP. When this becomes unreachable, the route is removed causing a syslog message to be logged. When the IP comes back up, a second syslog message is logged which then triggers an event in the EEM (embedded event manager). This event runs a cli command to clear the IPSEC SA connection for my backup ISP IP DDNS name. This is the only place that I have to use a specific IP address and so could potentially be a problem if my primary ISP IP changes on the Palo. If that happens, then when failing back to the primary VPN tunnels, there would be a delay until the idle timeout was reached on the secondary tunnels and DPD kicked in to bring down those SAs.

        Cutover to the backup ISP is fairly quick and only a few pings are lost. Failing back from the backup ISP to the primary ISP is seamless from an Internet traffic standpoint, but the VPN tunnels take a couple of minutes to properly fail back to the primary tunnels. I’m still working on the timing, but this seems to be pretty stable.

      2. Forgot to mention that PBF works fine with DHCP, you just have to put in the next hop statically. If the public IP ever changes, then the PBF would fail. In my case, since only guest traffic is steered to the backup ISP while the primary is still online, the worse that would happen is that guest traffic would start flowing out the primary ISP instead.

        1. Also forgot to mention that I’m using dynamic VPN tunnels on the ASA so I don’t have to worry about the IP addresses changing on the remote side. Since there’s always traffic coming from the remote side to the ASA, the tunnels from the primary ISP basically stay up all the time.

  5. Bless you. I’ve been beating my head on articles that don’t take into account multiple VRs. In hindsight, the return route between them makes total sense, but for the life of me, I could figure out why I was not getting return traffic. Great rite-up.

  6. Great article, I had to do something very similar with one of our PAN arrays and I had missed the static return route . Cheers!

  7. Hello!

    We have two ISPs (Comcast & AT&T) and would like for both of these ISP links to be able to route traffic. I have read and reviewed a few Palo Alto supported documentation & blogs from other sites. I’ve configured a few things on our firewall but I am not 100% clear on a few configurations that I have done to make sure it will work properly before going live.

    Here’s what I’ve done so far:

    • Interfaces:
    o Eth. 1/1 and along subinterfaces for LAN
    o Eth. 1/10 for AT&T ISP Link
    o Eth. 1/12 for Comcast ISP Link
    • Zones:
    o Created ISP_ATT & assigned Eth 1/9
    o Created ISP_Comcast & assigned Eth 1/12
    o Created Trust for LAN networks
    o Created Azure-S2S with Tunnel.1 & .2 (for failover)
    • Created two virtual routers one for each ISP link:
    o Primary-ISP-Comcast
    o Secondary-ISP-ATT
     Eth 1/9 is for AT&T ISP/Link
     Eth 1/10 is for PC connected directly
     Tunnnel.2 for Asure S2S VPN (as a backup route)
    o For Statis Routes for Primary-ISP-Comcast, do I create just Comcast’s network with its next-hop IP? But what about the other internal networks? Do I need to create for each internal/LAN network a route to point to the next VR, which in this case is AT&T ISP? (based on your explanation).
    o For the static routes for Secondary-ISP-ATT, besides configuring AT&T’s next-hop IP address, what else would I need to do?
    • Also, do I need to enable “Path Monitoring” on Comcast and/or AT&T too?
    • Finally, the PBF, is my understanding is that in the “Forwarding” tab, I need to enter Comcast’s IP address and monitor it as well so that if it fails, all traffic is routed out of the AT&T ISP link, right?

    1. Hey Freddy,

      some thoughts from me:
      1) I’m quite sure that you won’t have a fully working setup after your first commit. You should have a test environment or at least off-peak hours to test and troubleshoot the setup *before* going live.
      2) Do you have a sketch rather than text? ;)
      3) You’ve configured two different untrust zones for the two ISPs. While this is possible and valid, I don’t see the advantage here at a first glance. Personally, I’m using a single “untrust” zone for multiple ISPs, since they are all untrusted in the same manner.
      4) Your interface numbers are not consistent in your comment (but that’s probably just a typo?) Is it 1/10 or 1/9 for ATT?
      5) Here’s the general problem with scenarios like these: All of your internal networks MUST belong to a single virtual router. You can’t configure it differently. Hence they follow the default route from THIS single virtual router. You won’t be able to do an ECMP easily. My guide here shows how to use the 2nd ISP for certain types of IP connections (e.g., referenced by the destination IP or the destination port), while all other traffic will follow the default route. Or your 2nd ISP has some other routable IPv4 addresses for your DMZ. In this case, this DMZ must be place in the virtual router from the corresponding ISP.
      6) Unfortunately, I’m not showing here how to use those two ISPs for failover scenarios. Neither for normal outgoing traffic, nor for site-to-site VPNs. :( I’m sorry.

      So, in the end, I cannot help you that much right now. Anyway, please tell us what your setup looks like exactly. What is your scenario for using two ISPs? Load balancing? High availability?

      One idea, though: If you only want to load balance your *outgoing* traffic, you can try using one virtual router but with two default routes. However, note that you will run into problems if you have a DMZ with public IP addresses that are only available through one ISP. This won’t work with two default routes to two different ISPs.

      Cheers. ;)

  8. Thanks for your post but I really don’t understood the reason for the static route 192. I really lost where is this network

Leave a Reply to Dilip Ratna Cancel reply

Your email address will not be published. Required fields are marked *