I never grasped what’s so great about MPLS VPNs until I learned that core routers do not need to carry customer routes. This magic trick greatly reduces state in the MPLS network core. This article will follow a packet’s journey though an MPLS Layer 3 VPN.

We will follow an ICMP echo request from CUST2_CE1’s loopback to CUST2_CE2’s loopback. Source and destination address and the expected path shown in the topology below.

Source CE routing entry. Nothing special yet just regular IP forwarding.

Traffic comes into the PE (Provider Edge) on Eth1/1, which is a member of VRF CUST2.

If we examine the CE1 VRF CUST2 routing table for 44.44.44.44, we see a next hop of 3.0.0.3 in vrf default and that MPLS is required. We also see that MPLS label 36 will be used.

Another way to see this is the command shown below. The out label for 44.44.44.0/24 is 36. This is the VPN label. Traditional MPLS L3 VPN uses a stack of two labels in the core. The outer transport label and the inner VPN label. Make note of the next hop for 44.44.44.0/24, this will be important soon.

If we jump ahead momentarily, we can see that the egress PE (PE3) has incoming label 36 for 44.44.44.0/24.

So far, we know that PE1 needs to use MPLS to forward traffic to 44.44.44.0/24 and that the VPN label is 36. We also saw that PE1s next hop is 3.0.0.3. The next part is the magic to MPLS VPN forwarding.

PE1 will impose a transport label that corresponds to the IP next hop. The Label Forwarding Information Base (LFIB) entry below shows that the outgoing transport label for 3.0.0.3 will be 105. 105 will be the local label for 3.0.0.3 on the next hop router.

I started a pcap on the link between PE1 and P1 and ran a ping from CE1 to CE2. Notice the two MPLS headers between the ethernet and IP headers. The “outer” label is 105 and the “inner” label is 36. The outer label is the transport label, and the inner label is the VPN label.

Hopefully this will make sense after we examine the LFIB on the next hop, router P1. When P1 receives a packet with transport label 105, it will swap label 105 for label 32 and forward it out of Eth0/1. Forwarding with the MPLS core is based solely on label values, no IP lookup is required.

This means that the MPLS core routers do not need to know customer routes. In fact, the core routers are not even running BGP! We have an IGP running for loopback reachability and LDP, that’s it!

To further reinforce this, the output below confirms that P1 has no additional VRFs configured and only knows about connected, local, and OSPF routes. 44.44.44.0 is unknown to the core.

I ran the ping again and took a pcap on the next link, the P1-P3 link. As expected, the transport label is 32 and the VPN label is unchanged, still 36.

We’re ready to check the forwarding table on the next hop, P3. The local label we see here is the outgoing label we saw on P1, value 32. The MPLS routers are using Label Distribution Protocol (LDP) to allocate and advertise their local labels to their neighbors. By default, LDP routers allocate a label value for every entry in their routing table.

The outgoing label requires more explanation. “Pop Label” here means remove the transport label and forward without one. This is called Penultimate Hop Popping (PHP). This takes place at the router hop before the egress PE. This is done so that the egress router doesn’t have to do an extra lookup. Without PHP, the PE would receive a packet with a transport label, do a lookup on the transport label to realize it is the PE and then pop the transport label. Then, the PE would need to do a lookup on the VPN label and realize the destination belongs to a local VRF, and then finally do an IP route lookup within the target VRF. PHP removes the initial lookup, allowing the egress PE to look up the VPN label and VRF route lookup only.

I took another pcap, this time on the P3 – PE3 link. We can see that the outer label has been ‘popped’, leaving only the VPN label.

When PE3 receives a packet with bottom label 36, it can do a lookup and find that this is the VPN label for VRF CUST2 destination 44.44.44.0/24 with next hop 172.30.0.5.

PE3 will do a recursive lookup to find that the next hop is via Eth0/2.

Normal forwarding from here, check the ARP cache to construct the ethernet header and forward the traffic down to CE2 where the destination IP is on a loopback interface.

And that’s it! That is the fundamental magic behind MPLS VPNs. Not needing state for customer routes in the core simplifies service provisioning and reduces cost for core routers. The routers need label switching functionality at high speeds but can have less TCAM memory. High speed TCAM routing memory is very expensive, anything we can do to buy less of it is worth consideration.