Recently I’ve been doing a bunch of work around scaling our core network and in particular ensuring that CSPF has the right information to make intelligent decisions based on the amount of capacity available on different segments of our network and avoid putting primary and secondary paths on bearers that share common elements such as fibre paths etc.
This is all pretty simple with CSPF (Constrained Shortest Path First). Possessing a general aversion to using aggregate ethernet interfaces, and needing to balance around multiple geographically dispersed bearers this has provided some really good results.
It’s important to remember that RSVP was designed as a capacity reservation protocol long before it was used for signalling LSPs – therefore has some inherently good features for shuffling around LSPs as links get full.
Assuming that whatever the IGP there are on any given network point to the most preferred path between any two paths, we can assume that CSPF will just do it’s thing and by default automagically push the primary LSP path onto the best path (where possible) then avoid the primary path when building secondary paths. What I’ve been look at is a way to ensure that when we hit a threshold – say 70% of the capacity of any link on a 15min average – we’re shuffling off some traffic to decrease the load on that particular bearer. It may be worth noting here that a lot of these configuration tasks may be easier to apply in configuration groups with wildcards for interfaces or LSPs so that you save yourself from re-typing a lot of configuration into every LSP into your network!
Let’s assume for the purposes of this post that we’re starting with a LSP with two paths (remember the magic of CSPF in that it will automatically try to route the secondary path in a diverse manner to the primary path, and FRR detours will also be diverse where possible), and fast-reroute turned on. We’ll also assume that we’re optimizing every 5mins;
set protocols mpls label-switched-path A-to-B to x.x.x.x
set protocols mpls label-switched-path A-to-B fast-reroute
set protocols mpls label-switched-path A-to-B primary A-to-B-primary
set protocols mpls label-switched-path A-to-B secondary A-to-B-alternate standby
set protocols mpls path A-to-B-primary
set protocols mpls path A-to-B-alternate
set protocols mpls optimize-timer 300
The first thing to do here is to tell RSVP what the bandwidth of each link is
set protocols rsvp interface X bandwidth 10g
Following this, we need to tell the Juniper to start gathering statistics as to the utilisation of each link. This is pretty simple, all we need to decide on is number of files / size of files – plus how often to gather stats on utilisation per LSP;
set protocols mpls statistics file mpls_auto-statistics
set protocols mpls statistics file size 5m
set protocols mpls statistics file files 3
set protocols mpls statistics interval 300
set protocols mpls statistics auto-bandwidth
Now we need to tell the LSP to use the stats to reserve out the capacity it is using (in this example I’m doing it every 900 seconds). Note that it’s recommended to gather statistics (above config) 3x faster than you update the LSP (below config);
set protocols mpls label-switched-path A-to-B auto-bandwidth adjust-interval 900
At this point it should actually start working. A link will subscribe out capacity up to 100% then become unavailable for additional LSPs to reserve capacity on it. If the LSPs on the link creep up to over 100% one will be booted off the link – HOWEVER it’s actually quite (very very) unlikely that you’re going to go over 100% utilisation and boot a LSP off a link. Additionally I want to ensure that I’m only running my links up to 60-70% (based on a 15min average) before I free up capacity, therefore ensuring I have a good amount of burstability on my core bearers. While I could specify all links as having a bandwidth far lower than they actually do, there’s a far better way to do this, in that you can set a link to act like it only has xx% of the capacity it actually has;
set protocols rsvp interface X subscription 65
This is likely to produce some better results.
The next thing we want to do is ensure that we signal out our new path before tearing down our old path to prevent a brief loss of service. The “adaptive” command does exactly this;
set protocols mpls label-switched-path A-to-B adaptive
It’s worth noting that the “adaptive” command also changes the reservation type to SE (shared reservation between all paths/detours of the LSP) rather than FF (one reservation per member path/detour of the LSP – so a bit of doubling up potentially!)
Finally, despite having primary and secondary paths, plus fast reroute detours on both – I only actually want to reserve capacity at this point on the primary paths. While I can’t move the autobandwidth command into the more specific hierarchy, what I can do is leave it where it is and explicitly force the secondary and fast-reroute paths not to reserve any bandwidth;
set protocols mpls label-switched-path A-to-B secondary A-to-B-alternate bandwidth 0
set protocols mpls label-switched-path A-to-B fast-reroute bandwidth 0
At this point I have quite a cool setup – the primary path will only ever stand up where it has capacity, however in a disaster situation the secondary and fast-reroute paths will stand up anywhere and let CoS take care of the rest. Due to the fact that by default JunOS tries to re-signal the primary path every 30secs, I’m not going to be on a sub-optimal path for any great length of time (and I could tune this a bit later if I find I need to), but I’ve got my CSPF process passing intelligently selected routes (that have sufficient capacity available) to RSVP for reservations based on the available capacity on each link.
It’s also worth noting in the failure scenario that you are likely to have a fair bit of capacity available depending on your subscription ratios before you will need to use CoS.
I’ll post soon on some related fiddling with shared-fate which I’ve been looking at (as a part of the wider expansions I’m doing to multiple bearers) to ensure that I’m not doing silly things like signalling my primary and secondary paths on the same fibre (while keeping all the signalling automated!).
UPDATE – a few people have been asking me if there is a way to force an immediate auto-adjustment (i.e. if you get a big traffic spike). In JUNOS you can do this with the following command (? your way through it to specify an LSP name, or just run it to reoptimize every LSP that this PE is the ingress LSR for);
request mpls lsp adjust-autobandwidth
Note that this will update the RSVP reservation based on the latest statistics run but won’t gather fresh statistics (so it’s worth gathering statistics pretty frequently).
Additionally, it’s worth noting that you can change the path selection method of the LSP for routing the LSP choose links based on different criterion with the least-fill or most-fill options as per the below example. The default behaviour is to randomly select once it’s met the other criterion documented at http://www.juniper.net/techpubs/en_US/junos10.4/topics/concept/mpls-cspf-path-selection-method.html.
set protocols mpls label-switched-path A-to-B (least/most)-fill
Note that this is based on the percentage fill of the link, not the amount of megs/gigs available.
Additionally, if you have not played with RSVP-TE optimization before, have a look at http://blog.hoff.geek.nz/2013/11/02/rsvp-optimization-on-junos/.