Using apply-groups to insert policy in a predictable manner….

Recently I’ve been solving some interesting problems around how routing policy can be implemented. What I’ve been trying to do is have a way that enables me to quickly insert a policy at the start of a set of routing-policies on a BGP neighbor to stop sharing routes.

For those who are not aware, you can implement multiple policies in a chain on a BGP session. You do this like so;

[edit protocols bgp group R2]
[email protected]# show 
export [ policy-1 policy-2 policy-3 ];
neighbor {
    peer-as 2;

In this configuration, all policies will be evaluated sequentially. Policy with matching terms will take action as defined in the policy on the routes matched. Routes will continue to be processed by subsequent terms in each policy until a terminating action is hit. A “terminating action” is something like an “accept” or “reject” action (once you hit one of these you use the actions gathered so far including the terminating action, then stop processing)

Let us say for the sake of this test that the policies we have applied to our peer do 2x as-path prepends, add a MED value, then accept all routes;

[edit policy-options]
[email protected]# show 
policy-statement policy-1 {
    then as-path-prepend "4 4";
policy-statement policy-2 {
    then {
        metric 400;
policy-statement policy-3 {
    then accept;

This would result in the following routes being sent from this router (note the MED and as-path prepends);

[email protected]# run show route advertising-protocol bgp    

inet.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
*         Self                 400                4 4 [4] I
*          Self                 400                4 4 [4] I
*          Self                 400                4 4 [4] I

In the scenario I described at the start, the aim was to be able to easily (with one line of configuration) insert a policy at the start of the policy-set that rejected all routes. However, as described above, this peer might have existing policies that allow routes to be sent.

Assume we say that I have already created the following policy;

[edit policy-options]
+   policy-statement policy-reject {
+       then reject;
+   }

If I quickly activated this by adding it as an export policy, it would appear as the last policy on the policy-chain;

[email protected]# set protocols bgp group R2 export policy-reject 

[email protected]# show | compare 
[edit protocols bgp group R2]
-    export [ policy-1 policy-2 policy-3 ];
+    export [ policy-1 policy-2 policy-3 policy-reject ];

As we can see below, this does not reject any of the routes, because the reject action is processed after a terminating accept action in policy-3;

[email protected]# run show route advertising-protocol bgp              

inet.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
*         Self                 400                4 4 [4] I
*          Self                 400                4 4 [4] I
*          Self                 400                4 4 [4] I

In order to implement this in the manner I want with this way of doing it, I would have to do the following to achieve it;

[edit protocols bgp group R2]
[email protected]# delete export 
[email protected]# set export policy-reject
[email protected]# set export policy-1
[email protected]# set export policy-2
[email protected]# set export policy-3

You could shorten this by using the “insert” functionality, however you are still requiring a re-ordering of policies as opposed to a one liner to activate this.

The trick I have come up to solve this is quite cool. I am adding a term to an existing policy with an apply-group;

[email protected]# show | compare 
+ groups {
+     policy-reject {
+         policy-options {
+             policy-statement policy-1 {
+                 term reject-term {
+                     then reject;
+                 }
+             }
+         }
+     }
+ }

When we activate this apply-group, the term gets inserted into policy-1, which is already ordered to be applied prior to the terminating accept action in policy-3;

[email protected]# show | compare 
+ apply-groups policy-reject;

We can see here that it is implemented in policy-1;

[edit policy-options policy-statement policy-1]
[email protected]# show | display inheritance 
## 'reject-term' was inherited from group 'policy-reject'
term reject-term {
    ## 'then' was inherited from group 'policy-reject'
    ## 'reject' was inherited from group 'policy-reject'
    then reject;
then as-path-prepend "4 4";

As policy-1 is already the first policy to be processed, this ensures that without having to re-order our existing policies we can insert a reject policy at the start. We can now check the result;

[email protected]# run show route advertising-protocol bgp    

[email protected]#

And it is working!

You might ask why I did not just apply a new policy in an apply-group instead of a new term? The reason for this is that new policies are added at the end of the policy chain, while a new term in a pre-existing policy uses the existing place that policy has in the policy-chain.

For my purposes, this has enabled me to give out a one-line command which a large group of people can use to disable the advertisement of certain routes at certain times. As with everything in Junos though, there are many possible uses for this set of functionality. Hope this helps!

Junos CSPF for multiple dynamic paths

A quick one today, but a few people have recently asked me how the Junos CSPF process manages to always ensure the primary and secondary paths are kept diverse *automagically*. RSVP path calculation with CSPF is an interesting topic, and if you look through the archives, you’ll find I’ve touched on this as I’ve discussed SRLGs, RSVP auto-bandwidth, and other such topics.

Lets assume for the purposes of this post that we have a LSP that looks like this:

[edit protocols mpls]
[email protected]# show 
label-switched-path test-LSP-to-PE5 {
    primary primary-path-to-5;
    secondary secondary-path-to-5;
path primary-path-to-5;
path secondary-path-to-5;

This LSP is pretty simple. There are no constraints implemented for either of the primary or secondary paths, and no constraints on the LSP as a whole.

Unpon configuration of a LSP such as this, Junos will attempt to stand up the primary path. This will follow the best IGP metric subject to constraints (available bandwidth, admin-groups, SRLGs, etc). Only once this has stood up will the calculation for the secondary begin.

As Junos calculates the secondary path, it will first take a copy of the topology of the RSVP network and inflate the IGP metric of every link that has been used by the primary path by a factor of 8 million. This ensures that if at all possible, a diverse path is used – while still allowing the use of a shared path between the primary and secondary if this is our only option.

When the primary path re-optimizes to a new path, this process is repeated, thus ensuring that we at all times try to keep the primary and secondary as diverse as possible.

It’s worth noting that if you turn off CSPF for the LSP, the above functionality will not be implemented, and both paths will simply follow the best IGP path.

Hope this helps!

Two new certifications… and….. aspirations for another JNCIE!

Over the last month or so, I have been quietly working away at a couple more JNCIS certifications – JNCIS-QF (QFabric) and JNCIS-SEC (Security).

For those of you who have not done any Juniper certifications before – a JNCIS is roughly equivalent to a CCNP level certification. I’ve come off the back of fairly intensive study for my JNCIE-ENT, for which I was invited to sit the beta of some new test forms in February – so rather than doing 6+ an hour per night, I slowed this down to an hour or two per week. It’s actually been quite odd re-figuring out what to do with all this “free time” – so doing these certifications was a good thing to do to keep my brain occupied and learning something new!

I’m pleased to say that I passed both, getting my JNCIS-QF on 14 March, and my JNCIS-SEC on 8 April. I thought both did a really good job of establishing that the candidate had a reasonable knowledge of the subject areas covered, and would feel confident setting up either a QFabric system or a Juniper SRX in a live deployment (which I guess is the point, right?).

I’ve also decided that I am in fact going to work towards a third JNCIE – the JNCIE-SEC. In many ways, this is going to be far more interesting than the other two, given the fact that I have far less experience in the area of Network Security than I have in Service Provider or Enterprise (as I have spent most of my career working for large service providers!). I’m really looking forward to learning a bunch of new and different technologies – something which is always very enjoyable!

I do however plan to take this one significantly slower than the other two. Essentially I did all the study for my last two JNCIEs in one year – and while I am glad I did it, as I wanted to prove to myself that I could; I would not do it again as I had no life at all while I was doing it. My plan is to slowly work towards this with the aim of doing the JNCIE-SEC exam sometime in the next year. Over the next couple of months I plan to sit the JNCIP-SEC and the JNCSP-SEC – though I will be doing plenty of labbing for the JNCIE as I study for these two written exams. From there I’ll make a week-by-week study plan of what I want to learn and work out a pace to approach it, and only book the exam when I’m sure that I am entirely ready.

As I do this, I’ll be blogging regularly on some of the new technologies and concepts I will be learning – and would appreciate any feedback/corrections; much of this stuff will be very new and different for me!

I also am hoping to hear back on my JNCIE-ENT result in the next few weeks – and will post this as soon as I get it!


RPKI on Junos is easy!

My friend Michael Fincham did a great presentation a few weeks ago at NZNOG on RPKI. I spent a fair bit of time helping him get said presentation ready to go – as we did a bunch of testing on the MX implementation of RPKI. He made a good argument that as a community we are horribly bad at the security of our prefixes. We need to be ensuring we do more than just blindly trust our peers and check that they actually have the right to send us the prefixes they advertise. Of course we are all at some level aware of this, but this verification stuff is hard right… right?….surely?!?…

Michael and I spent some time prior to him doing his presentation playing around with the RPKI on a MX80 in my lab. One of the things that came out of this is how truly easy it is to get a basic RPKI deployment going. To  turn on validation in Junos only requires the following single line of config;

set routing-options validation group some-awesome-rpki-server session

Of course there are a bunch of other options available, such as the priority of different RPKI servers, timeouts, port numbers, source address, etc – but this is not complex stuff to configure….

At this point, you’ll start to see prefixes looking like this (if they are valid);

[email protected]_RPKI_DEMO_MX80> show route
[output omitted]         *[BGP/170] 1w3d 03:59:37, localpref 100
                      AS path: 23655 4648 2914 5511 3215 I, validation-state: valid
                    > to via ge-1/1/8.0
[output omitted]

Or this (if they are not valid);

[email protected]_RPKI_DEMO_MX80> show route
[output omitted]      *[BGP/170] 1w3d 04:01:05, localpref 100
                      AS path: 23655 4648 4134 I, validation-state: invalid
                    > to via ge-1/1/8.0
[output omitted]

Then you can start writing policy that matches on the following;

[email protected]# set policy-options policy-statement abc from validation-database ?  
Possible completions:
  invalid              Match for invalid database validation-state
  unknown              Match for unknown database validation-state
  valid                Match for valid database validation-state

The cool thing about the JUNOS implementation of this is that you take action based on policy – so you can do anything at all with this! This might just be attaching an “untrusted” community, or it might be doing more.

Once concern I had that we spent some time playing around with before his presentation was that verifying each route with RPKI would make the MX take longer to process routes. So we did a bit of testing. We had a MX80 in my lab (and we all know that the MX80 does not have a particularly awesome CPU!) with a full BGP feed from my network. We measured the time it took for the route-table to be populated, plus the time for it to push these routes to the forwarding-table before and after implementing RPKI (with a route policy applied on import from the BGP feed inspecting RPKI attributes). The result was that the load time for the full table increased by 300-400%. This is bad, but not unmanageable – especially if you are not validating the full table – but just a few specific peers.

If you are going to set this up in your network, you are likely going to want to use a local RPKI server. A good place to start getting info on some of this stuff would be the slide deck from Michael’s talk, which can be found here;

Please also see the video of the talk here;

Another bit of interesting reading is about the deployment of RPKI on the IX in Ecuador of all places! Read here;

One of the key things to note with RPKI is that while it is far from the final solution at this point (few have their routes signed yet), it’s a very good step in the right direction. And for that reason, I think we should all be looking into at least validating our routes, and perhaps assigning a better local-preference to validated routes. Like IPv6, it takes all of us ‘buying in’ to get this to the point where a large proportion of the internet routing table can be successfully validated!

CoS for RE sourced traffic

Many of you will have deployed CoS extensively on your networks. One area of a Junos CoS deployment that I am often asked about by friends is how to manipulate traffic that is sourced from the routing-engine. There are multiple catches and caveats in dealing with this traffic, and different ways to manipulate this.

At a 10,000 foot level, whenever we deploy CoS we generally want to be able to manipulate route-engine sourced traffic such as ISIS, BGP, OSPF, BFD, RSVP, etc to have various different DSCP, EXP, and 802.1p markings. Firstly, we can set a policy as to the marking used for traffic sourced from the route-engine;

set class-of-service host-outbound-traffic dscp-code-point 111000
set class-of-service host-outbound-traffic ieee-802.1 default 111

You can also specify the forwarding-class that is used for processing traffic sourced from the route-engine;

set class-of-service host-outbound-traffic forwarding-class hoffs-odd-class

It’s important to note that your rewrite rules will not take effect with this traffic (by default). Even if you have specified a forwarding-class, the “host-outbound-traffic” markings will be applied outbound for this route-engine sourced traffic.

However as of Junos 12.3, Juniper have implemented a new option for “host-outbound-traffic” on the MX, which causes the router to use the rewrite-rule for each unit to put markings onto traffic from the RE (based on the forwarding-class it is assigned). This is particularly helpful where you might have multiple fibre providers providing access to your customers, each with a different markings scheme that you are required to use. Note that this is only available for the 802.1p markings (not DSCP) This is done as follows;

set class-of-service host-outbound-traffic ieee-802.1 rewrite-rules

Of course a rewrite-rule must be configured on the outbound unit for this to have effect. So if we have a rewrite-rule to map “hoffs-odd-class” traffic to a marking of 010, the traffic will be now marked as 010 on egress.

This of course does not help us for DSCP markings (it only applies to 802.1p markings). Often we will want to manipulate these. Also how would we approach this problem if we were to wanted to assign different forwarding-classes to different types of traffic being sourced from the RE? A great example of this is that while we might want to ensure that BGP is prioritised, we probably don’t need prioritisation of http traffic sourced from the RE!

The solution for this is quite clever. Most of you will know that you can firewall off all traffic to the RE (regardless of the IP it is destined to – even if that IP is on a physical interface) by applying an inbound firewall filter to the loopback. The clever thing is that you can also apply a firewall filter to all traffic leaving the RE by applying an outbound firewall filter to the loopback. If we want to ensure that all http/https traffic is put into the best-effort forwarding-class, we could do the following;

set interfaces lo0 unit 0 family inet filter output RE-QOS
set firewall family inet filter RE-QOS term web from protocol tcp
set firewall family inet filter RE-QOS term web from port http
set firewall family inet filter RE-QOS term web from port https
set firewall family inet filter RE-QOS term web then dscp be
set firewall family inet filter RE-QOS term web then forwarding-class best-effort
set firewall family inet filter RE-QOS term web then accept
set firewall family inet filter RE-QOS term catchall then accept

This is a pretty handy tool and allows us to do a fairly fine-grained manipulation of how each traffic-type being sourced by the RE is treated. Obviously you could customise this in any way to suit your needs. However it’s worth noting that to my understanding you cannot manipulate the 802.1p markings with a firewall filter – hence why the “rewrite-rules” option becomes so important for host-oubound-traffic.

If you thought that this is all there is to marking/classifying traffic sourced from the RE, you would be wrong! On a MX router, the processing of certain control traffic is delegated to the individual line-cards (such as BFD). I have learned the hard way that the markings on this traffic are not modified by any configuration you apply to normal RE-sourced traffic.

The news is not all bad though, as there is an easy workaround for this, and not many protocols are distributed to the line cards. For this traffic, you can apply an outbound firewall filter to the interface you are doing this traffic on. As an example, here is how to ensure that BFD traffic which has been distributed to the line card is placed into the correct forwarding class and marked appropriately;

set interfaces ge-1/2/3 unit 0 family inet filter output cos-bfd-link
set firewall family inet filter cos-bfd-link term 1 from protocol udp
set firewall family inet filter cos-bfd-link term 1 from port 3784
set firewall family inet filter cos-bfd-link term 1 from port 3785
set firewall family inet filter cos-bfd-link term 1 then loss-priority low
set firewall family inet filter cos-bfd-link term 1 then forwarding-class network-control
set firewall family inet filter cos-bfd-link term 1 then dscp 111000
set firewall family inet filter cos-bfd-link term 2 then accept

Hope this helps!

Don’t forget the rest of your loopbacks!

With all that is going on on the internet currently around NTP reflection attacks and the like, it seemed timely to do a post on the logic of how router-protect filters are applied to loopbacks in JUNOS.

For those of you new to using Juniper gear, if you apply a firewall filter inbound on the loopback of a Juniper networks device, this will be applied to all traffic processed by the routing-engine. This includes traffic with a destination address of a physical interface (i.e. not the loopback). This provides a simple and convenient place to deploy firewall filters to protect the routing-engine on the Juniper device.

This generally looks something like this (where re-protect has the rules for what should talk to the RE);

set interfaces lo0 unit 0 family inet filter input re-protect

This includes VRF/Virtual Router interface traffic for VRFs/ Virtual routers that do not have their own loopback interfaces.

The catch that many people I have been helping over the last week have forgotten however, is the fact that this does not apply to traffic in VRFs or virtual-routers that have their own loopback. If the VRF or virtual-router has a loopback interface in it, you must apply the filter to this loopback as well for it to take effect. For example;

set interfaces lo0 unit 504 family inet filter input re-protect

The classic example where you may strike this is that you will generally require loopback interfaces in any VRF in which you wish to land BNG PPPoE subscribers on the MX routers.

However, a better way to implement firewall filtering to protect the routing engine would actually be to implement it in an apply group, in order that all future loopback interfaces are protected without any configuration being required. This could be done like so;

set groups re-protect interfaces lo0 unit <*> family inet filter input re-protect
set apply-groups re-protect

The only catch with deploying it like this is that if you ever do explicitly configure an input filter on a loopback unit directly (i.e. not through the apply-group to all), the group will cease to have any effect on this loopback (as it will see the group as having been overridden with local config).

Hope this all helps!

Regular expressions in JUNOS show commands

Most of you will be aware that you can use regular expressions to transform/filter the output of any command in JUNOS with a show blah | match “ge-*/0/[0-9]”, the one catch with this method is that it kills the headers for each column of output.

There’s actually another way to achieve this, which is to write the regexp directly into the show command. You can actually do this with most show commands in JUNOS (though the example below is a show interfaces, i’ve also tried this on ISIS, RSVP and OSPF output). The advantage of this way is that it preserves any information that is not related to a specific interface, making the command clearer to read.

To do this, just write any interface-specific command and put the regexp at the end. For example show rsvp interfaces ae* will show you all the RSVP AE interfaces. Or the below output will show you only the gigabit interfaces that match ge-*/0/* (i.e. over two line cards, but one set of interfaces within the line card);

[email protected]> show interfaces terse ge-*/0/* 

Interface               Admin Link Proto    Local                 Remote
ge-0/0/0                up    down
ge-0/0/1                up    down
ge-0/0/2                up    down
ge-0/0/3                up    down
ge-0/0/4                up    down
ge-0/0/5                up    down
ge-0/0/6                up    down
ge-0/0/7                up    down
ge-0/0/8                up    down
ge-0/0/9                up    down
ge-2/0/0                up    up
ge-2/0/1                up    up
ge-2/0/2                up    up
ge-2/0/3                up    up
ge-2/0/4                up    up
ge-2/0/5                up    down
ge-2/0/6                up    up
ge-2/0/7                up    up
ge-2/0/8                up    down
ge-2/0/9                up    up

I’ve explained this to a few people this week, so figured it would be a good thing to post here in case anyone hasn’t already picked this up.

Hope this helps!