18 Jun

Automated scripts can send commands faster than RP can process

Juniper automation with Ansible

When I was writing one Ansible playbook I faced an interesting situation. We all keep forgetting that automated tasks are executed faster than from CLI. Ane we do not take it into consideration when we write playbooks. Time is only a problem when playbook execution takes to much time.

On Juniper SRX you can enable VPN debugging using the traceoptions feature on IKE and IPSec processes. By default, it stores logs of all configured VPNs on the device in either /var/log/kmd file or one specified in traceoptions configuration. JunOS 11.4R3 brings additional enhancement to limit debugging to single VPN tunnel specified by local and peer IPs. You can turn it on by request security ike debug-enable command. This is very handy because in most cases you want to troubleshoot just the not operational connection.

There is a slight difference between the output of those commands in the log file. If you use the traceoptions you will see pretty much standard Juniper log output

[Mar 25 13:12:12]IPSec SA done callback called for sa-cfg VPN-V201 local:10.0.201.3, remote:10.0.201.4 IKEv1 with status Timed out

If you decide to perform troubleshooting using the second method you will get output like that

[Mar 25 13:20:12][10.0.201.3 <-> 10.0.201.4] IPSec SA done callback called for sa-cfg VPN-V201 local:10.0.201.3, remote:10.0.201.4 IKEv1 with status Timed out

The log now contains both peers IP addresses. It is really handy for my playbook. Because in one of the tasks I need to get specific information from the log output I can easily identify interesting lines using the peer IP addresses.

The Playbook

There are two tasks in my playbook divided by 60 seconds pause. In the first task, I send set of three commands to the router using the juniper_junos_command module here but it works the same in the one available in Ansible. I send three commands to save the time, and also provide the variables as a parameter – I gather them in previous tasks.

    - name: Prepare and start VPN debugging on remote device
      juniper_junos_command:
        commands:
          - "request security ike debug-disable"
          - "clear log VPN"
          - "request security ike debug-enable local {{ sa_tunnel_local_gateway }} remote {{ sa_tunnel_remote_gateway }}"
        format: xml

Then after one minute pause required to generate the logs I run another task where I simply read the log file via CLI command.

- name: Collect VPN logs
  juniper_junos_command:
    commands:
      - "show log VPN"
    format: xml
  register: result_VPN_log_xml

The problem

If you run those commands in this order from CLI you will get logs from troubleshooting the single VPN specified by IP parameters. But if you execute them from Ansible playbook you would not get the peer-specific logs output. It looks like the request command is ignored by the router. If you enable commands logging on the router you will see those from netconf as well. You can also check the debugging status directly from the router CLI. If no peer-specific debugging is running you will get following output

show security ike debug-status
Enabled
flag: all
level: 5

When the debug is running you will get information about its level and the peer’s addresses

show security ike debug-status
Enabled
flag: all
level: 7
Local IP: 10.0.201.3, Remote IP: 10.0.201.4

Because the delivery of command itself was not a problem then one of the workarounds could be a playbook, where I send only one command in each task. This workaround is working but it is not optimal. Remember that connection to the router is opened for each task and closed when the task completes. It takes time and the more command you want to send the longer it takes to finish the playbook.

Don’t send commands to fast

I asked JTAC for help in this case. It happened not to be standard customer problem, it was verified in their lab and at some point, the Engineering team was asked for support. I received the answer I like the best – “it is not a bug, it is a feature”!

Because there is no output when you execute any of the three commands from the first task the Ansible module sends the next command almost without the gap after the previous one. As Junos does not acknowledge that it has finished the first request command, the second comes in and gets ignored as the first one is not finished yet. Remember, you can never issue commands via CLI as fast as you can send them from the script.

I still want to send multiple commands in one task to save time, so here is other way to mittigate the problem

    - name: Prepare and start VPN debugging on remote device
      juniper_junos_command:
        commands:
          - "request security ike debug-disable"
          - "clear log VPN"
          - "show system uptime"
          - "request security ike debug-enable local {{ sa_tunnel_local_gateway }} remote {{ sa_tunnel_remote_gateway }}"
        format: xml

To slow down the task a little I put additional command before the ignored request command. It does not take much time to execute it and it generates the output that I simply ignore.

There is no documentation of how long the gap between commands should be but it would differ depending on platform.

 

04 May

How to setup VPN between Juniper SRX and AWS Cloud

I’ve said during several conferences where I had a privilege to be a speaker that clouds are one of the futures of computing along with DevOps/SysOps and Machine Learning. But there is no computing if you don’t have the data to compute or you have no way to send it to the cloud in reliable and secure way or you don’t have cloud infrastructure to perform computation. That’s why we need to take a look how to setup VPN between Juniper SRX and AWS Cloud.

I think that hybrid cloud will be the model how many of computer system will work in net few years. Private clouds are not scalable and public clouds cannot address all needs of current systems. So hybrid mode is a solution. But it requires reliable and easy to setup and maintain ways to connect on-premise resourced to public cloud. The technology is here and it’s called VPN.

Read More

21 Oct

Juniper error messages that says nothing

I wrote in the past that sometimes error messages are completely misleading and not connected to the problem that is blocking changes commit. Here is another example, I’m leaving it here because Google was not helpful on this 😉

I’ve been trying to configure DHCP server on SRX, firmware 12.1X46-D60.4, using the new approach that support both IPv4 and IPv6. DHCP parameters are now defined under access section

[email protected]# show access address-assignment pool LAN 
family inet {
    network 192.168.100.0/24;
    range Dynamic {
        low 192.168.100.101;
        high 192.168.100.140;
    }
    dhcp-attributes {
        maximum-lease-time 86400;
        domain-name lan;
        name-server {
            192.168.100.1;
        }
        router {
            192.168.100.1;
        }
    }
    host HOST1 {
        hardware-address d1:51:99:37:4d:79;
        ip-address 192.168.1.2;
    }
    host HOST2 {
        hardware-address 2d:f0:e2:51:74:55;
        ip-address 192.168.1.10;
    }
    host HOST3 {
        hardware-address d1:51:99:37:4d:79;
        ip-address 192.168.1.5;
    }
}

Attempt to commit the change result in error message

[email protected]# commit check 
error: Check-out failed for General authentication process (/usr/sbin/authd) without details
error: configuration check-out failed

Hmm.. yes, so the problem is…. no, that’s not the right guess.

The error message is not really helpful because the problem is that for two static assignments same MAC address was specified. Yes, error message was really helpful in this case….

25 Jan

No LACP on Juniper SRX650 built-in ports

Quite a surprise – I was configuring etherchannel on SRX650 to provide redundancy between SRX and EX switch. Quite natural is to use LACP as a control protocol for that. Unfortunately SRX650 does not support LACP on built-in ports. It is supported only on extension cards. You can see LACP counters of received LACP packets increasing but none are sent. That means if you want to build etherchannel with built-in interface (in example in case of extension module failure) you have to build bundle without LACP. It’s not the way I prefer…..

01 Jan

Prefix exports between routing-instances on Juniper

Juniper devices are quite flexible when it comes to routing tables and exporting prefixes between them, but all rib groups mechanism is not well explained in documentation. In general Juniper documentation is not really good but well….

Let’s first define two routing instances

routing-instance {
    CUSTOMER1 {
        instance-type virtual-router;
        interface ge-0/0/2.20;
    }
    CUSTOMER2 {
        instance-type virtual-router;
        interface ge-0/0/2.21;
    }
}

By default those two routing tables are totally separated so customers can’t communicate with each other.

CUSTOMER1.inet.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.1.0/27       *[Direct/0] 4w2d 10:52:08
                    > via ge-0/0/2.20
10.0.1.28/32      *[Local/0] 4w2d 10:52:08
                      Local via ge-0/0/2.20

CUSTOMER2.inet.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.2.0/27       *[Direct/0] 4w2d 10:52:08
                    > via ge-0/0/2.21
10.0.2.28/32      *[Local/0] 4w2d 10:52:08
                      Local via ge-0/0/2.21

Now if we want to let them communicate with each other we have to export prefixes between routing tables. To do that we have to use rib-groups and import/export policies.

routing-instance {
    CUSTOMER1 {
        instance-type virtual-router;
        interface ge-0/0/2.20;
        routing-options {
            interface-routes {
                rib-group inet CUSTOMER1-RG;
            }
        }
    }
}

for first customer we define rib-group for all routes directly connected. To make things little easier let’s say it’s just an identifier for prefixes of directly connected routes. Now we have to define export policies

routing-options {
        CUSTOMER1-RG {
            import-rib [ CUSTOMER1.inet.0 CUSTOMER2.inet.0 ];
        }
    }
}

What we see above is import-policy defined for for rib-group CUSTOMER1-RG. We tell the routers to import matching prefixes (in this case matching only CUSTOMER1-RG) from CUSTOMER1.inet.0 routing table to CUSTOMER2.inet.0 routing table. We can specify more destination routing tables within this command.

CUSTOMER2.inet.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.1.0/27       *[Direct/0] 4w2d 10:56:08
                    > via ge-0/0/2.20
10.0.1.28/32      *[Local/0] 4w2d 10:56:08
                      Local via ge-0/0/2.20
10.0.2.0/27       *[Direct/0] 4w2d 10:56:08
                    > via ge-0/0/2.21
10.0.2.28/32      *[Local/0] 4w2d 10:56:08
                      Local via ge-0/0/2.21

So now we have CUSTOMER1 prefixes in CUSTOMER2 routing table. To make configuration all working we have to perform same export from CUSTOMER2 to CUSTOMER1 routing-instance using same configuration way.