SSL Certificates and NetApp ActiveIQ Unified Manager

I had the opportunity to work a little bit with the new Nvidia SN2100 that NetApp have certified for Metrocluster use as an alternative to the Cisco and Broadcom switches.

The SN2100 comes with Cumulus linux installed and configuring them is more like a normal linux machine than a switch OS like the NXOS so that took some getting used you. Tab completion only works within a command, so one needs to know “net show [tab]” to have a starting point.

I have compiled here some of the commands that I used to check that the switch was correctly configured and ready to use.

There are a few things that I wanted to check before using the switches in production.

Version, hostname, management ip settings, port configs (vlan, mode, mtu, etc), port status (link, speed, etc), rcf file version.

Showing the version can mean a few things in cumulus but normally when we on a Cisco do a “show version” we just want the NXOS version so we can compare it to a support matrix, and that version number can be found either by “cat /etc/lsb-release” or with the hostnamectl command. The hostnamectl is not the obvious place to look, but if we want to check the hostname anyway then it can be a useful place.

ronnie@SWITCH-A1:mgmt:~$ cat /etc/lsb-release
DISTRIB_ID="Cumulus Linux"
DISTRIB_DESCRIPTION="Cumulus Linux 4.4.3"

4.4.3 is the number we need to look up in the support matrix.

ronnie@SWITCH-A1:mgmt:~$ hostnamectl
   Static hostname: SWITCH-A1
         Icon name: computer-desktop
           Chassis: desktop
        Machine ID: 4949e6ec559d507493bbc1eed319e570
           Boot ID: d595aa55efc825e5594eed52f61e1cfad
  Operating System: Cumulus Linux
       CPE OS Name: cpe:/o:cumulusnetworks:cumulus_linux:4.4.3
            Kernel: Linux 4.19.0-cl-1-amd64
      Architecture: x86-64

Here the version is listed as part of the CPE OS Name.

Normally it is pretty easy to find the hostname on a linux machine. It is right in the prompt. But sometimes the prompt is broken, or the system has not been rebooted since it was changed, or you want to put it into a script as a failsafe.

Seeing basic hardware status of psu, fans and temperature can be seen with “net show system sensors”

ronnie@SWITCH-A1:mgmt:~$ net show system sensors
Fan1      (Fan 1                                 ):  OK
Fan2      (Fan 2                                 ):  OK
Fan3      (Fan 3                                 ):  OK
Fan4      (Fan 4                                 ):  OK
PSU1                                              :  OK
PSU2                                              :  OK
Temp1     (Asic Temp Sensor                      ):  OK
Temp2     (Port Ambient Sensor                   ):  OK
Temp3     (Main Board Ambient Sensor             ):  OK
Temp4     (core0 Sensor                          ):  OK
Temp5     (core1 Sensor                          ):  OK
Temp6     (core2 Sensor                          ):  OK
Temp7     (core3 Sensor                          ):  OK

And the status of the LEDS on the front of the switch “net show system leds”

ronnie@SWITCH-A1:mgmt:~$ net show system leds
System: green
           Psu1: green
           Psu2: green
            Fan: green

The dedicated management interface on the SN2100 is eth0 and we can show a lot of information with “net show” like on modern linux.

ronnie@SWITCH-A1:mgmt:~$ net show interface eth0
    Name  MAC                Speed  MTU   Mode
--  ----  -----------------  -----  ----  ----
UP  eth0  11:71:fd:c1:d5:11  100M   1500  Mgmt

IP Details
-------------------------  ----------------
IP Neighbor(ARP) Entries:  1

cl-netstat counters
-------  ------  ------  ------  ------  ------  ------  ------
9597447       0       0       0  669746       0       0       0

  Interface eth0 is up, line protocol is up
  Link ups:       1    last: 2019/05/01 20:10:07.45
  Link downs:     0    last: (never)
  PTM status: disabled
  vrf: mgmt
  index 2 metric 0 mtu 1500 speed 100
  Type: Ethernet
  HWaddr: 10:70:fd:c0:d4:10
  inet6 fe80::1270:fdff:fec0:d410/64
  Interface Type Other
  protodown: off

It will also list a few things we normally don’t care about at first (like netstat counters) but gives a nice easy overview. If we just want the ip an net show interface will also list it, but if multiple address and interfaces has been configured it can be difficult to be sure what interface is for what.

The gateway can be found in /etc/network/interfaces under the interface.

ronnie@SWITCH-A1:mgmt:~$ cat /etc/network/interfaces | grep -A 3 eth0
auto eth0
iface eth0
    vrf mgmt.

NetApp has elected to not configure routing so some of the normal ways to also show gateway like net show route does not work in the default NetApp RCF configuration.

Now we have the basic information needed to management switch. So moving on to the port config. Most of the config we want can be seen with net show interface.

ronnie@SWITCH-A1:mgmt:~$ net show interface
State  Name           Spd   MTU    Mode        LLDP                   Summary
-----  -------------  ----  -----  ----------  ---------------------  -------------------------
UP     lo             N/A   65536  Loopback                           IP:
       lo                                                             IP: ::1/128
UP     eth0           100M  1500   Mgmt                               Master: mgmt(UP)
       eth0                                                           IP:
UP     swp1           100G  9216   Access/L2   clusterA-01 (e3a)      Master: bridge-CL-101(UP)
UP     swp2           100G  9216   Access/L2   clusterA-02 (e3a)      Master: bridge-CL-101(UP)
UP     swp7           100G  9216   Trunk/L2    clusterA-01 (e1a)
UP     swp7.10        100G  9216   Access/L2                          Master: bridge-DR-10(UP)
UP     swp8           100G  9216   Trunk/L2    clusterA-02 (e1a)
UP     swp8.10        100G  9216   Access/L2                          Master: bridge-DR-10(UP)
UP     swp13          100G  9216   BondMember  SWITCH-A2 (swp13)      Master: bond-DR(UP)
DN     swp14          N/A   9216   BondMember                         Master: bond-DR(UP)
UP     swp15          100G  9216   BondMember  SWITCH-B1 (swp15)      Master: bond-CL(UP)
UP     swp16          100G  9216   BondMember  SWITCH-B1 (swp16)      Master: bond-CL(UP)
UP     bond-CL        200G  9216   802.3ad                            Bond Members: swp15(UP)
       bond-CL                                                        Bond Members: swp16(UP)
UP     bond-CL.101    200G  9216   Access/L2                          Master: bridge-CL-101(UP)
UP     bond-DR        100G  9216   802.3ad                            Bond Members: swp13(UP)
       bond-DR                                                        Bond Members: swp14(DN)
UP     bond-DR.10     100G  9216   Access/L2                          Master: bridge-DR-10(UP)
UP     bridge-CL-101  N/A   9216   Bridge/L2
UP     bridge-DR-10   N/A   9216   Bridge/L2
UP     mgmt           N/A   65536  VRF                                IP:
       mgmt                                                           IP: ::1/128

When we have an port listed as .xx like swp8.10 that means port 8 is an trunk port with vlan 10 tagged. We can also see the MTU setting and if the port is up or down. If the port is part of a bond or bridge remember to also check that. The port might be down but the bond can be up, and then at least something is working.

We have ports swp1 and 2. Configured as access ports and ports swp7 and 8 as trunks with vlan 10 tagged.

This is as we would expect if the switch is configured for MetroCluster IP. Ports 1 and 2 are for cluster traffic that uses access ports and ports 7 and 8 are for the MetroCluster iSCSI connections. Remember to check the MetroCluster configuration is using the same vlan settings as the switch. NetApp change the iSCSI vlans in the RCF generator some time ago, but the default created vlan in ONTAP when doing metrocluster configuration has not changed yet.

Port 13, 14, 15 and 16 are ISL / uplink connections to the other switches. 13 and 14 are for local uplink to switch at same site. 15 and 16 are for the other site. They are configured with LACP(=802.3ad) for the bond.

All this config can also be found in the file /etc/network/interfaces some might find it easier to read.

Checking the NetApp RCF file version can be done in kind of around a bout way, not sure how else to check it. But the RCF will configure a banner to be displayed when logging with the SSH. This is configured in sshd_config. This banner will display the RCF version.

ronnie@SWITCH-A1:mgmt:~$ cat /etc/ssh/sshd_config | grep banner
# no default banner path
Banner /etc/banner.txt
Banner /etc/banner.txt
ronnie@SWITCH-A1:mgmt:~$ cat /etc/banner.txt
* NetApp Reference Configuration File (RCF)
* Switch    : SN2100 (direct storage, L2 Networks, direct ISL)
* Filename  : SN2100_v1.0_Switch-A1.txt
* Version   : v1.0
* Date      : Generator: v1.5a_2022-12-09_001, file creation: 2023-03-09, 15:42:18

If this is not as expected, then the RCF has not been applied correctly and I would recommend doing it again.

When troubleshooting a specific link or we can get a lot of information with the first command we used. Net show interface xx and if we add detail to the command we get more then we need.

ronnie@SWITCH-A1:mgmt:~$ net show interface eth0 detail
    Name  MAC                Speed  MTU   Mode
--  ----  -----------------  -----  ----  ----
UP  eth0  11:71:fd:c1:d5:11  100M   1500  Mgmt

IP Details
-------------------------  ----------------
IP Neighbor(ARP) Entries:  1

cl-netstat counters
-------  ------  ------  ------  ------  ------  ------  ------
9610386       0       0       0  672533       0       0       0

  Interface eth0 is up, line protocol is up
  Link ups:       1    last: 2019/05/01 20:10:07.45
  Link downs:     0    last: (never)
  PTM status: disabled
  vrf: mgmt
  index 2 metric 0 mtu 1500 speed 100
  Type: Ethernet
  HWaddr: 11:71:fd:c1:d5:11
  Interface Type Other
  protodown: off

Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 100Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: off (auto)
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

Ethtool Statistics
NIC statistics:
     rx_packets: 9610387
     tx_packets: 672533
     rx_bytes: 997768246
     tx_bytes: 198297704
     rx_broadcast: 7941578
     tx_broadcast: 0
     rx_multicast: 1648795
     tx_multicast: 660487
     multicast: 1648795
     collisions: 0
     rx_crc_errors: 0
     rx_no_buffer_count: 0
     rx_missed_errors: 0
     tx_aborted_errors: 0
     tx_carrier_errors: 0
     tx_window_errors: 0
     tx_abort_late_coll: 0
     tx_deferred_ok: 0
     tx_single_coll_ok: 0
     tx_multi_coll_ok: 0
     tx_timeout_count: 0
     rx_long_length_errors: 0
     rx_short_length_errors: 0
     rx_align_errors: 0
     tx_tcp_seg_good: 9
     tx_tcp_seg_failed: 0
     rx_flow_control_xon: 0
     rx_flow_control_xoff: 0
     tx_flow_control_xon: 0
     tx_flow_control_xoff: 0
     rx_long_byte_count: 997768246
     tx_dma_out_of_sync: 0
     tx_smbus: 0
     rx_smbus: 0
     dropped_smbus: 0
     os2bmc_rx_by_bmc: 0
     os2bmc_tx_by_bmc: 0
     os2bmc_tx_by_host: 0
     os2bmc_rx_by_host: 0
     tx_hwtstamp_timeouts: 0
     tx_hwtstamp_skipped: 0
     rx_hwtstamp_cleared: 0
     rx_errors: 0
     tx_errors: 0
     tx_dropped: 0
     rx_length_errors: 0
     rx_over_errors: 0
     rx_frame_errors: 0
     rx_fifo_errors: 0
     tx_fifo_errors: 0
     tx_heartbeat_errors: 0
     tx_queue_0_packets: 3609
     tx_queue_0_bytes: 402131
     tx_queue_0_restart: 0
     tx_queue_1_packets: 2360
     tx_queue_1_bytes: 237092
     tx_queue_1_restart: 0
     tx_queue_2_packets: 4582
     tx_queue_2_bytes: 703593
     tx_queue_2_restart: 0
     tx_queue_3_packets: 661982
     tx_queue_3_bytes: 194226174
     tx_queue_3_restart: 0
     rx_queue_0_packets: 7929212
     rx_queue_0_bytes: 718752118
     rx_queue_0_drops: 0
     rx_queue_0_csum_err: 0
     rx_queue_0_alloc_failed: 0
     rx_queue_1_packets: 1594679
     rx_queue_1_bytes: 214072200
     rx_queue_1_drops: 0
     rx_queue_1_csum_err: 0
     rx_queue_1_alloc_failed: 0
     rx_queue_2_packets: 2627
     rx_queue_2_bytes: 229561
     rx_queue_2_drops: 0
     rx_queue_2_csum_err: 0
     rx_queue_2_alloc_failed: 0
     rx_queue_3_packets: 83869
     rx_queue_3_bytes: 26272819
     rx_queue_3_drops: 0
     rx_queue_3_csum_err: 0
     rx_queue_3_alloc_failed: 0

In addition to the LLDP info that is shown in “net show interface” we can view more information with this command “sudo lldpcli show neighbors ports [port #]”

ronnie@SWITCH-A1:mgmt:~$ sudo lldpcli show neighbors ports swp1
LLDP neighbors:
Interface:    swp1, via: LLDP, RID: 5, Time: 0 day, 12:01:50
    ChassisID:    mac d1:30:ea:a1:c0:5a
    SysName:      clusterA-01
    SysDescr:     AFF-A400, NetApp Release 9.11.1P4: Thu Oct 27 07:37:48 EDT 2022
    Capability:   Station, on
    PortID:       ifname e3a
    TTL:          121
ronnie@SWITCH-A1:mgmt:~$ sudo lldpcli show neighbors ports swp15
LLDP neighbors:
Interface:    swp15, via: LLDP, RID: 8, Time: 0 day, 12:00:41
    ChassisID:    mac 11:71:fd:c1:d5:a9
    SysName:      SWITCH-B1
    SysDescr:     Cumulus Linux version 4.4.3 running on Mellanox Technologies Ltd. MSN2100
    MgmtIface:    2
    Capability:   Bridge, on
    Capability:   Router, on
    PortID:       ifname swp15
    PortDescr:    Intra-Cluster ISL Port
    TTL:          120

Depending on the equipment in the other end, we get different information. I really like that we can see the management ip of the switch here.