My blog; no-one else's

Category: Uncategorized (Page 1 of 2)

10G Ethernet NIC in a HP Microserver N40L

I’m thinking of moving my storage for my NAS (N40L) from DAS to iSCSI. My current set-up is trivial (and not performant): the N40L has 4 local SATA drives and an LSI2008 in IT mode connected to 6 SATA drives.

I think the current bottleneck is the controller:

# dd if=/dev/zero of=a bs=8192 count=1310720
1310720+0 records in
1310720+0 records out
10737418240 bytes transferred in 68.962335 secs (155699749 bytes/sec)

These results are targeting the 6 drives in a ZFS RAIDZ pool.

Anyway, by moving the pool to iSCSI I will be introducing IP and Ethernet to the transport, with 10G Ethernet a must. As a quick test I installed a cheap Intel 10G NIC into a spare N40L and connected it directly to a Cisco UCS with 10G with a twinax cable. I booted both machines from live USB OSes: FreeBSD 13 on the N40L and Ubuntu 20 Server on the UCS. FreeBSD 10 would hang at random times with the 10G NIC installed in the N40L nor would it recognise the Cisco 10G NIC in the UCS,

I installed iPerf3 on each machine, bumped the MTU on FreeBSD to 9000 (crazy that it defaults to 1500 for a 10G NIC) and ran a quick test. Here’s the results:

N40L:

root@:~ # iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 10.30.1.1, port 56382
[  5] local 10.30.1.2 port 5201 connected to 10.30.1.1 port 56384
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   1.00-2.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   2.00-3.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   3.00-4.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   4.00-5.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   5.00-6.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   6.00-7.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   7.00-8.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   8.00-9.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   9.00-10.00  sec  1.15 GBytes  9.90 Gbits/sec
[  5]  10.00-10.00  sec  1.32 MBytes  9.49 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec                  receiver

UCS:

root@ubuntu-server:~# iperf3 -c 10.30.1.2
Connecting to host 10.30.1.2, port 5201
[  5] local 10.30.1.1 port 56384 connected to 10.30.1.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.15 GBytes  9.91 Gbits/sec    0   1.39 MBytes
[  5]   1.00-2.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.39 MBytes
[  5]   2.00-3.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.39 MBytes
[  5]   3.00-4.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.39 MBytes
[  5]   4.00-5.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.39 MBytes
[  5]   5.00-6.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.39 MBytes
[  5]   6.00-7.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.39 MBytes
[  5]   7.00-8.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.39 MBytes
[  5]   8.00-9.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.39 MBytes
[  5]   9.00-10.00  sec  1.15 GBytes  9.90 Gbits/sec    0   1.39 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec                  receiver

iperf Done.

When reversing the direction the results were the same.

A bit later I ran some more tests. This time iSCSI with the target being the UCS server and the initiator being the N40L. The UCS is running ESXi 6.7 passing through the HBA to a FreeBSD 13.1 VM:
mpt0 Adapter:
Board Name: SAS3444
Board Assembly:
Chip Name: C1068E
Chip Revision: UNUSED
RAID Levels: none

The 10Gb adapter in the UCS is owned by ESXi and the VM is connected using a standard vSwitch. The N40L (also running FreeBSD 13.1) has an Intel 10Gb NIC. Both NICs are configured for 9000B MTU (verified during tests with tcpdump).

Here are some raw network tests similar to those above. The UDP results require further examination.

VM -> N40L (TCP)
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 11.5 GBytes 9.88 Gbits/sec receiver

VM -> N40L (UDP)
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 9.31 GBytes 8.00 Gbits/sec 0.000 ms 0/1115968 (0%) sender
[ 5] 0.00-10.00 sec 7.85 GBytes 6.74 Gbits/sec 0.007 ms 174995/1115968 (16%) receiver

N40L -> VM (TCP)
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 11.4 GBytes 9.81 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 11.4 GBytes 9.82 Gbits/sec receiver

N40L -> VM (UDP)
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 7.37 GBytes 6.33 Gbits/sec 0.000 ms 0/883094 (0%) sender
[ 5] 0.00-10.00 sec 2.68 GBytes 2.30 Gbits/sec 0.014 ms 562078/883064 (64%) receiver

Here is the VM writing to a 7K SAS drive on the HBA locally (no network involved):

dd if=/dev/zero of=/mnt/a bs=8192 count=1310720
1310720+0 records in
1310720+0 records out
10737418240 bytes transferred in 59.083653 secs (181732472 bytes/sec)

That’s a write speed of about 1.35GB/s. So we know the VM can access its local drives at that speed. I was hoping N40L hitting the same drive over iSCSI would get similar results:

dd if=/dev/zero of=/mnt/a bs=8192 count=1310720
1310720+0 records in
1310720+0 records out
10737418240 bytes transferred in 108.958877 secs (98545603 bytes/sec)

Unfortunately not. The write speeds over iSCSI were almost half of the local write speeds, 751MB/s.

When I get the time I will move the HBA to the N40L and test the local write speeds on that machine.

Juniper MX10003: Port speeds and ASICs

Each MX10003 comes with two MPC slots, each with two MIC slots. One slot is pre-populated with a fixed, 6-port PIC, while the other slot is free to accept a MIC. Keen-eyed readers will point out that a PIC goes into a FPC, and Juniper certainly don’t make life easy with their terminology. What the marketing team calls an MPC the Junos team calls an FPC; the same applying to MIC and PIC. These interpretations may not be perfect but will serve for this article. From here on I will be using the FPC/PIC nomenclature unless other terms better apply.

FPCs, PICs and ports use 0-based indexes, with FPCs numbered from bottom-up, PICs from right-to-left, and ports from top-to-bottom, left-to-right. The bottom-most FPC and right-most PIC provides ports (et|xe)-0/0/0 through 0/0/5 with ports numbered left-to-right. The top-most FPC and left-most PIC (when populated with a MIC) provides ports (et|xe)-1/1/0 through 1/1/11 with ports numbered left-to-right with odd numbers at the top, even at the bottom.

Each FPC provides three Eagle (EA) ASICs with ports from both PICs in the same FPC assigned to them. The diagram below shows the port assignments.

Each ASIC has four MACs* of which there are two types. Avoiding the terminology, let’s call these two types of MAC type-1 and type-2. Type-1 MACs support 100G-speed interfaces only, while type-2 MACs support all of the supported interface speeds: 100G, 40G, 4x10G and 4x1G, the latter two being channelised and requiring a fan-out cable to deliver each interface. The number of ports that each MAC type differs, too. Type-1 MACs support a single (100G) interface while type-2 MACs support a single 100G, or three 40G, or 12 10G, or 12 1G interfaces. The four MACs provided by each ASIC are in pairs: two type-1 MACs are available, as are two type-2 MACs.

So for each port group as shown above there are four MACs, two of type-1 and two of type-2. Interfaces are assigned to MACs in the following manner:

  1. The first two 100G interfaces are assigned to the two type-1 MACs
  2. Further 100G interfaces are assigned to the the type-2 MACs, one interface per MAC. Therefore if four 100G interfaces are configured in a port group no further interfaces are possible (all four MACs have been used)
  3. If any type-2 MACs are available each one can provide three 40G interfaces (or three 4x10G or three 4x1G).

One other consideration to this is the pic-mode. Each FPC operates in one of two modes: pic-mode and port-mode. I believe both PICs in an FPC must be in the same mode. In pic-mode, each port on a given PIC operates at the same speed and ports within the same group on different PICs can belong so the same MAC. In port-mode port speeds are independent, however ports on different PICs CANNOT belong to the same MAC.

MX10003 PFE Eagle ASIC MACs

In pic-mode the following configuration for a given port group is possible:

  1. PIC 0, ports 0 and 1 running at 40G (or 4×10 or 4×1). These consume two of the three 40G interfaces that one of the two type-2 MACs provides. There is now a single 40G port available on the first type-2 MAC, and three available on the second type-2 MAC.
  2. PIC 1, ports 0 – 3 running at 40G (or 4×10 or 4×1). Port 0 consumes the last interface on the first type-2 MAC, while port 1 -3 fully consume the second type-2 MAC.
  3. Both type-1 MACs are available and can be used for 0, 1 or 2 100G interfaces.

Note that the allocations above, PIC permitting, can be shuffled around.

In port-mode, however, the above configuration is not possible. This is due to the limitation that MACs cannot be shared between PICs in port-mode:

  1. PIC 0, ports 0 and 1 running at 40G. Two of the three 40G interfaces on the type-2 MAC are now consumed, the third being unavailable to PIC 1 (and therefore wasted). The remaining type-2 MAC can provide three 40G interfaces.
  2. PIC 1, ports 0 – 2 running at 40G. The second type-2 MAC is now fully used.
  3. Both type-1 MACs are available and can be used for 0, 1 or 2 100G interfaces.
  4. The sixth port cannot be used.

Juniper helpfully hides this complexity in a port checker tool (https://apps.juniper.net/home/port-checker/index.html) but hopefully the information above helps inform what happens in the background. The information in this post is mostly from https://kb.juniper.net/InfoCenter/index?page=content&id=KB35254&cat=MX10003&actp=LIST.

*I’m not sure what this initialism is short for.

Cisco WLC, MAC authentication and Passthrough WebAuth

While Passthrough WebAuth upon MAC authentication failure is not supported on Cisco WLCs, it is easy to workaround this by doing the following.

The first step is to create a user to be used as a dummy account that the web-form will silently submit when the user agrees. This user should be created depending on where you authentication happens (radius or local).

The second step is to modify the passthrough HTML file (login.html, for example) to include a hidden <div> that contains the dummy user’s login details:

<div id="hiddenForm" style="display: none;">
<h4 align=center><font color="#000000">Username<input type="TEXT" name="username" SIZE="25" MAXLENGTH="63" VALUE="webauthdummy"></font></h4>
<h4 align=center><font color="#000000">Password<input type="Password" name="password" SIZE="25" MAXLENGTH="63" VALUE="webauthdummy"></font></h4>
</div>

Just make sure the above is within the <form> block. The above can be simplified; I got lazy and just modified Cisco’s webauth examples.

<h4 align=center><font size="5" color="#000000"><input type="button" name="Submit" value="ACCEPT" class="button" onclick="submitAction();"></h4></font><p></p>

As above I also modified the submit button slightly to say “ACCEPT” as the passthrough form does.

And that’s it. You can now set the “On MAC Filter failure” when using the Web Policy Layer 3 Security feature.

FreeRADIUS, MySQL and Cisco WLC AAA Override

Here’s how to get AAA Override working on a Cisco WLC using FreeRadius backed with MySQL:

  • Enable AAA Override on the WLAN on the WLC
  • Create a Dynamic Interface on the WLC with a vLAN and IP address
  • Add the following to the radreply MySQL table:
idusernameattributeopvalue
nusernameTunnel-Type=VLAN
nusernameTunnel-Medium-Type=IEEE-802
nusernameTunnel-Private-Group-Id=vlanId
radreply table

The NAS (WLC) will receive these parameters in the Radius reply packet when user username authenticates and will place the user on the interface with the corresponding vLAN.

Cisco password type 7 Vigenère cipher seed/keyword

If you were ever wondering what the seed (keyword) is for Cisco’s password encryption type 7 (which uses the Vigenère cipher), in ASCII it’s:

dsfd;kfoA,.iyewrkldJKDHSUBsgvca69834ncxv9873254k;fg87

or in hex:

0x64, 0x73, 0x66, 0x64, 0x3b, 0x6b, 0x66, 0x6f, 0x41, 0x2c, 0x2e,
0x69, 0x79, 0x65, 0x77, 0x72, 0x6b, 0x6c, 0x64, 0x4a, 0x4b, 0x44,
0x48, 0x53, 0x55, 0x42, 0x73, 0x67, 0x76, 0x63, 0x61, 0x36, 0x39,
0x38, 0x33, 0x34, 0x6e, 0x63, 0x78, 0x76, 0x39, 0x38, 0x37, 0x33,
0x32, 0x35, 0x34, 0x6b, 0x3b, 0x66, 0x67, 0x38, 0x37

Private vLANs

Having just implemented private vLANs on a Cisco 3750X switch I thought I’d share some findings.

A private vLAN configuration consists of a primary vLAN, zero or one isolated vLANs, and zero or more community vLANs.

When associated with each other, primary, isolated and community vLANs limit the switch’s forwarding scope for frames.

For a normal vLAN the forwarding scope comprises all ports to which that vLAN is assigned.

For a pvLAN the scope changes:

  • the primary vLAN is a normal vLAN with a normal scope; frames are forwarded to any or all ports to which the primary vLAN is assigned
  • an isolated vLAN is a point-to-multipoint topology, much like the Frame Relay NBMA, where the scope is limited to a single port to which the isolated vLAN is assigned and all associated primary vLAN ports
  • a community vLAN’s scope is a hybrid of that of primary and isolated vLANs; it comprises all ports to which the community vLAN is assigned and all associated primary vLAN ports

Treatment of broadcast and unknown destination frames received on a given port is as follows:

  • primary vLAN: frames are forwarded to:
    • all other ports in the primary vLAN, all ports in the isolated vLAN, and all ports in all community vLANs
  • isolated vLAN: frames are forward to:
    • all ports in the primary vLAN, while having their vLAN translated to the primary vLAN number
  • community vLANs: frames are forwarded to:
    • all ports in the same community vLAN and all ports in the primary vLAN, again having their vLAN translated to the primary vLAN number

Another way to look at private vLANs is that they provide vLAN translation on ports depending on which way frames are flowing:

From/ToPrimaryIsolatedCommunity
PrimaryNo translationP->IP->C
IsolatedI->PNot permittedNot permitted
CommunityC->PNot permittedNo translation
pvLAN translation

You can see that ports, regardless of their assigned pvLAN type, receive frames with their own vLAN id.

An interesting consideration with pvLANs is with layer 3 SVIs and pvLAN trunks with a router connected. If there is no IP ACL on the SVI/router port denying such traffic, any host, regardless of the vLAN type assigned to its port, can reach any other host in the same private vLAN configuration, simply by creating a static ARP entry where the destination host’s IP is mapped to the SVI/router port’s MAC address.

This is because any host can reach a router on a primary vLAN (using its MAC address), which then is able to reach any other host within the pvLAN.

Using LetsEncrypt certificates with Cisco WLC WebAuth

Assuming you’ve got a nice fresh certificate from LetsEncrypt and are in the directory where it, your key, and the LE root certificate1 (TrustID X3) lives:

cat domain.com.cer ca.cer dst.cer > fullchain-plus-root.cer
openssl pkcs12 -export -in fullchain-plus-root.cer -inkey domain.com.key -out domain-fullchain-key.p12 -clcerts -passout pass:wlc
openssl pkcs12 -in domain-fullchain-key.p12 -out domain-fullchain-key.pem -passin pass:wlc -passout pass:wlc
cp -a domain-fullchain-key.pem /tftpboot/certs/
transfer download mode tftp
transfer download datatype webauthcert
transfer download serverip 
transfer download path /certs/
transfer download filename domain-fullchain-key.pem
transfer download certpassword wlc
debug pm pki enable
transfer download start

Check the logs to make sure the import was successful. If so you will be reminded to reboot. If so:

reset system

Cisco ACLs and the ‘established’ keyword

While writing software to convert Cisco ACLs to VyOS’s firewall syntax I got to wondering what the ‘established’ keyword meant on TCP ACLs. Although my Internet facing ACL is connected to CBAC (and therefore stateful), my inter-vLAN ACLs are stateless, and so the ‘established’ keyword comes up a lot. Additionally, I have yet to successfully get the VyOS stateful firewall working so I needed to convert the Cisco ‘established’ to the VyOS equivalent.

I wrote a script that called nmap with all 64 combinations of TCP flags and send the traffic from a host to a destination in another vLAN across my IOS router. The destination ran tcpdump to collect the data. Firstly the script ran without any access-list to verify that the destination saw all 64 packets, and secondly with an access-list with the ‘established’ keyword set.

The results were:

RSTRST,PSHRST,URGRST,PSH,URG
RST,FINRST,FIN,PSHRST,FIN,URGRST,FIN,PSH,URG
RST,ACKRST,ACK,PSHRST,ACK,URGRST,ACK,PSH,URG
RST,FIN,ACKRST,FIN,ACK,PSHRST,FIN,ACK,URGRST,FIN,ACK,PSH,URG
ACKACK,PSHACK,URGACK,PSH,URG
ACK,SYNACK,SYN,PSHACK,SYN,URGACK,SYN,PSH,URG
ACK,FINACK,FIN,PSHACK,FIN,URGACK,FIN,PSH,URG

Of 64 possible combinations 28 are permitted. Note that the second, third and fourth columns are the first column plus ‘PSH’, ‘URG’, and ‘PSH,URG’

The table can be summarised as:

  • ‘RST’ with all flags except ‘SYN’ (16 combinations)
  • ‘ACK’, ‘ACK,SYN’, ‘ACK,FIN’ with ‘PSH’,’URG’, and ‘PSH,URG’

Apple Watch and IPSEC/ESP

Something I noticed when my Apple Watch was associated to the wrong SSID (and was placed on a vLAN different to that of its paired iPhone) was the repeated attempts it was making to set up an IPSEC connection to the iPhone.

I saw this in the logs of the firewall that sits between the vLANs:

2019-10-29T16:56:20+11:00 10.1.1.67 kernel: [5242514.662174] [ALEXT-Zone-Media-340-R] IN=eth0.6 OUT=eth0 MAC=00:50:56:00:00:00:08:f4:ab:00:00:00:08:00 SRC=192.0.2.1 DST=192.0.2.2 LEN=120 TOS=0x00 PREC=0x00 TT
 L=63 ID=64399 PROTO=ESP SPI=0xabd8402

Here the Watch is 192.0.2.1 and the iPhone 192.0.2.2

Apple’s iOS Security Guide https://www.apple.com/business/docs/site/iOS_Security_Guide.pdf does not mention an IPSEC connection specifically but does discuss secure WiFi connections in case Bluetooth is unavailable.

avahi-daemon and incrementing Apple device names

For a long time on my network I noticed my various Apple devices incrementing their hostname (specifically the name advertised with mDNS/mdns-sd – as appears in the ‘Sharing’ section of ‘System Preferences’). I lazily put this down to a poorly configured (by me) WLC and/or avahi-daemon running on my FreeBSD NAS and moved on to bigger problems.

My recent purchase of an AppleTV and it’s placement on a different subnet to my wireless clients (iPhones, etc.) meant getting cross-subnet mDNS working, and so I started playing with the WLC, the Avahi daemon, as well the mDNS service routing on my Cisco 1941.

Hoping the Cisco IOS router would solve everything I removed all of the interesting mDNS configuration from the WLC, stopped the avahi-daemon, and concentrated on getting the router mDNS service working. After much Googling and perusal of configuration guides and CiscoLive! PDFs I applied what I thought was an appropriate configuration. Unfortunately the implementation proved to be quite buggy (although it appeared to have potential). I may revisit it with a newer IOS and some spare time.

Something common to Cisco’s IOS implementation of an mDNS cache and avahi-daemon (and possibly mDNS on the WLC) is that they can act as mDNS advertisement caches, responding on behalf of hosts on subnets other than that on which the request arrived (I assume they ignore requests seen on the same interface as the searched-for service/hosts). Cisco also mention that cache interaction with Apple’s sleep proxies can behave badly and helpfully describe how to block the sleep proxy mDNS service advertisements into the cache.

Another common feature is mDNS relaying across subnets, whereby advertisements and requests are simply forwarded across participating interface. In a CiscoLive! presentation Cisco warns against implementing this – I guess to avoid good-old broadcast loops should a second relay be present. Who knows whether IP TTL would even help here).

Back to the incrementing hostnames. While watching avahi-browse -a, tcpdump, and repeated restarts of the AppleTV I noticed a reliable +1 to the AppleTV Airpay name (per restart). Interestingly was that avahi-browse was not showing any withdrawal of the AppleTV’s services as it shutdown. So when it came back up it checked whether any other device was using the proposed name, to which request avahi-daemon helpfully replied. Of course, the AppleTV had to choose a different name, and incremented it by one.

The answer was to disable the cache, effectively limited the cache’s size to zero. The line “cache-entries-max=0” in the configuration file and a restart was all that was needed. Turning on the relaying with “enable-reflector=yes” and I was off the races.

A couple of notes:
the avahi-browse command only talks to the avahi-daemon, and only via DBUS, so if the daemon has yet to start, or it’s configured with “enable-dbus=no” avahi-browse will be useless
all of my devices talk happily to each other, although I had a couple of quirks. 1) I could not get my Mac and the AppleTV to pair using Apple Configurator 2 – I would be prompted with a password to enter into the Mac and once entered the AppleTV would say it was paired, but the Mac would time out with a “pairing failed (49)” error. Though it irked me so, I gave up and associated my Mac with the SSID on the same vLAN as the AppleTV and instantly they paired. Upon moving my Mac back to original vLAN the pairing remained. 2) sometimes the remote function on my iPhoneXS would allow gestures to control the AppleTV, but none of the buttons worked – sometimes that behaviour was reversed. I have no idea what caused/fixed it.

« Older posts

© 2022 Scott's Blog

Theme by Anders NorenUp ↑