Scott's Blog

The on-board Intel NICs support booting from an iSCSI target LUN, however they are not flashed to do so; rather they are ready for PXE booting.

The iSCSI configuration in the BIOS settings does not apply to iSCSI on these NICs. I’m not sure but maybe if you’re doing iSCSI via the MLOMs or 10GB cards those BIOS settings might come into play.

The first thing to do is re-flash the NICs to boot iSCSI. Using https://community.cisco.com/t5/data-center-and-cloud-knowledge-base/how-to-iscsi-boot-with-intel-i350-adapters-on-ucs-servers/ta-p/3123679 as the basis:

Download ucs-cxxx-utils-efi.3.1.3.iso from the Cisco website. It’s found under your specific C-series server. For example:

• Downloads Home
• Servers – Unified Computing
• UCS C-Series Rack-Mount Standalone Server Software
• UCS C220 M4 Rack Server Software
• Unified Computing System (UCS) Utilities
• EFI 3.1(3)

Mount the ISO on your local machine (not the server) and copy the file /Network/Intel/BootUtil/BootUtil-v1.6.57.0.zip to a folder somewhere. Extract it. Copy the file BootUtil-v1.6.57.0/BOOTUTIL64E.EFI to a USB stick (FAT or FAT32 formatted – maybe even ExFAT).

Now boot the UCS server into the EFI shell. Insert the USB and navigate to the USB: The map command is your friend. On my system the USB was automatically mapped to fs0.

You need to choose which NIC will perform the iSCSI mount. The thread above mentions people having issues with the iSCSI breaking once ESXi boots and steals the port for management. I didn’t have this issue, although I was using port 2 (only). One oddity is the IP address that the NIC is given via DHCP changes once ESXi boots. This isn’t an issue – but the initial iSCSI session drops and gives way to a second session when ESXi get an IP.

Anyhoo, now that you’ve booted EFI and mounted the USB, and chosen which NIC (or both) to configure for iSCSI, you need to play with BOOTUTIL64E.EFI:

Without any options you’ll get the state of both NICs:

The example configures port one (NIC 1) as the primary iSCSI boot NIC:

As I mentioned I ended up using port 2 so the following examples will show that. Once that’s done, unplug the USB stick and reboot. This time during the boot process you’ll get an opportunity to hit Ctrl-D to configure iSCSI on the NIC. When you hit Ctrl-D you’ll be presented with the Intel iSCSI Remote Boot utility:

Choose the primary NIC (or both, one after the other, if you have configured primary and secondary NICs). In the next window you’ll choose “iSCSI Boot Configuration” to configure IP and iSCSI initiator and target details. My setup used DHCP for the IP configuration. Since I also use authentication I also needed to configure the “iSCSI CHAP Configuration” page.

Save changes and exit, and you see the initiator attempt to connect to the iSCSI target.

As you can see I initially configured the incorrect target name. You get very little time to see such messages and the boot processes continues. Some screen recording of the KVM session and looking at the iSCSI target logs confirmed by initial mistake. The following log entry on the target confirms my initial misconfiguration on the iSCSI initiator:

ctld[12943]: a.b.c.d (iqn.1970-06.initiator): requested target "iqn.1970-06.target" not found

Once fixed, I see no errors in the ctld logs and I see a session in ctladm islist:

23 ip-addr-1 iqn.1970-06.initiator iqn.1970-06.target

And once ESXi boots (and the IP changes):

26 ip-addr-2 iqn.1970-06.initiator iqn.1970-06.target

When installing ESXi you’ll see a Remote disk (the iSCSI disk) on which you can install it.

I use NFS to mount the ESXi installer ISO via the CIMC with:

map-nfs esxi-installer <nfs-server>:</path/to/ISO/directory/> VMware-VMvisor-Installer-7.0U2-17630552.x86_64.iso ro

Other useful links:
https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/6-5/vsphere-installation-and-setup-6-5/installing-and-setting-up-esxi/installing-esxi/installing-esxi-interactively/install-esxi-on-a-software-iscsi-disk.html

Router configuration

Cisco IOS XE Software, Version 17.03.04a
cisco CSR1000V (VXE) processor (revision VXE) with 2071829K/3075K bytes of memory.
Processor board ID 9WL1PRT9H0W
Router operating mode: Autonomous
4 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
3978432K bytes of physical memory.
6188032K bytes of virtual hard disk at bootflash:.

crypto ikev2 keyring CRYPTO-IKEv2-KEYRING-01
 peer <freebsd-name>
  address <freebsd-ip>
  identity fqdn <freebsd-name>
  pre-shared-key <psk>

crypto ikev2 profile CRYPTO-IKEv2-PROFILE-02
 match identity remote address <freebsd-ip> 255.255.255.255
 match identity remote fqdn <freebsd-name>
 identity local fqdn <router-name>
 authentication remote pre-share
 authentication local pre-share
 keyring local CRYPTO-IKEv2-KEYRING-01

crypto ipsec transform-set CRYPTO-IPSEC-TS-01-TUN esp-aes 256 esp-sha256-hmac
 mode tunnel
crypto ipsec profile CRYPTO-IPSEC-PROFILE-02
 set transform-set CRYPTO-IPSEC-TS-01-TUN

crypto ikev2 nat keepalive 30
crypto ikev2 dpd 10 2 periodic

interface Tunnel0
 ip address <tunnel-net>.1 255.255.255.252
 ip mtu 1400
 tunnel source <router-IP>
 tunnel mode ipsec ipv4
 tunnel destination <freebsd-ip>
 tunnel protection ipsec profile CRYPTO-IPSEC-PROFILE-02 ikev2-profile CRYPTO-IKEv2-PROFILE-02

FreeBSD/StrongSwan configuration

# /etc/rc.conf
ifconfig_vtnet0="inet <freenbsd-ip>/<freebsd-mask>"
defaultrouter="<freebsd-gateway>"
strongswan_enable="YES"

# /usr/local/etc/strongswan.conf
charon {
        install_routes = no
}

# /usr/local/etc/strongswan.d/ike-updown
case "$PLUTO_VERB:$1" in
up-client:)
        if [ ${PLUTO_CONNECTION} = "<router-name>-vti-tunnel" ]
        then
                if_name=ipsec0
                ifconfig $if_name create >/dev/null 2>&1
                ifconfig $if_name reqid ${PLUTO_REQID}
                ifconfig $if_name inet tunnel ${PLUTO_ME} ${PLUTO_PEER}
                ifconfig $if_name inet <tunnel-net>.2/30 <tunnel-net>.1
        fi
        if [ $VPN_LOGGING ]
        then
                logger -t $TAG -p $FAC_PRIO \
                        "+++ ${PLUTO_CONNECTION} - ${PLUTO_REQID} - ${PLUTO_ME} (${PLUTO_MY_ID}) -- ${PLUTO_PEER} (${PLUTO_PEER_ID})"
        fi
        ;;
down-client:)
        if [ ${PLUTO_CONNECTION} = "<router-name>-vti-tunnel" ]
        then
                if_name=ipsec0
                ifconfig $if_name down >/dev/null 2>&1
        fi
        if [ $VPN_LOGGING ]
        then
                logger -t $TAG -p $FAC_PRIO -- \
                        "--- ${PLUTO_CONNECTION} - ${PLUTO_REQID} - ${PLUTO_ME} (${PLUTO_MY_ID}) -- ${PLUTO_PEER} (${PLUTO_PEER_ID})"
        fi
        ;;

# /usr/local/etc/swanctl/conf.d/router-vti.conf
connections {
        <router-name>-vti {
                keyingtries=0
                rekey_time=1h
                version=2
                remote_addrs=%any
                local_addrs=<freebsd-ip>
                children {
                        <router-name>-vti-tunnel {
                                mode=tunnel
                                policies=no
                                life_time=3600s
                                start_action=none
                                local_ts=%any
                                remote_ts=%any
                                updown = /usr/local/etc/strongswan.d/ike-updown
#                               reqid=100
                                local_ts  = 0.0.0.0/0
                                remote_ts = 0.0.0.0/0
                        }

                }
                local {
                        auth = psk
                        id = "<freebsd-name>"
                }
                remote {
                        auth = psk
                        id = "<router-mame>"
                }
        }
}

secrets {
        ike-<router-name> {
                id-fqdn-<router-name> = "<router-name>"
                secret = "<psk>"
        }
}

I’m thinking of moving my storage for my NAS (N40L) from DAS to iSCSI. My current set-up is trivial (and not performant): the N40L has 4 local SATA drives and an LSI2008 in IT mode connected to 6 SATA drives.

I think the current bottleneck is the controller:

# dd if=/dev/zero of=a bs=8192 count=1310720
1310720+0 records in
1310720+0 records out
10737418240 bytes transferred in 68.962335 secs (155699749 bytes/sec)

These results are targeting the 6 drives in a ZFS RAIDZ pool.

Anyway, by moving the pool to iSCSI I will be introducing IP and Ethernet to the transport, with 10G Ethernet a must. As a quick test I installed a cheap Intel 10G NIC into a spare N40L and connected it directly to a Cisco UCS with 10G with a twinax cable. I booted both machines from live USB OSes: FreeBSD 13 on the N40L and Ubuntu 20 Server on the UCS. FreeBSD 10 would hang at random times with the 10G NIC installed in the N40L nor would it recognise the Cisco 10G NIC in the UCS,

I installed iPerf3 on each machine, bumped the MTU on FreeBSD to 9000 (crazy that it defaults to 1500 for a 10G NIC) and ran a quick test. Here’s the results:

N40L:

root@:~ # iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 10.30.1.1, port 56382
[  5] local 10.30.1.2 port 5201 connected to 10.30.1.1 port 56384
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   1.00-2.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   2.00-3.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   3.00-4.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   4.00-5.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   5.00-6.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   6.00-7.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   7.00-8.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   8.00-9.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   9.00-10.00  sec  1.15 GBytes  9.90 Gbits/sec
[  5]  10.00-10.00  sec  1.32 MBytes  9.49 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec                  receiver

UCS:

root@ubuntu-server:~# iperf3 -c 10.30.1.2
Connecting to host 10.30.1.2, port 5201
[  5] local 10.30.1.1 port 56384 connected to 10.30.1.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.15 GBytes  9.91 Gbits/sec    0   1.39 MBytes
[  5]   1.00-2.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.39 MBytes
[  5]   2.00-3.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.39 MBytes
[  5]   3.00-4.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.39 MBytes
[  5]   4.00-5.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.39 MBytes
[  5]   5.00-6.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.39 MBytes
[  5]   6.00-7.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.39 MBytes
[  5]   7.00-8.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.39 MBytes
[  5]   8.00-9.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.39 MBytes
[  5]   9.00-10.00  sec  1.15 GBytes  9.90 Gbits/sec    0   1.39 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec                  receiver

iperf Done.

When reversing the direction the results were the same.

A bit later I ran some more tests. This time iSCSI with the target being the UCS server and the initiator being the N40L. The UCS is running ESXi 6.7 passing through the HBA to a FreeBSD 13.1 VM:
mpt0 Adapter:
Board Name: SAS3444
Board Assembly:
Chip Name: C1068E
Chip Revision: UNUSED
RAID Levels: none

The 10Gb adapter in the UCS is owned by ESXi and the VM is connected using a standard vSwitch. The N40L (also running FreeBSD 13.1) has an Intel 10Gb NIC. Both NICs are configured for 9000B MTU (verified during tests with tcpdump).

Here are some raw network tests similar to those above. The UDP results require further examination.

VM -> N40L (TCP)
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 11.5 GBytes 9.88 Gbits/sec receiver

VM -> N40L (UDP)
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 9.31 GBytes 8.00 Gbits/sec 0.000 ms 0/1115968 (0%) sender
[ 5] 0.00-10.00 sec 7.85 GBytes 6.74 Gbits/sec 0.007 ms 174995/1115968 (16%) receiver

N40L -> VM (TCP)
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 11.4 GBytes 9.81 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 11.4 GBytes 9.82 Gbits/sec receiver

N40L -> VM (UDP)
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 7.37 GBytes 6.33 Gbits/sec 0.000 ms 0/883094 (0%) sender
[ 5] 0.00-10.00 sec 2.68 GBytes 2.30 Gbits/sec 0.014 ms 562078/883064 (64%) receiver

Here is the VM writing to a 7K SAS drive on the HBA locally (no network involved):

dd if=/dev/zero of=/mnt/a bs=8192 count=1310720
1310720+0 records in
1310720+0 records out
10737418240 bytes transferred in 59.083653 secs (181732472 bytes/sec)

That’s a write speed of about 1.35GB/s. So we know the VM can access its local drives at that speed. I was hoping N40L hitting the same drive over iSCSI would get similar results:

dd if=/dev/zero of=/mnt/a bs=8192 count=1310720
1310720+0 records in
1310720+0 records out
10737418240 bytes transferred in 108.958877 secs (98545603 bytes/sec)

Unfortunately not. The write speeds over iSCSI were almost half of the local write speeds, 751MB/s.

When I get the time I will move the HBA to the N40L and test the local write speeds on that machine.

While Passthrough WebAuth upon MAC authentication failure is not supported on Cisco WLCs, it is easy to workaround this by doing the following.

The first step is to create a user to be used as a dummy account that the web-form will silently submit when the user agrees. This user should be created depending on where you authentication happens (radius or local).

The second step is to modify the passthrough HTML file (login.html, for example) to include a hidden <div> that contains the dummy user’s login details:

<div id="hiddenForm" style="display: none;">
<h4 align=center><font color="#000000">Username<input type="TEXT" name="username" SIZE="25" MAXLENGTH="63" VALUE="webauthdummy"></font></h4>
<h4 align=center><font color="#000000">Password<input type="Password" name="password" SIZE="25" MAXLENGTH="63" VALUE="webauthdummy"></font></h4>
</div>

Just make sure the above is within the <form> block. The above can be simplified; I got lazy and just modified Cisco’s webauth examples.

<h4 align=center><font size="5" color="#000000"><input type="button" name="Submit" value="ACCEPT" class="button" onclick="submitAction();"></h4></font><p></p>

As above I also modified the submit button slightly to say “ACCEPT” as the passthrough form does.

And that’s it. You can now set the “On MAC Filter failure” when using the Web Policy Layer 3 Security feature.

Assuming you’ve got a nice fresh certificate from LetsEncrypt and are in the directory where it, your key, and the LE root certificate¹ (TrustID X3) lives:

cat domain.com.cer ca.cer dst.cer > fullchain-plus-root.cer
openssl pkcs12 -export -in fullchain-plus-root.cer -inkey domain.com.key -out domain-fullchain-key.p12 -clcerts -passout pass:wlc
openssl pkcs12 -in domain-fullchain-key.p12 -out domain-fullchain-key.pem -passin pass:wlc -passout pass:wlc
cp -a domain-fullchain-key.pem /tftpboot/certs/

transfer download mode tftp
transfer download datatype webauthcert
transfer download serverip 
transfer download path /certs/
transfer download filename domain-fullchain-key.pem
transfer download certpassword wlc
debug pm pki enable
transfer download start

Check the logs to make sure the import was successful. If so you will be reminded to reboot. If so:

reset system