How to disable HP Proliant ML350p Gen8 P420i RAID controller, enable HBA mode (a.k.a. pass-through), and perform a FreeBSD root on ZFS installation

I recently purchased a second-hand HP Proliant ML350p Gen8 in order to be used as a home server and to my disappointment realized I won’t be able to disable the hardware RAID that comes with this model, at least at first glance. Well, there is a way to do that, which is supported by HP themselves. And, this is how to do it the easy way!

But, before we proceed any further, you might ask why do I need to disable the hardware RAID? The answer is because I need to install ZFS. There is already plenty of documentation why installing ZFS on a hardware RAID is a terrible idea. Thus, I won’t go over that since it’s not the focus of this article.

Again, before we go any further there’s a catch you should know about. If you’d enable HBA-mode, the server won’t be able to boot from any disk connected in HBA mode to the controller! You should consider this before converting to HBA mode. In case you need to perform a FreeBSD/Linux root on ZFS installation through this controller there are two solutions:

1. Installing in hardware RAID mode, but making each disk a RAID-0 array consisting of only one disk. For example, if you’ve got 8 disks, you’ll end up with 8 RAID-0 arrays. Then you’d perform a ZFS installation and your operating system boots as expected. Though this is not recommended and if you’d proceed with this approach, it renders the rest of this post useless.

2. HP Proliant ML350p provides an SD-Card slot, which can be used to install a full system, which is not recommended due to the wear and tear effect of SDCards with each write-operation on them. In addition to that, this storage type is costly and slow. For example, a SanDisk SDXC Extreme Pro 256GB, which provides a write speed of 90MB/s and read speed of 170MB/s, costs around €99,99 where I live. With today’s standards, this is not fast at all or even good enough, especially on a server. I was also able to find a Sandisk CF Express Extreme Pro 512GB type B, with a write speed of 800MB/s and a read speed of 1500MB/s. It only costs €629,-, which costs an arm and leg to buy!

So, then! What’s the solution one might ask? We are going to install only /boot partition on the internal SD Card. Note, that placing only a bootloader such as GRUB, rEFInd or other similar tools won’t work as they won’t be able to see the boot drive anyway! So, the only solution is to put the boot partition on the SDCard. This way, the system only reads it one time at boot and it does not even have to be an expensive SD-Card.

So, let’s begin!

Switching to HBA mode

Before performing this procedure, I highly recommend to update your server’s firmware to the latest version, or else this might not work!

In order to switch the P420i RAID controller on HP Proliant ML350p Gen8:

1. Download the latest version of SystemRescue. At the time of this writing, it’s 8.05.

2. Create a USB bootable from the downloaded iso. On Microsoft Windows, one could use Rufus. Or, the cross-platform free USB flash tool for Microsoft Windows, macOS, and Linux, balenaEtcher.

3. Before starting SystemRescue from the USB drive, make sure to delete all the RAID configurations on the controller by pressing F8 at boot time. This is mandatory! And, make sure to make a backup of everything for obvious reasons, otherwise, your data will be wiped!

4. After booting SystemRescue you could type startx in order to start a neat and clean Xfce session. This is optional in case you’d prefer a graphical environment or a web browser. Instead, you could do everything from the initial terminal SystemRescue boots to.

5. Initialize Pacman package manager, by issuing the following command:

$ Pacman -Sy

6. Install the required dependencies for preparing and installing HP Smart Storage Admin CLI, ssacli:

$ sudo pacman -S --needed base-devel git wget yajl

7. Enable the nobody user by changing login shell from /usr/bin/nologin to /bin/bash in the /etc/passwd in your favorite editor; e.g. vim, nano, etc. Just be careful with your modifications as this file is very sensitive. Although in case you mess up, you can simply reboot and your changes are reset to default since it’s a live cd. We need to run the makepkg command as a non-root user, otherwise, it refuses to work. Since user nobody is disabled we need to enable it first, otherwise it results in This account is currently not available error.

8. Get the package description file for HP ssacli from AUR and prepare the package:

$ cd /var/tmp
$ su nobody -c 'git clone https://aur.archlinux.org/ssacli.git'
$ cd ssacli
$ su nobody -c 'makepkg -si'

It may complain, about sudo at the end, but do not worry!

9. It’s time to copy the prepared ssacli package to the system:

$ cp -vr pkg/ssacli/opt /
$ cp -vr pkg/ssacli/usr/share /usr/
$ cp -vr pkg/ssacli/usr/bin/* /usr/bin/
$ rehash

10. Finally, it’s time to enable the HBA mode (pass-through) for the P420i controller:

$ ssacli controller slot=0 modify hbamode=on

You can re-enable hardware RAID and disable HBA mode at any time by starting over from step 1 and only set hbamode=off instead of hbamode=on.

11. Confirm the HBA mode has been enabled:

$ ssacli controller slot=0 show | grep -i hba

   HBA Mode Enabled: True

In case, you need to check more information about the controller, omit the part begging with the pipe character |:

$ ssacli controller slot=0 show

Smart Array P420i in Slot 0 (Embedded)
   Bus Interface: PCI
   Slot: 0
   Serial Number: 001438028080D00
   Cache Serial Number: PBKUC0BRH5OALM
   RAID 6 (ADG) Status: Enabled
   Controller Status: OK
   Hardware Revision: B
   Firmware Version: 8.32-0
   Firmware Supports Online Firmware Activation: False
   Cache Board Present: True
   Cache Status: Not Configured
   Total Cache Size: 1.0
   Total Cache Memory Available: 0.8
   Cache Backup Power Source: Capacitors
   Battery/Capacitor Count: 1
   Battery/Capacitor Status: OK
   Controller Temperature (C): 39
   Cache Module Temperature (C): 24
   Capacitor Temperature  (C): 25
   Number of Ports: 2 Internal only
   Driver Name: hpsa
   Driver Version: 3.4.20
   HBA Mode Enabled: True
   PCI Address (Domain:Bus:Device.Function): 0000:03:00.0
   Port Max Phy Rate Limiting Supported: False
   Host Serial Number: CZ23510556
   Sanitize Erase Supported: False
   Primary Boot Volume: Unknown (600508B1001CFFA55942D7A0BE8219D9)
   Secondary Boot Volume: None

12. We’re done! Reboot the server:

$ reboot

Installing FreeBSD

In order to perform a FreeBSD root on ZFS installation on the P420i RAID controller bundled with HP Proliant ML350p Gen8 in HBA mode:

1. Download a recent version of FreeBSD. At the time of this writing, it could be 13.0, 12.2, or even the upcoming 12.3. It would be best to download the .img or .img.xz (requires extraction), as it’s easier to make a bootable USB driver from.

2. Create a USB bootable from the downloaded image. On Microsoft Windows, one could use Win32 Disk Imager. Or, in case you are on macOS, Linux, or even FreeBSD, the disk dump utility, dd command comes in handy (replace X with your USB flash drive identifier, could be figured out by using fdisk utility):

$ sudo fdisk -l
$ sudo dd if=FreeBSD-13.0-RELEASE-amd64-memstick.img of=/dev/sdX bs=8M

3. Install FreeBSD as you’d usually do, use Auto (ZFS) Guided Root-on-ZFS option in order to perform the installation on your ZFS RAID level of choice. Remember to keep track of your HP iLO Internal SD-CARD disk number in the ZFS Configuration dialog. In my case with the first 8 disks, da0 to da7 will be actual SAS disks, the last one da9 is the USB flash drive used to boot the installer, and finally, the penultimate one da8 is the internal SD-Card (note that the order of the internal SD-Card and the installer USB drive could change, and this is not a general rule). So, for the rest of this post, I’d be using that. When done with the installation, DO NOT REBOOT YET!

4. After the installation is finished, the FreeBSD Installer will ask for any final manual modifications. So, choose Yes here in order to drop into a terminal. If you don’t do this, you’re installation simply won’t boot.

5. Since there might already be some data or partitions on the internal SD-Card, in my case da8, we are going to force destroy everything first:

$ gpart destroy -F da8

da8 destroyed

5. Then, create a GPT partition table on the internal SD-Card:

$ gpart create -s GPT da8

da8 created

6. Proceed to create a freebsd-boot partition. According to gpart manpage:

We then create a dedicatedfreebsd-boot partition to hold the second- stage bootloader,which will loadthe FreeBSD kernel and modules from a UFS or ZFSfilesystem. This partition must be larger thanthe bootstrap code (either /boot/gptbootfor UFSor /boot/gptzfsboot forZFS), but smaller than 545 kB since the first-stage loader will loadthe entire partition into memory during boot,regardless of how much data it actu- ally contains. Wecreate a 472-block (236kB) boot partition at offset 40, which is the size of the partition table (34 blocks or17 kB) rounded upto the nearest 4 kB boundary.

Thus, issue the following command:

$ gpart add -b 40 -s 472 -t freebsd-boot da8

da8p1 added

7. Now, it’s time to install the FreeBSD loader on the newly created partition and a protective MBR on the internal SD-Card:

$ gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da8

partcode written to da8p1
bootcode written to da8

8. After this, we need a ZFS partition on the internal SD-Card:

$ gpart add -t freebsd-zfs da8

da8p2 added

9. Time to create a ZFS pool on the newly created partition, in my case da8p2, and mount it. This will be our separate boot partition containing boot loader, kernel, etc. I also recommend enabling lz4 compression on the internal SD-Card since it improves the I/O performance, boot time, and its lifespan in exchange for negligible CPU usage:

$ zpool create -O mountpoint=/mnt/zboot -O compression=lz4 zboot /dev/da8p2

10. After creating and mounting the boot partition under /mnt/zboot, we need to copy everything from the /boot directory into the newly created boot partition:

$ cp -vr /boot /mnt/zboot/

11. Since we are going to replace /boot with /mnt/zboot/boot and we have already made a copy, we can now safely remove the /boot directory:

$ rm -rf /boot

12. OK, now we can create the symlink. Note that /mnt/zboot will remain mounted across reboots due to the fact that now it’s in the hand of ZFS to manage:

$ ln -s /mnt/zboot/boot /boot

13. As the last step we have to let the loader know where the operating system root is residing:

$ echo 'vfs.root.mountfrom="zfs:zroot/ROOT/default"' >> /boot/loader.conf

14. And, finally reboot the server:

$ reboot

Done :) If everything has been done correctly, your server should boot normally as expected.