Reviews

Pimoroni NVMe Base for Raspberry Pi 5

The Pimoroni NVMe Base for Raspberry Pi 5 is the first NVMe board I’ve obtained which uses the new PCIe port on the Raspberry Pi 5, to provide faster and less problematic access to fast storage. Supporting NVMe storage with the Raspberry Pi 4 relied upon the USB 3.0 in conjunction with USB to NVMe controllers which have had their challenges with performance and compatibility.

As noted from the Raspberry Pi 5 documentation the connection on the Raspberry Pi 5 is certified to PCIe 2.0×1, though a configuration option can be set to enable PCIe 3.0×1 but this isn’t officially supported and may not be stable, as we’ll see in the test results.

This board is designed specifically with NVMe storage cards in mind, whilst there are a range of cards which share the same connector they may not fit beneath the Pi. I’ll leave Jeff Geerling to explore PCIe device support for Raspberry Pi’s.

Design

The addition of the PCIe interface on the rear edge of the board is seeing manufacturers exploring add-ons that don’t use the approved HAT design, as they don’t use nor want to block the GPIO header. The Pimoroni NVMe Base for Raspberry Pi 5 opts to create a board which goes underneath, leaving all the top-mounted interfaces on the Raspberry Pi 5 unimpeded.

Pimoroni has released a useful video demonstrating how to construct and connect the Pimoroni NVMe Base for Raspberry Pi 5. The bolt to secure the top edge of the NVMe drive is a simple bolt and a pair of nuts, however, I’ve seen more elegant solutions, but it’s functional enough.

Enclosures

At the time of writing, there isn’t a commercial enclosure which accommodates the Pimoroni NVMe Base for Raspberry Pi 5. Leaving those that want to use this board with two choices a) Apply the stick-on feet to the NVMe base and b) Use a 3D printed case where the community have created a design which can accommodate the board plus a Raspberry Pi 5. For example, on Printables.com I found a 3D printed design for a Raspberry Pi 5 Case for Pimoroni NVMe Base, which may be of interest; assuming it works the first time, this 3D Printable case would cost < £2.

However, if you don’t have a 3D printer, and want a more professional-looking enclosure, be on the watchout for new cases announced by Argon Forty and others which I expect will be released soon.

Storage Performance

Testing Methodology

I’ve used a benchmarking script provided by Jeff Geerling to perform the testing which uses fio and iozone to measure the performance for 1MB sequential reads, 1MB random reads and writes and 4k random reads and writes. Each scenario was repeated 3 times and the results averaged.

The system is configured to boot from the microSD card, and NVMe storage is manually mounted.

  • OS: Raspberry Pi OS (64-bit) release 2023-12-05 with Desktop. All updates applied as of 3 Feb 2024
  • Kernel: Linux rpi5 6.1.0-rpi8-rpi-2712 #1 SMP PREEMPT Debian 1:6.1.73-1+rpt1 (2024-01-25) aarch64 GNU/Linux
  • Bootloader: Mon 22 Jan 14:44:36 UTC 2024 (1705934676)
  • OS configuration:
    • WiFi Disabled
    • SSH Enabled
    • System is configured to boot to the console
    • Samba + utilities installed by benchmarking script below
    • Python: 3.11.2
  • Storage performance benchmarking script by Jeff Geerling (GitHub) using:
    • iozone: 4.492
    • fileop: 3.33

Storage devices under test:

  • Sandisk Extreme Pro 32GB microSD
  • Kioxia Exceria 500GB (PCIe 3.0 x4) NVMe, purchased with the NVMe base from Pimoroni
  • Sunbow 128GB (PCIe 3.0 x2) NVMe
  • Western Digital WD Blue SN570 250GB (PCIe 3.0 x4) NVMe
    • Noting that Pimoroni had experience with the SN570 preventing the system from booting. I can confirm, that if you set the Pi to boot from NVMe then microSD it fails to start, just showing a black screen. So this drive can only be used as secondary storage not as boot media. I’d recommend you follow the guidance published by Pimoroni for drive compatibility.
    • Update 5 Feb 2023: An updated bootloader has been made available where the release notes contain an entry “Add a workaround for an issue seen when booting with WS Blue SN550 NVMe SSD”. Applying firmware dated 5 Feb 2024 that includes this change enabled my SN570 drive to boot successfully.

To update the bootloader, perform the steps below. If you want to see the date of your current firmware run sudo rpi-eeprom-update

sudo apt update && sudo apt upgrade -y
sudo rpi-eeprom-update -a
sudo reboot

Commands used to prepare and run the test:

sudo mkdir /mnt/nvme
sudo mount /dev/nvme0n1p2 /mnt/nvme
sudo DEVICE_UNDER_TEST=/dev/nvme0n1 DEVICE_MOUNT_PATH=/mnt/nvme ./disk-benchmark.sh

For tests with PCIe 3.0 instead of PCIe 2.0, the configuration needs to be updated via editing: /boot/firmware/config.txt to add an entry under [all]

 dtparam=pciex1_gen=3

Storage Performance Results

PCIe 2.0 x1

Comparing microSD card performance to the supplied Kioxia NVMe drive, demonstrates performance gains between 256-4354%, depending on the test. Switching from the Argon NVMe USB connected board using the same WD Blue drive, shows gains ranging from 18-327% once the drive is installed into the Pimoroni NVMe Base for Raspberry Pi 5. Showing the benefits of this PCIe interface.

PCI 3.0 x1

Enabling the unsupported PCIe 3.0 capability, shows significant gains are available given the capabilities of the NVMe drives. With the gains varying depending on the test, we can compare the same drive in PCIe 2.0 vs 3.0 configuration which shows:

  • fio 1MB sequential read: 93-97%
  • iozone 1MB random reads: 88-92%
  • iozone IMB random writes: 42-127%
  • iozone 4kB random read: 7-16%
  • iozone 4kB random writes: -23-32%

The clear outlier is the iozone 4kB random writes test with the Kioxia Exceria NVMe drive, where the data shows results from 85-306, with other tests and NVMe drives not showing the same variability in results.

PCIe 3.0 instability

Digging deeper into the varied performance of the Kioxia drive in the PCIe 3.0 configuration shows a large amount of timeout errors. Re-testing with the other drives did not repeat the timeouts with the other drives. I don’t appear to be unique in experiencing these issues as seen in the Pimoroni forum.

[ 110.660073] pcieport 0000:00:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
[ 110.660074] pcieport 0000:00:00.0: device [14e4:2712] error status/mask=00000040/00002000
[ 110.660076] pcieport 0000:00:00.0: [ 6] BadTLP
[ 110.668177] pcieport 0000:00:00.0: AER: Corrected error received: 0000:00:00.0
[ 110.668181] pcieport 0000:00:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID)
[ 110.668183] pcieport 0000:00:00.0: device [14e4:2712] error status/mask=00001080/00002000
[ 110.668185] pcieport 0000:00:00.0: [ 7] BadDLLP
[ 110.668186] pcieport 0000:00:00.0: [12] Timeout
[ 110.668188] pcieport 0000:00:00.0: AER: Corrected error received: 0000:00:00.0
[ 110.668195] nvme 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID)
[ 110.668197] nvme 0000:01:00.0: device [1e0f:0009] error status/mask=00001100/00006000
[ 110.668199] nvme 0000:01:00.0: [ 8] Rollover
[ 110.668200] nvme 0000:01:00.0: [12] Timeout

It’s possible different drives from different batches behave differently, as YMMV when using the unsupported PCIe 3.0 support.

Power Consumption

A benefit of the new PCIe interface and its elimination of USB 3.0 to NVMe controllers is a reduction in overall power consumption. The Western Digitial drive installed in an old Argon NVMe base for Raspberry Pi 4, showed idle power of 5W with a peak during the test of 6.7W. Popping the same drive into the Pimoroni NVMe Base for Raspberry Pi 5 shows an idle power of 3.5W (-29%) and a peak of 5.7W (-15%). With the Kioxia and Sunbow idling at around 3W and peaking at 5.9W.

Conclusion

One challenge faced with USB-connected NVMe storage was the need to jump through a few extra steps to enable trim support, as seen when testing the Argon NVMe board previously. With NVMe connected via PCIe, trim support works without any additional configuration.

Care needs to be taken when selecting what NVMe drive to use, the compatibility list from Pimoroni should be consulted. However, some compatibility issues are being addressed by RaspberryPi with updated bootloader firmware. PCIe 3.0 can offer additional performance, but even the recommended cards like the Kioxia card supplied as part of their 500GB kit appear to work for some people and not for others.

If you hope to put this in some sort of case, I’d recommend you wait to see what the new Argon Neo 5 NVMe or its big brother the Argon One V2 M.2 NVMe can offer. PiHut is showing the NEO 5 NVMe case with a pre-release price of £35, £17 more than the non-NVMe variant, which isn’t that much of a premium over the £13.50 NVMe Base.

Product Links

Amazon links are affiliate links which help support the site, where possible I’ve used links which should take you to the product in the Amazon store in your region. Links to other suppliers are included for your convenience.

2 thoughts on “Pimoroni NVMe Base for Raspberry Pi 5

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.