Raspberry Pi Official PoE-HAT – FAIL! If you want to use the USB Ports (last updated: 27 October 2018)

LATEST UPDATE: The PoE HAT has been updated, see my later post: Raspberry Pi PoE HAT – Official Modified Version

I’ve been awaiting the release of the official PoE HAT since the introduction of the Raspberry Pi 3 B+ (March 2018) where the introduction of the extra 4 pin header hinted that a PoE HAT would follow shortly after. It took a little longer than expected (availability announcement 24th August 2018) before the PoE HAT was released. I was keen to give it a try for one of my on going Home Automation Projects. As soon as I received notification it was in stock I placed an order with The Pi Hut along with replaced my existing Gigabit Switch with a PoE capable version (to avoid the need for lots of messy wires/PSU and dongles) so that I could take this new HAT for a spin.

I’m sad to report that so far my experience with this HAT has not been positive.

Issues

• (major) Attempting to use ANY of the USB ports with almost any USB devices caused the Pi to disable the ports due to over-current protection. This is being actively discussed in a forum post I started on 26th August 2018, where each day more uses join the thread reporting the same problems.

The problem can be seen clearly running the command dmesg -w which allows you to see real-time monitoring of driver messages. With and sometimes even without any USB devices connected then log shows messages like:

[ 11.661117] Bluetooth: RFCOMM socket layer initialized
[ 11.661151] Bluetooth: RFCOMM ver 1.11
[ 12.986076] usb 1-1.1-port2: over-current change 
[ 13.147084] usb 1-1-port2: over-current change 
[ 13.231877] usb 1-1.1-port3: over-current change 
[ 13.391693] usb 1-1-port3: over-current change 
[ 13.631851] usb 1-1-port4: over-current change 
[ 33.978119] usb 1-1.1-port2: over-current change 
[ 34.139091] usb 1-1-port2: over-current change 
[ 34.211769] usb 1-1.1-port3: over-current change 
[ 34.371756] usb 1-1-port3: over-current change 
[ 34.611750] usb 1-1-port4: over-current change 
[ 408.250020] usb 1-1.1-port2: over-current change 
[ 408.411057] usb 1-1-port2: over-current change 
[ 408.482948] usb 1-1.1-port3: over-current change 
[ 408.644602] usb 1-1-port3: over-current change 
[ 408.762133] usb 1-1.1-port2: over-current change 
[ 408.883014] usb 1-1-port4: over-current change 
[ 408.993105] usb 1-1.1-port3: over-current change 
[ 409.132964] usb 1-1-port2: over-current change 
[ 409.232949] usb 1-1.1-port2: over-current change 
[ 409.372938] usb 1-1-port3: over-current change

• (minor) When Raspberry Pi is connected to PoE just before initial boot there is a significant amount of what I suspect to be coil whine. This whine is also present when the device is shutdown, but left connected to the PoE network connection.
• (minor) The fan on-board the PoE-HAT spins up even when the Pi is idling (note, you need to update your Pi to the latest kernel to get support for controlling the fan on the HAT). Forum discussions have shown how to modify the trigger point from 45 DegC to a higher level, but it would be good to see this being offered via a simpler config file or better yet, the raspi-config utility. Also a bit of a buffer would be useful to stop the fan spinning up and down each time it crosses the 45 DegC point. According to the forum post, there is meant to be 5 DegC of hysteresis, however I’m not seeing that behaviour. (Update 27 Oct 2018: Issue has been fixed here: https://github.com/raspberrypi/linux/pull/2700 and verified included in update to 4.14.78 via rpi-update. Fan now turns on at 50 DegC and off at 45 DegC)
• (minor) The HAT mounts to the Pi via standoffs with screws in from the bottom of the Pi and the top of the PoE-HAT. These screws on the bottom of the Pi impede installation into some Pi cases, including the official Pi case. A quick modification care of a sharp knife to remove the plastic locator pins from the case, did allow the official case to the used. However, I suspect the PoE HAT was designed with being used in more industrial environments where it would be housed in a custom enclosure.

Investigations So Far

• Originally tested PoE HAT with a switch in my home office connected to a Cisco SG250-26HP smart switch via 1.5m cable. This should rule out any issues which could be attributed to cable lengths, condition, termination etc. The switch has since been relocated into my loft, and with the longer cable run the problems persisted.
• Purchased another RPi3 B+ and another PoE-HAT (from RS (item no longer sold by RS), this time). Brand new, never been used for anything else and problems persist (initially I thought it was fixed with the new HAT, but after a day the problems appeared on the new HAT. A replacement for the 1st HAT has also arrived from The Pi Hut and as expected it behaves the same as all the rest.
• Measured 5V and 3v3 GPIO voltage levels when running on RPi Micro USB official PSU vs PoE HAT including when adding and removing devices, nothing conclusive.
• Purchased an Active PoE Splitter from Amazon to verify that switch wasn’t the problem. Using the PoE Splitter which registered as a class 4 device, compared to class 3 for the PoE HAT, all worked just fine. Proving fairly conclusively that the issue resides firmly with these new PoE HATs.
• I also tried the non-updated Raspberry Pi Image 2018-06-27-raspbian-stretch.img so this lacked the kernel update needed to control the fan, however, the issue was still present.

I know from the replies to my forum post that the Raspberry Pi Engineering team are aware, but it would be good to see a bit more information flowing from the team. The PoE-HATs haven’t been pulled from sale, so I’m hoping that this is just going to be a software/firmware issue.

Conclusion (1st September 2018)

Advice to anyone considering the Raspberry Pi PoE-HAT, hold on. Unless you are sure you aren’t going to need to use any of the USB ports. If you need to use the USB ports and want to use PoE then continue to use an external Active PoE splitter.

Update (6 September 2018)

Relaying some of the discussions from the Forum Thread:

• The issue has been reported by many more users. Some with the experience and skills to do more measurements and experimentation with the hardware and analysing the spec sheet for the MP8007 DC-DC converter used on the PoE-HAT.
• Results show output stage of the DC-DC converter is very noisy.
• Workarounds:
  • Using longer leads between the RPi and the HAT seems to mitigate the issues. This led to speculation around RFI, however, it appears to the lead length that makes the problem go away. (@The_Raven)
  • Running the PoE-HAT Fan constantly (thought modifying the overlay file) seems to mitigate the issue in some cases, though load levels on RPi can trigger the problem. (@Daytona & @Roken)
  • Feeding the power output from the PoE-HAT back through the MicroUSB port, rather than directly onto the GPIO header, appears to solve it too (@grozzie)
  • Adding additional filtering (capacitors, and a coil, in the case of @The_Raven) to the output stage cleans up the output and appears to be closer to the real fix.
• @jdb at RaspberryPi.org has confirmed they have reproduced the issue in house and with the communities support are trying to determine why/how this problem has occurred.

Update (11 September 2018)

Finally, an update from the Raspberry Pi team, albeit it took the help of The Register get Eben Upton to admit that the current shipping version of the PoE Hat is fundamentally flawed, due to a compatibility issue with one (of two) vendors current limiting switch on the main RPi3 b+ board. Richard Speed from the Register posted this article today. A section was also then re-posted to the Raspberry Pi Forum thread where the community had been investigating the issues.

The choices ahead seem to be:

• Return the PoE HAT for a refund and await a new design to be released
• Remove some capacitors from the main RPI 3 b+ board
• Try and add some impedance between the HAT and the board (somehow).

I think I’ll wait a few more days before deciding what to do.

Update (13 September 2018)

EEVblog starts to investigate the RPi PoE HAT issue but lets the magic smoke escape. At least Dave was able to predict the issue is with the USB controller trigger the USB ports to turn off, due to the ripple from the PoE HAT. Which if he’d read Eben’s post or the rest of the forum thread (or here 😉 ) he’d already have known. Anyway always good to watch the EEVblog: https://www.youtube.com/watch?v=wopmEyZKnYo