Raspberry Pi PoE HAT – Official Modified Version

As I covered in depth in my previous post Raspberry Pi Official PoE-HAT – FAIL! If you want to use the USB Ports (last updated: 27 October 2018) the initial release of the PoE HAT was rather problematic leading to the Raspberry Pi Foundation making the difficult and costly decision to recall all the PoE HATs from the sales channel, stopping sales until a new version was ready. I understand that the modified version, like the prototype I received, is now making it’s way back in the sales channel ready for a re-launch.

As well as sharing some details from testing the new, or rather, modified PoE HAT, I also wanted to share a bit more around what happened since my initial reporting of the issue.

History

On 21 August 2018, I raised an issue with USB over-current warnings when powered a Raspberry Pi with the new PoE HAT via the Raspberry Pi Forum (Forum Post). This post sparked many corroborating reports from other customers, but it took a little while to get any acknowledgement from the Raspberry Pi Foundation that there was a real and widespread issue. I took to twitter to no avail, whilst at the same time the community were investigating the issue and posting back results from their experiments. These revealed more about the problem, ruling out RFI and instead, finding unexpected ways of working around the problem, such as updating the PoE Fan driver to always run and finding that extending the headers from the Pi to the PoE HAT also made the issue go away.

Eventually, The Register was made aware of the problem and were able to provoke a response from Eben Upton (Founder of the Raspberry Pi Foundation). He admitted there was an issue and believed they understood the cause and the likely fix for the problem. Going on to explain it was caused by “an interaction between the fairly low-frequency switching regulator on the HAT, and one of the two brands of USB current limiting switch that we use on the mainboard”. The reason it was wasn’t detected during their testing phase was said to be as high current load tests were performed with one brand and low current test with the other. This seemed to mostly explain the problem, except for a few edge cases being experienced by myself and a few others on the forum, where the warning could be trigger without any load at all.

The Edge Cases

Via the Forum the Foundation team asked for people experiencing an issue with over-current without any load connect to get in touch, as it didn’t fit with their understanding of the problem. It was great to see the Foundation team now being more proactive as they sought to fully understand the problems being reported in the field. I suspect these reports and subsequent investigation have led in part to delays in posting the promised blog post explaining the problem, along with details of how an enthusiast might be able to modify their HAT to fix the problem themselves; it’s unclear if this level of detail will be shared by the Foundation. I sent off one my HATs along with RPi 3B+ which demonstrated the problem along with details of how to reproduce the issue.

Within a few days the Foundation got back in touch to report they couldn’t reproduce the issue and wanted details of the specific Cisco switch model I was using, so they could source the same model to test with. I have to admit my email exchanges with Roger (one of the PoE HAT Engineers) and James (Engineer and Forum moderator) were great, helpful and transparent. James reported back via the Forum post, that in house none of the ‘Faulty’ Pi, HAT combos had shown any issues, which is interesting, but strange nonetheless. Given I had issues initially with the Cisco Switch and Pi on my desk separated by a 2m cable it’s hard to think that environmental factors would come in to play. I know my home network cables run a close to mains cables which might add more noise, but in such a simple setup that was all eliminated.

The Prototype

In early October I was sent a prototype modified PoE HAT to test. I was pleased to report back that it addressed the over-current issues and passed a series of load tests in my environment. I believe this prototype is the same design that is being released shortly, initially to use up all the original board that were manufactured. This new PoE HAT includes a small daughter-board hosting additional filtering capacitors and a small inductor on the output stage.

Based on some comments, I believe a second-generation version of the PoE HAT will be released once these modified versions have been sold through the channel.

Here are the two versions side by side:

Original PoE HAT	Modified (Prototype) PoE HAT

The Changes

Looking at the original board you’ll notice the bolts securing the fan have been moved over to accommodate the daughter-board. Though the shape and hole in the daughter-board suggest there might have been some thinking to try and keep the bolts in the same place, but the new daughter-board has a capacitor on the edge which would interfere with the bolt.

It’s interesting that there is so much free space where the daughter-board was able to be placed, sort of begs the question if some earlier variation of the design may have used this space? The area provides GND, 5V, 3v3 some presented on large contact pads, some via small test points. The edge of the daughter-board that is connected at one end to where the capacitors have been removed, with the other end connected to a large 5V pad and GND provided by a tiny (through-hole?) GND connection. The 5V rail on the original board has been cut just behind the fan connector so that the output voltage has to pass through the daughter-board, before rejoining the original board and onto power the fan and the header connector for the Raspberry Pi.

The Circuit

The PoE HAT is based around the Monolithic Power Systems (MPS) MP8007 PoE Power IC (datasheet). The reference diagram for a 5V 2.5A output is:

The final output stage of the reference diagram shows 2 x 47µF capacitors and 470µF polarised/electrolytic capacitor. When capacitors are connected in parallel, the level of capacitance is the SUM on each capacitor, as such the reference design has 564µF in the final output stage to help smooth/filter the output. Examining the original PoE HAT shows that instead there are only 2 x 22µF + 4 x 47µF capacitors, giving a total of just 232µF, ~41% of that expected in the reference design.

The new daughter-board removes 2 x 47µF capacitors from the original board but adds 6 x 47µF capacitors and 4.7µH inductor, my best guess of the revised circuit is below (looking at the final output stage only and needing to guess some of the capacitor values, as the numbers aren’t visible. (I’ve had confirmation from the PoE HAT Developer that the larger capacitors are 47µF):

Looking at the design, it appears to be a simplified series damped filter, to help clean up the output stage, with a total capacitance of 420µF, 75% of the original reference design. The inductor and resistor elements, in conjunction with the capacitors, provide low-pass filtering (based on my limited knowledge), thus providing a significantly cleaner 5V supply to the Raspberry Pi from the PoE HAT.

The Results

5V Noise

In order to help investigate/understand the issue with the PoE HAT, I bought an Owon VDS1022I USB/PC Oscilloscope (I don’t need much of an excuse to seek out a new toy to play with). With this I was able to measure the input voltage from the PoE switch, which is presented via the 4 pin header connector introduced on the RPi 3B+ model, each pin provided a 12.8-14.8Vac @ 50Hz supply from my Cisco SG250-26HP switch. The screenshots below show the measurements on the final 5V output stage of the PoE HAT.

5V output, with drops when PoE HAT fan engages	Noise level with drop when PoE HAT fan engages
Comparing Original (CH2 – Yellow) vs Modified (CH1 – Red) PoE HAT (No Fan)

Looking at the measurements above shows:

• The modified PoE HAT can a cleaner output voltage, with the spikes reduced to ~60mV from ~150mV and smoothed too.
• The modified PoE HAT shows a voltage drop from 4.98V to 4.72V when the Fan turns on.

USB Load Testing

The main issue with the original PoE HAT was a false trigger of the USB over-current detection which would shutdown the port. So I purchased a couple of extra bits of test equipment to characterise the problem. Measuring the output voltage as the load increased, using the following devices connected to a single USB port:

• Unit-T UT658 USB Tester (measuring load)
• Juwei 35W USB Adjustable Electronic Load

From the testing we can see that:

• Original PoE HAT triggered over-current at ~300mA, compared to ~1170mA with modified PoE-HAT
• Max Input voltage from PoE-HAT of 5.05V is significantly lower than the 5.23V measured when using the official RPi USB Power Supply.
• Reported load measured by the Cisco SG250-26HP PoE enabled network switch with the modified PoE-HAT connected:
  • 2500mW @ 0mA USB Load (PoE-HAT Fan not running)
  • 2900mW @ 0mA USB Load (PoE-HAT Fan Running)
  • 9600mW @ 1000mA USB Load (PoE-HAT Fan not running)
  • 10300mW @ 1170mA USB Load

Finally

I, like the rest of the community and no doubt the team at the Raspberry Pi Foundation were disappointed that this issue made its way through internal testing and into production. The PoE HAT is ultimately a very simple circuit in comparison to the Raspberry Pi, that getting power filtering wrong was a huge surprise. I wish the team had been a little bit more open and engaging with the community at the start, without The Register needing to get involved to add some weight behind the investigations. I’m sure behind the scenes it was being actively investigated, but keeping the community in the dark about what was happening was a poor way of handling it.

Hopefully, all the issues are now in the past and I can now get on with the project I intended to use the PoE-HAT.

Thanks for reading (if you got this far)!