Citrix Receiver H.264 Hardware Acceleration on Raspberry Pi 2
The information contained in this guide is mostly already out there in articles by Muhammad@Citrix and more recently Yi Lu. The purpose here is to simplify the process and hopefully reduce your frustration by creating a step by step guide.
Background
Citrix Receivers are able to support a number of graphics modes. In my previous post to setup Receiver, I recommended that the configuration be changed to disable H.264 mode, forcing compatibility mode, which can be further enhanced with Project Snowball. Software decoded H.264 doesn’t give a great experience, as such Muhammad/Citrix have released a proof of concept implementation using the Pi’s native hardware H.264 decoder. The Citrix source was released as part of the Linux Receiver’s Platform Optimization SDK, but you’ll need a bit more information to get this working.
Recently I saw a post on the Thin Client Project forum (no longer appears to exist), from a user Yi Lu that had shared the project on GitHub. Yi had taken the source that Muhammad/Citrix provided, and removed the watermark and added the necessary Makefile to allow others to compile it more easily. But there are more steps you need to follow to use or build a decoding plug-in for Receiver.
Using H.264 Hardware Acceleration
If you just want to use the capability, the GitHub project has a release which includes a simple installer with pre-compiled binaries. You still, however, have to modify /boot/config.txt and add the line:
gpu_mem=128
If you followed my previous setup guide you’ll also have to allow H.264 mode once again, editing /opt/Citrix/ICAClient/config/module.ini either comment out (add a ; prefix) or remove the line H264Enabled=False
Compiling it Yourself
If you prefer to know exactly what code is running on your system, or you just want to make changes, then you’ll need to know the steps to build the necessary libraries.
Step 1: Install pre-requisite packages
You will need some development packages to be added to your system:
sudo apt-get update && sudo apt-get install libx11-dev libxfixes-dev libxext-dev
Step 2: Compile Raspberry Pi VideoCore Dependent libraries
Your Raspberry Pi Raspbian image (and I assume other distributions) include Broadcom VideoCore API examples and source. The Citrix H.264 hardware acceleration implementation leverages two libraries from this, ilclient and vgfont.
You can either compile all the VideoCore example files in /opt/vc/src/hello_pi by running the command: /opt/vc/src/hello_pi/rebuild.sh
Or you can compile just the two libraries we need:
- Make ilclient library:
- (optional)
make -C /opt/vc/src/hello_pi/libs/ilclient/ clean make -C /opt/vc/src/hello_pi/libs/ilclient/
- (optional)
- Make vgfont
- (optional)
make -C /opt/vc/src/hello_pi/libs/vgfont/ clean make -C /opt/vc/src/hello_pi/libs/vgfont/
- (optional)
Step 3: Clone Yi Lu’s ctxh264_pi GitHub Project
I suggest you visit the GitHub project page to see if there is new information: https://github.com/luyi1888/ctxh264_pi
In order to keep things clean, I prefer to keep my projects and 3rd party projects in separate directories. It’s up to you how you structure your hierarchy.
- Create or change to the directory you wish to sync the github project down to. e.g:
cd ~/develop/3rdparty/ - Clone the project:
git clone https://github.com/luyi1888/ctxh264_pi.git - This will clone the project to a sub-directory:
ctxh264_pi
Yi Lu has updated the project on GitHub to implement my suggested changes, so Step 4 is no longer required. Thanks, Yi Lu.
Step 4: Modify the ctxh264_pi
I’m not sure how Yi Lu wishes to handle these suggested changes, I’ll update this section if these steps no longer are necessary.
4.1 Add de-initialization call to the the bcm_host
Edit:ctxh264_pi/bcm_init/bcm_init.cAdd the following function to the end of the file:
__attribute__((destructor)) static void deinit_bcm_omx(void) { bcm_host_deinit(); }
4.2 Update the Makefile.include File
The project makefile makes some assumptions about your directory layout and the relationship between the git project and the VideoCore libraries. We can eliminate this (assuming your VideoCore libraries are where in /opt/vc/src/hello_pi/libs/).
Editctxh264_pi/Makefile.includeUpdate the LDFLAGS and INCLUDES lines to provide the absolute paths doe ilclient and vgfont:
LDFLAGS+=-L$(SDKSTAGE)/opt/vc/lib/ -lEGL -lGLESv2 -lopenmaxil -lbcm_host -lvcos -lvchiq_arm -lpthread -lrt -lm -L/opt/vc/src/hello_pi/libs/ilclient -L/opt/vc/src/hello_pi/libs/vgfont INCLUDES+=-I$(SDKSTAGE)/opt/vc/include/ -I$(SDKSTAGE)/opt/vc/include/interface/vcos/pthreads -I$(SDKSTAGE)/opt/vc/include/interface/vmcs_host/linux -I./ -I$(SDKSTAGE)/opt/vc/src/hello_pi/libs/ilclient -I$(SDKSTAGE)/opt/vc/src/hello_pi/libs/vgfont
Step 5: Compile ctxh264_pi Project
- From the
ctxh264_pidirectory run the commands: - Compile the bcm_init library:
make -C bcm_init/ cleanmake -C bcm_init/
- Compile the ctxh264 library:
make -C H264_Pi_sample/ cleanmake -C H264_Pi_sample/
- The resulting files are:
bcm_init/bcm_init.soandH264_Pi_sample/ctxh264.so
Step 6: Install the Libraries
We can either copy the files to their required locations or if you plan on making further modifications you can link them.
- From the
ctxh264_pidirectory run the commands:sudo cp H264_Pi_sample/ctxh264.so /opt/Citrix/ICAClient/lib/sudo cp bcm_init/bcm_init.so /usr/lib/- You can copy this file to anywhere in the search path, I prefer usr/lib rather than /lib for non-core files.
- The installer that comes with the github project release puts this file in /lib. So you should check you only have one copy of this library where you expect it.
Testing Citrix Receiver H.264 Hardware Acceleration on Raspberry Pi 2
Ensure you have updated your config.txt to assign more video memory and that you haven’t got H.264 disabled in your module.ini, as described near the beginning of this post. Launch your receiver and playback a video, you should see a higher frame rate and if measuring it, a lower client CPU usage.
Client FPS Client CPU
A short (00:02:25) test video 1280 x 720 at 23 FPS was used as the source, played within VLC rendering the video server-side. The H.264 mode is great for playing back video. However, window movement and other interactions with the session felt laggy in comparison to the snowball compatibility mode.
Note: The reported FPS shown in the graph is higher than the source video frame rate, due to additional frames generated by Citrix graphics protocol to measure ICA round trip time.
Known Limitations
The current implementation was intended as a proof of concept for others to build upon. It uses an OpenGL surface for the sessions to be rendered on and a second surface to render the mouse.
- Wobbly mouse cursor – As the mouse moves near the edges of the display you’ll see double points as the mouse appears to wobble/flicker.
- Mouse cursor disappearing – This can happen from time to time. Yi Lu recommends running as root to try and avoid it happening.
- Fuzzy text – The current implementation lacks the text tracking feature used in most of the Citrix graphics modes. The lack of text tracking means that Text is also compressed/encoded along with the rest of the screen images and as a result can appear fuzzy. Perhaps the community can look at adding this support?
- Citrix instrumentation dialog – As discussed in my previous post you can set WFICA_OPTS to show various statistics about the session. This isn’t visible in H.264 hardware accelerated mode, however, if you’ve enabled it, it is there. As you’ll see if you try and send mouse clicks to that region of the screen. If you wish to review the performance metrics you’ll have to add an “l” to the list of options to have it log to a file.
Hi, Martin. We see cursor issues and fuzziness, as well, with the H.264 implementation. It would appear the Thinwire Plus option currently produces the cleanest video, wouldn’t you say, at the price of slightly higher CPU utilization? But at least Thinwire Plus is independent of the video protocol.
Regardless of client the different graphics modes have their pros and cons. IIRC the HDX 3D VDA may use H.264 without text tracking, though that may have changed. The ability to get clear text shouldn’t be “too hard”, as with the cursor overlay the RLE encoded text strings could be an another overlay. The mouse issues I think could also be addressed by reducing the size of the cursor overlay as is approaches the edge of the window. But if there a need, I’m not sure.
I should be clear this post, isn’t advocating the use of hardware accelerated H.264, just providing the steps for those that want to try it out. Hopefully to help someone.
Fuzzy text perhaps a network problem.
My environment is GRID K2 + HDX 3D Pro + VDA 7.1/7.5.
I only experiencing this when network is poor.
window movement and other interactions with the session felt laggy.
Perhaps EGL surface problem.
egl_render has higher latency compare to video_render.
I am testing HWCursor. It will always visible.
https://www.raspberrypi.org/forums/viewtopic.php?f=38&t=112366&p=769961
Maybe we can remove EGL surface to reduce latency because cursor is no longer a problem.
The following link is using video_render to render screen. egl_render is remove completely.
https://drive.google.com/file/d/0B70bYzs1FfZrcGpDRENMdGszWDA/view?usp=sharing
I’m not so sensitive to see window movement lag.
Hope martin or other people can test it.
I just tried your files on my development raspbian image and it broke my xserver. Perhaps we can follow up via email with the error I got? If you use the contact form linked at the top of the site I’ll get your address.
@martin
I updated github project. You mentioned issues has been fixed.
At John Nicholls’ recommendation (thank you, John!) we tried adding in the licensing for the MPEG2 and VC-1 decoders (£2 and £1, respectively, plus VAT where applicable) and they definitely make a big difference, in particular with full-screen video from youtube and WMV files. The licenses can be procured via PayPal from http://www.raspberrypi.com/ and while they say it may take up to 72 hours to process the order, we received our license codes within under three hours. It certainly seems like a small price to pay for the performance jump. It appears the adjustments for “Pi2” mode given in https://www.martinrowan.co.uk/2015/08/citrix-receiver-linux-raspberry-pi-2-using-raspbian-wheezy/ seems to help a little bit. After all, the Receiver doesn’t take up that much memory, so why not put more of the available memory to use elsewhere? We’d be interested to hear if others can experiment further to optimize these settings.
I’ll look into this. The licenses might help multimedia redirection which I’ll admit was terrible, hence why I disabled it. I’ll want to confirm it really works and that it works with Jessie too.
Sorry, I didn’t mean to say youtube videos in _general_ above, as they use a variety of formats/extensions
(MOV, MPEG4, MP4, AVI, WMV, MPEGPS, FLV, 3GPP, WebM) most of which are unaffected by MPEG2, while among other things, VC-1 is used for various DVD formats. However, for WMV files, the licensed MPEG2 component definitely makes a difference. The newest Citrix Receiver release also addresses a few issues.
So I’ve done some more testing and been in contact with ThinLinX. The Receiver on the Pi isn’t using multimedia redirection, as such any additional codec licenses won’t help. There were other changes to their latest image which improved performance but removing the license keys for the codecs didn’t change the performance.
The issue with multimedia redirection is that the current Linux Receiver uses gstreamer0.1 and for it to work well the hardware acceleration plugin gst-omx would need to be added, however that’s not available for this old gstreamer version. I spent a little time trying to compile it on my Pi, but due to it using a number of deprecated function calls it wasn’t going to be simple. Given the performance without redirection is pretty good I think there is much need in trying to go down that route. Maybe if the Receiver ever moves to using a newer gstreamer version it might be worth looking at again.
Thank you, Martin, for that update. Would going to Jessie help the gstreamer/gst-omx issue at all? In other words, how much of this is problematic because of the base OS? The lack of multimedia redirection doesn’t seem to be that important for the ThinLinX version as the various other tweaks appear to have readily compensated for the lack of the codec licenses. It would be interesting to see what difference would be there, if they could be tapped into with multimedia redirection enabled in the Receiver.
I think you are saying it’s the Citrix Receiver end that is missing the support for gstreamer. If so, not sure what could be done to lobby to get that built in at some point.
Hi Tobias, switching to Jessie doesn’t help the gst-omx issue. Also MM redirection may not be the answer to everything as you know it’s limited to content playing in Windows Media player and doesn’t redirect more common video playback scenarios, such as video in a web page like YouTube.
I imagine it’s possible for someone with more time and development skills than I to update the gstreamer plugins to not use the depricated calls and build a version that would work with gstreamer0.10, but it would be a fair amount of work.
Citrix Receiver does support gstreamer, it is just dependent on 0.10, version 1.0 and later use a new API which isn’t backward compatible with 0.10. I believe moving to the newer API is on the long to do list, but no idea if/when it might happen, nor if its the most important thing to do in comparison to all the other possible work the team could be doing.
Hi, Martin: OK, got it. Playing WWV files has never been a problem, so the MM redirection issue is really a non-issue. Best at this point to move on from that, You’re right — there are plenty of other things to address.
We see less stability with a XenDesktop 7,6 FP1 hosting compared to John’s using VDI-in-a-Box, which is probably the biggest issue at present. What VDI were you using for your tests of John’s build? He was saying some kernel may be responsible for other issues.
news about h264 implementation project?
Your timing is perfect. Your comment just arrived as I was being shown a demo of the latest from ThinLinX. There are still some things to iron out, but the performance was amazing.
Indeed, running under Windows 8.1 the ThinLinX thin client Receiver is remarkably good now. Windows 10 still seems to show that odd semi-periodic delay every few seconds, and we are going to give Windows 7 a test again to see if it behaves more like 8.1 or 10. Truly, just a few things to iron out and this will will one remarkable platform. Add to that the ability to create custom RPI2 boards: http://www.geek.com/chips/raspberry-pi-boards-can-now-be-customized-and-mass-produced-1637967/ !
what about h264 integration?
Denis, the ThinLinX solution is using hardware H264 acceleration using a library they have developed themselves based on the initial work from Muhammad. Likewise Yi Lu has been working on H264 hardware acceleration.
Maybe I’m not understanding your question?
ok thanks, I’m searching news about Yi Lu work; I’m not so interested about ThinLinX solution because it is a commercial product (is correct?)
Denis, yes it is a commercial undertaking. See the Raspian Jessie article on this site as well as above as to what has been incorporated from Yu Li’s efforts (which are also incorporated into the ThinLinX product). My understanding is that the ThinLinX Receiver will sell for a very reasonable price and incorporate a management interface. More details can be seen at their http://thinlinx.com Web site.
How to fix Mouse disappearing problem on jessie,when I’m using H264 HW Acceleration
I’m not sure I know of a fix. I believe the ThinLinx guys have resolve this in their implementation.
You have to use HWCursor in X, which is not supported by default.
Compile and install https://github.com/JamesH65/xf86-video-fbturbo/tree/raspi_hwcursor_pi2
Hell, what’s wrong with comments on this page. I tried to post several times a solution, but always got HTTP 404.
Add fbturbo config file to _etc_X11_xorg.conf.d and set option “HWCursor” “On”
Sorry, for posting so much waste, but please fix the system to accept slashes!!! Or at least implement a useful error/notify message.
Yes, sorry, on my todo list to figure out why WordPress is causing this problem. It’s been annoying me too. I have the luxury that it appears the problem is only with posting new comments, and if I post then edit the comment I can make it work. Odd, but will look into it. Sorry once again for the frustration it causes.
I’ve repaired the wordpress database, and posted a test with various slashes in it and it worked. Not sure if it’s fixed it. If you can confirm what you were trying to post that failed (I accept this maybe hard if it still fails, but will give me something to test against)
Is there any solution for Vmware horizon
Mohamad, I thought I replied to your email in the last few days. Perhaps you didn’t get it. I’d suggest looking at: rpitc.blogspot.com However I don’t think VMware have released a client which can use any hardware acceleration features on ARM based clients.
Hello, after many months you can have the version of the pre-compiled libraries with H.264 and hw cursor working?
Denis, at the moment the only supported solution I’m aware of for H264 offload on the Pi is from ThinLinx. I’m not aware of anyone else releasing a supported pre-compiled version.
HI Martin, Great article… when follow these steps the video plays fine but as you have mentioned the scrolling and windows movement feels laggy.. how can i fix this issue? and what is snowball compatibility mode?
Project Snowball was the creation of ThinWire+ also known as compatibility mode back in 2015. A variation of this mode is now the default on modern XA/XD versions. The current default is called Selective H264, or “Use codec for actively changing regions”. This will use ThinWire+ mode for most of the screen and only use H264 for an area of the screen where it would benefit from a video codec.
I don’t know if the hardware decoder supports Selective H264 or just fullscreen. But suggest you try TW+
the mouse disappeared completely when using hardware acceleration !! we tried to login to raspberry pi as root but still no mouse when opening VM via receiver ?? how to solve this issue?
Did you follow the recommendation “Mouse cursor disappearing – This can happen from time to time. Yi Lu recommends running as root to try and avoid it happening.”
Alternatively the best starting point would be using a ready tested and supported Raspberry Pi Client from one of the HDX Ready Pi vendors: https://citrixready.citrix.com/category-results.html?search=hdx-pi