Python and Godot engine on a Raspberry Pi: displaying physiological signals with LSL

Fancy open-source game engines? Godot is among the main alternatives to the big names such as Unity or Unreal engine. Compared to the latter, Godot is free and comes with a MIT licence attached. Although tt is not on par in terms of functionalities and optimizations and you might not use it for creating the next triple A game, it still has plenty to offer. Godot comes with examples for desktop or mobile apps, it can do AR or VR, there is a physics engine for 2D or 3D scenes, you can use it to program shaders like grown-ups. Most of all, Godot can run on the Raspberry Pi and projects can embed Python code. With some hacks, that this tutorial is about.

The first part covers a basic Godot tutorial, using a python script to fetch data from the network and control a sprite. Part two dwells in more technicalities to (force) assemble python and the aarch64 Raspberry Pi.

The code described below is available at https://github.com/UlloLabs/tutorial.GodotPythonLSL

Forewords

The use-case will be to animate a cube with signals coming from the Lab Streaming Layer protocol (LSL for short). In the past years LSL has been widely adopted in research and among hackers to exchange physiological data between programs and between computers. It is a network protocol that ensure precise synchronization between streams, a must-have when dealing with signals such as EEG. More often than not there is a bridge between the acquisition devices and LSL, either from the manufacturers (e.g. Brain products, Emotiv, Neuroeletrics, OpenBCI) or from the community (e.g. Bitalino, Muse, generic BLE smartwatches) − more extensive list here. LSL is also supported by several program aimed at real-time signal processing, that you can transparently incorporate between the raw signals and your application (e.g. OpenViBE, Timeflux), and many tutorials exist around the web, including one dealing with Unity if you want to stick with the cool kids. For the sake of the tutorial we will use generated data so you don’t need an actual device to follow through.

LSL comes with various bindings (C, C++, C#, Java, Python and so on). Beside C and C++ there are no “pure” implementation of LSL, meaning that all bindings rely on a native library, which has to be compile against each target platform (also a reason why at the moment a browser implementation is problematic). Godot main programming language is GDScript, developed especially for it. It is fortunately possible to incorporate external library through GDNative. Because Godot is in the process of integrating C# as a second scripting language, the easy solution would be to use the C# LSL binding for our project. Easy is no fun, and despite C# existing code base, for learners as well as in the research community Python is king, with many libraries in signal processing, image processing, statistics, audio, and so on. Hence the objective will be to harness the power of Godot with the flexibility of Python.

We are lucky, an unofficial and yet active project exists to do just, that godot-python. There are some caveats in terms of packaging, that we will tackle later on; apart from that it just works, you can interface Python code with the scene in Godot, to control objects for exemple, and several languages can co-exist in the same project. Godot-python relies itself on another project to provide a standalone Python environment: python-build-standalone. A standalone Python can be shipped alongside the Godot project to run on final user’s computer, and it will not interfere with any Python installation present on the system. All is fine, then? Almost. Remember the premise, aiming at running the software on a Raspberry Pi? Those single board computers are tiny and cheap, perfect to lower the entry barrier for computer science and, here, disseminate biofeedback applications. Except that it relies on a different CPU architecture than traditional PC, ARM (as the newer Apple M1) instead of x86.

We are again lucky (or, more accurately, we can benefit from a positive open-source ecosystem), an unofficial and yet active project exist to facilitate ports of Godot to such machines: FRT. There is a second project more focused on the Raspberry Pi, but it does not seem to support GDNative (hence third party libraries such as LSL), so we will stick with FRT. Even though FRT does not aim at providing the Godot editor (the environment used to program the application), it provides scripts and binaries for running the Godot platform, the component use to run the application. Think of this second binary as an “interpreter” for the scenes created in Godot. When a project is being exported, the interpreter specific to the target platform is packed with the scenes, assets and scripts.

On the one hand we have godot-python to run python code within Godot, on the other hand we have FRT to run Godot on the Raspberry Pi, all is fine, then? Almost. Never before godot-python ran on Raspberry Pi (or an any ARM system for what matters); we are about to change that.

The tutorial is tested on a Raspberry Pi 4 (2GB RAM), with Raspberry Pi OS (2020-08-20), itself a derivative of a Debian Buster 10.5. Programming within Godot Editor occurred on a more regular laptop, an ol’ (k)Ubuntu 16.04 x64.

Part 1: Python and LSL in Godot

In this first step, we will create our scene in Godot, install and configure Python so we can retrieve LSL signals, animate an objects with it.

First sprite

First things first, grab the Godot editor for your (desktop) platform (tested with release is Godot 3.3.2). You can save yourselves the extra space taken by C# and Mono integration and download the ~35MO of the archive (did I mention that Godot is very lightweight? Pleasant change compared to competitors). Note: if anyone reproduce this tutorial with the Mono version and run Godot plus Python plus C#, (“Godot babel”?), feedback is welcomed.

One binary in the zip file, launch Godot and start with a new project. Select “OpenGL ES 2.0” as the renderer, ensuring better compatibility with the Raspberry Pi we will use later on. Lacking inspiration when it comes to titles and names, the project will be called “LSLDemo”.

Other tutorials dedicated to Godot will to a better job at guiding you in creating a 3D project, for the sake of simplicity we will stay two dimensional on this one. Click on “2D scene” on the left-hand sideo of the window to create the root node of the scene. You will switch to a 2D plane in the main view. Right click on the Node2D root node, “Add Child Node”, look for and select “Sprite” in the list, an element used to represent a character or an object that can move on screen. We will associate an image to it. It can be anything, because we will have it turning and turning around, let’s download a PNG of a Bee on Wikimedia. Direct access to 320×240 version here. Save the PNG file on your disk. To add the image to the list of ressources available to the project, drag and drop the image file in the editor window. You should now see the image appear in the bottom left dock under the main folder res://.

To associate the image to the sprite, selecting the Sprite node, click on the “Texture” property situated on the left-hand side of the editor (at the moment marked “[empty]”) and select”Load” in the drop-down menu. A window will appear listing all the resources, including the bee image, select it, validate your choice and close the menu with “Open”.

Now is a good time to safe your scene, (“Scene”, “Save Scene” or the obvious Ctrl+S shortcut). Scenes have the tscn extension, we will only have one, let’s name it… “main”. Launch the scene to see how it looks for now (pres “F5” or click on the “play” arrow on the upper right). Godot will ask you confirmation that current scene is the main scene (it needs to point to one scene upon launching the game). In the new window, that shows what players would see upon lauching the game, only part of the sprite will be displayed. Indeed, by default the sprite was centered around the (0, 0) coordinates, and by convention the origin is the upper left corner of the screen. Go back to the editor, ove the bee so that you can see its entirety, either using the mouse (left click, drag and drop) or by changing the “Node2D” property of the sprite. You can also scale the size of the sprite or rotate it.

Installing and testing godot-python

On desktop and within the editor the simplest solution to install godot-python is to download it from the “AssetLib” tab (upper part of the screen). Search for “python” and pick “PythonScript”. It’s version 0.50.0 as for now. Select, click on “Download” in the new window, wait for the file to download in the AssetLib tab, click “Install” once done. The asset weights about 100MB as of v0.50.0, it contains the whole Python ecosystem for the four supported platforms. Upon installation, in the file list, you might want to un-check platforms you don’t plan to use during development and only keep yours (e.g. pythonscript/x11-64 for Linux), but it won’t hurt to keep everything. Wait for the asset to uncompress. It will reside in the addons sub-folder of the Godot project. As per instruction, you must close Godot and relaunch the project for godot-python to be properly installed. Do that, save if asked to.

To test the Python environment, go to the “Script” tab, click “File”, “New Script…”. Similarly to other engines, a script in Godot is attached to an object (a node). This one will change the position of the sprite, which is a property inherited from the “Node2D” class, that you will enter in the “Inherits” field. After setting the name of the script (here left_right.py, as we will translate the sprite back and forth), hit “Create”.

The python script comes with a default code. There should be one method _ready(), which will be called upon initialization of the class. The ready() method of each class will be called upon loading a scene, any blocking call here will delay or freeze the spashscreen when you first start the game. The should also be two class variables at this stage. For example a = export(int) declare a (integer) variable that will be accessible within the editor, very handy to easily set parameters to different nodes even if they share the same script. In our first code we will use an “exported” variable to set the speed of the sprite. To update the sprite we will need to add a new method _process(), that will be called upon each rendering loop by the engine. The python layer expose various variables, including the current position of the sprite. The idea is to save the initial position upon launch, and move the sprite from left to right with a configurable step between each loop. Coordinates will be handled through the Vector2 object and we will use a direction flag to switch from left to right. Borders will be hard-coded, let’s be a bit dirty. Simple enough to check that python runs correctly. The full code left_right.py is as follow:

from godot import exposed, export, Node2D, Vector2

@exposed
class left_right(Node2D):
	
	# translate node that many pixel each loop, variable exposed to editor.
	step = export(int, default=10)
	# initial position of the node
	init_pos = Vector2(0,0)
	# flag to switch from left to right
	direction = 1
	
	def _ready(self):
		"""
		Called every time the node is added to the scene.
		Initialization here.
		"""
		# save initial position
		self.init_pos = self.position

	def _process(self, delta):
		"""
		Called for each rendering. Main code here.
		"""
		# when the node goes to much on the left or on the right, revert direction
		if self.position.x > self.init_pos.x + 100:
			self.direction = -1
		elif self.position.x < self.init_pos.x - 100:
			self.direction = 1
		# apply translation change
		self.position += Vector2(self.step * self.direction, 0)

To associate the script to the sprite, within the editor drag and drop the file to the Sprite node (or right click on the Sprite node, select “Attach Script”, choose Python and select existing file). There are many area of improvements, for exemple delta could be use to smooth animation even if the framerate varies, but if you launch the scene, you should see a buzzing bee. Note that fore some reason the variables exposed by the Python script are not updated in the editor before you restart the project (“step” will not appear in the “Inspector” dock on the right). One bug, probably more to go.

Controlling a sprite with LSL

Those interested in next-gen biofeedack applications will have jumped right here. This time the objective is to fetch link the movement of the sprite with input coming from a LSL stream. To do that we will use one script to generate the data. This script is to be ran outside Godot, within your usual Python environment, or on another computer of the same network (Note: it should be possible to exchange signals through Internet as well, but it requires dwelling in LSL configuration, out of scope). If you never used LSL before, install it through pip: python3 -m pip install pylsl (possibly add the --user flag depending on desired scope). It should fetch the last version on desktop, 1.14 (which contains noticeable improvements in terms of CPU consumption compared to the previous ones).

Coordinates in a 2D space: two channels. To have the bee moving circles, sinus and cosinus of the same input. The simplest code to achieve this result (do not hesitate to grab more interesting bits from the official examples) to be included in a SendDataSin.py:

"""
Example program to demonstrate how to send time series to LSL
"""

import math, time

from pylsl import StreamInfo, StreamOutlet, local_clock

if __name__ == '__main__':
    # Sampling rate, on par with usual vsync framerate
    srate = 60
    # Identifier of the stream. Usually the name describe the device / model used, the type what data is being sent. Sticking with traditional examples values.
    lsl_name = 'BioSemi'
    lsl_type = 'EEG'

    # A stream info describe the meta data associated to the stream. We create two channels (x and y translations) of floats.
    # The last parameter is an ID that should be unique, used to automatically reconnect in case the stream is interrupted.
    info = StreamInfo(lsl_name, lsl_type, 2, srate, 'float32', 'myuid1337')

    # The outlet is the actual "pipe" outputing data, running in a separate thread
    outlet = StreamOutlet(info)

    print("now sending data...")
    # Infinity and beyond
    while True:
        # Prepare and send data
        mysample = [math.sin(local_clock()), math.cos(local_clock())]
        outlet.push_sample(mysample)
        # Wait before next push.
        # Note that relying on time.sleep to set the effective sampling rate (how many samples are sent per second) is sub-optimal, it will likely be slower and unstable
        # Check official LSL examples for a better way to steadily send data, e.g. compute how many samples are needed between two loops.
        time.sleep(1./srate)

There are many options related to LSL, the most important for performance being the buffer size (both on outlet and inlet) and the number of samples sent at once − check the well done documentation. If the system becomes laggy or unresponsive, the culprit is probably there. Run python3 SendDataSin.py and leave it live the big dream in the background. Now that we send data, we can take care to retrieve data within Godot.

First you need to also install LSL within the godot-python environment. Launch a terminal (we are still outside of Godot for now), and navigate to your Godot projet folder, e.g. LSLExemple. Still assuming that you are using Linux, Python binaries are located in addons/pythonscript/x11-64/bin (switch x11-64 for windows-64 or osx-64 accordingly to your situation). Most probably when the asset was installed the executable flag was not preserved. Fix that with chmod +x ./addons/pythonscript/x11-64/bin/* Now you can execute the standalone Python environment directly. Because the soft link were not preserved as well, we will have to aim at the exact Python version used (e.g. python3.8 rather than pyhton3). Install pip itself with ./addons/pythonscript/x11-64/bin/python3.8 -m ensurepip (you can also cd directly to the bin folder, I keep the path for clarity). Finally, as you did before with your own installation, to install LSL run ./addons/pythonscript/x11-64/bin/python3.8 -m pip install pylsl

On desktop not only pip will retrieve the corresponding .py files, but it will also fetch locally the .so library (or .dll or .dylib, each system its beloved extension). This subtlety will come handy in next part on Raspberry, because it won’t do that properly. Note that if you plan to use other libraries you will need to install them likewise, and you will have to do so for each target platform (there is no automatic process at the moment to do so). If you create another project, you can copy over the addons folder to keep all installed modules.

Now that everything is ready for Python to use LSL, head back to the Godot editor. Create a new script, still Pyhton, still inheriting from Node2D. Let’s call this one LSL_translations.py. The script will ressemble left_right.py, except that it will import the LSL module (you can import anything you have installed!) and fetch data from SendDataSin.py script. The translation will be applied to both X and Y axis.

from godot import exposed, export, Node2D, Vector2
from pylsl import StreamInlet, resolve_bypred

@exposed
class LSL_translations(Node2D):
	
	# magnitude of the applied translation
	factor = export(int, default=100)
	# initial position of the node
	init_pos = Vector2(0,0)
	# flag to switch from left to right
	direction = 1
	# LSL input stream
	inlet = None
	
	def _ready(self):
		"""
		Called every time the node is added to the scene.
		Initialization here.
		"""
		# save initial position
		self.init_pos = self.position
		
	def check_stream(self):
		"""
		Try to find the LSL stream on the network. Change predicate depending on target.
		WARNING: due to timeout option will block execution for the whole Godot engine upon each request.
		TODO: use threads to prevent blocking calls.
		"""
		if self.inlet is None:
			print("looking for stream init")
			streams = resolve_bypred("type='EEG'", timeout=0.1)
			if len(streams) > 0:
				# create a new inlet to read from the stream
				self.inlet = StreamInlet(streams[0])
				print("got stream")
				
	def _process(self, delta):
		"""
		Called for each rendering. Main code here.
		"""
		self.check_stream()
		if self.inlet is not None:
			# fetch data from inlet
			data, _ = self.inlet.pull_sample(timeout=0)
			# To maximize responsiveness, pull until last value in the buffer
			# Note: input with very high bandwidth might block forever execution here.
			while data is not None and len(data) >= 2:
				# expect two channels, translation from the initial position for X and Y.
				self.position = self.init_pos + Vector2(data[0]*self.factor, data[1]*self.factor)
				#print("got value: %s" % str(data))
				data, _ = self.inlet.pull_sample(timeout=0)

If you launch the scene and still SendDataSin.py is still running, you should see the bee hovering around its initial position. Well done! If you want to change the radius of the circle, tune the factor property (remember to restart Godot for any exported variables to be reflected in the editor the first time you add or change them). As indicated in the comments there some limitation and caveats with such simple code; we covered the basics, your turn to shine with actual physiological signals. Exporting the project (a toned down version of it due to the limitations of godot-python) is covered below.

Part 2: running a python Godot project on the Raspberry Pi

You mastered Godot, python and LSL. Next challenge: reproduce all of that on the single board computer. People only interested in having Python scripts with Godot on the Pi, welcome!

godot-python for Raspberry Pi and aarch64, long version

Godot-python needs to be modified to account for the “aarch64” platform that we will use (64-bit version of the ARM). The long route is to go back to the godot-python source and compile it on a Raspberry Pi. Fetch the source code, change lines in platforms/x11-64/SConscript, pythonscript/SConscript and site_scons/site_tools/cython.py to reflect that we update python-build-standalone from the Python 3.8 currently used to Python 3.9, for now the only version of Python supported by python-build-standalone on aarch64 Linux. Basically track down all “3.8” and “38” entries in those three files to substitute “3.9” and “39”.

Godot-python points to a python-build-standalone binary. Even by following carefully the instructions I did not manage to compile from scratch python-build-standalone (same symptoms as this bug issue), instead I retrieved automatic builds from its continuous integration (there is not yet official builds on the the “release” page). You will need to log in with a Github account to see the artifacts produced by the continuous integration. Fetch the output of the build of interest cpython-3.9-aarch64-unknown-linux-gnu, extract one of the tar.zst archive − I picked the version with optimizations enabled at compile time, suffixed “lto” −, place the archive in the file system and point godot-python to the right location by changing the variable PYTHON_PREBUILD_URL in platforms/x11-64/SConscript, pythonscript/SConscript, e.g. PYTHON_PREBUILD_URL = "file:///tmp/cpython-3.9.5-aarch64-unknown-linux-gnu.tar.zst".

The shorter route is to retrieve the patched code directly from our fork (in the future it will be a better option to create a dedicated version and merge changes back upstream to be in sync with latest modifications). Follow then the repository instructions to actually build godot-python. Compilation will still take hours, and stress the memory of the Raspberry Pi. I had to increase the swap size to its maximum for the compilation to success. To do that disable current swap sudo dphys-swapfile swapoff. Edit /etc/dphys-swapfile with your preferred editor (that would be Emacs and evil-mode), set CONF_SWAPSIZE=16384 and CONF_MAXSWAP=16384. 16GB of swap will take that much space on the SD card, check you have enough free space. Actually with these parameters we must hit a limit because on my system “only” about 12GB were allocated. And it barely sufficed to compile godot-python (I first tried and failed with both 2GB and 8GB) Allocate sudo dphys-swapfile setup and enable swap sudo dphys-swapfile swapon. (Best revert the changes after the compilation is done).

godot-python for Raspberry Pi and aarch64, short version

The shortest route is to grab a godot-python already compiled for our target.

Running a Godot project on the Raspberry Pi

At this stage you have a Godot project (no need to change the beautiful scene you came up with in the first part) and a godot-python addon for the Pi. Next: Godot on the Pi. This step will be easy, FRT proposes binaries ready to employ on its sourceforge repo. You can of course recompile if you with some specific options or optimization; I went not there. Download the frt_100_332.zip archive of the 1.0.0 version (last one as of today, tested with a project made with, indeed, Godot 3.3.2). Extract, pick for the Raspberry Pi 4 the file frt_100_332_arm64.bin (arm64 is synonyme to aarch64). Note: if you aim at 32bit distributions, you are on your own! The FRT project contains information graphic driver available, head there if you want to optimize user experience, especially if you go 3D.

frt_100_332_arm64.bin will be your “interpreter”, or “launcher”, for the Godot project. A normal way to proceed would be to use this file as a template in the Godot editor to neatly package everything in one binary, cipher a bit, but godot-python cannot be packaged that way. Instead we need to export the Godot project as a Zip package. So we will just do that. On the desktop, in the editor, select “Project”, “Export…”. Add a a preset, select Linux/X11. You don’t have to touch any option, hit “Export PCK/Zip”. Enter a filename ending with .zip, e.g. LSLDemo.zip. Export to a location outside the project folder to keep it tidy.

Transfer the zip file to the Raspberry Pi (I personally favor scp). Create a folder to hold your project, and extract the zip file in there. Do not make stupid mistakes like myself, such as erasing the .import folder (it is not decorative and you would loose like an hour getting over weird “ressource not found” errors). Copy frt_100_332_arm64.bin in the project folder. If you run the binary, it should detect the project automatically and try to launch it. It will fail because there is no addons folder at all in the zip file; except for pythonscript.gdnlib (the file indicating to Godot where to fetch the native library that is used as an interface with the standalone Python) the godot-python asset was not copied over during export. And even it it were, we would end up with the wrong architecture.

Importing custom godot-python to the project

In the previous episode the addons folder was obliterated. All hope in not lost: we happen to have retrieved a godot-python archive tailored for the Raspberry Pi. Extract godot-python-0.50.0+dev-x11-64_aarch64.tar.bz2 (it will be minus “aarch64” if you compiled it yourself) in the Godot project folder (reminder: the command line is tar xvf). It will create an addons folder with everything needed to run python in Godot, we are back on track. Because FRT declares a new platform upon compilation, we have to tune pythonscript.gdnlib so it can link our version of the Godot “interpreter” with our version of godot-python. Edit the file and under the section [entry] add the line FRT="res://addons/pythonscript/x11-64/libpythonscript.so" (you can leave the other entries). Now if you run ./frt_100_332_arm64.bin the program will start, the scene should load, but an error will be raised because LSL cannot load. Damn, we left it on the side of the road!

Note: on my system I am also spammed by messages about “ALSA lib pcm.c:8424:(snd_pcm_recover) underrun occurred”, probably some configuration to perform around ALSA or PulseAudio, you can disable audio altogether with the parameter --audio-driver Dummy

LSL: the last stand

To install LSL within the Python environment of our Godot project now on the Pi, we need to reproduce earlier steps. Namely, in the project folder on the Pi run ./addons/pythonscript/x11-64/bin/python3.9 -m ensurepip and ./addons/pythonscript/x11-64/bin/python3.9 -m pip install pylsl (notice the change to python3.9).

Did you think it would be enough? Of course not. Not only acute eyes will have seen that the LSL version downloaded on the Pi is an old 1.12.2 (and we do want this shiny 1.14, especially on the Pi), but even if you tried to go forward, you would face an error: current pip package of pylsl does not include a build for ARM. Hopefully this will be fixed in the future, in the meantime you need to grab the proper version.

Either retrieve the last stable release of the LSL sources, install some basic packages on the system to ensure compilation, such as build-essential and libboost-dev, run the standalone_compilation_linux.sh script and wait a couple minutes for the liblsl.so to be backed. arrive.

Or fetch directly the compiled file from there. No matter the solution, copy the .so file to the Python environment, overwriting the “faulty” file, e.g. in the project folder cp liblsl_aarch64_debian.so addons/pythonscript/x11-64/lib/python3.9/site-packages/pylsl/liblsl64.so (we mix a 1.12.2 API with the 1.14.0 library, not ideal but good enough for now, if necessary you can also manually update the python overlay).

[update] As of 22/05/05, latest versions of pylsl changed the way they look for the LSL library. They will try to load from the system install, not the local python install. If pylsl does not work for you on the Pi, with error messages related to the location of the library, here is a quick and ugly fix: force the installation of an older version that is still available on ARM: ./addons/pythonscript/x11-64/bin/python3.9 -m pip install pylsl==1.12.2 A better solution is for sure to install the compiled .so somewhere in LD_LIBRARY_PATH, I’ll have to test that, comments are welcomed if you do.

Now everything should finally be in place. If the SendDataSin.py script is running on the Pi (or on your desktop computer, should both machine be connected to the same Wifi), and if you launch the FRT binary (from ssh DISPLAY=:0.0 ./frt_100_332_arm64.bin), you should be met with excitation by the bee, happily circling forever after.

Conclusions

This is it, you tamed Godot, Python and LSL, conquered Raspberry Pi. The tutorial is still rough on the edges − finger crossed that it will become outdated soon, with Python and ARM officially supported in Godot at some point −, and yet it opens doors for creating quick prototypes. You can now add few assets, some lines of code and start creating environment that can respond in real time to physiological signals, plugging it to any screen or projector.

This tutorial is nothing more than gluing existing blocks produced by the community, thanks to those behind each one of them!

Do not hesitate to comment below or e-mail us if some instructions are unclear, if we blatantly missed something or if you cannot reproduce the tutorial. We do not aim at supporting the whole pipeline, but we will do our best to help. And if you produce something out of this page, we are also eager to know about it 😉