Multiprocessing and XPPython3

If you’re trying to use python’s subprocess or multiprocessing modules, you’ll quickly find out that X-Plane gets spawned, perhaps multiple times, rather than the python executable. The result is usually a crash.

Using subprocess, you commonly will spawn a python process using sys.executable:

subprocess.Popen([sys.executable, 'mypython.py'])

-or-:

subprocess.run([sys.executable, 'mypython.py'])

(You should have no problems if you are trying to launch a process other than python using subprocess.)

Using multiprocessing, you don’t specify the executable but know that it (also) defaults to sys.executable.

The problem is that X-Plane is the python executable, for example on a Mac:

>>> print(sys.executable)
'/Volumes/MyDrive/X-Plane-11.5/X-Plane.app/Contents/MacOS/X-Plane'

On a PC:

>>> print(sys.executable)
'D:\\X-Plane 11.50\\X-Plane.exe'

So, what to do?

Finding Python

Fortunately, you can specify the correct value rather than use the provided sys.executable. The trick is how to find the correct value on all platforms.

Short answer

Use XPPython.pythonExecutable.

We’ll calculate it for you at startup and make it available via this constant, for example:

subprocess.run([xp.pythonExecutable, 'mypython.py])

For multiprocessing, you need to explicitly set the value:

multiprocessing.set_executable(xp.pythonExecutable)
p = multiprocessing.Process(target=f, args=child_conn, ))
p.start()

Long answer

We’ll find python that’s compatible with the executing X-Plane, including the same set of libraries, but you may want to use a different version of python or a virtual environment with different libraries. Rather than using xp.pythonExecutable you can calculate your own.

On Windows, python is located at:

os.path.join(sys.exec_prefix, 'pythonw.exe')

python.exe will popup a console window while executing, pythonw.exe will not, so we use the latter.

On Mac, python is located at:

os.path.join(sys.exec_prefix, 'bin', 'python3')

Normally, there will be python3 symlinked to the specific minor version (e.g., python37 or python38) Because the Mac uses different directories for each version of Python, python3 will always match the version you’re running with X-Plane.

On Linux, python is also located at:

os.path.join(sys.exec_prefix, 'bin', 'python3')

But, because sys.exec_prefix is often something like /usr, the resolution of python3 is likely to be a symlink to a version in the same directory (which itself may contain python37, python38, and python39, you’re not guaranteed to get the correct version.

To make sure you get the same version use:

os.path.join(sys.exec_prefix, 'bin', 'python3{}'.format(sys.version_info.minor))

Or, as mentioned above, just use xp.pythonExecutable

Example - PI_MultiProcess.py

Copy the following into your PythonPlugins directory.

import os
try:
    import xp
except ImportError:
    pass

import multiprocessing


class PythonInterface:

    def XPluginStart(self):
        return 'PI_MultiProcess', 'xppython.demos.multiprocess', 'Example plugin using multiprocessing'

    def XPluginEnable(self):
        xp.log("Calling from PID {}".format(os.getpid()))
        parent_conn, child_conn = multiprocessing.Pipe()
        multiprocessing.set_executable(xp.pythonExecutable)
        p = multiprocessing.Process(target=f, args=(child_conn, ))
        p.start()
        xp.log('{}'.format(parent_conn.recv()))
        p.join()
        return 1

    def XPluginDisable(self):
        pass

    def XPluginStop(self):
        pass

    def XPluginReceiveMessage(self, *args, **kwargs):
        pass


def f(conn):
    conn.send([42, None, 'hello from PID: {}'.format(os.getpid())])
    conn.close()

This is modeled after the example in python documentation for multiprocessing with Pipes. All it does is spawn a separate process to run function f(), when this plugin is enabled.

Your log should show something like:

[PythonPlugins.PI_MultiProcess] Calling from PID 38087
[PythonPlugins.PI_MultiProcess] [42, None, 'hello from PID: 38167']

Couple of things to note:

1. We call multiprocessing.set_executable(). If you don’t, it will spawn a new copy of X-Plane with disastrous results.

2. The spawned function, f(), does not call X-Plane. It’s running in a regular python process, and therefore does not have access to Laminar SDK or XPPython3 calls. In our example, we use Pipes to send data back to the parent process. You should be able to use other multiprocessing features to exchange data.

3. Because the function f() is in this file, we need to guard against the import xp. Otherwise, when the child function starts, it will load this whole file and immediately get an ImportError. Because we’ll catch that error, the child process is able to continue. Python multiprocessing documentation describes how to guard against a similar problem using __main__. But the idea is the same: when the child process starts up, you want it to run only the code it needs to run.