Embedding Python – Tutorial – Part 1

This is a follow-up to my talk at EuroPython 2012, “Supercharging C++ Code with Embedded Python“. An embedded Python interpreter allows users to extend the functionality of the program by writing Python plug-ins. In this series of tutorials, I will give you step-by-step instructions on how to use the Python/C API to do this.

I assume that you know how to write and compile C/C++ programs. If you have prior experience with writing Python extension modules, it may be helpful, although it’s not required. In my article about extending Python, you can find instructions for setting up your Makefiles/workspaces when working with the Python/C API.

The Example Program

In this part, we’re going to add Python plug-ins to a simple C++ program. The program reads lines of text from STDIN and outputs them unmodified to STDOUT.

#include <iostream>
#include <string>

int main(int argc, char* argv[])
{
    std::clog << "Type lines of text:" << std::endl;
    std::string input;
    while (true)
    {
        std::getline(std::cin, input);
        if (!std::cin.good())
        {
            break;
        }
        std::cout << input << std::endl;
    }
    return 0;
}

The user should now be able to write a Python plug-in that modifies the string before it is printed. These plug-ins will look something like this:

# elmer_fudd_filter.py
def filterFunc(s):
    return s.replace("r", "w").replace("l", "w")

# shout_filter.py
def filterFunc(s):
    return s.upper()

To make this work, we will link the program to the Python interpreter and use the Python/C API to import the plug-in module and to invoke the “filterFunc()” function inside it.

As a first step in that direction, we simply initialize the Python interpreter with “Py_Initialize()” and shut it down with “Py_Finalize()”.

#include <Python.h>
#include <iostream>
#include <string>

int main(int argc, char* argv[])
{
    Py_Initialize();
    std::clog << "Type lines of text:" << std::endl;
    std::string input;
    while (true)
    {
        std::getline(std::cin, input);
        if (!std::cin.good())
        {
            break;
        }
        std::cout << input << std::endl;
    }
    Py_Finalize();
    return 0;
}

According to the Python documentation, the include directive for “Python.h” should appear first in the C/C++ file. It doesn’t sound like a good idea to require this, but if you don’t put “Python.h” first, you might get compiler warnings like these:

In file included from /usr/include/python2.7/Python.h:8:0,
                 from program.cpp:3:
/usr/include/python2.7/pyconfig.h:1161:0: warning: "_POSIX_C_SOURCE" redefined [enabled by default]
/usr/include/features.h:164:0: note: this is the location of the previous definition
/usr/include/python2.7/pyconfig.h:1183:0: warning: "_XOPEN_SOURCE" redefined [enabled by default]
/usr/include/features.h:166:0: note: this is the location of the previous definition

To compile and link the program, you need to have the Python headers and the static Python library on your machine. On Windows, the headers are in the “include” directory of the Python installation, and the .lib file is in the “libs” directory. On GNU/Linux, these files come with the Python development package. On my Ubuntu 12.04 system, I had to install “python-dev”: “sudo apt-get install python-dev”.

Here’s an example command line to build the program using GCC:

g++ -o program program.cpp -I/usr/include/python2.7 -Wall -lpython2.7

If you are on Windows and using Visual Studio, see my article about extending Python if you need help setting up your project.

Invoking the Plug-In

Once this works, we can start adding the code to call the Python plug-in. To find the appropriate Python/C API calls, it always helps to think about the equivalent Python code first. Written in Python, our finished program might look something like this:

PLUGIN_NAME = "shout_filter"

def CallPlugIn(ln):
    plugin = __import__(PLUGIN_NAME)
    filterFunc = getattr(plugin, "filterFunc")
    args = (ln,)
    ln = filterFunc(*args)
    return ln

while True:
    ln = raw_input()
    if not ln:
        break
    ln = CallPlugIn(ln)
    print ln

The “CallPlugIn()” function imports the “shout_filter.py” module, retrieves the “filterFunc()” function, and invokes it. The function could certainly be written in a more concise, more Pythonic way. However, it’s easier to find the corresponding Python/C API calls when the code is broken down into its basic building blocks like “__import__()” and “getattr()”.

By digging through the Python/C API reference manual, we can find the API calls for each piece of Python code:

Python Python/C API
__import__ PyImport_Import()
getattr PyObject_GetAttrString()
args = (ln,) Py_BuildValue()
filterFunc(*args) PyObject_CallObject()

Thus, the first attempt at writing the “CallPlugIn()” function in C++ looks like this:

static const char* PLUGIN_NAME = "shout_filter";

std::string CallPlugIn(const std::string& ln)
{
    PyObject* pluginModule = PyImport_Import(PyString_FromString(PLUGIN_NAME));
    PyObject* filterFunc = PyObject_GetAttrString(pluginModule, "filterFunc");
    PyObject* args = Py_BuildValue("(s)", ln.c_str());
    PyObject* result = PyObject_CallObject(filterFunc, args);
    return PyString_AsString(result);
}

In the main routine, call the function before printing the line of text:

std::cout << CallPlugIn(input) << std::endl;

In its current form, the “CallPlugIn()” function has two major problems:

  • There’s no error checking. When anything goes wrong (the module can’t be imported, “filterFunc()” raises an exception, etc.), the program will likely crash.
  • There are memory leaks. We create a number of objects, but we never decrement their reference counts. Eventually, the program will run out of memory.

We will fix the leaks later. For now, let’s at least return an error message if anything goes wrong. Most API calls that return a “PyObject*” return a NULL pointer if an error occurred. In this case, it is the caller’s responsibility to handle/report the error and to clear Python’s internal error indicator with “PyErr_Clear()”. To print the traceback of the last error, use “PyErr_Print()”, which has the side effect of also clearing the error indicator. It’s important to clear the error as soon as possible, otherwise subsequent Python calls might fail in unexpected ways or give you misleading error messages.

std::string CallPlugIn(const std::string& ln)
{
    PyObject* pluginModule = PyImport_Import(PyString_FromString(PLUGIN_NAME));
    if (!pluginModule)
    {
        PyErr_Print();
        return "Error importing module";
    }
    PyObject* filterFunc = PyObject_GetAttrString(pluginModule, "filterFunc");
    if (!filterFunc)
    {
        PyErr_Print();
        return "Error retrieving 'filterFunc'";
    }
    PyObject* args = Py_BuildValue("(s)", ln.c_str());
    if (!args)
    {
        PyErr_Print();
        return "Error building args tuple";
    }
    PyObject* result = PyObject_CallObject(filterFunc, args);
    if (!result)
    {
        PyErr_Print();
        return "Error invoking 'filterFunc'";
    }
    const char* cResult = PyString_AsString(result);
    if (!cResult)
    {
        PyErr_Print();
        return "Error converting result to C string";
    }
    return cResult;
}

Create the file “shout_filter.py” in the same directory that you run the program from and add a valid “filterFunc()” function:

def filterFunc(ln):
    return ln.upper()

When you run the program now, you get an error (most likely): “Error importing module”. Why is that?

Normally, when importing a module, Python tries to find the module file next to the importing module (the module that contains the import statement). Python then tries the directories in “sys.path”. The current working directory is usually not considered. In our case, the import is performed via the API, so there is no importing module in whose directory Python could search for “shout_filter.py”. The plug-in is also not on “sys.path”. One way of enabling Python to find the plug-in is to add the current working directory to the module search path by doing the equivalent of “sys.path.append(‘.’)” via the API.

Py_Initialize();
PyObject* sysPath = PySys_GetObject((char*)"path");
PyList_Append(sysPath, PyString_FromString("."));

Run the program again and type a few lines of text. Everything should work now. You can also try to deliberately introduce errors into the plug-in function and see whether they are caught by the program.

When you think about it, this is quite an achievement: You just added a full-fledged scripting language to your program with an amazingly small amount of code.

Reference Counting

As I mentioned, this program leaks memory pretty badly. For example, every time we run “Py_BuildValue()”, a tuple and a string object are created, but they are never freed. The reference count (refcount) of the tuple is initially 1 and we never decrement it, so the object remains alive forever. The next time we run “CallPlugIn()”, a new object is created.

Whenever you receive a “PyObject*” via the Python C/API, you need to figure out whether you are responsible for decrementing its refcount. The API docs distinguish between these cases:

  • New reference. The refcount has been incremented before the object was returned to the caller. The caller is reponsible for decrementing the refcount with “Py_DECREF()” when the object is no longer needed.
  • Borrowed reference. The refcount has not been incremented before the object was returned. For example, the “PyTuple_GetItem()” function, which returns an item of a tuple, returns a borrowed reference. You can work with the item normally, at least as long the item is still in the tuple. When the tuple is destroyed, though, the item may be destroyed (if the refcount reaches zero). If you need to keep a reference to the item for longer, you are responsible for incrementing the item’s refcount yourself with “Py_INCREF()”.

The docs also talk about “stolen references“. Sometimes when you pass an object to an API, the API will “steal” the reference, which means that the API will take care of decrementing the refcount at some point and that the caller must refrain from doing the same.

The “CallPlugIn()” function with added reference counting:

std::string CallPlugIn(const std::string& ln)
{
    PyObject* name = PyString_FromString(PLUGIN_NAME);
    PyObject* pluginModule = PyImport_Import(name);
    Py_DECREF(name);
    if (!pluginModule)
    {
        PyErr_Print();
        return "Error importing module";
    }
    PyObject* filterFunc = PyObject_GetAttrString(pluginModule, "filterFunc");
    Py_DECREF(pluginModule);
    if (!filterFunc)
    {
        PyErr_Print();
        return "Error retrieving 'filterFunc'";
    }
    PyObject* args = Py_BuildValue("(s)", ln.c_str());
    if (!args)
    {
        PyErr_Print();
        Py_DECREF(filterFunc);
        return "Error building args tuple";
    }
    PyObject* resultObj = PyObject_CallObject(filterFunc, args);
    Py_DECREF(filterFunc);
    Py_DECREF(args);
    if (!resultObj)
    {
        PyErr_Print();
        return "Error invoking 'filterFunc'";
    }
    const char* resultStr = PyString_AsString(resultObj);
    if (!resultStr)
    {
        PyErr_Print();
        Py_DECREF(resultObj);
        return "Error converting result to C string";
    }
    std::string result = resultStr;
    Py_DECREF(resultObj);
    return result;
}

Note that I try to decrement the refcount of each object as soon as possible. For example, after retrieving the “filterFunc” callable from the “pluginModule” object, we can immediately decrement the refcount of the “pluginModule” object. The underlying module will not go away, since its reference count is not zero yet.

We also need to make sure that the refcount is properly decremented even if we leave the function early due to an error. For example, when we fail to build the arguments tuple, we decrement the refcount of the “filterFunc” (the only object we have a reference to at that point in the code) before leaving the function.

At the end of the function, we must not decrement the refcount of the “resultObj” string object before we have created a copy of the underlying C string. (The pointer returned by “PyString_AsString()” is only valid as long as the string object has a refcount greater than zero.)

We created another temporary object when setting up “sys.path”. This must be freed as well:

Py_Initialize();
PyObject* sysPath = PySys_GetObject((char*)"path");
PyObject* curDir = PyString_FromString(".");
PyList_Append(sysPath, curDir);
Py_DECREF(curDir);

Note that “Py_DECREF()” is not called on “sysPath” since that one is a borrowed reference.

You might already see the problem with this: It is way too easy to make mistakes. If you forget to call “Py_DECREF()”, you have a leak. If you call “Py_DECREF()” on a borrowed reference, you probably cause a crash. With error handling mixed in, it’s very easy to cause both kinds of problems. Using a C++ library that wraps PyObjects and takes care of reference counting solves these issues (mostly). This will be the topic of a future tutorial.

Debugging Memory Leaks

Sometimes you will need to debug memory issues no matter whether you are using a C++ wrapper or not. For this, it is very useful to have a debug version of the Python interpreter. On GNU/Linux, you can usually just install it from the repositories. For example, on my Ubuntu 12.04 system, I have to “sudo apt-get install python-dbg”. I then build the program with debug options:

g++ -o program program.cpp -I/usr/include/python2.7 -Wall -DPy_DEBUG -g -lpython2.7_d

(Don’t forget the “Py_DEBUG” preprocessor definition when linking against the debug interpreter. Otherwise, you might see crashes and errors like: “Fatal Python error: UNREF invalid object”.)

On Windows, you might have to compile a debug version of the Python interpreter yourself.

One of the things that a debug interpreter allows you to do is query the total reference count of all objects. By calling the function “sys.gettotalrefcount()” at different points in your program, you can check whether this number remains stable.

void PrintTotalRefCount()
{
#ifdef Py_REF_DEBUG
    PyObject* refCount = PyObject_CallObject(PySys_GetObject((char*)"gettotalrefcount"), NULL);
    std::clog << "total refcount = " << PyInt_AsSsize_t(refCount) << std::endl;
    Py_DECREF(refCount);
#endif
}

...

int main(int argc, char* argv)
{
    ...
    std::cout << CallPlugIn(input) << std::endl;
    PrintTotalRefCount();
    ...
}

If you try to remove one of the “Py_DECREF()” calls in “CallPlugIn()”, you will notice that the total reference count goes up after each invocation.

Possible Improvements

The “CallPlugIn()” function in its current form is slightly inefficient. We don’t really have to re-import the plug-in module and retrieve the “filterFunc()” function each time a line of text needs to be transformed. (It’s not as bad as it may appear, though. Once the module has been imported, it remains in “sys.modules”, so each time we call “PyImport_Import()”, we receive a reference to the existing module object.) One possible optimization would be to keep a reference to the “filterFunc” object during the entire lifetime of the program. Then, for each invocation of “CallPlugIn()”, we’d merely have to build an arguments tuple and invoke “filterFunc()”.

If you implement this optimization, though, try to keep your regular C++ code separate from the parts that require knowledge of the Python/C API. It is nice to only use standard C++ types in the interface of the “CallPlugIn()” and be able to keep the PyObjects and Python/C API calls an implementation detail.

Next Time

In the next part, we’ll start with a more complex project. Among other things, we will give the Python plug-ins access to the application classes.

You can download the complete source code for this tutorial.

4 comments to Embedding Python – Tutorial – Part 1

Leave a Reply

  

  

  

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Ads