Boost C++ Libraries Home Libraries People FAQ More

PrevUpHome

General Techniques

ͨӃ켊彎h2>

Creating Packages
Extending Wrapped Objects in Python
Reducing Compiling Time
䴽谼
ԚPython֐)չ碗ൄ攏㼯a>
쵉ٱҫʱ줼/a>

Here are presented some useful techniques that you can use while wrapping code with Boost.Python.

բ/܁˒됩ӐӃ儼쇉㬿ɒԔړAoost.Python碗ഺ«ʱʹӃᣍ

Creating Packages

䴽谼

A Python package is a collection of modules that provide to the user a certain functionality. If you're not familiar on how to create packages, a good introduction to them is provided in the Python Tutorial.

Python༊ǒ븶ģ塚ﺏ㬌ṩ踓u璻樵Ĺ愜ᣍ ȧ黃겻ʬϤȧꎴ包ȭ쾰죬Ԛ Python팳̼/a> ֐Ӑһ趺ܺoĽ鉜ᣍ

But we are wrapping C++ code, using Boost.Python. How can we provide a nice package interface to our users? To better explain some concepts, let's work with an example.

嫊ǣ쎒CʇԚӃBoost.Python碗e+亂롣 ΒCԵѹ⅄܏⎒C儓u玀驒븶ƯA儈༽ӿڣ ΪK輺oؽ⊍һ腄ȃΒC怽דᣍ

We have a C++ library that works with sounds: reading and writing various formats, applying filters to the sound data, etc. It is named (conveniently) sounds. Our library already has a neat C++ namespace hierarchy, like so:

ΒCӐһ趃++ɹҴ궁跖ָኽᢶԉ鈷ʽ擃齂ˆ磬刵ȡ㍊ ˼㨷崩郼uΪ soundsᣍ ΒCѾ퓐һ趕뽠儃++uז䲣䎣쏱բѹ㺍

sounds::core
sounds::io
sounds::filters

We would like to present this same hierarchy to the Python user, allowing him to write code like this:

ΒCϫϲPythonӃ맳ʏ֕ⒻϠͬ儲㴎養ȃ˻ԕ6亂룺

import sounds.filters
sounds.filters.echo(...) # echo is a C++ function

The first step is to write the wrapping code. We have to export each module separately with Boost.Python, like this:

麼벽ʇ᠐䷢װ亂롣 ΒCᘐ듃Boost.Python皰ർ㶃︶ģ謹쏱բѹ㺍

/* file core.cpp */
BOOST_PYTHON_MODULE(core)
{
    /* export everything in the sounds::core namespace */
    ...
}

/* file io.cpp */
BOOST_PYTHON_MODULE(io)
{
    /* export everything in the sounds::io namespace */
    ...
}

/* file filters.cpp */
BOOST_PYTHON_MODULE(filters)
{
    /* export everything in the sounds::filters namespace */
    ...
}

Compiling these files will generate the following Python extensions: core.pyd, io.pyd and filters.pyd.

᠒땢΄쾻ቺ㉒ԏython)չ㺍 core.pydio.pydfilters.pyd

[Note] Note

The extension .pyd is used for python extension modules, which are just shared libraries. Using the default for your system, like .so for Unix and .dll for Windows, works just as well.

Python)չģ禎Aˀ久u .pyd㬍 嫋얻⻹튇鲏ᣍ ҲԊ鈴ģϵͳĬȏ儀久u㬍 ȧUnixɏʇ.so㬍 Windowsɏʇ.dll

Now, we create this directory structure for our Python package:

ϖԚ㬎҃ǎꎒC儐ython༽聢Kբ趄飺

sounds/
    __init__.py
    core.pyd
    filters.pyd
    io.pyd

The file __init__.py is what tells Python that the directory sounds/ is actually a Python package. It can be a empty file, but can also perform some magic, that will be shown later.

΄쾠__init__.py 擬Python㬍 sounds/ Ƥʵʇһ趐ython༡㍊ ˼Ԋǒ븶ļ嬡⿉Ҕִһħʵ㬽딚ɔ곕汞ᣍ

Now our package is ready. All the user has to do is put sounds into his PYTHONPATH and fire up the interpreter:

ϖԚΒC儈༒Ѿ헼Ḿ͐硣 Ӄ맖됨 sounds 煈닻儠 PYTHONPATH㬍 Ȼ곴⿪͆磺

>>> import sounds.io
>>> import sounds.filters
>>> sound = sounds.io.open('file.mp3')
>>> new_sound = sounds.filters.echo(sound, 1.0)

Nice heh?

ƯAɣ

This is the simplest way to create hierarchies of packages, but it is not very flexible. What if we want to add a pure Python function to the filters package, for instance, one that applies 3 filters in a sound object at once? Sure, you can do this in C++ and export it, but why not do so in Python? You don't have to recompile the extension modules, plus it will be easier to write it.

բʇ䴽舭쾰첣䎽ṹ׮첵嵄罷裬嫋첻ʇ꜁黮ᣍ ȧ黎҃Ǐ뒪Ԛfilters༖ӈ뒻趍 鬻/em> Python儺㬍 ለ磬Ԛsound攏㖐㬒봎ӦӃ3趹킋Ʒ㬍 胔僩׶Ę㿍 屈룬ģԓA++׶⢵쳶˼㬍 嫎ꊲo⻔ڐython֐׶Ę㿍 բѹ㬄㲻ᘖؐ±ҫ)չģ謹씙쓉ϻ᱈폈ݒא䡣

If we want this flexibility, we will have to complicate our package hierarchy a little. First, we will have to change the name of the extension modules:

ȧ黎҃Ǐ뒪բ֖i뮐ԣ썊 ΒC뵃⻈I҃ǵİ콡鹼䔓һ叡㍊ ʗψ㬎҃DZؐ븄ᤀ久ģ冀ă볆㺍

/* file core.cpp */
BOOST_PYTHON_MODULE(_core)
{
    ...
    /* export everything in the sounds::core namespace */
}

Note that we added an underscore to the module name. The filename will have to be changed to _core.pyd as well, and we do the same to the other extension modules. Now, we change our package hierarchy like so:

עҢ㬎҃Ǹ脣暑댭쓁ˏ»ᣍ ͬʱ΄쾃뽫⻵o븄Ϊ_core.pyd㬍 渴ң솤˻)չģ禍⍬ѹ䦀 ϖԚ㬎҃Lj༵IJ㴎鸄Ϊϳբѹ㺍

sounds/
    __init__.py
    core/
        __init__.py
        core.pyd
    filters/
        \_init__.py
        filters.pyd
    io/
        \_init__.py
        _io.pyd

Note that we created a directory for each extension module, and added a __init__.py to each one. But if we leave it that way, the user will have to access the functions in the core module with this syntax:

עҢ㬎҃ǎꃿ趀久ģ充包һ趄㬍 ⢇Ҏꃿ趄̭쓒븶__init__.pyᣍ 嫊ǣ숧黎҃Nj拼բѹ㬍 Ӄ맽벻仙듃բ֖ӯ稷IʺːĄ㿩֐儹愜㺍

>>> import sounds.core._core
>>> sounds.core._core.foo(...)

which is not what we want. But here enters the __init__.py magic: everything that is brought to the __init__.py namespace can be accessed directly by the user. So, all we have to do is bring the entire namespace from _core.pyd to core/__init__.py. So add this line of code to soundscore__init__.py:

բ⻊ǎ҃Ǐ뒪儡㍊ 嫠__init__.py ħʵ慠쁋㺍 렼tt class="literal">__init__.py uז䵄˹Ӑ櫎磬 Ӄ맶쿉Ĝֱ퓷Iʡ㍊ Ҳ䋣쎒CҪ׶齉͊ǣ콫ջ趃뗖䴓 _core.pyd ҽ擴 core/__init__.pyᣍ ˹ҔԚ soundscore__init__.py ֐쓈땢亂룺

from _core import *

We do the same for the other packages. Now the user accesses the functions and classes in the extension modules like before:

Ƥ˻།쑹䦀 ϖԚ㬓u翉Ҕꍒԇһѹ烎ʀ久ģ節ĺꍀK㺍

>>> import sounds.filters
>>> sounds.filters.echo(...)

with the additional benefit that we can easily add pure Python functions to any module, in a way that the user can't tell the difference between a C++ function and a Python function. Let's add a pure Python function, echo_noise, to the filters package. This function applies both the echo and noise filters in sequence in the given sound object. We create a file named sounds/filters/echo_noise.py and code our function:

殍ⵄꃴ抇㬎҃Ǐ֔ڿɒԇᒗ嘔ڈκ΄㿩֐쓈봿Pythonꯊ ȃӃ맲뱘Ǹ疃++ꯊPythonꯊ ȃΒCϲ filters ། ̭쓒븶俵ļ/em>echo_noiseᣍ 胺擽䈫儠sound 攏㍊ Ҁ䎓擃 echonoise 齂ˆ硣 ΒC䴽蒻趠sounds/filters/echo_noise.py ΄쾣첢ȧς᠐䎒C儺㺍

import _filters
def echo_noise(sound):
    s = _filters.echo(sound)
    s = _filters.noise(sound)
    return s

Next, we add this line to soundsfilters__init__.py:

퓗ţ쎒CѕⒻ쓈덊 sounds/filters/__init__.py

from echo_noise import echo_noise

And that's it. The user now accesses this function like any other function from the filters package:

Ǖ6ᣓu珖Ԛԏፊ filters ༖ĈκΆ䋻ꯊ풻ѹ烎ʕ⸶ꯊ큋㺍

>>> import sounds.filters
>>> sounds.filters.echo_noise(...)

Extending Wrapped Objects in Python

ԚPython֐)չ碗ൄ攏㼯h3>

Thanks to Python's flexibility, you can easily add new methods to a class, even after it was already created:

萐됹thon俊黮㬄㿉Ҕ꜈ݒ׵ؔڀ֐̭쓐μķ㬍 촊銇Ԛ 䴽藪곣ꍊ

>>> class C(object): pass
>>>
>>> # a regular function
>>> def C_str(self): return 'A C instance!'
>>>
>>> # now we turn it in a member function
>>> C.__str__ = C_str
>>>
>>> c = C()
>>> print c
A C instance!
>>> C_str(c)
A C instance!

Yes, Python rox. smiley

Ү㬐ython զഡ㼳pan class="inlinemediaobject">smiley

We can do the same with classes that were wrapped with Boost.Python. Suppose we have a class point in C++:

ΒCԶԓAoost.Python碗ൄ ׶ͬѹʂᣍ 왉莒CӐһ趠point C++ 㺍

class point {...};

BOOST_PYTHON_MODULE(_geom)
{
    class_<point>("point")...;
}

If we are using the technique from the previous session, Creating Packages, we can code directly into geom/__init__.py:

ȧ黎҃NJ鈴ɏһ횣썊 䴽谼 ֐儼습㬎҃ǿɒԔڍ geom/__init__.py ֐ֱ퓱«㺍

from _geom import *

# a regular function
def point_str(self):
    return str((self.x, self.y))

# now we turn it into a member function
point.__str__ = point_str

All point instances created from C++ will also have this member function! This technique has several advantages:

˹Ӑ 䓃++䴽資pointʵ=漻ᓵӐբ足ɔắʽ㡍 բ֖켊徟Ӑ츸擅ʆ㺍

  • Cut down compile times to zero for these additional functions
  • Reduce the memory footprint to virtually zero
  • Minimize the need to recompile
  • Rapid prototyping (you can move the code to C++ if required without changing the interface)
  • բ軾Ӻ儱ҫʱ줎ꁣ
  • Ě䦕쓃츺廑c
  • UəK֘᠒뵄Ҫ
  • ﬋ٔ퐍㨈瓐Ҫ㬄㿉ҔѴꂫ҆彃++㬶費ᘸ츄퓿ڣ鍊

You can even add a little syntactic sugar with the use of metaclasses. Let's create a special metaclass that "injects" methods in other classes.

ģɵցԊ鈴Ԫ 㨭etaclass㩀䌭쓉ِ퓯稌ǡ㍊ ȃΒC䴽蒻趌رൄԪ 㬓A䏲Ƥ˻ ᰗ∫ᱷᣍ

# The one Boost.Python uses for all wrapped classes.
# You can use here any class exported by Boost instead of "point"
BoostPythonMetaclass = point.__class__

class injector(object):
    class __metaclass__(BoostPythonMetaclass):
        def __init__(self, name, bases, dict):
            for b in bases:
                if type(b) not in (self, type):
                    for k,v in dict.items():
                        setattr(b,k,v)
            return type.__init__(self, name, bases, dict)

# inject some methods in the point foo
class more_point(injector, point):
    def __repr__(self):
        return 'Point(x=%s, y=%s)' % (self.x, self.y)
    def foo(self):
        print 'foo!'

Now let's see how it got:

ϖԚȃΒC4﴿佡黣ꍊ

>>> print point()
Point(x=10, y=10)
>>> point().foo()
foo!

Another useful idea is to replace constructors with factory functions:

mһ֖ӐӃ粲뷨ʇ㬓u䳧ꯊ툡亹錟ꯊ

_point = point

def point(x=0, y=0):
    return _point(x, y)

In this simple case there is not much gained, but for constructurs with many overloads and/or arguments this is often a great simplification, again with virtually zero memory footprint and zero compile-time overhead for the keyword support.

Ԛբ趼ⵥ儀헓֐㬋첢uӐʲoӃ㬍 嫈繻鷫캯ʽ֘Ԙ㬻╟⎊ բ͹͹ʇһ趾޴㵄첻 渿ꏺȔȻʇ㺼躵cĚ䦕쓃ꍁ㱠ҫʱ줡㍊

Reducing Compiling Time

쵉ٱҫʱ줼/h3>

If you have ever exported a lot of classes, you know that it takes quite a good time to compile the Boost.Python wrappers. Plus the memory consumption can easily become too high. If this is causing you problems, you can split the class_ definitions in multiple files:

ȧ黃㔸쳶KꜶ 㬍 ģ롖굀㬍 ᠒낯ost.Python碗Ҫꃳ䊱줡㍊ 빓Єڴ揻ꄻᇡҗ嘉햁Ꜹߡ㍊ ȧ黕⻡Ԭ㉎ʌ⣬ ģԽ룬ass_樒巖詎趎ļ

/* file point.cpp */
#include <point.h>
#include <boost/python.hpp>

void export_point()
{
    class_<point>("point")...;
}

/* file triangle.cpp */
#include <triangle.h>
#include <boost/python.hpp>

void export_triangle()
{
    class_<triangle>("triangle")...;
}

Now you create a file main.cpp, which contains the BOOST_PYTHON_MODULE macro, and call the various export functions inside it.

ϖԚģ䴽蒻趼tt class="literal">main.cpp΄쾣썊 ȃ˼༺켴t class="literal">BOOST_PYTHON_MODULEꪣ썊 ⢔ڀ巔o絼㶺ᣍ

void export_point();
void export_triangle();

BOOST_PYTHON_MODULE(_geom)
{
    export_point();
    export_triangle();
}

Compiling and linking together all this files produces the same result as the usual approach:

᠒뺍t퓋銬բ΄쾣쓫Ҕςͨ㣵ķϠሣ콡黊ǒ둹廣ꍊ

#include <boost/python.hpp>
#include <point.h>
#include <triangle.h>

BOOST_PYTHON_MODULE(_geom)
{
    class_<point>("point")...;
    class_<triangle>("triangle")...;
}

but the memory is kept under control.

嫄ڴ敼Ӄ롊ܵֆᣍ

This method is recommended too if you are developing the C++ library and exporting it to Python at the same time: changes in a class will only demand the compilation of a single cpp, instead of the entire wrapper code.

ȧ黃㕽Ԛ覆⃫+썊 ⢍슱쵼㶵퐹thon㬍 ͆춄㒲ʹӃ胸㺍 һ趀֐儸츄㬍 ֻ᠒뒻趣pp㬍 渲늇ջ趷◰亂롣

[Note] Note

If you're exporting your classes with Pyste, take a look at the --multiple option, that generates the wrappers in various files as demonstrated here.

ȧ黃八Ӄ Pyste 弳怠㬇뿴ﴍ --multiple ѡϮ㬍 ˼롉곉核淢װ΄쾣썊 ᕢ/˹չʾ億Ǒ顣

[Note] Note

This method is useful too if you are getting the error message "fatal error C1204:Compiler limit:internal structure overflow" when compiling a large source file, as explained in the FAQ.

屄㱠ҫһ趴㎄쾊ᣬ ȧ黓浽䭎㐅Ϣ "fatal error C1204:Compiler limit:internal structure overflow" 㨡ւu䭎204㺱ҫƷϞֆ㺄ڲ。鶊糶ᱣ養 胸ҲʇӐӃ廣썊 ͼ뼡 href="../../../../v2/faq.html#c1204" target="_top">FAQᣍ


PrevUpHome