Building Hybrid Systems with Boost.Python


Author: David Abrahams
Organization: Boost Consulting
Date: 2003-03-19
Author: Ralf W. Grosse-Kunstleve
Copyright: Copyright David Abrahams and Ralf W. Grosse-Kunstleve 2003. All rights reserved
筒뺼/th> ͵־ӂSlowness Chen켯a>
ҫ΄輐o 2008-05-29

Table of Contents




Boost.Python is an open source C++ library which provides a concise IDL-like interface for binding C++ classes and functions to Python. Leveraging the full power of C++ compile-time introspection and of recently developed metaprogramming techniques, this is achieved entirely in pure C++, without introducing a new syntax. Boost.Python's rich set of features and high-level interface make it possible to engineer packages from the ground up as hybrid systems, giving programmers easy and coherent access to both the efficient compile-time polymorphism of C++ and the extremely convenient run-time polymorphism of Python.

Boost.Pythonʇһ趿ꔴC++썊˼̡驁˒븶첃組IDLʽ儽ӿڣ썊ӃӚ++ ꍺ೶赽Pythonᣍ ꃫ+ǿ䳵ıҫʱĚʡĜfꍗ碕鴟Ԫᠳ̼습㬍 ೶蹤׷ͪȫӃ促++ʵϖ㬍 渃듐ҽȫЂ儓﷨ᣍ Boost.Python岸뵄̘ꍸ߼潓썊ʹ動ꈫ഻캏ϵͳɨ솈༳Ɏ꿉Ĝ㬍 ⢈o̐┬Ҕǡ¯l頁ķ튽㬍슱ʹӃ C++蟐組᠒늱櫨죬 ꍐython쫶˱〻儔ːЊᶠ̬ᣍ



Python and C++ are in many ways as different as two languages could be: while C++ is usually compiled to machine-code, Python is interpreted. Python's dynamic type system is often cited as the foundation of its flexibility, while in C++ static typing is the cornerstone of its efficiency. C++ has an intricate and difficult compile-time meta-language, while in Python, practically everything happens at runtime.

׷Ϊ}֖ӯє㬐ythonꍃ++䦔ںܶಮҬᣍ C++һࣱ뱠ҫΪ뺆炫㬍 漸ythonʇ͖䐐儡㍊Python攜 Ѝϵͳͨ㣱눏Ϊʇ˼i뮐ԵĻ鴡㬍 渃++齉⌬ ЍϵͳʇC++Š儻銯ᣍ C++Ӑһ֖贔Ӽ艮儱ҫʱԪӯє㬍 漁ڐython֐㬼躵һǐ漷≺Ԛԋʱᣍ

Yet for many programmers, these very differences mean that Python and C++ complement one another perfectly. Performance bottlenecks in Python programs can be rewritten in C++ for maximal speed, and authors of powerful C++ libraries choose Python as a middleware language for its flexible system integration capabilities. Furthermore, the surface differences mask some strong similarities:

Ȼ渶ԺܶೌԱ4˵㬍 բ⮒쒲ҢζׅPythonꍃ++ԍꃀ륲顣 ΪK̡蟔ːЋٶȣ쐹thon㌐ⵄĜƿɒԓA++֘㬍 渴㐍C++ė畟C㬍 ΪK뱵A黮粲即쯳Ʉ܁棬 ѡԱPython׷Ϊ֐줼єᣍ 䋍⣬Ԛᭃ沮Ҭњ臖㬶Ӑһ照㏠ˆ֮䦣ꍊ

  • 'C'-family control structures (if, while, for...)
  • Support for object-orientation, functional programming, and generic programming (these are both multi-paradigm programming languages.)
  • Comprehensive operator overloading facilities, recognizing the importance of syntactic variability for readability and expressivity.
  • High-level concepts such as collections and iterators.
  • High-level encapsulation facilities (C++: namespaces, Python: modules) to support the design of re-usable libraries.
  • Exception-handling for effective management of error conditions.
  • C++ idioms in common use, such as handle/body classes and reference-counted smart pointers mirror Python reference semantics.
  • 'C'-쒗嵄ƽṹ㨩f, while, for...㩼/li>
  • ֧㖃揲攏㡢ꯊ튽ᠳ̣쒔찷ꐍᠳ̍ 㨋샇漊Ǽem>樣抽㨭ulti-paradigm㩼/em>ᠳ̓ᣣ鼯li>
  • ȏͬӯ稿ɱ䐔㨳yntactic variability㩍 攓ڌḟ亂뿉恐Ժͱf監ؒꗷӃ㬍 ̡驁˶Ԕˋ㷻֘Ԙ儹㷺֧㖡㼯li>
  • 蟼清Į㬈繒ꏺ͵촺Ʒ᣼/li>
  • 蟼淢װ뺖ƣ脛+㺃뗖䣬Python㺄㿩㩣쒔֧㖿ɖؓyⵄɨ송㼯li>
  • Ҭ㣴怭㬌ṩӐ儴펳需
  • ͨӃ儃++韓÷裬ȧhandle/body 㬺͒퓃솊ևĜָի㬍 촐ython儒퓃ӯҥ᣼/li>

Given Python's rich 'C' interoperability API, it should in principle be possible to expose C++ type and function interfaces to Python with an analogous interface to their C++ counterparts. However, the facilities provided by Python alone for integration with C++ are relatively meager. Compared to C++ and Python, 'C' has only very rudimentary abstraction facilities, and support for exception-handling is completely missing. 'C' extension module writers are required to manually manage Python reference counts, which is both annoyingly tedious and extremely error-prone. Traditional extension modules also tend to contain a great deal of boilerplate code repetition which makes them difficult to maintain, especially when wrapping an evolving API.

ҲΪPythonӐׅ岸뵄'C'ӯє쯳ɁPI㬍 ԭԲɏ㬏␹thon弳憊+ Ѝꍺ퓿ړ渃ʇģ썊⢇ҵ쳶儽ӿړ붔ӦC++儽ӿړ渃ʇϠˆ儡㍊Ȼ渣쐹thonᾉ팡駔ă++쯳ɹ愜Ϡ庋Ƚψ塣 ꍃ++㬐ythonϠሣ썊'C'ֻӐ照㻹᾵ij錚Ĝf㬍 渴ҍꈫ⻖糖Ҭ㣴怭ᣍ 'C')չģ冀ė畟ᘐ늖糖܀퐹thon儒퓃솊 բ⻽浥巷捷㬁ĕ밣춸ǒ빼뒗㶴 䫍㵄)չģ祉鍹༺촳֘貴đ鰥亂룬 ʹ˼CđҔά뤣썊ӈƤʇ屒귢װ俊PIɐ䦓ڷ╹֮֐ᣍ

These limitations have lead to the development of a variety of wrapping systems. SWIG is probably the most popular package for the integration of C/C++ and Python. A more recent development is SIP, which was specifically designed for interfacing Python with the Qt graphical user interface library. Both SWIG and SIP introduce their own specialized languages for customizing inter-language bindings. This has certain advantages, but having to deal with three different languages (Python, C/C++ and the interface language) also introduces practical and mental difficulties. The CXX package demonstrates an interesting alternative. It shows that at least some parts of Python's 'C' API can be wrapped and presented through a much more user-friendly C++ interface. However, unlike SWIG and SIP, CXX does not include support for wrapping C++ classes as new Python types.

բϞֆ弖˶֖碗ϵͳ儷╹ᣍ SWIG ܊Ǘ儃/C++ꍐython쯳ɏ即ᣍ 빓碕鴟 SIP㬍 ˼ʇרAΪ Qt ͼЎӃ맽煩輆廣썊ӃӚ̡騮t儐ython퓿ڡ㍊ ΪK樖Ɠ줵İ㶨㬓WIGꍓIP漒툫K˼Cה캵ė藪ӯєᣍ բӐһ樵ĺo棬 嫊DŽ㲻仙눥Ӧ趈햖⻍쓯є㨐ythonᢃ/C++ꍽӿړ㩣썊˹ҔҲ一䁋ʂʵɏꍐĀ퉏儀焑ᣍ CXXȭ쾰썊չʾKmһ֖nȋ萐ˈ䵄ѡԱᣍ ˼ϔʾ㬖IٿɒԷ◰⿷֐ython 'C' API㬍 샇ᭊ輓Ѻoă++퓿ڡ㍊Ȼ渣첻ϱSWIGꍓIP㬍 CXX⻄ܽ냫+ 碗೉Ђ儐ython Ѝᣍ

The features and goals of Boost.Python overlap significantly with many of these other systems. That said, Boost.Python attempts to maximize convenience and flexibility without introducing a separate wrapping language. Instead, it presents the user with a high-level C++ interface for wrapping C++ classes and functions, managing much of the complexity behind-the-scenes with static metaprogramming. Boost.Python also goes beyond the scope of earlier systems by providing:

Boost.Python 儌ؐԺ̈́ﱪӫբϵͳӐꜶ֘御㍊ Boost.PythonŬf̡蟷◰儱〻ꍁ黮㬍 嬤뒽ȫ奶5ķ◰ӯєᣍ Ϡ綷싼ͨ齾⌬Ԫᠳ̣씚Ļ곹܀儸䔓㬍 土ָ藪맒븶蟼憊+퓿ڀ䷢װC++ ꍺᣍ Boost.PythonҲԚȧςlӲ㬔큋ԧƚ粲即㺍

  • Support for C++ virtual functions that can be overridden in Python.
  • Comprehensive lifetime management facilities for low-level C++ pointers and references.
  • Support for organizing extensions as Python packages, with a central registry for inter-language type conversions.
  • A safe and convenient mechanism for tying into Python's powerful serialization engine (pickle).
  • Coherence with the rules for handling C++ lvalues and rvalues that can only come from a deep understanding of both the Python and C++ type systems.
  • ֧㖃++ꯊ⢄ܔڐython֐貸ǡ㼯li>
  • 攓ڵͼ淨C++ָիꍒ퓃㬌ṩȫa儉ꃼƚ需ֆ᣼/li>
  • ֧㖰䐹thon༗閯)չģ謹써齖ėⲡ᭽萐ӯє준Ѝת뻡㼯li>
  • ͨ齒떖ಈ뷽ᣵĻꖆ㬒툫Pythonǿ䳵Đ⁐믒퇦㨰ickle㩡㼯li>
  • ӫC++䦀헳ֵꍓҖ嵄饌⏠һւ㬸O떂ֻĜ4הӚ攐ythonꍃ++ Ѝϵͳ儉-㍊

The key insight that sparked the development of Boost.Python is that much of the boilerplate code in traditional extension modules could be eliminated using C++ compile-time introspection. Each argument of a wrapped C++ function must be extracted from a Python object using a procedure that depends on the argument type. Similarly the function's return type determines how the return value will be converted from C++ to Python. Of course argument and return types are part of each function's type, and this is exactly the source from which Boost.Python deduces most of the information required.

һ趹ؼ쐔儷⏖ƴ毀˂oost.Python兒귢㬍 촀듃C++儱ҫʱĚʡ㬿ɒԏ볽䫍〩չģ節Ĵѹॴꂫᣍ ȧy趷◰儃++ꯊ⎊ʇ䓐ython攏㌡ȡ廣썊̡ȡʱᘐ븹Ί퀠Ѝ巔IӦ儹ᣍ ˆ崩캯ʽ絻ؖ崮C++ת뻳ɐythonʱ㬍 絻ؖ嵄 Ѝ聾ȧꎗ껻ᣍ ҲΪ⎊絻ؖ嵄 Ѝʇy趺 Ѝ儒벿疣썊˹ҔBoost.PythonԴӺ Ѝ͆弳洳⿷֋鐨儐ŏ⡣

This approach leads to user guided wrapping: as much information is extracted directly from the source code to be wrapped as is possible within the framework of pure C++, and some additional information is supplied explicitly by the user. Mostly the guidance is mechanical and little real intervention is required. Because the interface specification is written in the same full-featured language as the code being exposed, the user has unprecedented power available when she does need to take control.

բ֖罷趑ւK᰼em>Ӄ맖趑儷◰㨵ser guided wrapping㩼/em>ᱣꍊԚ促++兒⼜Ě㬍 䓴װ儔䴺«֐ֱ퓌ሡɄܶൄЅϢ㬍 卑됩殍ⵄЅϢӉӃ맏Ԋ팡驡㍊ͨ㣕▖ָ弊ǗԶ﵄㬺܉ِ蒪զս儸ɉ檔 ҲΪ퓿ڹ淶ꍵ쳶亂늇ӃͬһAȫ餛ܵē廣썊屭u爺ʵҪȡ僿ؖƊᣬ ˻˹ӵӐ儈聦ʇǿ䳵ġ㍊

Boost.Python Design Goals


The primary goal of Boost.Python is to allow users to expose C++ classes and functions to Python using nothing more than a C++ compiler. In broad strokes, the user experience should be one of directly manipulating C++ objects from Python.

Boost.Python儊גꄿ᪊ǣ썊ȃӃ맖듃C++᠒놷܏␹thon弳憊+ ꍺᣍ 䳌笨쓃맵Č嗜Ӧ胊ǣ서黴Ӑythonֱ퓲ٗ燙+攏㡣

However, it's also important not to translate all interfaces too literally: the idioms of each language must be respected. For example, though C++ and Python both have an iterator concept, they are expressed very differently. Boost.Python has to be able to bridge the interface gap.

Ȼ渣쓐һ壒⺜֘Ҫ㬍 ćDz뒪齔ڼ/em>ഗփ淭ҫ˹Ӑ퓿ڣꍊᘐ뿼‡y֖ӯє儹ߓ÷衣 =ȧ㬋䈻C++ꍐython漓촺Ʒ儸ń ᭴﷽ʽȴꜲ뒻ѹᣍ Boost.Pythonᘐ넜黏볽բ֖퓿ڵIJᣍ

It must be possible to insulate Python users from crashes resulting from trivial misuses of C++ interfaces, such as accessing already-deleted objects. By the same token the library should insulate C++ users from low-level Python 'C' API, replacing error-prone 'C' interfaces like manual reference-count management and raw PyObject pointers with more-robust alternatives.