Tales on Python: in praise of the GIL

November 28, 2011

Tales notes the trend towards Python in a recent post. Couple of points in response: firstly, slang at Goldman. I’m told by ex Goldmanites that slang is very close to Python anyway. Secondly, Python’s Global Interpreter Lock. The GIL always seems a disadvantage to developers accustomed to ‘traditional’ multi-threaded environments like C++, Java and C#. Those who criticise Python from that perspective are right to point out that the GIL prevents developers from achieving concurrency within a single process, because it puts locking around access to all memory owned by the Python runtime. That is a real constraint, but one that has benefits when dealing with mixed language environments. For example, if you’re implementing a pub sub API in C++ using Python’s C API, you’ll want a callback mechanism to handle incoming messages. Native pub sub APIs typically spawn another thread on which they dispatch new message callbacks. When that thread invokes the callback implementation, the callback implementation can invoke Python code without worrying about locking or mutexes or the fact that it’s invoking Python’s C API on a different thread than the main app thread. It’s not a concern, because the GIL serializes all access to Python’s memory. Which simplifies things hugely…

About these ads

2 Responses to “Tales on Python: in praise of the GIL”

  1. Alexei Z Says:

    John,

    I’m on the fence about your pub/sub example. On the one hand, the GIL does appear to slightly simplify things here. On the other hand, if there was no GIL, how hard would it be to stick a mutex wherever one needs mutual exclusion?

    What makes the GIL a real pain is that there’s no opt-out. As you point out, this means that it’s next to impossible to achieve parallelism using multiple CPython threads.

    This is a huge constraint in many situations, and one that can be hard to overcome.

    Recently, I’ve been using both CPython and Jython. One interesting thing about Jython is that it has no GIL. Having seen both the GIL world and the no-GIL world, I’d take the latter any day of the week!

    Alexei Z

  2. George Says:

    Lock-free concurrency is a very desirable thing, but GIL may be a red herring. The problem is whether there is a friendly lock-free high-level model of lightweight concurrency that developers can fall back on. The secondary problem is thread-safe memory management and safe interoperability with C. Twisted event-driven networking and NumPy/BLAS get us part of way there, but there is still a way to go. If Python were to include facilities for nested data parallelism and perhaps some other more ambitious model such as software transactional memory, there would probably be less of an issue.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s