Improve Python Performance with Cython

Improve Python Performance with Cython

Python is my favorite language. It is simple, readable, flexible, supports many programming paradigms, has an awesome standard library, tons of third-party packages and a great community. But, it is also slow and a memory hog. I use Python for many tasks: writing little scripts to make me more productive and automate repetitive tasks, one-off scripts to migrate data or convert files, and even to write full-fledged massive scale distributed systems. Guess what? Performance matters when you write large scale distributed systems.

Normally, this is not a problem because the performance of critical parts can be executed by native libraries or extensions, but sometimes your Python code is the bottleneck. When that is the case Python has a great interoperability story as far as C/C++ goes. This article will look into the various options.

Python Implementations

There are several Python implementations. I focus here on the primary CPython?implementation. Other implementations like PyPy, IRonPython and Jython have their own ways to handle performance issues, although some of them are compatible to various degrees with CPython C extensions. Those implementations are out of scope. For the rest of this article when I write “Python” I mean “CPython”.

Python and C

Python provides a C-based API for extending and embedding. Extending means you can call C/C++ code from Python code. Embedding means you call Python code from a C/C++ program (embed a Python interpreter in it). This API existed since the dawn of time and it gives you complete control, but it is pretty hairy and you need to have a pretty good knowledge of CPython internals to use it properly and more importantly to debug issues.

Cython

Cython?is an optimizing static compiler that allows you to write Python C extensions and call C and C++ code in a syntax that is a superset of Python. It is now the recommended way (by Python core developers) to write Python extensions. Cython will take your Python-like code, possibly annotated with type information and produce a high-performance extension you can use from your Python code. Let’s look at an example: Computing the n-th Fibonacci number. Here is the Python code in fib_python.py

def fib_py(n):    """Return the n-th Fibonacci number."""    a, b = 0, 1    for i in range(n):        a, b = b, a + b    return a

The Cython code is identical except that it is in a file called fib_cython.pyx and I renamed the function to fib_cython, so I can import both without name conflicts

def fib_cython(n):    """Return the n-th Fibonacci number."""    a, b = 0, 1    for i in range(n):        a, b = b, a + b    return a

Finally, another Cython variation with type information in a file called fib_cython_with_types.pyx. Here the a,b and i variables are annotated with Cython’s ‘cdef int’ to let Cython know those variables should be treated as C integers

def fib_cython_with_types(n):    """Return the n-th Fibonacci number."""    cdef int a = 0    cdef int b = 1    cdef int i    for i in range(n):        a, b = b, a + b    return a

Before using Cython code you either need to create a setup.py and use some special build instructions or take the highroad and use pyximport as follows:

import pyximportpyximport.install()

Now, you can import .pyx Cython modules just like normal Python modules and they will be compiled on the fly when you import them. I then proceeded to import all 3 functions from the Python module and two Cython modules:

from fib_py import fib_pyfrom fib_cython import fib_cythonfrom fib_cython_with_types import fib_cython_woth_types

Here is the output when import fib_cython:

/Users/gigi/.pyxbld/temp.macosx-10.9-x86_64-2.7/pyrex/fib_cython.c:1639:28:warning: unused function '__Pyx_PyObject_AsString' [-Wunused-function]static CYTHON_INLINE char* __Pyx_PyObject_AsString(PyObject* o) {                           ^/Users/gigi/.pyxbld/temp.macosx-10.9-x86_64-2.7/pyrex/fib_cython.c:1636:32:warning: unused function '__Pyx_PyUnicode_FromString'[-Wunused-function]static CYTHON_INLINE PyObject* __Pyx_PyUnicode_FromString(const char* c_str) {                               ^/Users/gigi/.pyxbld/temp.macosx-10.9-x86_64-2.7/pyrex/fib_cython.c:325:29:warning: unused function '__Pyx_Py_UNICODE_strlen' [-Wunused-function]def fib_cython(n):static CYTHON_INLINE size_t __Pyx_Py_UNICODE_strlen(const Py_UNICODE *u)                            ^/Users/gigi/.pyxbld/temp.macosx-10.9-x86_64-2.7/pyrex/fib_cython.c:1701:26:warning: unused function '__Pyx_PyObject_IsTrue' [-Wunused-function]static CYTHON_INLINE int __Pyx_PyObject_IsTrue(PyObject* x) {                         ^/Users/gigi/.pyxbld/temp.macosx-10.9-x86_64-2.7/pyrex/fib_cython.c:1751:33:warning: unused function '__Pyx_PyIndex_AsSsize_t' [-Wunused-function]static CYTHON_INLINE Py_ssize_t __Pyx_PyIndex_AsSsize_t(PyObject* b) {                                ^/Users/gigi/.pyxbld/temp.macosx-10.9-x86_64-2.7/pyrex/fib_cython.c:1813:33:warning: unused function '__Pyx_PyInt_FromSize_t' [-Wunused-function]def fib_cython_with_types(n):static CYTHON_INLINE PyObject * __Pyx_PyInt_FromSize_t(size_t ival) {                                ^/Users/gigi/.pyxbld/temp.macosx-10.9-x86_64-2.7/pyrex/fib_cython.c:1172:32:warning: unused function '__Pyx_PyInt_From_long' [-Wunused-function]static CYTHON_INLINE PyObject* __Pyx_PyInt_From_long(long value) {                               ^/Users/gigi/.pyxbld/temp.macosx-10.9-x86_64-2.7/pyrex/fib_cython.c:1223:27:warning: function '__Pyx_PyInt_As_long' is not needed and will not be      emitted [-Wunneeded-internal-declaration]static CYTHON_INLINE long __Pyx_PyInt_As_long(PyObject *x) {                          ^/Users/gigi/.pyxbld/temp.macosx-10.9-x86_64-2.7/pyrex/fib_cython.c:1407:26:warning: function '__Pyx_PyInt_As_int' is not needed and will not be      emitted [-Wunneeded-internal-declaration]static CYTHON_INLINE int __Pyx_PyInt_As_int(PyObject *x) {                         ^9 warnings generated.

The warnings are harmless and you can see that Cython has gone ahead and built a Python C extension. To test the performance of the three versions I used IPython’s %timeit magic function to compute the 100th Fibonacci number using each function and here are the results:

In [18]: %timeit fib_cython_with_types(100)10000000 loops, best of 3: 140 ns per loopIn [19]: %timeit fib_cython(100)100000 loops, best of 3: 2.75 ?s per loopIn [20]: %timeit fib_py(100)100000 loops, best of 3: 7.85 ?s per loop

The pure Python version took 7.85 Microseconds. The Cython version with no type information to 2.75 Microseconds (about 3X faster). However, the Cython version with type information took only 140 Nanoseconds, which is about 56X faster than the pure Python version.

This is, of course, just an example, but it hints at the capabilities of Cython to improve the performance of Python programs with very little effort.

Python and C++

C++ was very difficult traditionally to integrate with Python. Solutions like SWIG?or Boost.Python?were not the most user-friendly. Now there are some easier solutions that take advantage of modern C++ such as pybind11.

Conclusion

There are many ways to optimize Python and improve its performance. Cython is very easy to use and recommended to get the low-hanging fruits of performance. If you want to interface with C++ there are several options as well, and in case you need total control you can always use the Python C API directly. But, don’t forget the premature optimization is the root of all evil in computer science. Before using any of these techniques, first verify that Python is indeed the bottleneck.

devx-admin

devx-admin

Share the Post:
Development Project

Thrilling East Windsor Mixed-Use Development

Real estate developer James Cormier, in collaboration with a partnership, has purchased 137 acres of land in Connecticut for $1.15 million with the intention of

USA Companies

Top Software Development Companies in USA

Navigating the tech landscape to find the right partner is crucial yet challenging. This article offers a comparative glimpse into the top software development companies

Software Development

Top Software Development Companies

Looking for the best in software development? Our list of Top Software Development Companies is your gateway to finding the right tech partner. Dive in

India Web Development

Top Web Development Companies in India

In the digital race, the right web development partner is your winning edge. Dive into our curated list of top web development companies in India,

USA Web Development

Top Web Development Companies in USA

Looking for the best web development companies in the USA? We’ve got you covered! Check out our top 10 picks to find the right partner

Clean Energy Adoption

Inside Michigan’s Clean Energy Revolution

Democratic state legislators in Michigan continue to discuss and debate clean energy legislation in the hopes of establishing a comprehensive clean energy strategy for the

Development Project

Thrilling East Windsor Mixed-Use Development

Real estate developer James Cormier, in collaboration with a partnership, has purchased 137 acres of land in Connecticut for $1.15 million with the intention of constructing residential and commercial buildings.

USA Companies

Top Software Development Companies in USA

Navigating the tech landscape to find the right partner is crucial yet challenging. This article offers a comparative glimpse into the top software development companies in the USA. Through a

Software Development

Top Software Development Companies

Looking for the best in software development? Our list of Top Software Development Companies is your gateway to finding the right tech partner. Dive in and explore the leaders in

India Web Development

Top Web Development Companies in India

In the digital race, the right web development partner is your winning edge. Dive into our curated list of top web development companies in India, and kickstart your journey to

USA Web Development

Top Web Development Companies in USA

Looking for the best web development companies in the USA? We’ve got you covered! Check out our top 10 picks to find the right partner for your online project. Your

Clean Energy Adoption

Inside Michigan’s Clean Energy Revolution

Democratic state legislators in Michigan continue to discuss and debate clean energy legislation in the hopes of establishing a comprehensive clean energy strategy for the state. A Senate committee meeting

Chips Act Revolution

European Chips Act: What is it?

In response to the intensifying worldwide technology competition, Europe has unveiled the long-awaited European Chips Act. This daring legislative proposal aims to fortify Europe’s semiconductor supply chain and enhance its

Revolutionized Low-Code

You Should Use Low-Code Platforms for Apps

As the demand for rapid software development increases, low-code platforms have emerged as a popular choice among developers for their ability to build applications with minimal coding. These platforms not

Cybersecurity Strategy

Five Powerful Strategies to Bolster Your Cybersecurity

In today’s increasingly digital landscape, businesses of all sizes must prioritize cyber security measures to defend against potential dangers. Cyber security professionals suggest five simple technological strategies to help companies

Global Layoffs

Tech Layoffs Are Getting Worse Globally

Since the start of 2023, the global technology sector has experienced a significant rise in layoffs, with over 236,000 workers being let go by 1,019 tech firms, as per data

Huawei Electric Dazzle

Huawei Dazzles with Electric Vehicles and Wireless Earbuds

During a prominent unveiling event, Huawei, the Chinese telecommunications powerhouse, kept quiet about its enigmatic new 5G phone and alleged cutting-edge chip development. Instead, Huawei astounded the audience by presenting

Cybersecurity Banking Revolution

Digital Banking Needs Cybersecurity

The banking, financial, and insurance (BFSI) sectors are pioneers in digital transformation, using web applications and application programming interfaces (APIs) to provide seamless services to customers around the world. Rising

FinTech Leadership

Terry Clune’s Fintech Empire

Over the past 30 years, Terry Clune has built a remarkable business empire, with CluneTech at the helm. The CEO and Founder has successfully created eight fintech firms, attracting renowned

The Role Of AI Within A Web Design Agency?

In the digital age, the role of Artificial Intelligence (AI) in web design is rapidly evolving, transitioning from a futuristic concept to practical tools used in design, coding, content writing

Generative AI Revolution

Is Generative AI the Next Internet?

The increasing demand for Generative AI models has led to a surge in its adoption across diverse sectors, with healthcare, automotive, and financial services being among the top beneficiaries. These

Microsoft Laptop

The New Surface Laptop Studio 2 Is Nuts

The Surface Laptop Studio 2 is a dynamic and robust all-in-one laptop designed for creators and professionals alike. It features a 14.4″ touchscreen and a cutting-edge design that is over

5G Innovations

GPU-Accelerated 5G in Japan

NTT DOCOMO, a global telecommunications giant, is set to break new ground in the industry as it prepares to launch a GPU-accelerated 5G network in Japan. This innovative approach will

AI Ethics

AI Journalism: Balancing Integrity and Innovation

An op-ed, produced using Microsoft’s Bing Chat AI software, recently appeared in the St. Louis Post-Dispatch, discussing the potential concerns surrounding the employment of artificial intelligence (AI) in journalism. These

Savings Extravaganza

Big Deal Days Extravaganza

The highly awaited Big Deal Days event for October 2023 is nearly here, scheduled for the 10th and 11th. Similar to the previous year, this autumn sale has already created

Cisco Splunk Deal

Cisco Splunk Deal Sparks Tech Acquisition Frenzy

Cisco’s recent massive purchase of Splunk, an AI-powered cybersecurity firm, for $28 billion signals a potential boost in tech deals after a year of subdued mergers and acquisitions in the

Iran Drone Expansion

Iran’s Jet-Propelled Drone Reshapes Power Balance

Iran has recently unveiled a jet-propelled variant of its Shahed series drone, marking a significant advancement in the nation’s drone technology. The new drone is poised to reshape the regional

Solar Geoengineering

Did the Overshoot Commission Shoot Down Geoengineering?

The Overshoot Commission has recently released a comprehensive report that discusses the controversial topic of Solar Geoengineering, also known as Solar Radiation Modification (SRM). The Commission’s primary objective is to

Remote Learning

Revolutionizing Remote Learning for Success

School districts are preparing to reveal a substantial technological upgrade designed to significantly improve remote learning experiences for both educators and students amid the ongoing pandemic. This major investment, which

Revolutionary SABERS Transforming

SABERS Batteries Transforming Industries

Scientists John Connell and Yi Lin from NASA’s Solid-state Architecture Batteries for Enhanced Rechargeability and Safety (SABERS) project are working on experimental solid-state battery packs that could dramatically change the