NumPy ufuncs — The Magic Behind Vectorized Functions | by Diego Barba | Jul, 2022
Learn about NumPy universal functions (ufuncs) and how to create them. Code your own vectorized functions.
Have you ever wondered about the origin of NumPy’s magical performance? NumPy powers the performance, under the hood, of many daily drivers of the data scientist, such as pandas, among an extensive list. Of course, you’d be right to think about optimized arrays written in C and Fortran. Half right, at least. The other half is not the arrays but NumPy’s functions themselves. NumPy universal functions (henceforth ufuncs) are the other building block other than arrays (ndarray).
When we want to write some sleek and performant numerical code, there is no way around it; we use NumPy. Once we start using NumPy arrays, it is intuitive to use NumPy’s built-in functions to manipulate and operate on them. The performance would disappear quickly if we used Python’s ordinary functions on the arrays.
In this story, we will discuss some insights about ufuncs and, more importantly, learn the basics about custom ufunc creation. Being able to implement our own ufuncs is what will take out NumPy projects to the next level.
Often we can spend lots of time looking in NumPy’s documentation for a way to implement the functionality we need; sometimes, we find it, sometimes, we don’t. In the latter case, we might end up implementing some hack or falling back to plain Python and sacrificing performance. Coding our custom ufuncs can simplify our life greatly.
story structure:
- What are ufuncs?
- Unary and binary ufuncs
- NumPy ufunc creation
- Numba ufunc creation: the vectorize decorator
- ufunc creation: NumPy vs. Numba
- ufunc.reduce
- ufunc.accumulate
- Final comments
What are ufuncs?
NumPy ufuncs operate on arrays (ndarray) element-wise. Some of them support more complex features such as array broadcasting. While NumPy’s build-in ufuncs are vectorized (vectorized computations either in C or Fortran), it is not true that all ufuncs are vectorized in the same sense, as we will see in the following sections.
All ufuncs take the same optional keyword arguments and expose the same attributes and methods; you can check them out here.
Most functions in NumPy are ufuncs; however, some are not, but rather methods of a ufunc. Methods of a ufunc are ordinary Python functions but derived from a ufunc nonetheless. That is where they get their magic. For example, np.sum is a regular function that comes from np.add, a ufunc.
However, methods do not work with all ufuncs. There are two main kinds of ufuncs, binary and unary. Methods only work with binary ufuncs.
Unary and binary ufuncs
In mathematics, the most common operations are binary. The sum (a + b) and the product (a * b) are some of the simplest and most famous examples. Some of these operations commute; others do not. There are, however, some transformations that operate on a single element. We could argue whether these transformations can, in fact, be derived by using binary operations, maybe infinite sums — for example, the exponential function.
One of the reasons why binary operations are central in mathematics is because all operations with more than two operands can be converted to a sequence of binary operations.
NumPy inherits this behavior in its functions. There are two types of ufuncs:
- unary ufuncs: take one array (ndarray) as the argument
- binary ufuncs: take two arrays (ndarray) as arguments
For example:
<class ‘numpy.ufunc’>, 1
<class ‘numpy.ufunc’>, 2
where the nin attribute is the number of inputs.
NumPy ufunc creation
Creating custom ufuncs is very easy. NumPy has a utility, np.frompyfunc, which takes a normal Python function, the number of input arguments, and the number of output arguments and returns a ufunc. For example, we can create a unary ufunc:
<class ‘function’>, <class ‘numpy.ufunc’>
3, 3
We can also create a binary ufunc:
<class ‘function’>, <class ‘numpy.ufunc’>
2, 2
Numba ufunc creation: the vectorize decorator
Numba also has a way to create a ufunc. The vectorize decorator. This decorator, however, does more than merely create a ufunc; it also optimizes it for performance. As per Numba’s documentation, it can compile a Python function in such a way its speed is comparable with ufuncs written in C (NumPy built-ins).
There are many ways to use the decorator; the most straightforward way is to use all defaults in the decorator (i.e., no arguments):
<class ‘numba.np.ufunc.dufunc.DUFunc’>
2
[2. 2.]
Here we do not specify input and output datatypes for compilation, therefore lazy mode.
We can also pass the datatypes explicitly:
<class ‘numba.np.ufunc.dufunc.DUFunc’>
2
[2. 2.]
We are saying here that there is one input (the type outside the brackets) and two outputs (the tuple in brackets). If both inputs are int64, the return type will also be int64. If both inputs are float32, the output is float32, and so on. Specifying the datatypes optimizes the compilation.
Finally, we can also change the other decorator’s default arguments to optimize for computation performance:
<class ‘numba.np.ufunc.dufunc.DUFunc’>
2
[2. 2.]
ufunc creation: NumPy vs. Numba
In this section, we will compare the computation times from several approaches to add two 2D NumPy arrays (ndarray). We will use the following ufuncs:
- NumPy built-in add (
np.add) np_my_sum, ufunc created with np.frompyfuncnumba_lazy_sum, ufunc created with Numba’s vectorize decorator, with default arguments and no dtypes informationnumba_dtype_sum, ufunc created with the vectorize decorator, stating dtype conversion for compilation- numba_
dtype_opt_sum, ufunc created with the vectorize decorator, stating dtype conversion for compilation plus optimizing arguments for performance
Coding the talk:
The results are:
We can see that the ufunc created with NumPy frompyfunc is in another league entirely; the computation time is orders of magnitude larger than the other functions. However, if we ignore it, we can see that Numba’s ufuncs performance is comparable to NumPy’s built-in add function and is similar among them.
Numba is the way to go if we want to create performant ufuncs. Unless the performance is critical, the vectorize decorator can be used without arguments.
One of the strengths of NumPy is its performance, bringing speed to Python. Let’s face it; speed is not one of Python’s strengths. Hence, it is essential to distinguish between C or Fortran vectorized computations vs. pure Python vectorization. The speed will not be the same.
ufunc.reduce
There are some functions in NumPy that are not ufuncs; most of them accept the axis keyword. These functions are fantastic since they allow us to specify whether we want to apply the function operation over the whole array or just a subset of axes— for example, np.sum.
Suppose we have a 2D array and do not specify the axis (or specify the two axes at once). In that case, the function returns a scalar with the sum of the whole array, but if we specify the axis zero (one), the function will return a vector of the sum of the columns (rows).
We can check that np.sum is a regular Python function, not a ufunc:
<class ‘function’>
This is what we can do with the function and the axis argument:
printing the sums:
36
36
[ 9 12 15]
[ 3 12 21]
Under the hood, np.sum is essentially the application of the reduce method of the np.add function:
36
36
[ 9 12 15]
[ 3 12 21]
So if we can recover the NumPy sum function from the add ufunc, what about, Numba’s add ufunc? Sure we can use reduce too. Let’s define the function with the vectorized decorator:
Testing it:
36
36
[ 9 12 15]
[ 3 12 21]
we are golden.
It is crucial to talk about NumPy’s apply_along_axis function. It can reduce a Python regular function:
[ 9 12 15]
[ 3 12 21]
However, doing this will not take advantage of NumPy’s performance. It will be much slower than using a compiled ufunc. So be warned.
ufunc.accumulate
Another useful method of ufuncs is accumulate. In the case of the sum, it is similar to what the function (ordinary Python) np.cumsum does:
[[ 0 1 2]
[ 3 5 7]
[ 9 12 15]]
[[ 0 1 3]
[ 3 7 12]
[ 6 13 21]]
We can use the accumulate method in the add ufunc and get the same results:
[[ 0 1 2]
[ 3 5 7]
[ 9 12 15]]
[[ 0 1 3]
[ 3 7 12]
[ 6 13 21]]
We can also use it in the Numba’s ufunc:
[[ 0 1 2]
[ 3 5 7]
[ 9 12 15]]
[[ 0 1 3]
[ 3 7 12]
[ 6 13 21]]
Final comments
It is essential to highlight that not all ufuncs are created equal, as we’ve seen in this story. It is crucial to spot performance bottlenecks in your code when applying custom functions over ndarrays, even when using NumPy tools such as frompyfunc or apply_along_axis.
So next time you use a function over arrays, ask yourself if it is properly optimized for vectorized operations.
Regarding ufunc creation, Numba’s utilities are the way to go in terms of performance.
Learn about NumPy universal functions (ufuncs) and how to create them. Code your own vectorized functions.
Have you ever wondered about the origin of NumPy’s magical performance? NumPy powers the performance, under the hood, of many daily drivers of the data scientist, such as pandas, among an extensive list. Of course, you’d be right to think about optimized arrays written in C and Fortran. Half right, at least. The other half is not the arrays but NumPy’s functions themselves. NumPy universal functions (henceforth ufuncs) are the other building block other than arrays (ndarray).
When we want to write some sleek and performant numerical code, there is no way around it; we use NumPy. Once we start using NumPy arrays, it is intuitive to use NumPy’s built-in functions to manipulate and operate on them. The performance would disappear quickly if we used Python’s ordinary functions on the arrays.
In this story, we will discuss some insights about ufuncs and, more importantly, learn the basics about custom ufunc creation. Being able to implement our own ufuncs is what will take out NumPy projects to the next level.
Often we can spend lots of time looking in NumPy’s documentation for a way to implement the functionality we need; sometimes, we find it, sometimes, we don’t. In the latter case, we might end up implementing some hack or falling back to plain Python and sacrificing performance. Coding our custom ufuncs can simplify our life greatly.
story structure:
- What are ufuncs?
- Unary and binary ufuncs
- NumPy ufunc creation
- Numba ufunc creation: the vectorize decorator
- ufunc creation: NumPy vs. Numba
- ufunc.reduce
- ufunc.accumulate
- Final comments
What are ufuncs?
NumPy ufuncs operate on arrays (ndarray) element-wise. Some of them support more complex features such as array broadcasting. While NumPy’s build-in ufuncs are vectorized (vectorized computations either in C or Fortran), it is not true that all ufuncs are vectorized in the same sense, as we will see in the following sections.
All ufuncs take the same optional keyword arguments and expose the same attributes and methods; you can check them out here.
Most functions in NumPy are ufuncs; however, some are not, but rather methods of a ufunc. Methods of a ufunc are ordinary Python functions but derived from a ufunc nonetheless. That is where they get their magic. For example, np.sum is a regular function that comes from np.add, a ufunc.
However, methods do not work with all ufuncs. There are two main kinds of ufuncs, binary and unary. Methods only work with binary ufuncs.
Unary and binary ufuncs
In mathematics, the most common operations are binary. The sum (a + b) and the product (a * b) are some of the simplest and most famous examples. Some of these operations commute; others do not. There are, however, some transformations that operate on a single element. We could argue whether these transformations can, in fact, be derived by using binary operations, maybe infinite sums — for example, the exponential function.
One of the reasons why binary operations are central in mathematics is because all operations with more than two operands can be converted to a sequence of binary operations.
NumPy inherits this behavior in its functions. There are two types of ufuncs:
- unary ufuncs: take one array (ndarray) as the argument
- binary ufuncs: take two arrays (ndarray) as arguments
For example:
<class ‘numpy.ufunc’>, 1
<class ‘numpy.ufunc’>, 2
where the nin attribute is the number of inputs.
NumPy ufunc creation
Creating custom ufuncs is very easy. NumPy has a utility, np.frompyfunc, which takes a normal Python function, the number of input arguments, and the number of output arguments and returns a ufunc. For example, we can create a unary ufunc:
<class ‘function’>, <class ‘numpy.ufunc’>
3, 3
We can also create a binary ufunc:
<class ‘function’>, <class ‘numpy.ufunc’>
2, 2
Numba ufunc creation: the vectorize decorator
Numba also has a way to create a ufunc. The vectorize decorator. This decorator, however, does more than merely create a ufunc; it also optimizes it for performance. As per Numba’s documentation, it can compile a Python function in such a way its speed is comparable with ufuncs written in C (NumPy built-ins).
There are many ways to use the decorator; the most straightforward way is to use all defaults in the decorator (i.e., no arguments):
<class ‘numba.np.ufunc.dufunc.DUFunc’>
2
[2. 2.]
Here we do not specify input and output datatypes for compilation, therefore lazy mode.
We can also pass the datatypes explicitly:
<class ‘numba.np.ufunc.dufunc.DUFunc’>
2
[2. 2.]
We are saying here that there is one input (the type outside the brackets) and two outputs (the tuple in brackets). If both inputs are int64, the return type will also be int64. If both inputs are float32, the output is float32, and so on. Specifying the datatypes optimizes the compilation.
Finally, we can also change the other decorator’s default arguments to optimize for computation performance:
<class ‘numba.np.ufunc.dufunc.DUFunc’>
2
[2. 2.]
ufunc creation: NumPy vs. Numba
In this section, we will compare the computation times from several approaches to add two 2D NumPy arrays (ndarray). We will use the following ufuncs:
- NumPy built-in add (
np.add) np_my_sum, ufunc created with np.frompyfuncnumba_lazy_sum, ufunc created with Numba’s vectorize decorator, with default arguments and no dtypes informationnumba_dtype_sum, ufunc created with the vectorize decorator, stating dtype conversion for compilation- numba_
dtype_opt_sum, ufunc created with the vectorize decorator, stating dtype conversion for compilation plus optimizing arguments for performance
Coding the talk:
The results are:
We can see that the ufunc created with NumPy frompyfunc is in another league entirely; the computation time is orders of magnitude larger than the other functions. However, if we ignore it, we can see that Numba’s ufuncs performance is comparable to NumPy’s built-in add function and is similar among them.
Numba is the way to go if we want to create performant ufuncs. Unless the performance is critical, the vectorize decorator can be used without arguments.
One of the strengths of NumPy is its performance, bringing speed to Python. Let’s face it; speed is not one of Python’s strengths. Hence, it is essential to distinguish between C or Fortran vectorized computations vs. pure Python vectorization. The speed will not be the same.
ufunc.reduce
There are some functions in NumPy that are not ufuncs; most of them accept the axis keyword. These functions are fantastic since they allow us to specify whether we want to apply the function operation over the whole array or just a subset of axes— for example, np.sum.
Suppose we have a 2D array and do not specify the axis (or specify the two axes at once). In that case, the function returns a scalar with the sum of the whole array, but if we specify the axis zero (one), the function will return a vector of the sum of the columns (rows).
We can check that np.sum is a regular Python function, not a ufunc:
<class ‘function’>
This is what we can do with the function and the axis argument:
printing the sums:
36
36
[ 9 12 15]
[ 3 12 21]
Under the hood, np.sum is essentially the application of the reduce method of the np.add function:
36
36
[ 9 12 15]
[ 3 12 21]
So if we can recover the NumPy sum function from the add ufunc, what about, Numba’s add ufunc? Sure we can use reduce too. Let’s define the function with the vectorized decorator:
Testing it:
36
36
[ 9 12 15]
[ 3 12 21]
we are golden.
It is crucial to talk about NumPy’s apply_along_axis function. It can reduce a Python regular function:
[ 9 12 15]
[ 3 12 21]
However, doing this will not take advantage of NumPy’s performance. It will be much slower than using a compiled ufunc. So be warned.
ufunc.accumulate
Another useful method of ufuncs is accumulate. In the case of the sum, it is similar to what the function (ordinary Python) np.cumsum does:
[[ 0 1 2]
[ 3 5 7]
[ 9 12 15]]
[[ 0 1 3]
[ 3 7 12]
[ 6 13 21]]
We can use the accumulate method in the add ufunc and get the same results:
[[ 0 1 2]
[ 3 5 7]
[ 9 12 15]]
[[ 0 1 3]
[ 3 7 12]
[ 6 13 21]]
We can also use it in the Numba’s ufunc:
[[ 0 1 2]
[ 3 5 7]
[ 9 12 15]]
[[ 0 1 3]
[ 3 7 12]
[ 6 13 21]]
Final comments
It is essential to highlight that not all ufuncs are created equal, as we’ve seen in this story. It is crucial to spot performance bottlenecks in your code when applying custom functions over ndarrays, even when using NumPy tools such as frompyfunc or apply_along_axis.
So next time you use a function over arrays, ask yourself if it is properly optimized for vectorized operations.
Regarding ufunc creation, Numba’s utilities are the way to go in terms of performance.
Denial of responsibility! Techno Blender is an automatic aggregator of the all world’s media. In each content, the hyperlink to the primary source is specified. All trademarks belong to their rightful owners, all materials to their authors. If you are the owner of the content and do not want us to publish your materials, please contact us by email – [email protected]. The content will be deleted within 24 hours.