Parallel and High Performance Programming with Python
National University of Singapore

Textbook:Parallel and High Performance Programming with Python: Unlock Parallel and Concurrent Programming in Python using Multithreading, CUDA, Pytorch, and Dask

Author(s): Fabio Nelli

Description:
This coursebook provides you everything about the powerful techniques and applications of parallel computing, from the basics of parallel programming to the cutting-edge innovations shaping the future of computing.
The cooursebook starts with an introduction to parallel programming and the different types of parallelism, including parallel programming with threads and processes. The book then delves into asynchronous programming, distributed Python, and GPU programming with Python, providing you with the tools you need to optimize your programs for distributed and high-performance computing.
The coursebook also covers a wide range of applications for parallel computing, including data science, artificial intelligence, and other complex scientific simulations. You will learn about the challenges and opportunities presented by parallel computing for these applications and how to overcome them.
By the end of the coursebook, you will have insights into the future of parallel computing, the latest research and developments in the field, and explore the exciting possibilities that lie ahead.


Chapter 1 will start with an introduction to parallelism in Python and then will explain how the operating system handles multiple processes and threads. Its purpose will be to introduce you to the concept of parallel programming by talking about all the fundamental concepts that are involved, and which are necessary to fully understand the features and uses. Once these general concepts have been introduced, you will see the peculiarities of Python in this area, especially with threads, talking about the GIL ( Global Interpreter Lock ) and the problems it introduces. We will also learn about standard Python library modules such as threading and multiprocessing .
Chapter 2 will cover parallel programming in Python that uses Threads as concurrent elements. Specifically, the chapter will deal with the threading module that allows us to easily implement threads and provide a whole series of useful tools for their synchronization.
Chapter 3 will cover parallel programming in Python that uses Processes. There are two main approaches by which processes can be implemented in parallel programming: the multiprocessing module of the standard library and the mpi4py library which extends the MPI protocol also for the Python language. Both libraries will be treated in detail, with a series of examples that will show how they work and their main characteristics .
Chapter 4 will cover aspects of asynchronous programming in Python including the AsyncIO library as an example library.
Chapter 5 will cover the distributed systems as they fall within the scope of parallel and concurrent programming and can prove to be a valid solution.

The Celery library is the reference point in Python for the realization of distributed systems. With a series of examples, we will see how to use this library to perform multiple concurrent operations, called tasks, and how these are distributed and executed in parallel on a Celery-based system. In addition, we will subsequently move on to other alternative solutions, starting from a very similar one, such as Dramatiq to a simpler one, but which conceptually differs from these, such as SCOOP.
Chapter 6 will cover GPU programming since these processors are designed to process vector data extremely quickly and efficiently for image rendering, 3D engines and manipulation of polygonal primitives. Python offers good solutions with various libraries such as Numba (CUDA) and PyOpenCL.
Chapter 7 will introduce you to the world of parallel computing applications, showing how this approach is now present in many scientific and professional disciplines.
Chapter 8 will cover parallel computing specifically with regard to Data Science. Many of the libraries commonly used in Data Science such as Numpy, Pandas and Scikit-learn can be extended to parallel computing thanks to the Dask library. This library provides objects such as nparrays, dataframes and machine learning api designed to work in parallel. There are many other scientific libraries that can be integrated with Dask.
Chapter 9 will cover another very interesting field of application, which in part extends to what was said in Data Science: Artificial Intelligence. Machine Learning and Deep Learning are subsets of Artificial Intelligence and they provide us with a good tool to explore data. Lately, these techniques are expanding enormously by exploiting parallel and distributed computing.
Chapter 10 will show you how developments and trends of this way of programming are fitting perfectly with the most innovative new technologies.