Understanding Threads

To embark on parallel computing, we must first grasp the concept of threads. Even in parallel computing scenarios that utilize hardware beyond the CPU, comprehending threads is vital for task distribution. A thread represents an ongoing task that a processor is executing. The kernel of our operating system initiates a new thread to connect our application to a processor, allowing every application on our system to run asynchronously across all available threads—provided the software is designed to leverage them effectively.

When we state that a processor has _ threads, we imply that it can execute _ small sequences of tasks simultaneously. The kernel interfaces these threads with our hardware, enabling us to observe this activity through a system monitor. Each process launched by the kernel allows us to manage a new thread. The primary way to initiate additional workloads concurrently is to deploy them on new threads. Asynchronous operations differ in that they do not execute tasks simultaneously; instead, they pause certain tasks to allow others to complete.

Exploring Parallel Processes in Julia

To illustrate, let’s start Julia with multiple threads and spawn some processes using the ParametricProcesses library:

julia --threads 8

Now we can see the initiation of four Julia processes.

Trade-offs in Parallel Computing

A keen observer may notice a significant limitation in parallel computing. Due to thread dynamics, data must be transmitted between processes. While data can be shared through arguments, the functions and dependencies associated with those arguments must also be shared, leading to a substantial memory overhead. For instance, merely launching four threaded workers can consume over 700 MB of memory before any processing occurs.

Another critical point to remember is that transferring data between threads involves a call to the kernel, which inherently takes time. This is a vital trade-off to consider; as we increase the number of threads, the time taken for data movement escalates. Consequently, starting more processes can prolong the application's launch time. Effective software development hinges on accurately identifying your target and recognizing your hardware constraints. Key questions include: What is your goal, and what are the limitations of your hardware? How many threads best suit your application’s needs?

Use-Cases for Parallel Computing

A common challenge when exploring parallel computing is the unexpected outcomes. For instance, a developer might multi-thread a function anticipating improved performance, only to find that it runs slower than its single-threaded counterpart. This can be perplexing for newcomers, who may assume that more threads always equate to better performance.

There are two primary scenarios where parallel computing shines. The first involves larger, more intensive calculations. While this isn't applicable to all algorithms, there are instances where distributing computations across a cluster, available threads, or a GPU is beneficial. This is particularly relevant in specialized fields like Data Science, although rendering tasks in software engineering can also benefit from this approach.

The second significant scenario involves callbacks—functions that are registered for later execution. This is a common pattern in Graphical User Interfaces (GUIs), where events trigger functions to provide graphical responses. Callbacks represent an excellent opportunity for multi-threading, as they enable simultaneous processing of multiple tasks.

While there are numerous applications for parallel computing, it's essential to ensure that the tasks justify the hardware resources allocated. The trade-offs involved can be considerable, and if the tasks don't warrant the distribution of resources, the result may be subpar compared to executing them on a single thread.

GPU Parallel Computing Technologies

Another vital consideration in parallel computing is the variety of platforms available for Graphics Processing Units (GPUs). Indeed, the most prevalent form of parallel computing, aside from multi-threading, often involves tasks ideally suited for GPUs. Before selecting a GPU for parallel computing, it’s crucial to understand the technologies that are compatible with specific cards. The three main technologies for distributing tasks to GPUs are CUDA, OpenCL, and ROCm.

CUDA, developed by Nvidia, is a licensed software that utilizes its proprietary CUDA cores. For many programmers, CUDA is the preferred choice due to its dominance in the GPU parallel computing landscape. However, it is exclusive to Nvidia GPUs, making it a prerequisite for those wishing to use it.

In contrast, OpenCL is a third-party technology designed for distributing processes across various hardware. OpenCL serves as a programming language compatible with a wide range of devices, making it an ideal solution for applications that require GPU-agnostic functionality.

AMD's ROCm is another parallel computing technology aimed at competing with CUDA. Although it may not be as widely adopted in the industry, the AMDGPU graphics driver for Linux users is quite robust, offering a viable option for those seeking alternatives to Nvidia.