How Do Computers Solve Problems?
Traditionally, computers (much like people) break a problem into a discrete series of instructions (an instruction stream) and then execute these instructions one after the other. Retrieving these instructions, decoding these instructions, and executing these instructions is the job of the Central Processing Unit or the CPU in every modern computer. This is the pattern that the CPU follows from boot until it is powered off, and the pattern itself is commonly referred to as the instruction cycle.
When the CPU executes these instructions on a single computer that has a single Central Processing Unit (CPU) where each instruction is executed sequentially, it is called serial execution.
Up until the early 2000s, single CPUs would yield about 50% more performance each year due to design changes (i.e., smaller transistors). The conclusive way to solve problems faster at this point was:
smaller transistors = faster processors = increased power consumption = solve problems faster
For software developers, this meant to wait until the next generation of CPUs were released from the CPU chip manufacturers in order to get more performance out of scientific applications.
However, since 2002, single-processor performance improvement has slowed to about 20% per year (Ch.1, pg. 1) and the increased power consumption from designing smaller transistors began to lead to increased heat and unreliable processors. Thus, around this time chip manufacturers began adopting the approach of putting multiple processor cores (each with their own instruction stream) on a single circuit as opposed to trying to make single-processors themselves faster. In effect, this was the birth of multi-core CPUs often referred to as "multi-core processors"
Really it wasn't until around 2005, when microprocessor manufacturers had decided that the movement away from single-core CPU systems to multi-core processors was the new road to increasing performance (Ch.1, pg. 1).
core = a distinct execution unit of a CPU with its own instruction stream
TO DO: Single Core sum array vs multi-core sum array, explain concurrency vs parallelism
The caveat here is that adding more CPU processors did not magically improve the performance of the vast majority of serial programs. For instance, <--- serial code vs multicore code --->
Instead, exploiting the increased power of multiple processors on a single chip for increased performance required parallelism. Hence, an entirely new programming paradigm was established for writing applications to be run on multiple CPUs/cores. A very good parallel programming tutorial can be found here.
Put simply, parallel computing is the simultaneous use of multiple compute resources to solve a computational problem.
This requires an additional step at the beginning, when compared to serial execution.
-
Break the computational problem into discrete parts that can be solved concurrently
-
Reduce each part of the problem further into a series of instructions
-
Execute the instructions from each part simultaneously on multiple cores/CPUs
A computer cluster is a set of connected computers that perform as a single system for the purpose of providing an inexpensive way to obtain even more parallel computing capabilities.When we create a cluster, the CPU cores and GPU accelerator(s) of a computer become the basic units of a much bigger system, which is called a node. Note that a cluster can exist anywhere between two personal computers connected in a simple two-node system, or a supercomputer with a complicated architecture such as the supercomputers listed here
Ultimately, the unifying characteristic of a cluster is the integration of highly replicated compute and communication components into a single system, with each node still able to operate independently. To see more about the cluster this project was built on, see this page
Next: Serial to Parallel speedup example
[1] An Introduction to Parallel Programming by Peter S. Pacheco, Ch.1
[2] Lawrence Livermore National Laboratory : An Introduction to Parallel Computing Tutorial