I’ve been thinking about this for several months now, for work that has thousands of tasks, each task being massively data parallel. Traditionnally, one launches a job through one’s favorite batch scheduler (favorite or mandatory…) with fixed resources and during an estimated amount of time. This may work well in research, but in the industrial world, there often a new job that arises and that needs part of your scarce resources. You may have to stop your work, loose your current advances and/or restart the job with less resources. And then the cycle goes on.

Static resource allocation

How can resource allocation work? Let’s start with a simple case where you have 2 applications with different priorities. One of them has a priority of 70 (it’s supposed to finish in three days) whereas the other one has a priority of 50 (four days left). They share the cluster so that 66% is allocated to the first application and 33% to the second one.

What happens if a third application must be launched with a higher priority, because it has to ne finished by tomorrow? You may stop the other two programs, you may loose a lot of work if you didn’t implement checkpoints (besides, one of them may be an of-the-shelf program you bought yesterday) or suspend it. Either way, this is what you will get:

In fact, even if you use dynamic resource allocation, this is what you must get to have your results by the time you need them, but obviously, you have lost your two other applications. Some batch schedulers allow applications to be suspended, but this is a double-edge sword:

• your cluster must support job suspension, and thus have access to drives to save the job state (which is not possible for medium to large-scaled clusters)
• if your application does not scale to your entire cluster (it happens), although one of the other two applications could go on, it is not possible, all processes are put to sleep

So all things considered, you have to implement dynamic resource allocation.

Dynamic resource allocation

How does this work? Each application must be aware that it can be allocated more resources or deallocated some at all time. To be portable on all clusters, you cannot suspend part of your program, it must really go away. The batch scheduler must also notice that your application has freed some of its resources. You thus have to allocate small jobs that will communicate together (this can be done with MPI-2).

This means that you will have hundreds or thousands of small works. All of them will not have to be connected to the scheduler, only one master must be. Of course, this can easilly be done by using a specific queue. Each application on this queue will thus receive orders from the batch scheduler and act upon it. Another advantage is that also the application gets no resource at one point, it still has a saved state that enable the continuation of a run.

Of course, this is not easy to do. How can this be applied to an of-the-shelf application? Well, in this case, you may create a bogus application on the master queue that will at least allow other applications to be allocated resources beside it.

You do not have to implement this on top of MPI. It can be really hard to do (handling data moves between processors, change the decomposition, …), and you may implement another solution. In my case, I have thousands different tasks that can be run on very few cores, so this is my elementary unit. I don’t need all tasks to communicate between them, so I create each time brand new independent jobs and I also can tell the scheduler it can kill jobs that are not responding before the next allocation phase.

Conclusion

To finish, I’ll say that I know that LSF allows plugins that help dispatch jobs on specific hosts of your cluster (to have the best communication location). There seems to be a way of implementing the needs gathering and the resource assignment, but the documentation is not clear (at all). A specific daemon may be needed. I don’t know if other batch scheduler allow plugins to modify their behavior, if you know of them and their API, please do tell

Content and opinions

First of all, it’s not a book on programming massively parallel processors, it’s a book about CUDA. One of the authors is a nVidia fellow, so it’s no wonder. I think there are three parts in the book: an introduction of CUDA, two examples and then general considerations and the future.

The first 6 chapters (I don’t count the first chapter as a real chapter, it’s more of an introduction to the massively paralell processors and their use in a few pages) are the main CUDA tutorial. I say tutorial because it feel like all beginner courses I’ve taken in CUDA. The content can be found in all Internet classes, so the only advantage is that you have everything in a book. Nothing less, nothing more.

I had a feeling of “deja vu” for the MRI example, the second was unknown to me. There is not much code, only for the relevant parts, but you won’t be able to test the different implementations with what is provided in the book. Besides, several times during the writting flow, new techniques are introduced, but one can’t know what speed-up they provide. Perhaps this is because this speedup cannot be generalized, but still, with proper warnings, the different timings through the GPU port of woth examples would have been great.

The last part is, as I’ve said, more general. It starts with a workflow to help parallelizing with GPUs, then an introduction (too short IMHO) of OpenCL and the future of CUDA with Fermi and the SDK 3.0. The workflow chapter is too small. Of course, the goal isn’t to be like The Art of Concurrency, and at least there is a chapter about the process of selecting the algorithm, … but it is too small. The OpenCL introduction is really an introduction. I’ve seen one small complete OpenCL call, but that’s it. I couldn’t program a single kernel right now. Of course it’s a CUDA book, not an OpenCL one, but the chapter is useless. Perhaps it would be better to merge it with the “future” chapter, as OpenCL is not widely available. Finally, the last chapter states what can be expected of Fermi (really interesting) and of the SDK 3.0.

What I miss in this book is some explanations of the texture memory. The obvious matrix example uses constant memory for caching the memory accesses. Why isn’t texture memory used in this example? It’s far bigger than constant memory and also has a cache, so why not use it? It’s a CUDA book, but a lot of content is freely available in several tutorials that are sometimes better shaped than the book, so why isn’t there some special content, like how the cache works? How can you manage grid sizes that are no a power of two? (it’s explained in one of the example, with zero padding, but there are no protection in the first chapters, which is dangerous) What is coalescing memory and how can I optimize the memory bandwidth with coalescing in mind? (the actual real explanation and appropriate picture is in the last annexe!)

Conclusion

I don’t say that the book is not useful, it’s really interesting as a companion book for a CUDA course or for a beginner. If you’re used to electronic papers, you will not be interested. If you buy this book, don’t expect to know everything about CUDA, or even less massively parallel processors. You will have to dig deeper for specific topics, but at least you will have a good basis.

Programming Massively Parallel Processors: A Hands-on Approach
Price: $62.95

Programming Massively Parallel Processors: A Hands-on Approach (Paperback)
by David B. Kirk, Wen-mei W. Hwu
ISBN: 0123814723

Price: USD 51.99
40 used & new available from USD 46.52

| 3.5 | 8

Today’s accelerators are not there yet. The tools are not ready yet (code translators) and usual programming practices may not be adequate. All the ecosystem will evolve, accelerators will change (GPUs are the main trend, but they will be different in a few years), so what you will do today needs to be shaped with these changes in mind. How is it possible to do so? Is it even possible?

Available code translators

Code translators are the easiest path to solution. I know two of them.

The first is the PGI compiler. It only supports CUDA and the Fortran and C99 language. I didn’t use it yet, also I plan of testing it in the near future. It is based on pragmas, and the compiler generates the CUDA microcode.

The second solution is HMPP. It supports more than just CUDA (also CAL/IL or OpenCL) and Fortran/C (also Java now). As the PGI compiler, it is based on pragmas, and a excellent thing is that it detects the available accelerators and launches the correct kernel (if you authorized it) or the original code. You can also modify the generated code to put your own (you can tune the code for instance, which may give you an additional x2 factor). Unfortunately, it is not possible to call functions inside the parallelized kernels, which means that only simple or badly-written (too many lines or duplicated code) kernels can be called. I think this is the same for the PGI compiler.

It seems that code translators still need work:

• only few accelerators are supported (CUDA, and sometimes CAL/IL or OpenCL),
• almost no langage (Fortran/C/Java, a lot of Virtual Machines should be able to use them natively, without developers using specific tools),
• only one function can be parallelized at a time.

The last point is currently the biggest issue. You need to cut your function int pieces to have clean code and a good portability/evolutivity for the future.

This is why one still need to program a lot for those accelerators, and so we need to adapt our programming practices, develop in the accelerators’ native langages (even if we know that they may disappear in a few years).

Developping your own “tool chain” for accelerators

For accelerators, there are a lot of things that needs to be done each time: copying some data, computing and getting some data back. These are the steps that code translators automate, in fact it is a common practice to use tools to automate stuff. The issue is that complex kernels are not supported by those translators. So what?

Creating automatic functions that will copy the data you need is in fact very common in metaprogramming. Coding the kernel on an accelerator is in fact not that difficult: the manufacturers provide the needed compilers (that’s what nVidia does and the success of the tool chain cannot be denied), and this is really the cornerstone. One has to write more code, some parts are less portable (because they are written in one of the accelerator’s languages), but in the end, with metaprogramming, the code can be better tuned, enhanced and read. This is the leverage of the accelerators.

Conclusion

Why do we care developing for accelerators? We know that they will go away. Before they do, they are the only way of speeding up our software. Code translators are the best tools to develop in a portable way, but they need time to support more accelerators, languages and method of programming. When CPUs will be on a par with accelerators, their progress will help compilers to target them correctly. It’s just a matter of time.
Meanwhile, metaprogrammin is the next best solution to automate processes that code translators cannot support yet.

Test on parallelizing a custom library

I’ve decided to test Advisor Lite on my Interactive Raytracer. This is a test to verify if Advisor Lite finds the adequate parallelization and the memory sharing issues. It is a simple raytracer, so it can be parallelized for each pixel in the image. The only memory sharing issue that I know of is in the kd-tree ray traversal.

Profiling the library

First, I will profile the library. For the complete Advisor Lite workflow, I have to use Intel Compiler, and as it is faster than Microsoft’s compiler, I will use the timeit_image.py script instead of the measure_image.py I’ve used when profiling with Valgrind or Visual Studio.

Amplifier can show the results in a bottom-up or in a top-down manner. Unfortunately, you only have the exclusive timing that is displayed. In my case, when displaying bottom-up results, the method getEntryExitDistances() is the most costly one. In the top-down view, unfortunately, I can’t have a simple tree, as it can be seen in the following view:

In Visual Studio, I have the same results – more or less -, but with a correct top-down call-tree:

The method getEntryExitDistances() cannot be parallelized: it is recursively called, several times per pixel, which would lead to a lot of memory contention. The simpler task is thus to parallelized the pixel rendering, a perfect data-parallel problem.

Annotation of the code

OK, now I can annotate my code. I had to dig inside the help for this, as it was not mentionned in the Starting Guide that Intel provides a header, annotate.h, which mimics the issues you may encounter in a multithreaded application.

So you need to read at least once the online help so that you know the available annotation macros, how you can get them and how they will retrieve what you need. Once the code is annotated, it must be recompiled and then the sharing issues can be detected.

Detection of sharing issues

As expected, Inspector detected errors in the kd-tree traversal:

The solution in this case is to have a ray-traversal stack per thread, which will have to be implemented in whichever parallel library will be chosen, or simply to put the stack in the actual traversal algorithm and not in the instance.

Using TBB

I’ve decided to go for Thread Building Blocks, as it was already used for game development. This seemed to me a good idea, as it is a Open Source solution. So now, I will split the screen in 2D pieces, and add a thread-specific storage in the kd-tree class. Of course, I will have to add a flag to disable this paralellization if TBB is not available.

The actual parallelization will be in a future post in the Interactive Raytracer category. It is pretty straightforward once I had the different elements Parallel Studio gave me.

Conclusion

In fact Advisor is mainly the annotate.h header, as you have to know your program to put the macros at correct locations. The parallelization must be done by hand, as well as correcting the memory sharing issues.

The only problem I had is that annotate.h includes window.h. This header is not C++ compliant and declares some macros as max() (in fact I got the same issue with TBB headers!). As I use a max() function declared in std::numerical_limits, I had to explicitely undefine this macro.

Safe from this, Advisor Lite is a good plugin, and I’m looking forward to seeing Advisor in a next Parallel Studio release.

Intel Parallel Studio is a set of tools geared toward multithreaded applications. It consists of three Visual Studio plugins (so you need a fully-fledged Visual Studio, not an Express edition):

• Parallel Inspector for memory analysis
• Parallel Amplifier for thread behavior and concurrency
• Parallel Composer for parallel debugging

This is an update of the review I’ve done for the beta version. Since this first review, I’ve tried the official first version.

Since the beta phase, Intel added a lot of documentation, online help, as well as additional samples. This was my main complaint at that time, and now, I can say that Intel provides a complete tool with appropriate help. There is still room for improvement, but not much. For instance, here are videos presenting Parallel Studio.

There is a simple sample to show how all the plugins can be used simultaneously, the NQueens solution, that is also the main Composer example. For Composer though, different parallelization solutions are proposed. According to the Starting Guide and other documents, Intel’s workflow consists of using Advisor (I’ll try to use it in an other post), then Composer to debug the parallelization, Inspector to check for contentions, … and then Inspector to profile the application. One of the videos is dedicated to showing how to use the plugins with the NQueens sample.

As a final point, each plugin has a specific, parametrable toolbar, with a distinct icon.

Parallel Composer

Parallel Composer is mainly an parallel extension to Visual Studio’s debugger. It is based on an Intel runtime, which means you have to use Intel C++ Compiler, which is provided, as well as IPP (a primitives library) and TBB (a parallel library), but not MKL, the scientific library. The 11.1 version of the compiler provides OpenMP 3.0 (Visual Studio compiler only provides 2.5) and thus task parallelism. Intel’s goal is to provide this to C developers (C++ programmers can use TBB, for instance).

The goal of the extension is to detect shared data and its implication on reentrancy (can this function be simultaneously called by different threads ?) or the task and thread tree with OpenMP.

Parallel Composer: The additional options during debugging

The OpenMP panels are not only for OpenMP. They are for every extension that needs /qopenmp (for instance for the parallel extension like __par), in which case useful information is displayed for the state of existing threads. It is also possible to suppress the multithreading and use a monothread execution.

Parallel Composer: Task and thread views

It seems that it is possible to debug several process simultaneously, like TotalView does, but there are no example and no tutorial to explain how to do this.

Parallel Composer is a powerfull debugger extension, with a lot of information that you can get. On one hand, Intel did also a good job to provide tutorials and an online help. On the other hand, the documentation for the most important plugin is perhaps the shortest compared to the two other ones.

Parallel Inspector

Parallel Inspector is in charge of detecting general memory issues as well as thread memory issues. Depending on the inspection level, the execution time can be several times longer. Each time a problem is detecting, it is assigned a gravity degree and registered in a list where you can then have access to its location and the source code.

The first analysis is the general memory one. It detects, for instance, memory leaks. Here is a result that it can give:

Parallel Inspector: Memory report

Parallel Inspector: Location of a memory leak

Usually, this kind of detection needs to modify your code, or with Linux, you have to preload a library that will detect memory leaks (or use valgrind). Here, the really great point is that there are no modification to do on the code and you can use the compiler of your choice.

The real addition of Inspector is of course not checking for memory leaks. Parallel Inspector is not titled “Parallel” for nothing. It can check concurrent memory accesses, and thus warn the developers that some threads can read or write concurrently. Of course, once you’ve checked the access is not dangerous, you can indicate Inspector to skip it (so the inspection is faster next time).

Parallel Inspector: Concurrent memory access

Parallel Inspector: source code of a memory access

Inspector is, in my opinion, the easiest-to-use plugin of Parallel Studio. I find it easy to use because memory checks is something developers always care, so we know what to expect from it.

Parallel Amplifier

Parallel Amplifier is a profiler (I don’t know if it is instrumentation- or sampling-based) like the one you can found in Visual Studio Team edition, or like VTune, the fully-fledged profiler Intel sells as a stand-alone product. Here, you can only get the execution time, but it is still valuable information (if you need more, go and get VTune or Visual Studio Team). Then, for the Parallel profile, you can get the concurrency quality as well as waiting time.

Hotspot is the first profile you can get. The goal is to find where the application sends most of its time, which is in fact called the “hotspot”. In the next example, it is algorithm2, and by double-clicking on it, an annotated source code is displayed.

Parallel Amplifier: Hotspot profil

Parallel Studio: Hotspot annotated source code

How scalable is my program? This is what the second profile tries to answer to. In this case, the scalability is given in the panel at the lower right of the screen (here, for two processors, I get 1.57, which means 78.4% of use, or efficiency). Source code can then be displayed with the annotations, here the lack of concurrency comes from the display routines. On the other hand, algorithm2 scales well. To optimize your concurrency, what you need is to reduce the red/”poor” part of the bar, and maximize the other ones.

Parallel Amplifier: Concurrency

Parallel Amplifier: Concurrency annoted source code

Finally, a crucial issue is waiting and locks. Here again, Amplifier has a specific profile. Here, the main thread only waits for the subthreads to return.

Parallel Amplifier: Waits and locks

Parallel Amplifier: Annotated source code for waits and locks

Profiling should be done anytime, and it is interesting to see whether one optimization enhances the program or not. Amplifier can help you do this.

Parallel Amplifier: Profils comparison

Amplifier comes with several examples, and a good online help. It is not meant to be a full guide to optimization (there are complete books dedicated to this topic), but it gives you access to the tools you need and some leads to use them correctly.

Conclusion

If Amplifier and Inspector are intuitive and simple to use, it is perhaps not the same for Composer. Intel provides several videos as tutorials to help you use all the plugins, as well as complete guides and samples. Parallel Composer is perhaps less documented, but it is mainly more complicated to use, at least from my point of view.

This product is very helpful, in my opinion, not code intrusive (I’m thinking about Amplifier and Inspector for detecting issues without additional libraries) and efficient. The tackled issues are not easy ones to solve, and it does it brilliantly. Since the beta phase, Intel did a tremondous job at providing better documentation for its tool, and now it is the best tool for multithreaded development.

Dr. Dobbs publish some days ago a small post on what is needed for multithreaded application development, and it said Parallel Studio is the perfect tool to help this.

Intel Parallel Studio
Price: $799.95
Designed for today s serial applications and tomorrow’s software innovators

TotalView

Inside the main TotalView window, each program with its threads and processes can be accessed, reopened, even if you closed the application window. The only drawback is that it is not possible to remove an application from this window…

TotalView main window

Launching Totalview raises a window allowing to launch a new program, attach to a running one or analyze a core dump. If the application uses MPI, it must be indicated (several implementations are available).

TotalView open a program windows

Once the application is launched, it is possible to actually debug it. The interface shows which process and thread is currently selected (the list of processes and threads is available in the lowest tab window). Unfortunately, there is no way to browse the code, so you have to go through your code (you can “dive” into a function by double-clicking on a call) to put a breakpoint somewhere.

For TotalView, breakpoints are a special case of action points. On action points, you can stop the program, or execute a simple code. You can also tell Totalview to stop the program when the program went a specific number of time through an instruction (efficient when the error shows up at the hundredth-or-so iteration of a loop).

There are several ways of stopping when arriving at an action point: stopping as soon as one thread/process arrives, when all arrived at it, a group, … There also a lot of other functions that are quite usefull.

TotalView debugging window

Exploring variables is one of the obvious uses of a debugger. Without it, debugging is often useless. TotalView allows to “dive” into a variable, and then explore it. A multi-dimensional variable can be sliced, and then compared between processes. When a variable is modified, it appears in yellow. It is then possible to compare an MPI communication result (for instance).

When comparing to other parallel debugger (like DDT), the array display is not as beautiful. TotalView has other advantages, as having its own C/C++/Fortran debugger, without relying on gdb.

TotalView variable display window

MemoryScape

MemoryScape is TotalView’s memory tool. It captures OS memory calls and watches what the application does.

The first option is to quard memory blocks. It’s less efficient that Fortran’s bound checks, but it is less costly (as the memory guards are only checked when the program stops). Other options include paint blocks (a pattern is “painted” inside the block, and if it shows up somewhere else in the code, it’s that the block wasn’t worrectly initializd, for instance), hoarded memory (deallocated memory is not immediatelly freed, which can then lead to detect memory corruption) and of course leak detection.

Several graphs can be drawn, but some are misleading (as the memory pie, which does not show the truth).

ReplayEngine

Replay Engine is a reverse debugger. When the program crashed, it is possible to rewind the execution to find where the problem first showed up.

Of course, the rewind option is based on snapshots, which means that you cannot replay a really big program (that uses several GB), that ReplayEngine chooses when to do a snapshot, and it is possible that the instant you want was not captured. I never used the ReplayEngine because of these pitfalls (no reverse debugger can escape them).

Conclusion

Although it is pretty much expensive, TotalView is very helpfull. When I had to parallize with MPI a scientific code, it was simple to use the MPI library I used, and the variable display helped me fix the communications in no time.

I never had a real use for MemoryScape. The leak detection is efficient, but like Valgrind, some detected leaks are not real leaks. The guarded memory could have been useful, but as I had read issues, it couldn’t help me.

In the end, I would recommand TotalView as a parallel debugger. With an efficient parallel profiler, it is one of the need-to-have tools in one’s toolbox.

Link to the official TotalView website: http://totalviewtech.com/

Content and opinions

The global content of the book is nothing new. What is really great is the progression thoughout the book (the table of contents is perhaps obvious but it is easy to miswrite it). The decision one has to make to write a parallel application really follow the book’s flow.

Before the actual patterns, two chapters are spent explaining what is meant by parallel programming and the associated patterns. The implication on the OS and the environment are clear and simple. There isn’t much to say about these chapters, as it is pretty much basic knownledge.

The first interesting chapter is the third one. It starts the pattern show with how to achieve concurrency in one’s application. Data, task parallel? Then, how do data and task interact? Once this is done, the fourth chapter comes in. How can one use the preceding findings inside one’s algorithm? Pipeline, divide and conquer, … the usual solutions are thoroughly explained.

Once the interactions are set, they can be used to choose the best tool: one program or several (SPMD or MPMD)? Master/Worker, fork/join, … There are several ways of setting up the application to use the patterns of chapter four (what is called the structure space inside the book). Then, the final chapter is dedicated to the implementation of those structures. Threads or processes? It also depends on the parallel tool that one can use (MPI or OpenMP, for instance), and also how they can interact through synchronization and communication.

At this point, the last step is achived and the program may be written without additional thought about how parallelism can be achieved.

Conclusion

Of course, since the book was first published, additional parallel patterns were “found” (more exactly described), but the ones from the book are the most used ones. Nothing impedes you from using additional ones insode your own parallel workflow.

Ralph Johnson (from the Gang of Four) gave a talk about parallel patterns, and this book. It can be found there: http://media.cs.uiuc.edu/Apresos/seminars/UPCRC/2008-09-19/UPCRC__2008-09-19_02-58-PM_files/flash_index.htm Now that parallelism is the only way to go to really speed up programs, I hope the book will become more popular.

If you don’t like to read and if you have some sense of fun, you can view this Intel video. Enjoy

Patterns for Parallel Programming(Software Patterns Series)
Price: $38.7

Patterns for Parallel Programming(Software Patterns Series)
Price: $30.96

Patterns for Parallel Programming (Hardcover)
by Timothy G. Mattson, Beverly A. Sanders, Berna L. Massingill
ISBN: 0321228111

Price: USD 52.29
42 used & new available from USD 30.00

| 3.5 | 6

Content and opinions

The book can be split in two parts: the first is about the electromagnetic equations, the second on its parallel implementation.

The first chapter deals with the basics of FDTD. Stability analysis, dispersion, … Nothing fancy, but it does its work. Then different kind of boundary conditions are presented in the second chapter. A lot are specific to electromagnetism, but the one I use (CPML) is also part of this one.

The three next chapters deal mainly with electromagnetism specifics, so I didn’t read them much. The third is about some FDTD optimizations, the fourth introduces the different source solutions, mainly electromag-specific. The last chapter on FDTD is about data collection and what can be computed from them. Some information are worth readign, as we can forget, for instance, that an FDTD computation does not output results on the same grid (electric field and magnetic field are interlaced).

The five last chapters are dedicated to the parallel FDTD. After a parallel system introduction (not outstanding, but it present the architecture, the different techniques, how speedup is computed, …), the actual FDTD method is dissected through the different exchange techniques (although I do not use one of them, but the differences and implications of each are correctly described) and the actual exchange code (independent of the technique used). Drawings explain what must be communicated, and the MPI code is given (in fact, it’s an extract of the program given in the beginning of the chapter). Then other electromagnetism topics are addressed.

The eighth chapter presents some results and finally the last two chapters FDTD when using a polar representation (which is not my case, so I’ve skipped them).

Conclusion

As usual for such a book, you have to make your firm buy it for you. At 100$ the 260-pages book, it’s not cheap.
Now, the real question was if I thought it would help a trainee (or a beginner) understand the issues that arise with FDTD. In that matter, I think it does, also a lot can be skipped if electromagnetism is not the application field. It will not teach the specifics of rotated grids or higher orders, but if you’re explained the basics, you can understand more complex issues.

Parallel Finite-Difference Time-Domain Method
Price: $101.6

Parallel Finite-Difference Time-Domain Method (Artech House Electromagnetic Analysis) (Hardcover)
by Wenhua Yu, Raj Mittra, Tao Su, Yongjun Liu, Xiaoling Yang
ISBN: 1596930853

Price: USD 127.00
16 used & new available from USD 94.35

| | 0

Raw computational power needs

With the raw computational power we have, it is possible to simulate more and more complicated models. So the newest processors will never be enough, more precise models will be used (http://www.ddj.com/cpp/205900309)

Still, more complicated models means more memory, more communications and more I/Os. For some applications, this is not a problem. For instance, Folding@Home sends a small data set that a computer will process and then an answer is sent to the server. In this model, there is not much communications, as each data set can be processed by itself.

Memory bandwidth and latency needs

For other applications, memory bandwidth and its latency are the real bottleneck: the application spends more time waiting for data than actually computing. And there are different levels in the bottleneck.

Sometimes, all data can fit inside the L2 cache, so the CPU can almost be fully used. Then, perhaps the data fit inside the RAM, and there are a lot of exchanges between RAM and L2 (for instance in finit difference schemes). In this case, the CPU can sometimes wait for data. There are strategies to enhance this, but the COU will always be idle at some point.

I don’t even talk about cases where the model cannot fit inside the memory, and still is needed at every stage of the computation.

Communication needs

When the computation cannot fit in one computer or one cluster node, applications may face another bottleneck. This is where you have to choose between a grid, a cluster of workstations or a real cluster. A massive grid, like the one used for Folding@Home, solves problems that can’t be solved by a cluster of workstations (in an acceptable time). The latter can solve for instance some medical image processing problems (like detecting differences between 3D brain images of different populations), as the amount of work for one step (like the normalization), can be done on one node but the communication amount is too big to be achieved on the Internet (sending hundreds of megabytes for each result). Then, other medical imaging processings, like the evolution of an artery during a cardiac cycle, need a real cluster with low-latency communications (during each iteration, data must be transmitted to different nodes, and this can only be achived through fast and low latency network interfaces).

What is really needed?

In the end, before you choose the architecture you need, you have to write down the actual problem you want to solve. Buying a cluster if your program won’t benefit from it will be a waste of money. The same can be said for the processor you will use. A processor with a high number of FLOPS, but with a low memory bandwidth can be worse than a processor not so fast but with a high memory bandwidth. It all depends on our problem.