I’ve been thinking about this for several months now, for work that has thousands of tasks, each task being massively data parallel. Traditionnally, one launches a job through one’s favorite batch scheduler (favorite or mandatory…) with fixed resources and during an estimated amount of time. This may work well in research, but in the industrial world, there often a new job that arises and that needs part of your scarce resources. You may have to stop your work, loose your current advances and/or restart the job with less resources. And then the cycle goes on.

Static resource allocation

How can resource allocation work? Let’s start with a simple case where you have 2 applications with different priorities. One of them has a priority of 70 (it’s supposed to finish in three days) whereas the other one has a priority of 50 (four days left). They share the cluster so that 66% is allocated to the first application and 33% to the second one.

What happens if a third application must be launched with a higher priority, because it has to ne finished by tomorrow? You may stop the other two programs, you may loose a lot of work if you didn’t implement checkpoints (besides, one of them may be an of-the-shelf program you bought yesterday) or suspend it. Either way, this is what you will get:

In fact, even if you use dynamic resource allocation, this is what you must get to have your results by the time you need them, but obviously, you have lost your two other applications. Some batch schedulers allow applications to be suspended, but this is a double-edge sword:

• your cluster must support job suspension, and thus have access to drives to save the job state (which is not possible for medium to large-scaled clusters)
• if your application does not scale to your entire cluster (it happens), although one of the other two applications could go on, it is not possible, all processes are put to sleep

So all things considered, you have to implement dynamic resource allocation.

Dynamic resource allocation

How does this work? Each application must be aware that it can be allocated more resources or deallocated some at all time. To be portable on all clusters, you cannot suspend part of your program, it must really go away. The batch scheduler must also notice that your application has freed some of its resources. You thus have to allocate small jobs that will communicate together (this can be done with MPI-2).

This means that you will have hundreds or thousands of small works. All of them will not have to be connected to the scheduler, only one master must be. Of course, this can easilly be done by using a specific queue. Each application on this queue will thus receive orders from the batch scheduler and act upon it. Another advantage is that also the application gets no resource at one point, it still has a saved state that enable the continuation of a run.

Of course, this is not easy to do. How can this be applied to an of-the-shelf application? Well, in this case, you may create a bogus application on the master queue that will at least allow other applications to be allocated resources beside it.

You do not have to implement this on top of MPI. It can be really hard to do (handling data moves between processors, change the decomposition, …), and you may implement another solution. In my case, I have thousands different tasks that can be run on very few cores, so this is my elementary unit. I don’t need all tasks to communicate between them, so I create each time brand new independent jobs and I also can tell the scheduler it can kill jobs that are not responding before the next allocation phase.

Conclusion

To finish, I’ll say that I know that LSF allows plugins that help dispatch jobs on specific hosts of your cluster (to have the best communication location). There seems to be a way of implementing the needs gathering and the resource assignment, but the documentation is not clear (at all). A specific daemon may be needed. I don’t know if other batch scheduler allow plugins to modify their behavior, if you know of them and their API, please do tell

Content and opinions

First of all, it’s not a book on programming massively parallel processors, it’s a book about CUDA. One of the authors is a nVidia fellow, so it’s no wonder. I think there are three parts in the book: an introduction of CUDA, two examples and then general considerations and the future.

The first 6 chapters (I don’t count the first chapter as a real chapter, it’s more of an introduction to the massively paralell processors and their use in a few pages) are the main CUDA tutorial. I say tutorial because it feel like all beginner courses I’ve taken in CUDA. The content can be found in all Internet classes, so the only advantage is that you have everything in a book. Nothing less, nothing more.

I had a feeling of “deja vu” for the MRI example, the second was unknown to me. There is not much code, only for the relevant parts, but you won’t be able to test the different implementations with what is provided in the book. Besides, several times during the writting flow, new techniques are introduced, but one can’t know what speed-up they provide. Perhaps this is because this speedup cannot be generalized, but still, with proper warnings, the different timings through the GPU port of woth examples would have been great.

The last part is, as I’ve said, more general. It starts with a workflow to help parallelizing with GPUs, then an introduction (too short IMHO) of OpenCL and the future of CUDA with Fermi and the SDK 3.0. The workflow chapter is too small. Of course, the goal isn’t to be like The Art of Concurrency, and at least there is a chapter about the process of selecting the algorithm, … but it is too small. The OpenCL introduction is really an introduction. I’ve seen one small complete OpenCL call, but that’s it. I couldn’t program a single kernel right now. Of course it’s a CUDA book, not an OpenCL one, but the chapter is useless. Perhaps it would be better to merge it with the “future” chapter, as OpenCL is not widely available. Finally, the last chapter states what can be expected of Fermi (really interesting) and of the SDK 3.0.

What I miss in this book is some explanations of the texture memory. The obvious matrix example uses constant memory for caching the memory accesses. Why isn’t texture memory used in this example? It’s far bigger than constant memory and also has a cache, so why not use it? It’s a CUDA book, but a lot of content is freely available in several tutorials that are sometimes better shaped than the book, so why isn’t there some special content, like how the cache works? How can you manage grid sizes that are no a power of two? (it’s explained in one of the example, with zero padding, but there are no protection in the first chapters, which is dangerous) What is coalescing memory and how can I optimize the memory bandwidth with coalescing in mind? (the actual real explanation and appropriate picture is in the last annexe!)

Conclusion

I don’t say that the book is not useful, it’s really interesting as a companion book for a CUDA course or for a beginner. If you’re used to electronic papers, you will not be interested. If you buy this book, don’t expect to know everything about CUDA, or even less massively parallel processors. You will have to dig deeper for specific topics, but at least you will have a good basis.

Programming Massively Parallel Processors: A Hands-on Approach
Price: $62.95

Programming Massively Parallel Processors: A Hands-on Approach (Paperback)
by David B. Kirk, Wen-mei W. Hwu
ISBN: 0123814723

Price: USD 51.99
40 used & new available from USD 46.52

| 3.5 | 8

Today’s accelerators are not there yet. The tools are not ready yet (code translators) and usual programming practices may not be adequate. All the ecosystem will evolve, accelerators will change (GPUs are the main trend, but they will be different in a few years), so what you will do today needs to be shaped with these changes in mind. How is it possible to do so? Is it even possible?

Available code translators

Code translators are the easiest path to solution. I know two of them.

The first is the PGI compiler. It only supports CUDA and the Fortran and C99 language. I didn’t use it yet, also I plan of testing it in the near future. It is based on pragmas, and the compiler generates the CUDA microcode.

The second solution is HMPP. It supports more than just CUDA (also CAL/IL or OpenCL) and Fortran/C (also Java now). As the PGI compiler, it is based on pragmas, and a excellent thing is that it detects the available accelerators and launches the correct kernel (if you authorized it) or the original code. You can also modify the generated code to put your own (you can tune the code for instance, which may give you an additional x2 factor). Unfortunately, it is not possible to call functions inside the parallelized kernels, which means that only simple or badly-written (too many lines or duplicated code) kernels can be called. I think this is the same for the PGI compiler.

It seems that code translators still need work:

• only few accelerators are supported (CUDA, and sometimes CAL/IL or OpenCL),
• almost no langage (Fortran/C/Java, a lot of Virtual Machines should be able to use them natively, without developers using specific tools),
• only one function can be parallelized at a time.

The last point is currently the biggest issue. You need to cut your function int pieces to have clean code and a good portability/evolutivity for the future.

This is why one still need to program a lot for those accelerators, and so we need to adapt our programming practices, develop in the accelerators’ native langages (even if we know that they may disappear in a few years).

Developping your own “tool chain” for accelerators

For accelerators, there are a lot of things that needs to be done each time: copying some data, computing and getting some data back. These are the steps that code translators automate, in fact it is a common practice to use tools to automate stuff. The issue is that complex kernels are not supported by those translators. So what?

Creating automatic functions that will copy the data you need is in fact very common in metaprogramming. Coding the kernel on an accelerator is in fact not that difficult: the manufacturers provide the needed compilers (that’s what nVidia does and the success of the tool chain cannot be denied), and this is really the cornerstone. One has to write more code, some parts are less portable (because they are written in one of the accelerator’s languages), but in the end, with metaprogramming, the code can be better tuned, enhanced and read. This is the leverage of the accelerators.

Conclusion

Why do we care developing for accelerators? We know that they will go away. Before they do, they are the only way of speeding up our software. Code translators are the best tools to develop in a portable way, but they need time to support more accelerators, languages and method of programming. When CPUs will be on a par with accelerators, their progress will help compilers to target them correctly. It’s just a matter of time.
Meanwhile, metaprogrammin is the next best solution to automate processes that code translators cannot support yet.

TotalView

Inside the main TotalView window, each program with its threads and processes can be accessed, reopened, even if you closed the application window. The only drawback is that it is not possible to remove an application from this window…

TotalView main window

Launching Totalview raises a window allowing to launch a new program, attach to a running one or analyze a core dump. If the application uses MPI, it must be indicated (several implementations are available).

TotalView open a program windows

Once the application is launched, it is possible to actually debug it. The interface shows which process and thread is currently selected (the list of processes and threads is available in the lowest tab window). Unfortunately, there is no way to browse the code, so you have to go through your code (you can “dive” into a function by double-clicking on a call) to put a breakpoint somewhere.

For TotalView, breakpoints are a special case of action points. On action points, you can stop the program, or execute a simple code. You can also tell Totalview to stop the program when the program went a specific number of time through an instruction (efficient when the error shows up at the hundredth-or-so iteration of a loop).

There are several ways of stopping when arriving at an action point: stopping as soon as one thread/process arrives, when all arrived at it, a group, … There also a lot of other functions that are quite usefull.

TotalView debugging window

Exploring variables is one of the obvious uses of a debugger. Without it, debugging is often useless. TotalView allows to “dive” into a variable, and then explore it. A multi-dimensional variable can be sliced, and then compared between processes. When a variable is modified, it appears in yellow. It is then possible to compare an MPI communication result (for instance).

When comparing to other parallel debugger (like DDT), the array display is not as beautiful. TotalView has other advantages, as having its own C/C++/Fortran debugger, without relying on gdb.

TotalView variable display window

MemoryScape

MemoryScape is TotalView’s memory tool. It captures OS memory calls and watches what the application does.

The first option is to quard memory blocks. It’s less efficient that Fortran’s bound checks, but it is less costly (as the memory guards are only checked when the program stops). Other options include paint blocks (a pattern is “painted” inside the block, and if it shows up somewhere else in the code, it’s that the block wasn’t worrectly initializd, for instance), hoarded memory (deallocated memory is not immediatelly freed, which can then lead to detect memory corruption) and of course leak detection.

Several graphs can be drawn, but some are misleading (as the memory pie, which does not show the truth).

ReplayEngine

Replay Engine is a reverse debugger. When the program crashed, it is possible to rewind the execution to find where the problem first showed up.

Of course, the rewind option is based on snapshots, which means that you cannot replay a really big program (that uses several GB), that ReplayEngine chooses when to do a snapshot, and it is possible that the instant you want was not captured. I never used the ReplayEngine because of these pitfalls (no reverse debugger can escape them).

Conclusion

Although it is pretty much expensive, TotalView is very helpfull. When I had to parallize with MPI a scientific code, it was simple to use the MPI library I used, and the variable display helped me fix the communications in no time.

I never had a real use for MemoryScape. The leak detection is efficient, but like Valgrind, some detected leaks are not real leaks. The guarded memory could have been useful, but as I had read issues, it couldn’t help me.

In the end, I would recommand TotalView as a parallel debugger. With an efficient parallel profiler, it is one of the need-to-have tools in one’s toolbox.

Link to the official TotalView website: http://totalviewtech.com/

Raw computational power needs

With the raw computational power we have, it is possible to simulate more and more complicated models. So the newest processors will never be enough, more precise models will be used (http://www.ddj.com/cpp/205900309)

Still, more complicated models means more memory, more communications and more I/Os. For some applications, this is not a problem. For instance, Folding@Home sends a small data set that a computer will process and then an answer is sent to the server. In this model, there is not much communications, as each data set can be processed by itself.

Memory bandwidth and latency needs

For other applications, memory bandwidth and its latency are the real bottleneck: the application spends more time waiting for data than actually computing. And there are different levels in the bottleneck.

Sometimes, all data can fit inside the L2 cache, so the CPU can almost be fully used. Then, perhaps the data fit inside the RAM, and there are a lot of exchanges between RAM and L2 (for instance in finit difference schemes). In this case, the CPU can sometimes wait for data. There are strategies to enhance this, but the COU will always be idle at some point.

I don’t even talk about cases where the model cannot fit inside the memory, and still is needed at every stage of the computation.

Communication needs

When the computation cannot fit in one computer or one cluster node, applications may face another bottleneck. This is where you have to choose between a grid, a cluster of workstations or a real cluster. A massive grid, like the one used for Folding@Home, solves problems that can’t be solved by a cluster of workstations (in an acceptable time). The latter can solve for instance some medical image processing problems (like detecting differences between 3D brain images of different populations), as the amount of work for one step (like the normalization), can be done on one node but the communication amount is too big to be achieved on the Internet (sending hundreds of megabytes for each result). Then, other medical imaging processings, like the evolution of an artery during a cardiac cycle, need a real cluster with low-latency communications (during each iteration, data must be transmitted to different nodes, and this can only be achived through fast and low latency network interfaces).

What is really needed?

In the end, before you choose the architecture you need, you have to write down the actual problem you want to solve. Buying a cluster if your program won’t benefit from it will be a waste of money. The same can be said for the processor you will use. A processor with a high number of FLOPS, but with a low memory bandwidth can be worse than a processor not so fast but with a high memory bandwidth. It all depends on our problem.

I think it goes even further than that, and it will be a part of this post. What I see is that trainees and newly-hired people (to some extent myself included) lack a lot of basic Computer Science knowledge, and even IT knowledge.

General knowledge

A crucial issue is the computer architecture knowlegde:

• What endinanness do I use on my computers? A lot of scientific code was programmed in big endian, whereas current mainstream computers are little endian (safe for the Power architecture, thus Cell, IBM clusters, …). Although this is taught during the first programming courses, some students tend to forget about it.
• How should I loop on data? If you loop on discontiguous data block, the CPU cache is likely to be too small to contain all your data. This will lead to cache misses, and those can cost a lot.
• More generally how much memory can I use and what is its speed (bandwidth and latency)? On current computers you have access to several GB of RAM, MB of cache. On GPUs (which is more and more used for HPC), you have hundreds of MB, but access to the graphic global RAM is costly (the same applies for the CELL).

At least students must learn that they will have to think about those issues. I don’t know each memory size of each processor (I roughly know how much, and if I need more details, I search the Internet for the answer).

Parallel knowledge

Then there is the parallel part. Even for scientific research on small datasets, the advent of multicores is a challenge for students (and to a lot of teachers as well). We can’t expect them to know how to deal with multithreading or multiprocessing if they don’t even know the difference between an Intel chip and an IBM one.

In medical imaging, we could simply split our workflow in tasks (we had several registrations to do, so we can simply split the work and make on each core one registration), but even that was not easy. Some algorithms were not thread-safe (who said that global variables should be avoided?), so we had to launch several different processes. But when it comes to parallelize the registration itself because it was long, and also because the problem size began to be too big for a 32bits application (yes, we were still in 32bits because of legacy GTK1 applications), nobody could do it, because nobody had the time and the knowledge to use all the CPU power.

In bigger applications like sismic ones, it’s even more obvious. Copy ‘n’ Paste can be a sport, if someone one day parallized efficiently his program. All things considered, it’s not a problem, but if the paste is not done smartly and if the developer didn’t learn from the analyze of the first program, it’s useless, because he didn’t learn AND because the pasted program will not be optimal and even subject to bugs.

Some additional thoughts

I found a post on Intel’s Community page about questions that will be discussed at SuperComputing ‘08. It shows that there is no clear answer to this problem (at this time of writing). We should teach parallelism, HPC, … to students. But are the tools ready? Is every student even able to cope with parallelism?