I’ve been thinking about this for several months now, for work that has thousands of tasks, each task being massively data parallel. Traditionnally, one launches a job through one’s favorite batch scheduler (favorite or mandatory…) with fixed resources and during an estimated amount of time. This may work well in research, but in the industrial world, there often a new job that arises and that needs part of your scarce resources. You may have to stop your work, loose your current advances and/or restart the job with less resources. And then the cycle goes on.

Static resource allocation

How can resource allocation work? Let’s start with a simple case where you have 2 applications with different priorities. One of them has a priority of 70 (it’s supposed to finish in three days) whereas the other one has a priority of 50 (four days left). They share the cluster so that 66% is allocated to the first application and 33% to the second one.

What happens if a third application must be launched with a higher priority, because it has to ne finished by tomorrow? You may stop the other two programs, you may loose a lot of work if you didn’t implement checkpoints (besides, one of them may be an of-the-shelf program you bought yesterday) or suspend it. Either way, this is what you will get:

In fact, even if you use dynamic resource allocation, this is what you must get to have your results by the time you need them, but obviously, you have lost your two other applications. Some batch schedulers allow applications to be suspended, but this is a double-edge sword:

• your cluster must support job suspension, and thus have access to drives to save the job state (which is not possible for medium to large-scaled clusters)
• if your application does not scale to your entire cluster (it happens), although one of the other two applications could go on, it is not possible, all processes are put to sleep

So all things considered, you have to implement dynamic resource allocation.

Dynamic resource allocation

How does this work? Each application must be aware that it can be allocated more resources or deallocated some at all time. To be portable on all clusters, you cannot suspend part of your program, it must really go away. The batch scheduler must also notice that your application has freed some of its resources. You thus have to allocate small jobs that will communicate together (this can be done with MPI-2).

This means that you will have hundreds or thousands of small works. All of them will not have to be connected to the scheduler, only one master must be. Of course, this can easilly be done by using a specific queue. Each application on this queue will thus receive orders from the batch scheduler and act upon it. Another advantage is that also the application gets no resource at one point, it still has a saved state that enable the continuation of a run.

Of course, this is not easy to do. How can this be applied to an of-the-shelf application? Well, in this case, you may create a bogus application on the master queue that will at least allow other applications to be allocated resources beside it.

You do not have to implement this on top of MPI. It can be really hard to do (handling data moves between processors, change the decomposition, …), and you may implement another solution. In my case, I have thousands different tasks that can be run on very few cores, so this is my elementary unit. I don’t need all tasks to communicate between them, so I create each time brand new independent jobs and I also can tell the scheduler it can kill jobs that are not responding before the next allocation phase.

Conclusion

To finish, I’ll say that I know that LSF allows plugins that help dispatch jobs on specific hosts of your cluster (to have the best communication location). There seems to be a way of implementing the needs gathering and the resource assignment, but the documentation is not clear (at all). A specific daemon may be needed. I don’t know if other batch scheduler allow plugins to modify their behavior, if you know of them and their API, please do tell

Content and opinions

First of all, it’s not a book on programming massively parallel processors, it’s a book about CUDA. One of the authors is a nVidia fellow, so it’s no wonder. I think there are three parts in the book: an introduction of CUDA, two examples and then general considerations and the future.

The first 6 chapters (I don’t count the first chapter as a real chapter, it’s more of an introduction to the massively paralell processors and their use in a few pages) are the main CUDA tutorial. I say tutorial because it feel like all beginner courses I’ve taken in CUDA. The content can be found in all Internet classes, so the only advantage is that you have everything in a book. Nothing less, nothing more.

I had a feeling of “deja vu” for the MRI example, the second was unknown to me. There is not much code, only for the relevant parts, but you won’t be able to test the different implementations with what is provided in the book. Besides, several times during the writting flow, new techniques are introduced, but one can’t know what speed-up they provide. Perhaps this is because this speedup cannot be generalized, but still, with proper warnings, the different timings through the GPU port of woth examples would have been great.

The last part is, as I’ve said, more general. It starts with a workflow to help parallelizing with GPUs, then an introduction (too short IMHO) of OpenCL and the future of CUDA with Fermi and the SDK 3.0. The workflow chapter is too small. Of course, the goal isn’t to be like The Art of Concurrency, and at least there is a chapter about the process of selecting the algorithm, … but it is too small. The OpenCL introduction is really an introduction. I’ve seen one small complete OpenCL call, but that’s it. I couldn’t program a single kernel right now. Of course it’s a CUDA book, not an OpenCL one, but the chapter is useless. Perhaps it would be better to merge it with the “future” chapter, as OpenCL is not widely available. Finally, the last chapter states what can be expected of Fermi (really interesting) and of the SDK 3.0.

What I miss in this book is some explanations of the texture memory. The obvious matrix example uses constant memory for caching the memory accesses. Why isn’t texture memory used in this example? It’s far bigger than constant memory and also has a cache, so why not use it? It’s a CUDA book, but a lot of content is freely available in several tutorials that are sometimes better shaped than the book, so why isn’t there some special content, like how the cache works? How can you manage grid sizes that are no a power of two? (it’s explained in one of the example, with zero padding, but there are no protection in the first chapters, which is dangerous) What is coalescing memory and how can I optimize the memory bandwidth with coalescing in mind? (the actual real explanation and appropriate picture is in the last annexe!)

Conclusion

I don’t say that the book is not useful, it’s really interesting as a companion book for a CUDA course or for a beginner. If you’re used to electronic papers, you will not be interested. If you buy this book, don’t expect to know everything about CUDA, or even less massively parallel processors. You will have to dig deeper for specific topics, but at least you will have a good basis.

Programming Massively Parallel Processors: A Hands-on Approach
Price: $62.95

Programming Massively Parallel Processors: A Hands-on Approach (Paperback)
by David B. Kirk, Wen-mei W. Hwu
ISBN: 0123814723

Price: USD 51.99
40 used & new available from USD 46.52

| 3.5 | 8

Content and opinions

Thee parts, three views how what debugging actually is.

The first is the pragmatic debugging: reproduce your bug, diagnose it, fix it, and make it definitely go away. Each step is actually not mandatory done by a debugger, and in fact the book advocates the TDD approach. Making it go away is in fact thinking about what this bug meant: was it a design issue, a feature, and if it is a bug, how can all the consequences be erased?

The second is managing your bugs. The main action is to have a bug database that is healthy, meaning it must be taken care of. As the book is about a pragmatic approach, one may think the advice are pure logic, and that no one would not follow them. If this book serie has so much success, it is because those advice are in fact not that obvious for everyone!

Finally, the first two parts are in fact the theory. Practice is always a little bit different. What do you do of leagacy releases? Of third-parties libraries? Of bugs that you need to fix in a hurry in parallel? Tests is another facet of software closelly related to debugging. Once you’ve debugged something, you want your tests to assert that it won’t come again unless you see it. How can you be sure that you will have enough information when debugging your software? Finally, there are some pitfalls in software that hinder debugging. As those anti-patterns, this part is about concrete actions that must be implemented in every software project.

Conclusion

Pragmatic Bookshelf has the habit of having a lot of experience feedback in their books, and this one is not an exception. Sadly, there are mainly present in the first part of the book. Still, it’s not the main value of the book. I think the main value is placing debugging in the new landscape of agile software. Too many times, the word debugging is missing in agile books, replaced by only testing. They are the two sides of the same coin. The last part has its virtues as well. Debugging is not a simple workflow, there are differences for each software project.

As usual, the publisher has an excellent book on a practical subject that answers practical questions on debugging.

Debug It!: Find, Repair, and Prevent Bugs in Your Code
Price: $29.88

Debug It!: Find, Repair, and Prevent Bugs in Your Code (Pragmatic Programmers) (Paperback)
by Paul Butcher
ISBN: 193435628X

Price: USD 23.07
38 used & new available from USD 12.94

| 4.5 | 8

Today’s accelerators are not there yet. The tools are not ready yet (code translators) and usual programming practices may not be adequate. All the ecosystem will evolve, accelerators will change (GPUs are the main trend, but they will be different in a few years), so what you will do today needs to be shaped with these changes in mind. How is it possible to do so? Is it even possible?

Available code translators

Code translators are the easiest path to solution. I know two of them.

The first is the PGI compiler. It only supports CUDA and the Fortran and C99 language. I didn’t use it yet, also I plan of testing it in the near future. It is based on pragmas, and the compiler generates the CUDA microcode.

The second solution is HMPP. It supports more than just CUDA (also CAL/IL or OpenCL) and Fortran/C (also Java now). As the PGI compiler, it is based on pragmas, and a excellent thing is that it detects the available accelerators and launches the correct kernel (if you authorized it) or the original code. You can also modify the generated code to put your own (you can tune the code for instance, which may give you an additional x2 factor). Unfortunately, it is not possible to call functions inside the parallelized kernels, which means that only simple or badly-written (too many lines or duplicated code) kernels can be called. I think this is the same for the PGI compiler.

It seems that code translators still need work:

• only few accelerators are supported (CUDA, and sometimes CAL/IL or OpenCL),
• almost no langage (Fortran/C/Java, a lot of Virtual Machines should be able to use them natively, without developers using specific tools),
• only one function can be parallelized at a time.

The last point is currently the biggest issue. You need to cut your function int pieces to have clean code and a good portability/evolutivity for the future.

This is why one still need to program a lot for those accelerators, and so we need to adapt our programming practices, develop in the accelerators’ native langages (even if we know that they may disappear in a few years).

Developping your own “tool chain” for accelerators

For accelerators, there are a lot of things that needs to be done each time: copying some data, computing and getting some data back. These are the steps that code translators automate, in fact it is a common practice to use tools to automate stuff. The issue is that complex kernels are not supported by those translators. So what?

Creating automatic functions that will copy the data you need is in fact very common in metaprogramming. Coding the kernel on an accelerator is in fact not that difficult: the manufacturers provide the needed compilers (that’s what nVidia does and the success of the tool chain cannot be denied), and this is really the cornerstone. One has to write more code, some parts are less portable (because they are written in one of the accelerator’s languages), but in the end, with metaprogramming, the code can be better tuned, enhanced and read. This is the leverage of the accelerators.

Conclusion

Why do we care developing for accelerators? We know that they will go away. Before they do, they are the only way of speeding up our software. Code translators are the best tools to develop in a portable way, but they need time to support more accelerators, languages and method of programming. When CPUs will be on a par with accelerators, their progress will help compilers to target them correctly. It’s just a matter of time.
Meanwhile, metaprogrammin is the next best solution to automate processes that code translators cannot support yet.

Content and opinions

The book starts with two chapers on what actions to take before parallelizing and what can and what cannot. Before the usual algorithms that can be parallelized, the author takes three chapters to explain how you may achieve your goal. Ensuring correctness is a difficult task, so the book gives 8 rules to help and then an explanation of several support libraries that can be used.

The biggest part of the book, as I’ve hinted, is dedicated to simple but usual algorithms that may be parallelized: sums and scans, mapreduce, sorts, searches, and graph algorithms. Each time, several different algorithms are first coded in a serial way and then parallelized with possibly different support libraries. Also each time, the efficiency, the simplicity, the portability and the scalability conclude the sub art: it helps standing back.

The last chapter is a small overview of the additional tools that you may use (but they are not mandatory). They are mainly Intel’s tools, but it’s mainly because Intel provides the developer with some of the best tools.

Conclusion

Although the author works for Intel, he doesn’t expose Intel tools more than others. The book tone is adequate, not too much serious, not like a “For Dummies”, so just enjoyable.

If you need advices to parallelize your applications and you don’t want to buy Patterns for Parallel Programming, buy this one (well, buy it anyway).

The Art of Concurrency: A Thread Monkey’s Guide to Writing Parallel Applications
Price: $40.49

The Art of Concurrency: A Thread Monkey’s Guide to Writing Parallel Applications (Paperback)
by Clay Breshears
ISBN: 0596521537

Price: USD 38.48
40 used & new available from USD 17.99

| 3.5 | 6

Mercurial (also called hg as the Mendeleiev symbol for mercurial) is one of the three DVCS (Distributed Version Control System) that are in the mood nowadays. Written in Python, its life started at the time as git’s when BitKepper was dumped as the Linux kernel’s VCS. Now it is a mature product, and the book tries to explain how to use it and also the differences with Git. Bazaar, the third DVCS, is not even mentionned, although it is also written in Python.

Content and opinions

The book starts with a small introduction to revision control, and then gets to the basics of version control with Mercurial. This chapter and the following one really are the only things to fully master before starting with hg. With them, you can track your changes, send them to other repositories, …, and the third chapter helps you merging different changes together. In this case, you may encounter some small “glitches” of Mercurial, and the book explains them correctly.

The remainign of the book is more dedicated to small additions that can make your life better. For instance, the fourth chapter is about the efficient repository storage. If a repository with Subversion could get really big, Mercurial keeps things small and doesn’t use garbage collectors like Git. The next chapter that was interesting was the eigth (the other deal with daily use, that is, copy, rename, …, web servers, selecting several files through regexp). It deals with organizing the branches and giving names to revisions. I think it could have deserved a better place, as I find the last three chapters of less interest (in fact, other book on DVCS deal with the web server part at the end, so I think it’s not just me).

The next chapter is about fixing some mistakes you may make. Mercurial has some useful tools, and there is also the bisect command. I had to use it several times, and it is really efficient. Some tips are given to help using this command efficiently, and it may be the most important topic in this chapter. Reverting or “removing” a commit (they are never really removed, they will stay in a hidden branch of some sort) can help sorting a situation, but you shouldn’t use it anyway. The book should have big red/orange warnings in this chapter (which is obviously has not). An important part of VCS is everything related to automation with hooks. Mercurial delivers several places where you can launch your scripts (for instance to update your web site), and the book has a good tutorial on how to write simple hooks.

Another not so important part is how you can customize Mercurial’s output. Interesting, but not that much. A topic that helped Mercurial gaining momentum is MQ (Management Queues). I don’t have the need myself, so the two chapters on that subject were a little bit boring. Besides, to use MQ, you have to set up another repository inside the hidden folder of Mercurial, which is not as efficient as having the queues directly inside the first repository. A good point is that it shows that extensions can really change the way Mercurial can be used (which is the subject of the last chapter).

Conclusion

All things considered, the book managed browsing through the capabilities of Mercurial. The contents may not be organized in the best way, but one can still find what it needs, and even more. If you want to use a DVCS, try Mercurial with this book, you won’t be disappointed (by neither of them).

Mercurial: The Definitive Guide: Modern Software for Collaboration (Animal Guide Series)
Price: $35.99

Mercurial: The Definitive Guide (Animal Guide) (Paperback)
by Bryan O’Sullivan
ISBN: 0596800673

Price: USD 34.07
33 used & new available from USD 17.93

| 5 | 4

Content and opinions

The book has three main parts: a general introduction to Distributed Version Control Systems (DVCS) and Git, using Git everyday and what is called administration.

The introduction covers the different choices made by Git relative to Version Control. Without the actual commands, it is a gentle introduction, and if the choices does not suit you, well, you can walk away here. Its mainly an introduction to DVCS, but with a Git biais. If you find the book in a real library, you can browse this single chapter first. Then, another chapter helps you install the software package on Linux, OS X or even Windows.

The actual fun part comes next. How do you actually use Git to add and commit changes? Git has some some specifics in that matter compared to Bazaar or Mercurial. What are branches and how can they be used, how they should be used? What is the purpose of history and how it can be changed (fundamentaly changed)? Git is one of the few VCS that can actually rewrite history. There are of course enough warnings because this can be dangerous. Once history is known, it can be shared through remote repositories, and then how should a repository be organized? Finally some additional tools are presented. It’s really easy to follow the evolution, and it draws a small picture of the commands you will use every day.

The last part is abut migrating from CVS and SVN, and then how to set up a Git server. The connection between the commands in SVN and CVS is also described when you switch to Git. Of course, you will need more than CVS and SVN knowlegde to really use Git to its maximum power, the book is not about the theory of VCS, only about how to use Git. You will find a good introduction, but you will have to learn by other means the complete philosophy behind DVCS.

Conclusion

I didn’t know much about Git before reading the book, and the approach proposed here is efficient (although simple): from the basics to advanced workflows. You need to know a little about version control, as the Git specific parts are well explained in the book.

Pragmatic Version Control Using Git (Pragmatic Starter Kit) (Paperback)
by Travis Swicegood
ISBN: 1934356158

Price: USD 23.07
46 used & new available from USD 16.00

| 4.5 | 20

Test on parallelizing a custom library

I’ve decided to test Advisor Lite on my Interactive Raytracer. This is a test to verify if Advisor Lite finds the adequate parallelization and the memory sharing issues. It is a simple raytracer, so it can be parallelized for each pixel in the image. The only memory sharing issue that I know of is in the kd-tree ray traversal.

Profiling the library

First, I will profile the library. For the complete Advisor Lite workflow, I have to use Intel Compiler, and as it is faster than Microsoft’s compiler, I will use the timeit_image.py script instead of the measure_image.py I’ve used when profiling with Valgrind or Visual Studio.

Amplifier can show the results in a bottom-up or in a top-down manner. Unfortunately, you only have the exclusive timing that is displayed. In my case, when displaying bottom-up results, the method getEntryExitDistances() is the most costly one. In the top-down view, unfortunately, I can’t have a simple tree, as it can be seen in the following view:

In Visual Studio, I have the same results – more or less -, but with a correct top-down call-tree:

The method getEntryExitDistances() cannot be parallelized: it is recursively called, several times per pixel, which would lead to a lot of memory contention. The simpler task is thus to parallelized the pixel rendering, a perfect data-parallel problem.

Annotation of the code

OK, now I can annotate my code. I had to dig inside the help for this, as it was not mentionned in the Starting Guide that Intel provides a header, annotate.h, which mimics the issues you may encounter in a multithreaded application.

So you need to read at least once the online help so that you know the available annotation macros, how you can get them and how they will retrieve what you need. Once the code is annotated, it must be recompiled and then the sharing issues can be detected.

Detection of sharing issues

As expected, Inspector detected errors in the kd-tree traversal:

The solution in this case is to have a ray-traversal stack per thread, which will have to be implemented in whichever parallel library will be chosen, or simply to put the stack in the actual traversal algorithm and not in the instance.

Using TBB

I’ve decided to go for Thread Building Blocks, as it was already used for game development. This seemed to me a good idea, as it is a Open Source solution. So now, I will split the screen in 2D pieces, and add a thread-specific storage in the kd-tree class. Of course, I will have to add a flag to disable this paralellization if TBB is not available.

The actual parallelization will be in a future post in the Interactive Raytracer category. It is pretty straightforward once I had the different elements Parallel Studio gave me.

Conclusion

In fact Advisor is mainly the annotate.h header, as you have to know your program to put the macros at correct locations. The parallelization must be done by hand, as well as correcting the memory sharing issues.

The only problem I had is that annotate.h includes window.h. This header is not C++ compliant and declares some macros as max() (in fact I got the same issue with TBB headers!). As I use a max() function declared in std::numerical_limits, I had to explicitely undefine this macro.

Safe from this, Advisor Lite is a good plugin, and I’m looking forward to seeing Advisor in a next Parallel Studio release.

Content and opinions

TDD is about testing while developing, so it is of no wonder that the first part of the book which expose the methodology is done with the help of an example. Small steps, one of the main ideas behind TDD, in the explanations result in a lot of small chapters that explain how to do one thing.

The second part is about the main tol for TDD: xUnit. Its goal is to provide a standard tool for all languages.Well, in fact, it is different implementation, but some are not as easy to use as JUnit, the Java implementation. Some languages are more easy to cope with TDD than others.

The third part is a formal list of patterns to efficiently develop with TDD. What do you need to test something completly? What do you need to go from a test to the actual associated solution? This is the kind of questions the patterns try to answer.

Conclusion

There are a lot chapters with simple ideas in each one. What bothers me is that they could have been hierarchized a little bit more, but really, this is not an issue here. TDD is something that everyone should do, but there are some domains where it is more difficult than in other. For instance, hw would you test a complex stochastic scientific module? This is a question that remains open at the end of the book. For general IT, it answers every question on testing though.

Test Driven Development: By Example
Price: $44.99

Test Driven Development: By Example (Paperback)
by Kent Beck
ISBN: 0321146530

Price: USD 40.53
54 used & new available from USD 24.99

| 4 | 34

Using the profiler

First of all, the code must be compiled with /Zi or /Z7. Then, the link edition must be done with /DEBUG. Without these options, it won’t be possible to measure anything.

Performance Tool must be used in a Visual Studio Solution, which I don’t have in the case of the Interactive Raytracer. I will have to create an empty project, and I will create a bogus project, just to be able to browse the code in the IDE.

Once in Performance Tool, I create a new target which will be my generated library:

Then, I have to add it to the targets to profile:

Then I modify the launch properties to launch python with a sample script that is supposed to be descriptive enough. If you profile a Visual Studio project, you can use the debugging project properties (there are more environment parameters that you may modify).

Sampling

To use the sampling profiler, you have to select it in the session parameters. Once done, you can launch the profiler.

This is the results I got from the IRT:

This view helps finding the function to optimize. It display the current function at the middle of the panel, and then the caller functions at the top and the callee/called functions at the bottom. The displayed costs are inclusive and exclusive costs.

A more general view, the call-tree, can help finding the hotspot as well. The caller can obviously not be displayed.

It is easy to get access to the code by a left click, which is one of the reasons I’ve created a bigus Visual Studio project.

Instrumentation

Instrumentation-based profiling is somewhat different than sampling-based profiling. Here, the executable is modified to count cycles or counters, whereas in sampling-based profile, the program is sampled at regular intervals to get where the program is, the counters, …

Instrumentation results are quite like the sampling ones

Conclusion

According to the different results I got, I know that I have some work to do on the math library. Well, this was some months ago, so now this code optimized as well. Based on Performance Tool profile, a lot has been optimized, as not creating rays inside the loop, …

I cannot really compare Performance Tool to Valgrind, as they are not the same kind of profilers. They each give more or less precise information on some parts of your program.
Compared to KCacheGrind, the callee map is missing, but Microsoft did a great job with the UI. The information may not be visualized as a 2D image, but it is still easy to extract it from the different views.

As a conclusion, I’d simply say that Performance Tool is an excellent tool Mixrosoft provides in its high-end Visual Studio versions. To bad it is not acceissble in the lower versions.