• you can interpolate the data in the original space based on an interpolation in the reduced space, or
• you create an approximation of the mapping with a multidimensional function (B-splines, …)

When using the first solution, if you map one of the reduced point used for the training, you get the original point. With the second solution, you get a close point. If the data set you have is noisy you should use the second solution, not the first. And if you are trying to compress data (lossly compression), you can not use the first one, as you need the original points to get new interpolated points, so you are not compressing your data set.

The solution I propose is based on approximation with a set of piecewise linear models (each model being a mapping between a subspace of the reduced space to the original space). At the boundaries between the models, I do not assert continuity, contrary to hinging hyperplanes. Contrary to Projection Pursuit Regression and hinging hyperplane, my mapping is between the two spaces, and not from the reduced space to one coordinate in the original space. This will enable projection on the manifold (which is another subject that will be discussed in another post).

Position of the problem

In the literature, several papers were published in order to create a piecewise linear function. Their main advantage is that they can compute the reduced space when computing the mapping. Their issue is that the number of models is fixed at the beginning of the optimization. What I propose now is an adaptive number of models, depending on the manifold.

Each point has a set of neighbors, the k nearest ones are used here.

Basic approach

In fact, the basic algorithm is straightforward: put a new model where you can, optimize all models, label the points you can label, loop and stop at some point.

The first algorithm I used is the following:

1. Start with no model.
2. Find a point whose neighborhood is not labeled, if you can’t, stop.
3. Create a new model there and label the points accordingly.
4. Estimate all models with regards to the labels.
5. Label each point to the nearest model if the point is too far (that is further than a factor times the mean error), do not label it.
6. If one model has not enough points (twice the dimension of the reduced space), the model is deleted.
7. Go back to step 2.

This algorithm is very simple, but cannot model precisely a manifold, you cannot tune it. But it can give a good first impression on the manifold.

Here are some steps of the algorithm for a SwissRoll (the left figure indicates the label in the reduced space and the left figure is the approximation of the manifold in the original space):

At the end of the iteration, the manifold is roughly approximated:

Maximize a likelihood

As I’ve said, there are no ways for the first method to get a given precision. So instead of adding models where it is available, the second algorithm adds models where the current function is the least likely. Here are the steps for this algorithm:

1. Start with one model (every point is labeled to this model).
2. Compute the likelihood of each point.
3. Get the n points for which the neighborhood is the least likely
4. Get one point from this set.
5. Create a new model and label the point and its neighborhood to it.
6. Update all plans.
7. Update all labels (label a point to the most likely model).
8. If one model has not enough points (twice the dimension of the reduced space), the model is deleted.
9. Go to step 6 until the labels are stable.
10. Go to step 2 if a criterion is not met (usually, I choose the Aike information criterion).

I’ve added later an additional step that asserts that the points assigned to a model are connected (that is the subgraph is connex). If it is not, the model is split in several parts, one for each connected component.

I didn’t state it before, but the error can be modeled by different random variables. I’ve chosen an isotropic Gaussian variable in every training.

There an immediate improvement of the algorithm, as it can be seen in the following figures:

Here, after the introduction of the second model, a third was introduced. Indeed, the second model took the middle part of the reduced space thus splitting the graph with the points of the first model in two components, thus a new model was added.

At the end of the global optimization, the result is the following one:

The differences between the two algorithms are obvious in the quality of the reconstruction. The first one had a 20% reconstruction error, instead of 3% for the second one. Although it is more complicated, it is more capable of optimizing the problem of finding linear models that will minimize the reconstruction error.

Coming next

After this training, a complete manifold model is available, with the precision one need. I’ll present how it can be used in the following posts. Stay tuned !

Content and opinions

After a small introduction chapter (installation, the TBB philosophy of tasks instead of threads), the second chapter exposes what parallelism is. In every C++ book (that I know of), the programmer is taught how to think one instruction after another. The trouble with parallel thinking is that you are executing several things together that can interact, and this is not something the human mind handles correctly. So every useful concept is presented, locks, scaling, …

The next chapter is perhaps the most useful one, as it presents basic parallel loops that you will use almost everywhere if you have data parallelism. The concept of range is introduced, and it’s the basic tool for TBB. The problem is split thanks to these ranges and when the ranges are small enough, tasks are created for each of these small ranges. The drawback of this chapter is the code quality (I hate when the code uses a mix of C/C++ when the library is 100% C++, that is, don’t use the climits header and a macro when there is the limits header!) and the lack of some explanations of the expected speedup of the algorithms. For instance, parallel_reduce is different than the parallel reduction proposed as a sample by CUDA. Then, parallel_scan (it computes y[n] = f[n](y[n-1])) is not clear either. The speedup is achieved by computing several time the same values, but in one case, the result is not saved in the memory. So what you can gain is the fact that for the last part of the computation, only the cost of computing the results for the first parts matters. So if your function is complicated, your speedup is zero. A small explanation for this would have been great.

Fortunately, this chapter was the only one that had such drawbacks. Unfortunately, these drawbacks occur in the most import part of the book. After these basic blocks, task parallelism is introduced. Contrary to data parallelism for which you split your dataset into chunks that can be computed on a lot of cores, task parallelism only splits the work into a fixed number of predefined tasks. This means that the work cannot scale as much as for data parallelism, but the task themselves can be data parallel. This is done with a pipeline. Other algorithms for other kinds of loops are exposed and complete the toolbox for basic/usual processing.

Some algorithms need to update some variables in containers (like a sort algorithm), and as the STL containers are not thread-safe, specific containers must be used. TBB proposes a queue, a vector and an hash map for those purposes. The operations on these containers are not the same as the STL operations. New thread-safe methods are available with some examples.

The following chapter is about memory allocation. TBB exposes its own functions so that there cannot be cache conflicts between two threads. Basic examples show how to use them in overloaded new and delete instructions.

Then, the books gets one step further inside TBB with mutual exclusion. The underlying OS mutex is wrapped, but additional specific mutexes are available. Their specifications is well explained. A small chapter is dedicated to timing in a safe way. It is really useful for task timing (before using Intel tools for thread profiling).

The last important chapter is about the task scheduler. This is the basis of the whole library, and using it directly can generally be avoided, but if you don’t have a choice, TBB allows its direct use. The different choices Intel made when designing it are clearly stated, although some paragraphs are not that easy to understand. This is because the task scheduler is very modular and a lot of things can be done. This chapter talks about every possibility, but don’t forget to state precisely what you need so that your application is as simple as possible.

The next to last chapter echoes the second one. It sums up the different things you have to remember when write parallel algorithms with TBB. Although some points are pragmatic and very basic, they are all sound.

Finally, the last chapter exposes several examples with increasing difficulty (following the book actually). Additional examples are available with the library but these ones are explained and decrypted.

Conclusion

I had trouble at first with this book, because of the mix between C and C++ in the second chapter and then some trouble understanding how parallel_scan could even been sped up. But when I went on reading, I enjoyed it. Sometimes I would have liked additional example inside each chapter, instead of writing the interface of a class and explaining each function. In fact, this book, is a reference manual for TBB, and a good one. My troubles with the code was only at the beginning. At the end, everything is OK.

So if you want to use a more usable C++ thread library, go, get TBB (Open Source) and this book.

Intel Threading Building Blocks: Outfitting C++ for Multi-core Processor Parallelism (Paperback)
by James Reinders
ISBN: 0596514808

Price: USD 31.49
37 used & new available from USD 15.32

| 3.5 | 3

I’ve written a small tutorial. It is not an usual tutorial (there is a user tutorial and then what developers should know to enhance it), and some results of the techniques are exposed in my blog. It provides the basic commands to start using the scikit yourself (reducing some data, projecting new points, …) as well as the expoed interface to enhance the scikit.

If you have any question, feel free to ask me, I will add the answers to the tutorial page so that everyone can benefit from it.

Be free to contribute new techniques and additional tools as well, I cannot write them all ! For instance, the scikit lacks some robust neighbors selection to avoid short-cuts in the manifold…

Tutorial and the learn scikit mainpage.

Content and opinions

The book is split in three parts. The first defines peer-to-peer, the second presents the then-usual applications of peer-to-peer and the third exposes the different issues that looms large with this technology (well, it’s more a group of technologies).

So the first part is about what peer-to-peer really is. It is really interesting to follow the history of Internet and see that it is so closely related to peer-to-peer. Then, the reason of the emergence of these technologies is presented, though ICQ and Napster. Perhaps not the best peer-to-peer applications (in the technological sense), but they were one of the first, so… The authors do not underestimate the legal/economic war that was and is still waged, but they ask the real questions about this war. Then, Tim O’Reilly proposes a new taxonomy of peer-to-peer, how to make it popular technologies for everyone, (music/cinematographic) industries included. The fourth chapter was not that interesting (very short), but showed some additional reasons to the success of everything that is peer-to-peer.

The second part is about projects. The first is SETI@Home, a distributed computer. It shows how it started and became the project we know, with the different problems it had to tackle. Then the next application is Jabber, a conversation tool. Based on XML, it can talk to other clients (MSN, ICQ, …), but is not only geared to people conversations, but also application conversations, which is somewhat different. Mixmaster is next. I didn’t know that some people used remailers to ensure anonymity (mails are sent through several mail servers). Besides, it seems that only few remailers are left. Then Gnutella and Free net are presented. Those two applications are perhaps the most known ones, as their goal is file-sharing. The underlying technology and approaches are very different, and thus it is interesting to read those two chapters. Red Rover, Publius and Free Haven are dedicated to avoid the censorship of some files. This can be very interesting for people located in countries where Internet is censored. I have to say that those applications were the least interesting for me, but they may interest others.

The third part is dedicated to some theoretical thoughts. Or at least that was what the title said. The first chapter, Metadata, says that files should have metadata. Yes. That is kind of logic. Then, there are some interesting thoughts about applications performance. The four next chapters are geared toward trust, anonymity, reputation or accounatbility. All in all, they talk about the same topics, and it gets boring in the end. Finally, the last chapter presents how all those applications may talk together.

Conclusion

I thought that the book would teach me a lot about peer-to-peer technology. In fact, I learned about the history of the Internet, some issues about performance, but a lot on file sharing. If the editor wanted to show that we can do much more than file-sharing with peer-to-peer, it didn’t show up in the actual book. Yes, accountability is important, as security, but speaking about how to remain anonymous when there is a illegal-peer-to-peer witch hunt (and it was already the case when the book was written) is not going to help the reputation of the technologies.
I would like to have an updated version of the book speaking about the current peer-to-peer applications : Napster is dead, Gnutella is not widely used, … Finally, I would have liked to have more applications of the peer-to-peer technologies for the industry (distributed computing, web services, …). Seti@Home is a start, but only a start, it cannot be used by firms for their own computations.

Peer-to-Peer : Harnessing the Power of Disruptive Technologies (Hardcover)
by Andy Oram
ISBN: 059600110X

Price: USD 21.86
92 used & new available from USD 0.32

| 4.5 | 21

I will publish additional figures here in a short time. The scikit is almost completed as well, I’m finishing the online tutorial for those who are interested in using it and/or enhancing it.

If some of you are interested in my French tutorials, I may translate them from time to time, depending on their content (I don’t want to translate an article on Boost for instance, the documentation does provide everything). But I’ll do that only if people tell me “Go on”. So I’m all ears…

Before

My current job is to find a model for datasets.

A lot of datasets can be explained by a small number of parameters. For instance identity photos of a single person can be explained by 3 translations and 3 rotations. So my algorithms did that: find the parameters (or something that is close enough) and create a mapping between the parameters and the original space.

During this research, I learnt what is scientific computing. I did not explore everything in this field, but I covered the basics. That’s where I found about Python, but also C++ (which is the first language I really used). My thirst for information lead me to read a lot of books on several matters (architectural design, process, but also parallel computing and its different flavors). This led me to search for a job that would interest me the most.

After

So starting from September I’ll move to Pau, a town in the South of France. This is where the biggest research center of Total S.A. is located. I will work on oil exploration.

Although the theory behind this are well known (acoustic wave propagation and inverse problem), this does not mean that research in this field is over. For instance, the power needed for solving these problems are enormous. So their implementation must be well thought. And even if you managed to find a solution to your problem, you are not done. Total’s goal is not to be able to see if acoustic waves propagate fast in some places and slowly in others. Its goal is to find oil and gas. So now that one has an acoustic model, one must see with the geologists if there are some odds that there is oil or gas. And that’s also a big interesting challenge.

Meanwhile

For those who were interested in manifold learning, don’t worry, I’m not finished in exposing my research. I will go on with some new posts about the mapping between the two spaces and how it can be used to test new samples. The scikit is now almost available. I still have to finish the tutorial and test if everything is OK.

I hope I will be able to continue with other subjects on this blog, there is no reason I cannot do this. Although what I’ll be doing at Total is secret, there are a lot of fields I’d like to talk about.

Content and opinions

The book is split in four parts. The first chapter is an introduction to distributed systems, with the definitions and some examples of what is and what is not a centralized (by opposition to decentralized) application or framework. For people novice to P2P, the examples and the problems are well presented.

The first part goes into details of a distributed environment. The P2P solution is thus exposed, with its specific aspects, social (P2P has not a good reputation, to say the least) or routing (accessing peer behind firewalls, routers). As web services are one of the main subjects of the book, they are presented next. The last chapter tackles grid computing through its evolution, the current definition of a grid and the Globus Toolkit 2 architecture. Those chapters are really interesting because they lay down the ground for the remaining of the book.

The second part is about several P2P technologies that can be used, as well as some specific issues. Jini and Gnutella are the first ones to be exposed. They were not developed to answer to the same questions: Jini is about Remote Objects and Gnutella is about file sharing. These technologies introduce the issues of scalability and security ; the first tackles the use of more nodes in the grid, the second the protection of the grid. Finally, Freenet and JXTA are exposed. The first is dedicated to fiel storage on a distributed data grid, the second is a generic P2P framework. The chapters on the different technologies do not bring more information than what can be found on tutorials on the net, but they explain them in a clear way. Scalability and security are aspects that are sometimes forgotten in the design of a distributed system, so their presence in this part of the book helps remembering them.

Part three tackles Jini, JXTA and web service deployement. The first two chapters have some code samples that can be used ; for web service, there is only some XML fragments. Several formats are exposed in this chapter with their advantages and drawbacks.

Finally, the fourth chapter presents web services applied to grid systems. This gives grid services. The Globus toolkit 3 can be used for those grid services, and the future version 4 is introduced as well. This part is the shortest, maybe because these special services are not widely used, and a lot should still be explored to have a clear software designed (which may be used by the Globus toolkit 4, according to the book).

Conclusion

The good writting style of the book helps reading the book, as some pages can be difficult to understand. The final goal is to present grid services, with the underlying frameworks and tools that are grid (and P2P) systems and web services. The beginner is taken from the basics to advanced concepts, which can be applied to concrete grids.

If grid computation, how they can be done, and web services are of interest to you, I suggest you read this book.

From P2P to Web Services and Grids: Peers in a Client/Server World (Paperback)
by Ian J. Taylor, Andrew Harrison
ISBN: 1852338695

Price: USD 55.60
22 used & new available from USD 31.50

| 4.5 | 4

In fact, the registry is one of the best replacement for the switch statement. Indeed, it is far more modular as new cases can be introduced and deleted, and it is more readable as well.

I used a registry in several pieces of code:

• when I’m playing with manifold learning, I have a registry of the dimensionality reduction tools and I can choose the one that fits my needs (given as a commandline argument);
• I refactored scikits.openopt so that a registry contains the different solver wrappers, and now people can add their own wrapper in the dictionary, the key being the name of the solver and the value is a string with the package hierarchy to use;
• I’m trying to develop a JXTA implementation in Python, and the registry shows up everywhere, or almost. For instance, as in the original, Java, implementation, the advertisements are stored in a dictionary, and new kind of advertisements can be registered efficiently. The parsing of each advertisement is also done with a class registry.

Python, with is dynamic and duck typing, is naturally inclined to use registers, IMHO. This is more scientific-oriented coding, but the ease of use of a registry is very helpful in my everyday work.

Here is a sample of the automatic use in pyP2P:

from advertisement_core import *
from peer_advertisement import *

class Advertisement(object):
  pass
 
registry = {}

import advertisement_core
 
class PeerAdvertisement(advertisement_core.Advertisement):
  pass
 
advertisement_core.registry["PA"] = PeerAdvertisement

This way, when the module is imported, the registry is also automatically populated.

Content and opinions

Twisted is a network library aiming at simplifying the developement of network applications. It is based on an event loop for all processing (unfortunately, no word in the book about managing several event loops, as it is the case with GUI-based applications).

After an introduction of what is event programming with simple clients and servers, the reader will be introduced to basic web clients and servers. Twisted proposes a lot of bridges to create webservices with XMLRPC or SOAP. The explanations and the code is pretty clear, and it is easy to do one’s own small distributed application with these blocks.

When authentification is introduced, it is hard to understand at first. Zope interfaces are used, but I didn’t find the explanation of what they are and what the function implements() is and does. One can find out with the context, but a complete introduction of these techniques should be done at this point. Once authentification is understood, other services are exposed, like mail clients and servers (how to send a mail, process the information in the mail to send an answer), as well as NNTP.

SSH is only introduced towards the end of the book. And it is not simply explained as it is mixed with shells. Finally, network applications often are services or dameons, how to create them is done in the last chapter.

Conclusion

This book is good, a lot of explanations and of code (some mistakes can be found here and there) helps understanding the use of the library. Some parts of the book are outdated, so I hope that a new edition will be published soon, and some software tools should be more explained. Every aspect of Twisted is not developed in the book, it’s only Networking Programming Essentials, but once the basics are known, the rest can be learnt with the documentation.

Twisted Network Programming Essentials (Paperback)
by Abe Fettig
ISBN: 0596100329

Price: USD 19.77
46 used & new available from USD 14.44

| 3.5 | 9