I’ve started a branch available on github.com, and I will some examples in the scikit as well. I may explain them here, but I won’t rewrite what is already published. A future post will explain the changes, and I hope that interested people will understand the modifications and apply them to my former posts. It’s just that I don’t have much time to change everything…

Projection

Finding the projection of a point on the manifold can be done by searching in the reduced space the point that will minimize some criterion. As the dimension of this space is small, this can be efficiently done.

Another way is to use the multidimensional regression. It consists of several linear models, so one can project the new sample on each model and choose the best one. This can lead to somewhat different projections than the search in the reduced space. Indeed, a projection on one model can in fact be reconstructed by another model. We did not do this, because we’ve added another way of ensuring that we are coosing a good projection. The reason is that to correctly project on each model, you have to track on which model you currently are, which means optimizing a function that is not smooth, which is more difficult and thus longer.

So we are using two cost functions, one is simply the Maximum Likelihood, so for a Gaussian random variable it is an orthogonal projection. But as figure 1 tells us, sometime, the projection is not on the manifold. So to enhance this result, we use a Maximum A Posteriori function, which adds a regularization term. It simply is a Gaussian Mixture in the reduced space, one Gaussian for each learning point labelled to the considered model. This way, the projection will be attracted to the subspace where the model is used to reconstruct points.

Figure 1: Different kind of projections

I’ve projected new samples of the SwissRoll and the SCurve with both methods. The SwissRoll shows that projecting points on models without a regularization feature or without tracking the correct model during the projection does not give valid results in the reduced space. In the original space, the points are correctly reconstructed. With the SCurve, the results are better.

SwissRoll: Projected coordinates with ML	SwissRoll: Projection with ML
SwissRoll: Projected samples with MAP	SwissRoll: Projection with MAP
SCurve: Projected samples with ML	SCurve: Projection with ML
SCurve: Projected samples with MAP	SCurve: Projection with MAP

As an other benchmark, I’ve taken the 20 datasets from the COIL-20 database. I’ve added an occlusion to each image, and I’ve projected the images with and without occlusion on the corresponding manifold (I’ve used a Laplacian random variable to correctly describe the noise made by the occlusion). I’ve compared the results to a robust projection on the 2 or 15 first eigenvectors of the datasets. First, without occlusions, the projection on 15 eigenvectors manages to reproduce the images, as does my method. Of course, only 2 eigenvectors is not enough to describe the dataset correctly (although I’m also using only 2 coordinates to describe them). With 40% of occlusion, my method gets on a par with the 15 eigenvectors, and even better if we consider only the reconstruction error. Indeed, my method yields better images than the 15 eigenvectors.

COIL-20 dataset 1: 40% occlusion
Projection on 2 eigenvectors
Projection on 15 eigenvectors
Exact pose estimation (0% occlusion)	Approximate pose estimation (0% occlusion)
Exact pose estimation (40% occlusion)	Approximate pose estimation (40% occlusion)

Classification

Now, if we use the 20 datasets as 20 manifolds and if we project all images on all manifolds and select the best projection for each image, we have a new of doing a classification. This leads to the following graphic representation of the confusion matrix (hotter colors indicate higher percentage).

Confusion matrix for the whole COIL-20 database (0% occlusion)

Confusion matrix for the whole COIL-20 database (10% occlusion)

Without occlusion, the confusion matrix is pretty good, but with even 10% occlusion, some test class samples are misclassified for an other class. This is because those classes are darker than the actual test sample class, and with an occlusion, these classes fit better the occlusions.

Now, for the last dataset, I’ve taken samples from the Outex database. Each texture image is cut is 16 samples, then one half of the samples is used as training, the other half as the test samples. Then, the sample are transformed in cooccurrence matrices.

One texture sample

Multichannel cooccurrence matrices

There 72 textures, thus 72 classes. Here are the resulting confusion matrices for the training and test samples.

Confusion matrix for the training samples

Confusion matrix for the test samples

In this case, the results are better than the litterature (Generalization of the cooccurrence matrix for colour images : Application to colour texture classification, Image Analysis & Stereology, 2004).

The end

This is the last post I’ll be doing on manifold learning. It is very long because I wanted to write my last results (some can’t be found in the litterature), and I didn’t feel writing two or three posts. I’m not researching mainfold learning anymore, so it needed to be finished clearly.

I hope you enjoyed the different posts in this category. There is still much to do in the field, but one cannot do everything…

Here is the evolution of the mapping with the first algorithm. Only a part of the manifold is regressed each time, and there is no way of tuning the result.

The following video was made with the second algorithm. Each time, the whole manifold is regressed and the model enhanced. I didn’t put the whole optimization, at the end nothing could be seen on such a video.
After the first iteration, the original model is split in two other models, and this happens again after the second iteration.

I’ll make other videos on the other parts of my work.

• you can interpolate the data in the original space based on an interpolation in the reduced space, or
• you create an approximation of the mapping with a multidimensional function (B-splines, …)

When using the first solution, if you map one of the reduced point used for the training, you get the original point. With the second solution, you get a close point. If the data set you have is noisy you should use the second solution, not the first. And if you are trying to compress data (lossly compression), you can not use the first one, as you need the original points to get new interpolated points, so you are not compressing your data set.

The solution I propose is based on approximation with a set of piecewise linear models (each model being a mapping between a subspace of the reduced space to the original space). At the boundaries between the models, I do not assert continuity, contrary to hinging hyperplanes. Contrary to Projection Pursuit Regression and hinging hyperplane, my mapping is between the two spaces, and not from the reduced space to one coordinate in the original space. This will enable projection on the manifold (which is another subject that will be discussed in another post).

Position of the problem

In the literature, several papers were published in order to create a piecewise linear function. Their main advantage is that they can compute the reduced space when computing the mapping. Their issue is that the number of models is fixed at the beginning of the optimization. What I propose now is an adaptive number of models, depending on the manifold.

Each point has a set of neighbors, the k nearest ones are used here.

Basic approach

In fact, the basic algorithm is straightforward: put a new model where you can, optimize all models, label the points you can label, loop and stop at some point.

The first algorithm I used is the following:

1. Start with no model.
2. Find a point whose neighborhood is not labeled, if you can’t, stop.
3. Create a new model there and label the points accordingly.
4. Estimate all models with regards to the labels.
5. Label each point to the nearest model if the point is too far (that is further than a factor times the mean error), do not label it.
6. If one model has not enough points (twice the dimension of the reduced space), the model is deleted.
7. Go back to step 2.

This algorithm is very simple, but cannot model precisely a manifold, you cannot tune it. But it can give a good first impression on the manifold.

Here are some steps of the algorithm for a SwissRoll (the left figure indicates the label in the reduced space and the left figure is the approximation of the manifold in the original space):

At the end of the iteration, the manifold is roughly approximated:

Maximize a likelihood

As I’ve said, there are no ways for the first method to get a given precision. So instead of adding models where it is available, the second algorithm adds models where the current function is the least likely. Here are the steps for this algorithm:

1. Start with one model (every point is labeled to this model).
2. Compute the likelihood of each point.
3. Get the n points for which the neighborhood is the least likely
4. Get one point from this set.
5. Create a new model and label the point and its neighborhood to it.
6. Update all plans.
7. Update all labels (label a point to the most likely model).
8. If one model has not enough points (twice the dimension of the reduced space), the model is deleted.
9. Go to step 6 until the labels are stable.
10. Go to step 2 if a criterion is not met (usually, I choose the Aike information criterion).

I’ve added later an additional step that asserts that the points assigned to a model are connected (that is the subgraph is connex). If it is not, the model is split in several parts, one for each connected component.

I didn’t state it before, but the error can be modeled by different random variables. I’ve chosen an isotropic Gaussian variable in every training.

There an immediate improvement of the algorithm, as it can be seen in the following figures:

Here, after the introduction of the second model, a third was introduced. Indeed, the second model took the middle part of the reduced space thus splitting the graph with the points of the first model in two components, thus a new model was added.

At the end of the global optimization, the result is the following one:

The differences between the two algorithms are obvious in the quality of the reconstruction. The first one had a 20% reconstruction error, instead of 3% for the second one. Although it is more complicated, it is more capable of optimizing the problem of finding linear models that will minimize the reconstruction error.

Coming next

After this training, a complete manifold model is available, with the precision one need. I’ll present how it can be used in the following posts. Stay tuned !

I’ve written a small tutorial. It is not an usual tutorial (there is a user tutorial and then what developers should know to enhance it), and some results of the techniques are exposed in my blog. It provides the basic commands to start using the scikit yourself (reducing some data, projecting new points, …) as well as the expoed interface to enhance the scikit.

If you have any question, feel free to ask me, I will add the answers to the tutorial page so that everyone can benefit from it.

Be free to contribute new techniques and additional tools as well, I cannot write them all ! For instance, the scikit lacks some robust neighbors selection to avoid short-cuts in the manifold…

Tutorial and the learn scikit mainpage.

I will publish additional figures here in a short time. The scikit is almost completed as well, I’m finishing the online tutorial for those who are interested in using it and/or enhancing it.

First of all, my test is about respecting the geodesic distances in the reduced space. This is not possible for some manifolds like a Gaussian 2D plot. I used the SCurve to create the test, as the speed on the curve is unitary and thus the distances in the coordinate space (the one I used to create the SCurve) are the same as the geodesic ones on the manifold. My test measures the matrix (Froebenius) norm between the original coordinates and the computed one up to an affine transform of the latter.

I tested several noise levels :

• no noise
• 5% of Gaussian noise
• 2% of Laplacian noise (only 2% because there are many outliers in the Laplacian law)
• Impulsive noise on 12.5% elements of the distance matrix with 200% Laplacian noise (quantified with respect to the variance of the noise-free d1-d2, where d2 was estimated with my robust cost function)

Here are the results:

The geodesic-based algorithms perfom obviously and logically better than every other algorithms. In my case, I want this to happen as I want to estimate a mapping function between the reduced space and the original space. This estimation and the effect of the reduction algorithm on it will be the subjects of future tickets.

It is worth mentioning that I only implemented Laplacian Eigenmaps with a sparse matrix, due to the lack of generalized eigensolver for sparse matrix, but it will be available in a short time, I hope.

The Laplacian Eigenmaps are the most known technique using the similarity graph (safe for LLE, which is nothing more than a special case of the Laplacian Eigenmaps). The similarities are computed between neighboors (neighboors meaning the samples that are near one from another in a distance way or samples that are close, like pixels in an image), generally with a Gaussian kernel. The trick here is to choose the correct width of the kernel. Then, the similarities matrix is weighted (each column and line must sum to one, this is the Laplacian of the graph) and then eigenvectors are extracted from it. The first eigenvalue is one and must not be used.

Here is what I get :

One may wonder why the reduction is so poor, but I’m not the only one to get this result. I tried every width for the kernel to no avail. The literature says that Laplacian Eigenmaps tendto cluster points, which is easily explained by the algorithm. The eigenproblem extracts the main eigenvectors so that the weighted similarities matrix is preserved (in a quadratic way). This means that even if points should be close, if they are not close enough, they have a similarity of 0 so the eigenproblem will separate them.

Diffusion maps are another similarity graph technique. Although there is a Markovian/probabilistic interpretation, diffusion maps are basically Laplacian Eigenmaps with similarities computed between every pair of points. This means that they have the same drawbacks that Laplacian Eigenmaps except for the clustering. The width of the kernel is still difficult to estimate.

Here is the result :

The fact that every similarity is used explains the fact that diffusion maps cannot reduce the SwissRoll correctly. In this precise case, the kernel width was obviously too big, but smaller width gives a result similar to the Laplacian Eigenmaps, which is not correct either.

The other technique I will present is Hessian Eigenmaps. Instead of estimating the Laplacian of the similarities graph, it tries to estimate the Hessian. This gives very good result for the SwissRoll :

Unfortunately, the technique is not robust to noise, as I will show you in the result ticket. Safe for this fact, the technique is robust to holes in the manifold (not uniformly sampled manifolds for instance), which is one of the biggest drawback in techniques based on the geodesic distances.

Stay tuned.

Some cost functions have been proposed, also known as stress functions as they measure the difference between the estimated geodesic distance and the computed Euclidien distance in the “feature” space. Every metric MDS can be used as stress functions, here are some of them.

The oldest function is Sammon’s NonLinear Mapping. Originally based on Euclidien distances, I implemented it with the approximated geodesic distances described in the Isomap ticket. The goal of this function is to add a weight (the inverse of the geodesic distance), leading to less weight for the greatest distances, but also an important weight for small distances.

Here is the cost function for the distances (y are the coordinates in the original space and x in the feature/reduced space) :

Optimizing this function with a conjugate-gradient descent from a random start can give this result :

Another function that is present and cited in the litterature is Desmartines’ one from the Curvilinear Component Analysis :

The F() function is 1 when the argument is small (less than an arbitrary value lambda), and else 0.

As a consequence, this function is not convex, not even continuous. The algorithm proposed in the associated paper is not great, I never managed to make it work on a SwissRoll, even with few points. So here are the step I use to optimize the function :

• Start from a random position
• Optimize 10 points (lambda infinite)
• Optimize one more point :
  • Move only this point with lambda infinite (gradient descent)
  • Move every point with a decreasing lambda (I start with lambda = greatest geodesic distance in the data set and linearly decrease it until lambda is the limit of 5% of he smallest geodesic distances)

Each time, the new point is moved according to every already placed point, then when every point is moving, only the local stresses are used. But the optimization can still go wrong and some points that should be close can end far one from another, because their associated stress is zero.

Here is the result for this optimization :

The cost function I use is a robust one, not “recursive” as Desmartines qualifies it (the weight is not a function of the estimated distance as it is the case from the CCA cost function) :

The first term is the robust term, derivable when the (geodesic estimated and Euclidien computed) distances are equals, the second term allows for a fast convergence when the distances are not correctly estimated (useful at the beginning of the optimization, less afterwards) and the last term gives a small weight for small distances, as they can be polluted by noise for noisy manifolds. Gamma should only be a small value, Tau is set to be equal to 80% of the geodesic distances and sigma to 5%. This gives good results in every case.

Here is the result with this cost function :

Its optimization is not easy as it can give folded reduced space as an answer. I proposed two algorithms to solve the issue :

The first one :

• Optimize every point
  • Add some noise to the computed coordinates depending on the global cost and the iteration (at the end, the added noise must be very small)
  • Optimize with a simple gradient descent and a Fibonacci line search

The second one :

• Optimize 10 points with a gradient descent from a random start
• Optimize with one new point
  • Move only this point with a gradient descent
  • Move every points

The second algorithm is slower than the first, but it works every time.

Stay tuned for the results…

I’m in the process of refactoring it so that it depends less on some of our libraries here. In two weeks, there is a nipy sprint in Paris I will attend, and machine learning is one of the topic we will discuss, so this may indicate where and how I’ll contribute the code I will keep going on showing some results next week.