As Blei, the author of LDA, points out, the topic proportions in pLSA are tied with the training documents. This is problematic: 1) the number of parameters grows linearly with the number of training documents, which can cause serious overfitting 2) it is difficult to generalize to new documents and requires so-called “folding-in”. LDA fixes those issues by being a fully generative model: where pLSA uses a matrix of P(topic|document) probabilities, LDA uses a distribution over topics.

To date, there exists several parameter estimation schemes for LDA: variational Bayes, expectation propagation and Gibbs sampling. I’ve chosen to implement the latter. It has first been described in a paper entitled “Finding scientific topics”, by Griffiths and Steyvers.

Artificial data

As with all model-based algorithms, during the early development phase, it is useful to work with artificial data, generated by following the model assumptions. In the case of LDA (and pLSA), the core assumption is that words (w) in documents are generated by mixture of topics (z). In other words, the probability of a word is:

The generative process can be summarized as follows: 1) set the topic proportions once for all when the collection is instantiated and 2) for each document and for as many words as needed, sample a topic from the topic distribution and sample a word from the word distribution of the selected topic. Obviously, this is only an approximation of how documents are created in reality.

To generate an artificial dataset, we can fix the word distribution of each topic and then generate documents as explained above. Since we generated documents by sticking to the generative assumption of the model, if the algorithm is correctly implemented, it should be able to recover the word distribution of each topic, from the generated documents.

Graphical example

To gain insight and intuition, we can reuse the graphical example from Griffiths and Steyvers’ paper.

In the bag-of-words model, documents are represented by vectors of dimension , where is the vocabulary size. Moreover, an image of size has pixels: it can thus be stored as a string/vector of length/size . This means that a document in the bag-of-words model can be represented as an image, where pixels correspond to words and pixel intensities correspond to word counts!

As put previously, we first need to fix the word distribution of each topic. Let’s arbitrarily create 10 topics.

5 with “vertical” bars:

and another 5 with “horizontal” bars:

Each topic distribution is represented by a 5×5 image, so the vocabulary is of size 25. Black pixels correspond to words that the topic will never possibly generate. White pixels correspond to words that the topic can generate with probability 1/5.

Now let’s generate 500 documents using the generative process previously described. Here are 3 examples of such generated documents.

We clearly see bars emerging from the documents and can thus confirm that documents are mixtures of topics.

We can now use the generated documents as training data. If the Gibbs sampler is correctly implemented, we should be able to recover the original topics. Here are the results for the 1st, 6th and 26th iterations. The number between brackets is the log-likelihood.

1st iteration (-278541.7835):

5th iteration (-165139.56193):

[...]

26th iteration (-129272.328181):

After a few iterations, we see that the algorithm recovered the topics correctly. Also, the log-likelihood increases: as the number of iterations increases, it becomes more and more likely that the model generated the data. The fact that it works pretty well is not surprising: the data used were generated by sticking to the model assumptions.

Gibbs sampling

The Gibbs sampler used is said to be collapsed: the parameters of interest are not sampled directly. Instead we sample the topic assignments and the parameters can be computed in terms of those.

It is not necessarily obvious from the equation of the full conditional distribution (from which the topic assignments are sampled) but the sampler is naturally sparse: it doesn’t need to iterate over words with zero-count. This is a nice property, given that sampling algorithms are often considered slow.

Source code

http://gist.github.com/542786

Fairly readable and compact code but to be considered a toy implementation.

Useful Resources

MCMC

- “MCMC lecture at MLSS09” (Iain Murray). Nice for a first general overview and the insights.

- “Gibbs sampling for the uninitiated” (Resnik and Hardisty). Nice for a first general overview and the insights.

- “Pattern Recognition and Machine Learning” (Bishop), Chapters 8 and 11 on graphical models and sampling methods. Excellent chapters.

- “Review Course: Markov Chains and Monte Carlo Methods” (Cosma and Evers). Very nice free online course and solutions to exercises in Python and R!

LDA

- “Latent Dirichlet Allocation” (Blei et al, 2003). By Blei himself.

- “Finding scientific topics” (Griffiths and Steyvers). Insightful comments and nice intuitive graphical example.

- “Parameter Estimation for text analysis” (Heinrich). Very nice introduction to Bayesian thinking. Pseudo-code for the LDA Gibbs sampler.

- “On an equivalence between PLSI and LDA” (Girolami and Kaban). Connections between pLSA and LDA.

- “Integrating Out Multinomial Parameters in Latent Dirichlet Allocation and Naive Bayes for Collapsed Gibbs Sampling” (Carpenter). Very detailed, step-by-step derivation of the collapsed Gibbs samplers for LDA and NB.

- “Distributed Gibbs Sampling of Latent Dirichlet Allocation: The Gritty Details” (Wang). Insightful comments and pseudo-code of the LDA Gibbs sampler.

Other Python implementations

- nrolland’s pyLDA. Works fine but mixes Python-style and Numpy-style.

- alextp’s pylda. Numpy-style but not tested.

The Expectation-Maximization (EM) algorithm is a popular algorithm in statistics and machine learning to estimate the parameters of a model that depends on latent variables. (A latent variable is a variable that is not expressed in the dataset and thus that you can’t directly count. For example, in pLSA, the document topics z are latent variables.) EM is very intuitive. It works by pretending that we know what we’re looking for: the model parameters. First, we make an initial guess, which can be either random or “our best bet”. Then, in the E-step, we use our current model parameters to estimate some “measures”, the ones we would have used to compute the parameters, had they been available to us. In the M-step, we use these measures to compute the model parameters. The beauty of EM is that by iteratively repeating these two steps, the algorithm will provably converge to a local maximum for the likelihood that the model generated the data.

Naive Bayes trained with EM

In their paper “Semi-supervised Text Classification Using EM”, Nigam et al. describe how to use EM to train a Naive Bayes classifier in a semi-supervised fashion, that is with both labeled and unlabeled data. The algorithm is very intuitive:

• Train a classifier with your labeled data
• While the model likelihood increases:
  • E-step: Use your current classifier to find P(c|x) for all classes c and all unlabeled examples x. These can be thought as probabilistic/fractional labels.
  • M-step: Train your classifier with the union of your labeled and probabilistically-labeled data.

The hope is that using (abundantly available) unlabeled data, in addition to (labor-intensive) labeled data, improves the quality of the classifier.

Code

I made a simple implementation of it in Python + Numpy. The code is fairly optimized.

$ git clone http://www.mblondel.org/code/seminb.git

web interface

Implementation details

Here are implementation details that were not mentioned in the original paper and that I found necessary to get a correct implementation.

Naive Bayes is called naive because of the (obviously wrong) assumption that words are conditionally independent given the class:

However, since the vocabulary size V can be pretty big and the probabilities P(w|c) can be pretty small, P(x|c) can quickly exceed the precision of the computer and become zero. The solution is to perform the computations in the log domain:

To turn around P(x|c), we use Bayes’rule:

By posing , we get:

This is the softmax function. However, we are back to our initial problem because, since the are likely to tend to -inf, the exponentials are likely to in turn underflow. The trick is to multiply the numerator and denominator by the same constant :

Setting m to , the values will get closer to zero. The rationale for this is that computation of the exponential overflows earlier and is less precise for big values (positive or negative) than for small values.

This trick will improve the situation quite a lot but in case this is not enough:

This sets the exponentials to zero when . For t=-10, this is 0.000045. Equivalently this corresponds to setting the exponentials to zero when . Since both t and and m are negative, this shows that subtracting the maximum m, as explained before, does help improving the precision.

Reference

Kamal Nigam, Andrew McCallum and Tom Mitchell. Semi-supervised Text Classification Using EM. In Chapelle, O., Zien, A., and Scholkopf, B. (Eds.) Semi-Supervised Learning. MIT Press: Boston. 2006.

Note: LSA and pLSA are also known in the Information Retrieval community as LSI and pLSI, where I stands for Indexing.

Comparison

1. LSA stems from Linear Algebra as it is nothing more than a Singular Value Decomposition. On the other hand, pLSA has a strong probabilistic grounding (latent variable models).

2. SVD is a least squares method (it finds a low-rank matrix approximation that minimizes the Frobenius norm of the difference with the original matrix). Moreover, as it is well known in Machine Learning, the least squares solution corresponds to the Maximum Likelihood solution when experimental errors are gaussian. Therefore, LSA makes an implicit assumption of gaussian noise on the term counts. On the other hand, the objective function maximized in pLSA is the likelihood function of multinomial sampling.

The values in the concept-term matrix found by LSA are not normalized and may even contain negative values. On the other hand, values found by pLSA are probabilities which means they are interpretable and can be combined with other models.

Note: SVD is equivalent to PCA (Principal Component Analysis) when the data is centered (has zero-mean).

3. Both LSA and pLSA can handle synonymy but LSA cannot handle polysemy, as words are defined by a unique point in a space.

4. LSA and pLSA analyze a corpus of documents in order to find a new low-dimensional representation of it. In order to be comparable, new documents that were not originally in the corpus must be projected in the lower-dimensional space too. This is called “folding-in”. Clearly, new documents folded-in don’t contribute to learning the factored representation so it is necessary to rebuild the model using all the documents from time to time.

In LSA, folding-in is as easy as a matrix-vector product. In pLSA, this requires several iterations of the EM algorithm.

Implementation in Python

LSA is straightforward to implement as it is nothing more than a SVD and Numpy’s Linear Algebra module has a function “svd” already. This function has an argument full_matrices which when set to False greatly reduces the time required. This argument doesn’t mean that the SVD is not full, just that the returned matrices don’t contain vectors corresponding to zero singular values. Scipy’s Linear Algebra package unfortunately doesn’t seem to have a sparse SVD. Likewise, there’s no truncated SVD (there exists fast algorithms to directly compute a truncated SVD rather than computing the full SVD then taking the top K singular values).

pLSA’s source code is a bit longer although quite compact too. Although the Python/Numpy code was quite optimized, it took half a day to compute on a 50000 x 8000 term-document matrix. I rewrote the training part in C and it now takes half an hour. Keeping the Python version is quite nice for checking the correctness of the C version and as a reference as the C version is a straightforward port of it.

The implementation is sparse. It works with both Numpy’s ndarrays and Scipy’s sparce matrices.

$ git clone http://www.mblondel.org/code/plsa.git

web interface

Next, I would like to explore Fisher Kernels as there seems to have nice interactions with pLSA. I would also like to implement Latent Dirichlet Allocation (LDA), although it’s more challenging. LDA is a Bayesian extension of pLSA : pLSA is equivalent to LDA under a uniform Dirichlet prior distribution.

I’ve been having this idea on my mind for quite some time: wouldn’t it be nice to write a book about Machine Learning where each chapter is a literate program?

From Wikipedia:

The literate programming paradigm, as conceived by Knuth, represents a move away from writing programs in the manner and order imposed by the computer, and instead enables programmers to develop programs in the order demanded by the logic and flow of their thoughts.

From the PyLit homepage:

The idea is that you do not document programs (after the fact), but write documents that contain the programs.

There are plenty of great textbooks about Machine Learning out there, so the point would not be to write yet another one, but write something different. Here’s what I had been thinking.

• Each chapter written as a literate program, organized so as to maximize understanding
• Code in Python (+Numpy + Scipy but without any additional dependencies)
• Readability over Performance
• Intuitions, nice figures, useful tips or tricks
• Real-world applications at the end of each chapter
• Don’t shy away from the maths, especially if at high-school or undergraduate level…

I bet that quite a few algorithms can be written this way, yet remain very concise!

Except for the maths part, the closest book to this idea that I know of is probably “Programming Collective Intelligence: Building Smart Web 2.0 Applications”, by Toby Segaran.

An example with logistic regression

So, in order to experiment with what such a book could look like, I’ve decided to write a chapter about Logistic Regression. Topics I cover include Maximum Likelihood Estimation, Regularization and Cross-validation. At the end, I use heart disease prediction as an example of real-world application. Probably many things could be improved or added but the point for now is mainly to show what it could look like.

• HTML
• PDF
• Code

Tools

For the documentation tool, I’ve decided to go for Sphinx, which seems to be emerging as the de-facto documentation tool in the Python community. It has nice features like syntax highlighting, latex support and matplotlib plots support and can output to HTML and PDF.

Normally, in literate programming, there’s the literate source, which uses some kind of markup-language and tools are used to generate either code or documentation from it. I took a different approach. In my case, the source file is the code and the documentation is extracted from the comments in the code. Technically, it’s therefore closer to extensively documented code than actual literate programming. It has some limitations but the main advantages are that the program is runnable directly (since Python is interpreted) and the programmer can benefit from syntax highlighting. I wrote a simple program that converts Python source code to reStructuredText, as necessary for integration in Sphinx.

Interested?

It took quite some time to collect the information and do the actual writing but I feel like I improved my own understanding in the process, so I’m thinking of writing a chapter from time to time. If I do so, at the end of my PhD, I may have gathered enough material to make it a real book! The book could affectionately be entitled “The Little Machine Learner”, hence the title of this post.

Since Machine Learning is a very large field and to write a better book than I could possibly write alone, I’m also thinking that it could actually be a collaborative effort (by researchers, students and practitioners). If you’re interested, please leave a comment. I will create a discussion group if there’s enough interest.

As usual, the source code is available in my git repo:

$ git clone http://www.mblondel.org/code/tlml.git

web interface

Miyako Island, Okinawa, Japan.

The principle is very simple. Find the connected paths of low energy pixels (”the seams”). This can be done efficiently by dynamic programming (see my post on DTW).

Same image in the gradient domain showing the vertical and horizontal seams of lowest cumulated energy.

The seams of lowest cumulated energy can be seen as the pixels contributing the least to an image. By repeatedly removing or adding seams, it is thus possible to perform “content-aware” image reduction or extension. The resulting images feel more natural, less “streched”.

Height reduced by 50% by seam carving.

Height reduced by 50% by traditional rescaling.

Although seam carving doesn’t need human intervention, in the original paper, a graphical user interface (GUI) was also developed to let the user define areas that can’t be removed, or conversely, that must be removed.

In my opinion, seam carving is simple and elegant. No sophisticated object recognition algorithm was used, yet the results are quite impressive.

You can find my implementation in 250 lines of Python in my git repo:

$ git clone http://www.mblondel.org/code/seam-carving.git

web interface

Unfortunately, it’s too slow to be real-time.

preprocessing -> feature-extraction -> training -> evaluation

If I make a modification in my training algorithm and want to re-evaluate it, I do need to re-run the “training” and “evaluation” tasks, but I don’t need and don’t want to re-run the “processing” and “feature-extraction” tasks, especially if they take time to compute.

At first, I tried to save and load task results manually. This quickly proved unmanageable so I started to think of ways to automate this. Since I had quite a precise idea of what I wanted, I’ve decided to write my own tool, at the risk of reinventing the wheel. (I suspect it’s quite hard to come up with a universal tool, though) To keep things simple, I’ve decided to limit the tool’s scope to projects that can be run on a single computer, typically with multi-cores. In particular, it won’t support any kind of distributed computing.

Dependency resolution & Object persistence

Basically, the tool boils down to dependency resolution and object persistence. make is an obvious possibility for dependency resolution, but it can only use file modification time to decide whether to recompute tasks or not. In my tool, I check cache availability for a task based on the task inputs (these can be outputs from previous tasks, files, algorithm parameters…) as well as source code. The source code is also taken into consideration because a task’s result is likely to change if the task source code has changed.

To store objects on the filesystem or in a database, you need a way to serialize and deserialize objects. In the python world, the obvious choice is pickle, which is also used by the module shelve to store objects in a dbm database with a dict-like interface. Pickle is quite slow to load and save big lists of objects though, so I created two sqlite-based stores called KeyListStore and ListStore, to address this issue. A difficulty is how to efficiently compute a hash that identifies objects uniquely. To make things simple, I just took the hash of pickled objects. This is wrong, since pickle doesn’t guarantee to return twice the same strings for two same objects. However, while this can lead to incorrectly invalidating a cache, forcing the task to be recomputed, hopefully, it’s very unlikely that a cache is mistaken for the cache of another object. In practice, I haven’t had any problem with cache so far.

One feature in Python that was particularly useful for this tool was decorators. They can be used to change the behavior of a function, in a declarative style.

Example

Here’s a concrete example of a program written with my tool:

import numpy as np
import taskmanager as tm
 
DFLT_TRAIN = "/path/to/..."
DFLT_EVAL = "/path/to/..."
 
# Preprocessing
 
def preprocess(img_folder, normalize):
    return [preprocess_img(img, normalize) \ 
                  for img in img_folder.get_files()]
 
@tm.task(tm.directory("*.jpg"), bool)
def preproc_train(img_folder=DFLT_TRAIN, normalize=False):
    return preprocess(img_folder, normalize)
 
@tm.task(tm.directory("*.jpg"), bool)
def preproc_eval(img_folder=DFLT_EVAL, normalize=False):
    return preprocess(img_folder, normalize)
 
# Feature extraction
 
@tm.task(preproc_train):
def fextract_train(images):
    return [...]
 
@tm.task(preproc_eval):
def fextract_eval(images):
    return [...]
 
# Training
 
@tm.task(fextract_train, int, float)
def train(features, maxiter=10, esp=0.0001):
    return [...]
 
# Evaluation results
 
@tm.task(fextract_eval, train)
def evaluate(features, models):
    return [...]
 
@tm.task(evaluate)
@nocache
def results(eval_res):
    print [...]
 
def main():
    try:
        tm.TaskManager.OUTPUT_FOLDER = "./tmp"
        tm.run_command(sys.argv[1:])
    except tm.TaskManagerError, m:
        print >>sys.stderr, m
 
if __name__ == "__main__":
    main()

• The tool is quite unobtrusive.
• There’s no need to deal with file names or file versions, this is all done transparently for you.
• You get a command-line interface for free. Here, since all tasks have a default parameter, you could just run “./mypgm.py results” and it would work. If you wanted to try out different parameters for, e.g., “train”, you could run “./mypgm.py train:5 results”

Code

Code available here.

Everything is kept in one file to make it easy to copy the tool to another project. The tool is quite usable already but of course, it’s a work in progress.

The multi-core trend

Moore’s law states that:

The density of transistors on chips doubles every 24 month.

Although Moore’s law, contrary to what is often thought still holds true, the exponential processor transistor growth predicted by Moore does not always translate into exponentially greater practical computing performance. Therefore parallel computation has recently become necessary to take full advantage of the gains allowed by Moore’s law. This explains the recent multi-core trend: most recent computers are now equipped with 2 or more cores.

The problem is that you can’t just use multi-core equipped computers and hope that your programs will run faster on them. Programs need be modified to operate in a parallel fashion as opposed to a sequential fashion.

At the same time, languages like Ruby and Python are famous for their GIL (Global Interpreter Lock). Because of the GIL, even programs that are designed to be parallel can effectively use only one core at a time, resulting in no speed improvement. Parallelism here is just an illusion: the processor switches between threads but does so frequently that the user perceive the operations as being performed in parallel.

The novelty of the multiprocessing module in Python 2.6 is that is uses processes instead of threads (see Threads compared with processes) and it does not suffer from the GIL. Programs running on multi-cores can therefore operate in a truly parallel fashion.

Parallelizing programs

To make things simpler, let me quote the excellent blog post How-to Split a Problem into Tasks.

The very first step in every successful parallelization effort is always the same: you take a look at the problem that needs to be solved and start splitting it into tasks that can be computed in parallel. [...] what I am describing here is also called problem decomposition. The goal here is to divide the problem into several smaller subproblems, called tasks that can be computed in parallel later on. The tasks can be of different size and must not necessarily be independent.

And, about data decomposition:

When data structures with large amounts of similar data need to be processed, data decomposition is usually a well-performing decomposition technique. The tasks in this strategy consist of groups of data. These can be either input data, output data or even intermediate data, decompositions to all varieties are possible and may be useful. All processors perform the same operations on these data, which are often independent from one another. This is my favorite decomposition technique, because it is usually easy to do, often has no dependencies in between tasks and scales really well.

Data decomposition is so straightforward that it can without any doubt be called embarrassingly parallel.

Map

If you are a Python user, you most probably know list comprehensions:

>>> from math import sqrt
>>> [sqrt(i) for i in [1, 4, 9, 16]] 
[1.0, 2.0, 3.0, 4.0]

In this example, sqrt is applied to each element of the list and a list is returned. The resulting list and the input list are therefore the same size.

Probably less known are generator comprehensions, which can be written by replacing the outer brackets with parentheses:

>>> gen = (sqrt(i) for i in [1, 4, 9, 16]) 
<generator object at 0xb7cec56c>
>>> for i in gen: print i
1.0
2.0
3.0
4.0

The difference between list and generator comprehensions is that list comprehensions are evaluated entirely before returning, while generator comprehensions yield results one by one. Generators are therefore more “lazy” and can results in big memory savings when iterating over large lists.

The outer parentheses can even be omitted when calling functions with only 1 argument:

>>> print(sqrt(i) for i in [1, 4, 9, 16]) 
<generator object at 0xb7cec68c>

For those more familiar with functional programming, list comprehensions are similar to the map higher-order function.

>>> (map sqrt '(1 4 9 16))
(1.0 2.0 3.0 4.0)

In fact, Python has a built-in map function too.

>>> map(sqrt, [1, 4, 9, 16])
[1.0, 2.0, 3.0, 4.0]

Reduce

While map applies a function to each element of a list and returns the resulting list, reduce is a higher-order function that uses another function to combine the elements of a list in some way.

>>> plus = concatenate = lambda x,y: x+y
>>> reduce(plus, [1,2,3,4])
10
>>> reduce(concatenate, [[1,2], [3,4]])
[1, 2, 3, 4]

multiprocessing.Pool ’s map

Applying a function to each element of a list with map kind of assumes that the function is pure, i.e. that the result output by the function is only function of its input arguments. Although nothing prevents you from giving an impure function as argument to map, it is dirty, potentially dangerous and not the functional philosophy. Concretely, it means that you’d better not use global variables or anything the state of which may be changed during the program execution, in your functions. This thus also includes instance methods (an object, in essence, encapsulates a state).

To reuse the terminology above, if we think of applying our function to each element of the list as tasks, then our tasks are independent from each other and so there’s is no reason to operate over the list sequentially. Independence is also very nice because communication and collaboration between threads/processes happen to be one of the most difficult aspect of concurrent programming. Here, no communication between threads/processes is required.

And here comes the new multiprocessing module and more particularly its Pool class. This class represents pools of worker processes and has a map method, which is similar to the map built-in function.

>>> from multiprocessing import Pool
>>> pool = Pool(processes=4)
>>> pool.map(sqrt, [1,4,9,16])
[1.0, 2.0, 3.0, 4.0]

The difference with the built-in map here is that 4 processes are used. This will result in about a 4x speedup if the computer running the program has at least 4 cores. Of course, sqrt is a toy example but here’s a real-life example in a Machine Learning context.

>>> image_sets = [set1, ..., setn]
>>> preprocessed = pool.map(preprocess, images_sets)
>>> feat_sets = pool.map(feat_extract, preprocessed)
>>> models = pool.map(train, feat_sets)

As long as you can write your code as list comprehensions, you can apply the data decomposition approach. It’s easy, abuse it!

However, spawning a process has a cost because of context switching. Therefore, when the function to be applied on each element returns quasi instantaneously, it may be worth splitting the data into larger chunks, run each chunk in a separate process and then recombine the results with reduce. (See also MapReduce)

Helpers

Here are some helpers which make parallelizing your list comprehensions even more straightforward and easy to read.

As mentioned before, the blog post that introduced me to this new multiprocessing module also came with a smart code snippet. I reworked it to fit my liking and this is what it looks like now:

>>> sqrtd = delayed(sqrt)
>>> powd = delayed(pow)
>>> squares = [1, 4, 9, 16]
 
>>> pool_parallelize([sqrtd(i) for i in squares], njobs=4)
[1.0, 2.0, 3.0, 4.0]
 
>>> pool_parallelize([powd(i, 0.5) for i in squares], njobs=4)
[1.0, 2.0, 3.0, 4.0]

Contrary to Pool’s map, this supports parallelizing functions of any arity.

Then I came up with this solution, which reduces the typing and is quite elegant.

>>> sqrtp = parallelized(sqrt)
>>> powp = parallelized(pow)
 
>>> sqrtp(squares, njobs=4)
[1.0, 2.0, 3.0, 4.0]
 
>>> powp([(i, 0.5) for i in squares], njobs=4)
[1.0, 2.0, 3.0, 4.0]

Code available here.

Conclusion

You really gotta love functional programming in Python! (By the way, see also Charming Python: Functional programming in Python part 1 and part 2)

Lately, Python and C/C++ are becoming my language combination of choice for my research. It’s a pragmatical choice.

Regarding Python:

- It has interesting packages for scientific computing such as NumPy (fast multi-dimensional arrays and vectorized code), SciPy (reusable scientific packages), Matplotlib (plotting), IPython (Matlab-like interactive environment).
- It has many libraries and many bindings/wrappers for C/C++ libraries, including in my fields of interest such as Machine Learning, Natural Language Processing and Image Processing.
- It has many users, meaning that more people can contribute to your projects.
- It’s a full-fledge language, with powerful features and a large standard library.

Regarding C/C++:

- They are the most commonly used languages to write native extensions for Python. Even though it’s possible to get huge speedups by vectorizing your code with NumPy (avoid for loops like the plague!), you can never get anywhere close to native programs speed.
- They are pretty much the fastest languages out there, although Fortran can be faster.

In a nutshell, I try to use Python and NumPy as much as possible and when necessary, I rewrite selected portions in C or C++.

Wrapping with SWIG

Wrappers/bindings need be created in order to be able to call C/C++ code from Python. Although it’s possible to write such bindings by hand, using the Python C API, it’s quite a tedious task and a lot of it can be automated.

SWIG is a tool that reads a .i interface file and can generate a wrapper automatically. The interface file defines the signature of the functions to wrap, together with hand-made rules to handle cases that SWIG cannot process automatically. SWIG can generate bindings for many languages besides Python.

SWIG can usually manage to generate bindings automatically for functions with simple types. However, when functions require arrays, pointers or structures, hand-made rules usually need be added to the interface file to tell SWIG how to process these types. In that case, SWIG becomes, in my opinion, a complicated tool and needs quite some time to master.

So far I never had to wrap existing third-party libraries and the only code I had to wrap is my own, to speed up selected portions of my Python code. In that case, my approach has been to specifically design my code to be easy to wrap by SWIG. Basically, the trick is to avoid all types that SWIG cannot handle directly and instead use C++ objects as a facade to your data. For example, to represent a handwritten stroke, instead of using a 2-dimensional array of integer coordinates (x, y), I can create a Stroke class, which defines a add_point(int x, int y) method.

C++ side:

void do_something (int **stroke); 
void do_something2 (Stroke *stroke);

Python side:

mydata = [(x1,y1), ..., (xn,yn)]
 
# won't work because need to instruct SWIG
# how to convert a list of int pairs to a int **
do_something(mydata)
 
# OK
stroke = Stroke()
for x,y in mydata:
    stroke.add_point(x, y)
do_something2(stroke)

As can be seen, this approach is far from ideal but at least, the code can be wrapped without adding any hand-made rules to SWIG’s interface file. However, it does require you to design your code specifically to make it easy to wrap for SWIG. The resulting Python code feels objected-oriented but not necessarily as pythonic as it could be.

Wrapping with Cython

Cython (not to be confused with CPython, the default implementation of the Python interpreter) is a tool which allows to compile Python into native Python extensions. It generates a C file which is then compiled to a Python extension with gcc. The compilation is completely transparent to the user thanks to the integration of Cython in distutils (you know, the module used by setup.py).

Compiling Python to native extension in itself won’t provide any speed up compared to purely interpreted code, although it does provide an effective way to do close-source in Python. However, Cython also provides additional features to the Python language such as static types that can give hints to the compiler to optimize the extension (Cython is therefore a new language, similar to Python). Concretely, Cython feels like Python with additional C-like features. The programmer can thus optimize selected portions of code by providing additional hints to the compiler, while still being able to use Python objects in other portions if willing to pay the speed cost.

Moreover, there are mechanisms in Cython to easily call external C/C++ functions. So with Cython, one can write extensions running at native speed either by writing them entirely in Cython, or, in a combination of C/C++ and Cython.

To try Cython, I’ve written a small extension to perform Dynamic Time Warping (see my recent post). Rather than writing my extension entirely in Cython, I’ve opted for the solution of writing it in C and wrapping it in Cython. There were several reasons for that:

- Cython is a language in its own right meaning that you need to learn it. Although Cython does seem easy to learn, I didn’t want to spend too much time for now.
- If I am to write C-style code, I prefer to write it in C directly. Mixing C-like code and Python-like code in the same functions seemed to be confusing me. I found it much easier to reason about my program if I split the C-like code and the Python-like code into two separate files.
- My editor doesn’t have Cython syntax highlighting ;-)

When wrapping C code with Cython, a useful trick is to leave memory allocation of both your input and output data to the Cython side. For example, my dtw C function takes a matrix as input and outputs a new matrix as result. Rather than allocating the output matrix on the C side, I allocate it from the Cython side and pass it as an additional argument to my function. It makes the C code clearer because it can focus on processing the data and you don’t need to care about memory de-allocation since objects are taken care of by the garbage collector just like other objects.

When using Numpy to allocate your arrays and matrices, it’s important to know the internal ndarray memory model: how elements are organized in memory. The Guide to Numpy has all the information you need for that.

This DTW implementation is trivial: it doesn’t support features commonly found in other implementations like window constraints. However, it was a nice way to try out Cython and I got a 100x speed-up compared to the pure Python and Numpy version. The problem of DTW and dynamic programming algorithms in general is that they are difficult to vectorize so writing a native extension makes a lot of sense here.

The source code is available here.

For completeness, finally let me mention ctypes and Weave, which are respectively a way to call C functions in pure Python and a way to inline C in your Python programs.

1. Macports

Install Macports.

2. X11

Install X11 for Mac OS X.

3. Fantasdic

Install dependencies:
$ sudo port install rb-gtk2 rb-libglade2 git-core

Retrieve latest source code:
$ git clone git://git.gnome.org/fantasdic

Install fantasdic:
$ cd fantasdic/
$ ruby setup.rb config
$ ruby setup.rb setup
$ sudo ruby setup.rb install

You can now launch fantasdic by running the “fantasdic” command.

You can use Platypus to make it a dock application. In that case, you need to input the full path to the ruby interpreter and fantasdic: /opt/local/bin/ruby and /opt/local/bin/fantasdic, respectively.

4. Kinput2 and canna

You can safely skip this if you don’t need to input Japanese.

Install kinput2 and canna (kana-kanji conversion server):
$ sudo port install kinput2 canna

Activate canna on startup:
$ sudo launchctl load -w /opt/local/etc/LaunchDaemons/org.macports.canna/org.macports.canna.plist

Activate kinput2 on X’s startup:
$ cp /usr/X11/lib/X11/xinit/xinitrc ~/.xinitrc
$ vi ~/.xinitrc

And add the following line below “# start some nice programs”:
test -x /opt/local/bin/kinput2 && /opt/local/bin/kinput2 &

The command to launch fantasdic is now:
XMODIFIERS=”@im=kinput2″ GTK_IM_MODULE=”xim” LANG=”ja_JP.UTF-8″ fantasdic

And the obligatory screenshot ;-)

Dynamic Time Warping (DTW) is a well-known algorithm which aims at comparing and aligning two sequences of data points (a.k.a time series). Although it was originally developed for speech recognition (see [1]), it has also been applied to many other fields like bioinformatics, econometrics and, of course, handwriting recognition.

Consider two sequences A and B, composed respectively of n and m feature vectors.

Each feature vector is d-dimensional and can thus be represented as a point in a d-dimensional space. For example, in handwriting recognition, we could directly use the raw (x,y) coordinates of the pen movement and that would make us sequences of 2-dimensional vectors. In practice however, one would extract more useful features from (x,y) and create vectors of dimension possibly greater than 2. It’s also worth noting that the sequences A and B can be of different length.

Time warping

DTW works by warping (hence the name) the time axis iteratively until an optimal match between the two sequences is found.

In the figure above, which is an example of two sequences of data points with only 1 dimension, the time axis is warped so that each data point in the green sequence is optimally aligned to a point in the blue sequence.

Best path

We can construct a n x m distance matrix. In this matrix, each cell (i,j) represents the distance between the i-th element of sequence A and the j-th element of sequence B. The distance metric used depends on the application but a common metric is the euclidean distance.

Finding the best alignment between two sequences can be seen as finding the shortest path to go from the bottom-left cell to the top-right cell of that matrix. The length of a path is simply the sum of all the cells that were visited along that path. The further away the optimal path wanders from the diagonal, the more the two sequences need to be warped to match together.

The brute force approach to finding the shortest path would be to try each path one by one and finally select the shortest one. However it’s apparent that it would result in an explosion of paths to explore, especially if the two sequences are long. To solve this problem, DTW uses two things: constraints and dynamic programming.

Constraints

DTW can impose several kinds of reasonable constraints, to limit the number of paths to explore.

• Monotonicity: The alignment path doesn’t go back in time index. This guarantees that features are not repeated in the alignment.
• Continuity: The alignment doesn’t jump in time index. This guarantees that important features are not omitted.
• Boundary: The alignment starts at the bottom-left and ends at the top-right. This guarantees that the sequences are not considered only partially.
• Warping window: A good alignment path is unlikely to wander too far from the diagonal. This guarantees that the alignment doesn’t try to skip different features or get stuck at similar features.
• Shape: Aligned paths shouldn’t be too steep or too shallow. This prevents short sequences to be aligned with long ones.

These constraints are best visualized in [3].

Dynamic Programming

Taking advantage of such constraints, DTW uses dynamic programming to find the best alignment in a recursive way. Previously, the cell (i,j) of the distance matrix was defined as “the distance between the i-th element of sequence A and the j-th element of sequence B”. In the dynamic programming way of thinking, this definition is changed, and instead, the cell (i,j) is defined as the length of the shortest path up to that cell. Assuming local constraints like below,

it allows us to define the cell (i,j) recursively:

cell(i,j) = local_distance(i,j) + MIN(cell(i-1,j), cell(i-1,j-1), cell(i, j-1))

Here, recursively means that the shortest path up to the cell (i,j) is defined in terms of the shortest path up to the adjacent cells. A lot of different local constraints can be defined (see this table) and thus there are many variations in the way DTW can be implemented.

DTW as a distance metric

Once the algorithm has reached the top-right cell, we can use backtracking in order to retrieve the best alignment. If we’re just interested in comparing the two sequences however, then the top-right cell of the matrix just happens to be the length of the shortest path. We can therefore use the value stored in this cell as the distance between the two sequences. DTW has the nice property to be symmetric so DTW(a,b) = DTW(b,a). Also, DTW doesn’t fulfill the triangle inequality but it isn’t a problem in practice.

Related algorithms

DTW looks almost identical to the Levenshtein algorithm, an algorithm to compare strings, and is very similar to the Smith-Waterman algorithm, an algorithm for sequence alignment.

References

[1] Sakoe, H. and Chiba, S., Dynamic programming algorithm optimization for spoken word recognition, IEEE Transactions on Acoustics, Speech and Signal Processing, 26(1) pp. 43- 49, 1978

[2] DTW algorithm @ GenTχWarper

[3] PowerPoint presentation by Elena Tsiporkova