According to Essential Science Indicators SM from Thomson Reuters, the paper “Fast and robust fixed-point algorithms for independent component analysis,” (Hyvarinen A, IEEE Trans. Neural Networks 10: 626-34, May 1999) is ranked at #7 among Highly Cited Engineering papers published over the past decade, with 636 citations to its credit up to April 30, 2009. Author Professor Aapo Hyvarinen is based at the University of Helsinki, where he is Professor of Computational Data Analysis jointly in the Department of Mathematics and Statistics and the Department of Computer Science.
Here is a short interview with him:
What factors prompted you to undertake this study, and how was it conducted?
In August 1995, I started my Ph.D. at the Helsinki University of Technology, Finland, in a research group focusing on the method called independent component analysis (ICA). ICA is a statistical method for analyzing multidimensional data, which can find underlying components in the data based on independence of the components. Such linear decompositions had been used before; factor analysis and principal component analysis are two well-known methods. However, it was also well-known that those methods couldn’t find the actual underlying components which generated the data. Each small square is one feature of small colour image…
I was really fascinated by ICA. In fact, when I first saw the description of the model, I found it difficult to believe the model could be estimated (i.e., the problem could be solved). I thought that if it really were possible, I certainly would have heard about it during my undergraduate studies! But the trick is that ICA is based on the rather unconventional framework on non-normality (non-Gaussianity), which is why it became well-known only in the 1990s. During the first year of my Ph.D., my advisor, Prof. Erkki Oja, suggested that I consider fixed-point algorithms to solve the demanding computational problems encountered in ICA. I was able to find one that worked quite well, and we soon published it (Hyvarinen A and Oja E, “A Fast Fixed-Point Algorithm for Independent Component Analysis,” Neural Computation 9:1483-92, 1997).
However, when I was writing that paper I already thought that this could not be the final answer because the method was statistically not very good. The ICA problem has two parts, statistical (how to get maximum information out of your data) and computational (how to do the required computations as efficiently as possible). The initial fixed-point algorithm in our 1997 paper provided a good solution for the computational part, but it was rather bad from the statistical viewpoint. In particular, it suffered from being extremely non-robust, i.e. very sensitive to outliers. This means that a single bad measurement point—for example, due to a faulty sensor—can ruin the whole analysis. Thus, I struggled for a few months more before I found a fixed-point algorithm which was not only fast but also robust. The algorithm was initially published in the proceedings of the International Conference in Acoustics, Speech, and Signal Processing in 1997. Then, I spent some more time analyzing the algorithm and polishing the theory, and finally at the end of 1997, I had finished the manuscript. However, as is typical in many engineering and mathematical sciences, it took more than 18 monthsbefore the article was actually published.
How was the paper received by the community?
There were some initial problems due to the fact that the utility of ICA was not widely appreciated in the 1990s. In my initial conference presentation I mentioned above, a senior scientist came to my poster and basically explained how I had got it all wrong!
But when the paper was published in 1999, people were starting to appreciate ICA more and more, and many new application areas were being found, so my paper was published at the right time. Essential for the success of the algorithm was the development, by Dr. Jarmo Hurri, of an excellent software package that enabled people to use it with minimal effort.
What are the applications for these algorithms?
The reason for the popularity of the algorithm is that ICA can be applied in almost any discipline of science, as well as in technology and even some of the humanities. Whenever you measure several variables at many different time points or for many individuals, your data can potentially be analyzed by ICA. Often, of course, the analysis does not give anything useful, but because of the great generality of the model, cases where it actually is useful are numerous. Probably the most successful application field is biomedical engineering, where the
assumptions of the model (linear mixing, non-Gaussian source signals) are quite often approximately fulfilled.
Where have you taken this work since the publication of this paper?
ICA has been the basis of most of my research ever since. Regarding this particular algorithm, I have published some extensions, but in general, the extensions are not nearly as important as the original algorithm. This has actually been rather annoying: I have worked really hard to develop an even better algorithm for ICA, but I have never found anything that beats this algorithm, which I developed in my first year of Ph.D. studies! Fortunately, there is much more to this direction of research than developing better algorithms. I have written a book simply called Independent Component Analysis with Prof. Juha Karhunen and Prof. Erkki Oja, I have developed models which generalize the ICA model, and I have applied these models in computational neuroscience (see, e.g., our new book).
Recently Prof. Hyvarinen had a visit to Kyoto University and we enjoyed his talk about his new research themes in Ishii Lab:
- Testing the ICA mixing matrix based on inter-subject or inter-session consistency, NeuroImage, 2011. Matlab code.
- Estimation of a Structural Vector Autoregression Model Using Non-Gaussianity, JMLR, 2010.
- Noise-contrastive estimation: A new estimation principle for unnormalized statistical models, AISTATS2010.
I’m also really delighted to have him signed my copy of his book:
And here are some photos I’ve taken during this opportunity (My favorite is Prof. Hyvarinen and Prof. Ishii talking talking. It depicts two great guys, with two completely different background and culture yet in the same front: the science):
This is one of my favorite online videos about ICA, take a look: