Swimming in a sea of data

A recent NY Times article highlights the issue of data overload. It points out some of the approaches gathering momentum to keep up with the pace of data given the enormous computational requirements. Modern algorithms are not just enormously complex to design they are ravenously processor hungry and require some clever juggling to compute within a reasonable cost. However approaches such as Hadoop (as pointed out in the article), HBase, Hypertable etc. offer a quantum leap in capabilities.

At Cognika we have been leveraging Hadoop extensively for many of our machine-learning activities. Lately we have developed some image-processing and feature detection algorithms which normally would require significant investments in hardware. However, w have managed to achieve comparable performance using commodity hardware and squeeze out some impressive results. Please post your experiences and/or lessons learned and we are happy to share ours.