The proof is in the pudding, as they say. At Yahoo, the pudding is quite murky at present, what with Jerry Yang out as CEO, a disastrous 2008 in terms of stock price and failing to be taken over by Microsoft. Yet, just in the past few weeks, as Carol Bartz took over as CEO, it's as though the dark cloud has lifted.
Truth be told, Yahoo has just as many Stanford grads and doctorate researchers on staff as Google, and has always tried to push the envelope in terms of web innovation. And it must: future technologies on the web - such as location-awareness, social networking, cluster computing and advanced web search algorithms - do not just spring to life, they must be born and bred in a research lab.
In a recent visit to the Yahoo facilities, we met with the company's top researchers and innovators, including the VP of engineering and a very smart Oxford alumnus working on mapping technologies. It was an illuminating experience, because we realized that Yahoo is not just another me-too search giant or a company that acquires start-ups like Flickr.com. Here's the most exciting projects we found...
1. Y!OS
Yahoo seeks to make more of its web properties as open as possible. Today, disparate web entities within Yahoo - such as Answers and Search - do not share data well. And Yahoo doesn't do it well with other sites, either. Y!OS intends to rectify that problem. And we're already seeing the earliest signs of this kind of interoperability with sites such as Twitterfeed.com that let you log in to the site with your Yahoo account. Neal Sample, the Vice President of Engineering, says there shouldn't be a multitude of logins while APIs should be more open (so developers can build features into Yahoo), standards like XML should be more portable, and standalone apps should be easier to create.
"Users should be able to bring other experiences into Yahoo," says Sample, who we met with at Yahoo HQ in Sunnyvale. "With Y!OS, developers could add to existing services, such as Yahoo Mail or take an existing product and expand it. We allow developers onto the network and give them the ability to augment services. By opening data and preferences, we want to platform-ize the Yahoo experience."
If this all sounds familiar to you, it's because operating systems went through the same phase early on – unifying platforms and data models, making tools more widely available, and encouraging application development. Will it lead to a Yahoo web OS someday? Sample would not make any specific predictions, but did agree that web standards such as OpenID are paving the way for a web OS.
2. Hadoop
Yahoo has a keen interest in cluster computing, which uses commodity hardware to handle the most difficult and complex computing tasks. The goal of the Hadoop project is to mask the complexity of the hardware but to allow parallel processing tasks to run quickly and efficiently.
"We enable people to explore problems they could not address in the past, and to innovate with data," says Ajay Anand, part of the Yahoo Grid Computing initiative. "Yahoo Search was one of the first projects that benefited from this scalable structure. The intent is to do this processing as fast as possible and as cheaply as possible. With thousands of machines, something will break almost every single day" which is why the Hadoop project tends to focus on a scalable infrastructure.
In September of last year, Hadoop was scaled up to 4,000 nodes, each powered by two quad-core Intel Xeon processors running at 2.5GHz, 4x1TB SATA disks and 8G RAM.
