Skip to main content

IBM to use Apache Spark to underpin pretty much everything it does

Audio player loading…

Apache Spark has attracted its biggest supporter yet, with IBM promising to run most of the company's services on the open source cluster computing framework; that includes Bluemix, Watson Health Cloud and all its analytics and commerce platforms.

In addition, Big Blue said that more than 3,500 of its staff will work on Spark-related projects and will create a Spark Technology Center in San Francisco, California.

The company will also release its SystemML machine learning technology to the Spark open source ecosystem, and pledged to help educate more than one million data scientists and data engineers on Apache cluster computing technology.

A new Spark

IBM said that Spark could shake up the data analytics market just like Linux did to the internet years ago. The company draw parallel with the popular OS, calling Spark an analytics operating system.

Spark thrives partly because it excels at analysing data when it is stored in computer memory, with performance improvements of up to 100x in some cases.

Such technology became a reality as the average price of system memory has decreased to the point where for large organisations, it doesn't break the bank to have terabytes of memory if the gains are tangible.

Desire Athow

Managing Editor, TechRadar Pro

Désiré has been musing and writing about technology during a career spanning four decades. He dabbled in website building and web hosting when DHTML and frames were en vogue and started writing about the impact of technology on society just before the start of the Y2K hysteria at the turn of the last millennium. Then followed a weekly tech column in a local business magazine in Mauritius, a late night tech radio programme called Clicplus and a freelancing gig at the now-defunct, Theinquirer, with the legendary Mike Magee as mentor. Following an eight-year stint at where he discovered the joys of global techfests, Désiré now heads up TechRadar Pro. He has an affinity for anything hardware and staunchly refuses to stop writing reviews of obscure products or cover niche B2B software-as-a-service providers.