4 questions to consider before running Hadoop

Sean Suchter Pepperdata Hadoop analytics
Sean Suchter, CEO of Pepperdata

Hadoop is no longer hype; it's a real business tool that can deliver real value. If you're like most companies, you have (at least) tested it. If you've tested Hadoop, you're familiar with – and excited about – its benefits: flexibility, scalability, and the value of the results it delivers.

For businesses across industries, Hadoop has already become a critical piece of infrastructure, and there are countless success stories.

Is the elephant already in the room?

For many businesses, Hadoop starts out as an experiment. One team or individual tries it out and realizes the results are not only interesting, but valuable to the line of business. Soon other teams start running jobs on Hadoop and all of a sudden you realize you depend on something that started as an experiment. Does this sound familiar?

Since the importance of Hadoop to your business can sometimes just sneak up on you, one of the key factors to optimize your investment in and take advantage of the benefits delivered by Hadoop is understanding whether or not it's a tool you can live without.

Consider the following scenarios:

  • Do business users notice when clusters get tied up and your hourly/daily reports don't run as they should?
  • Does someone carry a pager to be notified immediately if a Hadoop cluster goes down?
  • Are you scrambling to find data center space or get new hardware fast enough to keep up with the expected growth in demand on a cluster?
  • Does a C-level executive get an email when a Hadoop job that's being used in your product or daily business process is late?

If any of the above ring true for you, Hadoop is already an important part of your business – and you should be paying close attention and allocating the right resources to it.

Looking ahead

Although Hadoop is set up to have a huge impact on driving business results, it still comes with limitations (which you may have already experienced first-hand). For example, a single user or job can suddenly slow down an entire cluster by saturating the network or disk, so high-priority jobs can't get through.

However, as new tools are introduced to help businesses rely on Hadoop, the future looks bright. Consider, for example, pairing Hadoop's benefits with the ability to let multiple teams run diverse applications on a single cluster (predictably) and deploy a new distributed application on a thousand-node cluster as seamlessly as installing a new app on your phone.

Not only are these things possible, but they're right around the corner.