Storing big data: why compliance is key

The big data explosion may be offering organisations new insights and a wealth of business advantages, but how much control do we have over the storage and handling of this data?

If the answer for your business is not 'absolute control', then you are entering a minefield of laws, regulations and possible fines over the mishandling of data, as compliance issues continue to grow.

With this in mind, we spoke to Deidre Mahon, VP of big data software firm Rainstor, who told us about the data storage challenges facing organisations right now, and what compliance managers can do to steer clear of hefty fines.

TechRadar Pro: With respect to the storage of data, which compliance issues are most pertinent right now?

Deirdre Mahon: Rules are particularly stringent for financial services at the moment, particularly investment banks, who face SEC 17a-4(f) and CFTC (commodity futures trade commission) which mandate that transaction and trade data be held for specific time frames in an immutable store. In other words tamper-proof and providing the highest level of protection against malicious damage or breach.

Other regulations for banking include PCI, which is for the protection of sensitive personally identifiable information (PII) following a twelve-step data-related policy.

Additional compliance coverage includes Dodd Frank reform in financial services and Sarbanes Oxley for public corporations, which is mostly related to the retention and storage of structured and semi-structured data.

In other sectors, you have lawful intercept and mass intercept rulings for the retention of communications data from all mobile networks. This, of course, varies by country and region.

TRP: What are some of the potential problem areas that organisations should be aware of when determining their storage and compliance needs?

DM: Most compliance mandates specify how the data should be stored, verified, accessed and disposed of, and unfortunately, most traditional database systems don't address these requirements out-of-the-box.

Often, when databases and data warehouses reach capacity limits, the data is offloaded to tape which may be WORM compatible and initially lower cost, but it doesn't satisfy SEC 17a-4 mandates and you simply cannot query it.

Manual intervention and days-to-weeks of effort ensue to bring tape data back to life, which is error-prone and highly risky. By not meeting these compliance mandates, banks face heavy fines and, of course, bad PR which can result in lost revenue and credibility.

IT teams need to work closely with the data governance and compliance officers to determine which data sets need to be retained and for how long.

By institutionalising policies which help you avoid fines and proactively monitor for potential breaches, you lay the foundation and the rest is really about which technology platform you wish to deploy.

The technology solution is by no means trivial, but there are proven and battle-tested solutions that solve this problem and really prevent you from having to build it from scratch.

Built-in security, data disposition, legal hold, and audit trails are all critical elements of what is required in order to stay compliant and within the law.

TRP: Where do you see data storage compliance heading in the next five to ten years?

DM: Standard & Poor's recently conducted research among financial services chief compliance and risk officers showed who they now collectively spend $50 Billion on compliance mandates alone.

These costs are expected to rise over the coming years. As data continues to grow from a wider variety of sources, proliferates and is shared across global networks, the level of stringency and controls is bound to increase.

Concerns of privacy, identity theft and credit card fraud are everyday occurrences, but once you start to consider other data-related threats around healthcare insurance, pornography, and other criminal activities, you can very quickly see how there must be standards and policies adhered to in order to protect the innocent.