Why you should be digging up your Dark Data

What is Dark Data?

Dark Data is data that is being stored, processed and collected but that has no recognised value or purpose attached to it. It’s data that has been temporarily forgotten about or else purposefully put to the back of the storage container as it “might be needed one day”. It’s the keys to the shed that have been lost behind the back of the sofa.

Ironically, a lot of Dark Data may actually have some serious importance and may include business-critical information that can eventually be utilised for substantial and long-lasting benefit – you just don’t know it yet! However, this doesn’t count for all of it. In fact, the vast majority of Dark Data is what is known as Redundant, Obsolete or Trivial (ROT) data, and is generally useless.

It’s not enough, though, to overlook this and pretend that the problem doesn’t exist, because the reality of it is that this data is costing businesses a lot of money, in storage, protection and management. This is using precious resources that could be redeployed for more important tasks. What’s more, there’s a staggering amount of risk that comes with it – legal, financial, security, intelligence – which can all be easily mitigated if time and resources were to be invested in sifting through the swathes of accumulated data.

With the arrival of the General Data Protection Regulation (GDPR) legislation just around the corner, it’s certainly true that Dark Data is more frequently being illuminated as a growing concern for businesses.  In fact, recent reports from Gartner have indicated that as much as 80% of the world’s Big Data is Dark. But what exactly does that mean? What really is Dark Data and why should companies start to become more aware of it?

The dark side of Dark Data

International software company, Veritas conducted a number of reports on Dark Data. According to their ‘Databerg Report’ of 2015, the UK is the second worst Dark Data offender in the world. 59% of data in the UK and Ireland is Dark, with only 12% of data being business-critical.

The unfortunate thing is that the constant influx of data means that its management becomes a significant chore, and as such a lot of data is simply dumped into corporate systems ‘for the time-being’, data management strategies gravitating around the need to manage volume rather than value. This kind of thinking, coupled with unhelpful user behaviour that assumes storage is limitless and free for personal use, can lead to serious issues further down the line, and as long as the data is ignored, the bigger these issues grow. Veritas’ report claims that Dark Data could cost EMEA customers €784bn by 2020.

Nevertheless, there is a light at the end of the Dark tunnel. By finding and extracting Dark Data, identifying it, and eradicating any untapped areas of risk, you could gain some very valuable insight, and start influencing business decision-making for the better.

The silver lining

The largest challenge when it comes to Dark Data is the fact that so much of it is just that – in the dark. Making your Dark Data visible and exposing it gains you back some significant control over it, allowing you the freedom to separate it from any redundant data and derive real value.

A host of steps will help extract value from conducting regular audits to introducing information governance strategy through to encryption of data.

Where is it finding success?

Dark Data is already being dealt with rather well by a number of recent technologies. IBM’s supercomputer, Watson, has found success in the sorting of Dark Data, and software framework Apache Hadoop is also making light of it for users. Data Warehousing leaders Teradata and big data solutions company Nuix are also currently working together to turn unstructured data into searchable text, applying analysis techniques to aid better understanding of the data.

So more and more ways are being found to combat Dark Data and turn it into a revenue.  It’s now more imperative than ever to begin dealing with Dark Data, and we hope that this article has provided some guidance on the subject.