Dark data is defined as the information assets organisations collect, process and store during regular business activities, but generally fail to use for other purposes (for example, analytics, business relationships and direct monetizing).
Dark data is all of the unused, unknown and untapped data across an organisation, generated as a result of users’ daily interactions online with countless devices and systems — everything from machine data to server log files to unstructured data derived from social media.
Similar to dark matter in physics, dark data often comprises most organisations’ complete universe of information assets and organisations often retain dark data for compliance purposes only. Storing and securing data typically incurs significant expense (and sometimes greater risk) than it does value.
Organisations may consider this data too old to provide value, or see it as incomplete or redundant, or limited by a format that can’t be accessed with available tools. All too often, they don’t even know it exists.
However, dark data may be one of an organization’s biggest untapped resources and greatest areas of potential risk. Data is increasingly a major organisational asset, and, more stringent data and privacy regulations may require complete management of all data within an organisation.
One key area of challenge is to manage dark data in order to minimise risks and liabilities in information governance, that is, regulatory compliance, litigation, records-keeping, privacy and records management.