Data De-Duplication

Is your data growing out of control?

For reasons of internal compliance, client and supplier relationships or regulatory requirements, we are all retaining growing volumes of data and this is a trend that is likely to continue.

Increased storage of data leads to increased costs of retention and management.
Increase demand for real time access to ‘retained’ data is leading organisations to implement disk to disk backup, with tape increasingly being seen as a last line of defence (archive).
Primary disk, be that Fibre Channel, SCSI and SAS that is directly attached to a server (DAS) or part of Network Attached Storage (NAS) - or more typically in today’s environment a Storage Area Network (SAN) is often expensive in real terms.

Therefore, by nature, most organisations implementing a disk to disk storage strategy lean towards slower, higher capacity, but lower cost per terabyte (Tb) Serial ATA (SATA) disk solutions.
This does start to help address the need to keep a second copy on an available, but reduced cost, disk solution.

But how much of our stored data is actually in operational use on a daily basis? One comparison we can make is that once data has been produced and read it is like a book that was produced and read. We do not want to dispose of it; we wish to keep it for future reference, so we put it in a library. Once in the library, it needs to be filed for future reference and of course it still takes up considerable shelf space even if the book is never used again.

How do we reduce the amount of shelf space we need for this book and our stored (but rarely accessed) data?

One such way is called data de-duplication.

So how does it work and what are the benefits?

In simple terms, data de-duplication removes duplicate copies of documents and images from primary, secondary and archived data so that you still retain the ability to recover entire documents, files and folders, but without the need to store multiple copies of the same material.

Data de-duplication solutions can sometimes come as part of an integrated storage solution (such as NetApp’s ASIS solution, or the Cryoserver email archiving solution), or in a more sophisticated format they are available for purchase as standalone solutions.

The measure of the benefit you get from a purchased solution is relatively simple. If the amount of storage you can reclaim in capital cost terms is greater than the cost of the de-duplication software, then you are winning back storage and preserving further capital expenditure.

There are other operational benefits such as reduced backup times, reduced recovery times, reduced need for second tier disk capacity and reduced use of tape media and off-site storage of tape media.