The storage system in most of the organizations comprises duplicate copies of many pieces of data. In this case, de-duplication technique can be used to eliminate all these extra copies by saving just one copy of the data and replacing the other copies with pointers that lead back to the original copy.
Some of the most important benefits of data de-duplication are:
- The lower storage space requirements allow organisations to save money on disk expenditures
- The more proficient utilization of disk space also allow for longer disk retention periods that provides better recovery time objectives for a longer time
- It reduces the need for tape backups
- Companies often use deduplication software in backup and disaster recovery applications
- It can also be used to free up space in primary storage
• Look at the broad propositions of de-duplication: One must consider using the broader implications of de-duplication within the context of their entire data management and storage strategy. Moreover, this technique can be performed at different levels including the file, block and byte levels.
It is necessary to consider the trade-offs for each method that include computational time, accuracy, index size, level of duplication detected and in some cases, the scalability of the solution. Furthermore, in order to eliminate tape, consider how you can use de-duplication.
• Find out what data does not de-dupe properly: It is essential to learn about what data doesn’t de-dupe very well and consider not de-duping it. With most of the de-dupe systems, data that is formed by human such as database entries, office documents, transactions, etc., de dupes well.
On the other hand, the data that is automatically created by computers such as audio, video, photos, seismic data etc., don’t de-dupe very well. In such situations, you may consider storing it on non de-duped storage systems. Also, you can consider using a de-duplication solution that can selectively avoid certain sets of data.
• Don’t fixate over space diminution ratios: A data de-duplication ratio usually refers to the number of bytes input into the de-duplication process divided by the number of bytes output from the process. However, the length of time that data is retained affects data de-duplication ratios in two ways:
• The more the scope of data de-duplication, the advanced the data de-duplication proportion will likely be. You are more likely to find duplicate data, if more data is examined when de-duplicating new data.
• Comparatively small de-duplication ratios can produce significant space savings
Furthermore, it is better to consider increasing your backup retention period for your on-disk data store, rather than performing more frequent full backups just to get a better data de-duplication ratio.
• Avoid employing multiplexing if you’re backing up to a VTL: You must avoid using multiplexing, if you’re storing to a virtual tape library (VTL). Restoring data from multiplexed backup may take longer time. People carrying over this practice from physical tape to virtual tape, may face devastating effects on their de-dupe ratio.
• Check for multiple systems before picking out your solution: In order to best determine the optimum approach for de-duping your data, be sure to pilot several de-duplication systems in your environment. As soon as you choose your data de-duplication solution, be certain to follow the best practices suggested by your de-duplication solution vendor.
Thus, Data de-duplication plays an imperative role in managing large amount of data in any organization. It can be very well used to provide an efficient data storage infrastructure for optimizing data storage.
ChrisAbout the Author:
Chris is a software engineer and is sharing some of the useful tips related to de-duplication technology and de-duplication software benefits in visualized environment.