With the increasing amount of digital data, data deduplication has become an increasingly popular method for reducing data in large-scale storage systems. Generalized deduplication is an alternative technique for reducing the cost of data storage by identifying similar data chunks. This paper proposes TL-GD, a method for improving cloud storage efficiency using generalized deduplication focusing on textual datasets. The core concept of this study is to develop an efficient deduplication system that combines an alternative technique for splitting data into smaller pieces and a new approach for transforming data pieces into bases and deviations. The performance of the system has been validated using two real-world datasets. We also compare the results to state-of-the-art deduplication methods. Our evaluation results show that TL-GD achieves nearly 67% lossless compression for textual navigation instructions datasets, which is a 25% improvement on average compared to existing deduplication techniques.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Deduplication of Textual Data by NLP Approaches


    Contributors:


    Publication date :

    2023-06-01


    Size :

    1137614 byte





    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English





    Analysis of Textual Data with Multiple Classes

    Sakurai, S. / Goh, C. / Orihara, R. | British Library Conference Proceedings | 2005