Please wait a minute...
FITEE
Front. Inform. Technol. Electron. Eng.  2016, Vol. 17 Issue (10): 982-993    DOI: 10.1631/FITEE.1500332
    
TextGen：用于新型存储系统基准测试的真实文本数据集生成方法
Long-xiang Wang, Xiao-she Dong, Xing-jun Zhang, Yin-feng Wang, Tao Ju, Guo-fu Feng
School of Electronic and Information Engineering, Xi'an Jiaotong University, Xi'an 710049, China; Shenzhen Institute of Information Technology, Shenzhen 518172, China; College of Information Technology, Shanghai Ocean University, Shanghai 201306, China
TextGen: a realistic text data content generation method for modern storage system benchmarks
Long-xiang Wang, Xiao-she Dong, Xing-jun Zhang, Yin-feng Wang, Tao Ju, Guo-fu Feng
School of Electronic and Information Engineering, Xi'an Jiaotong University, Xi'an 710049, China; Shenzhen Institute of Information Technology, Shenzhen 518172, China; College of Information Technology, Shanghai Ocean University, Shanghai 201306, China
 全文: PDF 
摘要： \n 概要：新型存储系统通过内置数据压缩功能提高性能，并节省存储空间。因此，数据内容会显著影响存储系统基准测试结果。由于真实数据集规模庞大，难以复制到目标测试系统，并且大多数数据集由于隐私性无法进行共享。因此，基准测试程序需要人工生成测试数据集。为了保证测试结果的准确性，需要根据影响存储系统性能的真实数据集特征信息生成数据。现有方法SDGen在字节级别上分析真实数据集内容分布特征，并以此生成数据集，因此能够保证内置字节级压缩算法的存储系统测试结果准确。但是SDGen并未分析真实数据集的词级别内容分布特征，因此不能保证内置词级别压缩算法的存储系统测试结果准确，本文提出了一种基于Lognormal概率分布模型的文本数据集生成方法TextGen。该方法根据真实数据集的词切分结果建立语料库，分析语料库中词的分布特征，利用最大似然估计得到词分布的Lognormal模型参数，根据模型采用蒙特卡洛方法生成数据内容。该方法生成数据集所消耗的时间只与生成数据集规模相关，具有线性的时间复杂度O(n)。本文收集了四种数据集验证方法有效性，并通过一种典型的词级别压缩算法——ETDC（End-Tagged Dense Code）进行测试。实验结果表明：相比SDGen，TextGen生成文本数据集性能更高，并且，生成数据集用于压缩测试后与真实数据集的压缩速率、压缩率相似程度更高。
\n
关键词： 基准测试;  存储系统;  基于词的压缩算法    
Abstract: Modern storage systems incorporate data compressors to improve their performance and capacity. As a result, data content can significantly influence the result of a storage system benchmark. Because real-world proprietary datasets are too large to be copied onto a test storage system, and most data cannot be shared due to privacy issues, a benchmark needs to generate data synthetically. To ensure that the result is accurate, it is necessary to generate data content based on the characterization of real-world data properties that influence the storage system performance during the execution of a benchmark. The existing approach, called SDGen, cannot guarantee that the benchmark result is accurate in storage systems that have built-in word-based compressors. The reason is that SDGen characterizes the properties that influence compression performance only at the byte level, and no properties are characterized at the word level. To address this problem, we present TextGen, a realistic text data content generation method for modern storage system benchmarks. TextGen builds the word corpus by segmenting real-world text datasets, and creates a word-frequency distribution by counting each word in the corpus. To improve data generation performance, the word-frequency distribution is fitted to a lognormal distribution by maximum likelihood estimation. The Monte Carlo approach is used to generate synthetic data. The running time of TextGen generation depends only on the expected data size, which means that the time complexity of TextGen is O(n). To evaluate TextGen, four real-world datasets were used to perform an experiment. The experimental results show that, compared with SDGen, the compression performance and compression ratio of the datasets generated by TextGen deviate less from real-world datasets when end-tagged dense code, a representative of word-based compressors, is evaluated.
Key words: Benchmark    Storage system    Word-based compression
收稿日期: 2015-10-13 出版日期: 2016-10-08
CLC:  TP311.1  
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
RSS
作者相关文章　　
Long-xiang Wang
Xiao-she Dong
Xing-jun Zhang
Yin-feng Wang
Tao Ju
Guo-fu Feng

引用本文:

Long-xiang Wang, Xiao-she Dong, Xing-jun Zhang, Yin-feng Wang, Tao Ju, Guo-fu Feng. TextGen: a realistic text data content generation method for modern storage system benchmarks. Front. Inform. Technol. Electron. Eng., 2016, 17(10): 982-993.

链接本文:

http://www.zjujournals.com/xueshu/fitee/CN/10.1631/FITEE.1500332    或    http://www.zjujournals.com/xueshu/fitee/CN/Y2016/V17/I10/982

[1] Nan-nan Zhao, Ji-guang Wan, Jun Wang, Chang-sheng Xie. 一种针对基于一致性哈希的键值存储系统的低能耗副本布局策略[J]. Front. Inform. Technol. Electron. Eng., 2016, 17(10): 994-1007.