Please wait a minute...
浙江大学学报(工学版)
Journal of ZheJiang University (Engineering Science)  2022, Vol. 56 Issue (11): 2224-2231    DOI: 10.3785/j.issn.1008-973X.2022.11.013
    
Histogram based tone mapping algorithm using image segmentation and fusion
Hao JIANG(),Hai-song XU*()
State Key Laboratory of Modern Optical Instrumentation, College of Optical Science and Engineering, Zhejiang University, Hangzhou 310027, China
Download: HTML     PDF(5565KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

The tone mapping operators (TMO) aims at reproducing the visual perception of high dynamic range (HDR) images on the low dynamic range media. A histogram based tone mapping algorithm was proposed to use image segmentation and fusion. The HDR images were segmented into non-overlapping rectangle blocks. In each block, the tone mapping problem was regarded as the K-means clustering problem based on the image histogram with a proposed loss function. The derived mapping function was adaptively adjusted considering the uniformity of the block. The detail-enhancing effect of the mapping function was then restrained by the block uniformity for less arising artefacts. An image fusion scheme was presented to deal with the artefacts on the segmentation boundaries. With the form of a bilateral filter, the fusion scheme took into account both the difference of position and luminance, ensuring the naturalness of the tone-mapped results. The proposed algorithm was evaluated to use the tone-mapped image quality index and the results illustrated a better image quality compared with the typical tone mapping methods. A balance between the detail enhancement and the global appearance preserving has also been achieved with?an?impressed?execution?efficiency?among?local?TMOs.

Key words： image processing      tone mapping      high dynamic range image      histogram adjustment      image fusion      K-means clustering     
Received: 08 February 2022      Published: 02 December 2022
CLC:  TN 27  
  TP 751  
Corresponding Authors: Hai-song XU     E-mail: zjujhh@163.com;chsxu@zju.edu.cn
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Hao JIANG
Hai-song XU
Cite this article:

Hao JIANG,Hai-song XU. Histogram based tone mapping algorithm using image segmentation and fusion. Journal of ZheJiang University (Engineering Science), 2022, 56(11): 2224-2231.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2022.11.013     OR     https://www.zjujournals.com/eng/Y2022/V56/I11/2224


基于直方图与图像分块融合的阶调映射算法

阶调映射算法（TMO）旨在将高动态范围（HDR）图像以最符合人眼感知的形式复现于低动态范围显示设备上. 为此基于图像分块与融合，提出一种直方图调整的阶调映射算法. 高动态范围图像被分为若干非重合的长方形区域，在每一块区域中阶调映射问题被视为基于直方图的K均值聚类问题并且建立相应的求解目标函数. 各区域中解得的映射函数根据该区域的均匀性进行调整，以避免对比度增强过度，从而减少伪影. 最后，提出一种双边滤波形式的图像融合策略以保证区域边界的平滑性，兼顾位置及亮度差，进而提升映射结果的自然性. 在实验中，采用阶调映射图像质量指标来对算法进行客观评估，结果表明所提算法相对于经典的阶调映射方法有更好的阶调映射效果，在细节增强与全局外貌保留间达到平衡，并在局部阶调映射算法中具有较好的运算效率.


关键词： 图像处理,  阶调映射,  高动态范围图像,  直方图调整,  图像融合,  K均值聚类 
Fig.1 Flowchart of histogram based tone mapping algorithm using image segmentation and fusion
Fig.2 Tone mapped results of different curves for same block
Fig.3 Fusion results corresponding to different σd values
Fig.4 Fusion results corresponding to different σs values
图像序号 Q
k均值聚类方法 本研究算法
1 0.67 0.88
2 0.53 0.82
3 0.65 0.87
4 0.65 0.87
5 0.76 0.98
6 0.70 0.90
7 0.76 0.95
8 0.75 0.97
9 0.59 0.84
均值 0.67 0.90
Tab.1 TMQI scores of proposed algorithm with and without image segmentation
Fig.5 Tone mapped results of proposed algorithm with and without image segmentation
图像 Q
Drago et al. [4] Kimet al. [22] Reinhard et al. [23] Khan et al. [8] Krawcyzk et al. [24] Shan et al. [25] Liang et al. [11] Liet al. [26] 本研究算法
1 0.76 0.72 0.72 0.81 0.73 0.73 0.81 0.91 0.88
2 0.70 0.68 0.65 0.79 0.68 0.60 0.75 0.89 0.82
3 0.77 0.78 0.73 0.88 0.74 0.75 0.82 0.87 0.87
4 0.76 0.78 0.73 0.85 0.71 0.74 0.80 0.85 0.87
5 0.88 0.84 0.79 0.88 0.85 0.88 0.86 0.73 0.98
6 0.85 0.84 0.73 0.88 0.77 0.76 0.86 0.81 0.90
7 0.91 0.88 0.79 0.83 0.88 0.87 0.84 0.71 0.95
8 0.86 0.87 0.78 0.87 0.82 0.84 0.85 0.72 0.97
9 0.72 0.82 0.71 0.79 0.70 0.72 0.83 0.80 0.84
均值 0.80 0.80 0.74 0.84 0.76 0.77 0.82 0.81 0.90
Tab.2 TMQI scores of different algorithms
图像 S
Drago et al. [4] Kimet al. [22] Reinhard et al. [23] Khan et al. [8] Krawcyzk et al. [24] Shan et al. [25] Liang et al. [11] Liet al. [26] 本研究算法
1 0.78 0.70 0.69 0.70 0.73 0.72 0.66 0.73 0.87
2 0.61 0.55 0.49 0.70 0.55 0.37 0.57 0.68 0.74
3 0.79 0.81 0.73 0.75 0.74 0.77 0.64 0.72 0.83
4 0.77 0.79 0.71 0.69 0.64 0.74 0.62 0.71 0.82
5 0.87 0.88 0.85 0.73 0.90 0.88 0.70 0.69 0.93
6 0.70 0.71 0.65 0.73 0.64 0.67 0.59 0.67 0.78
7 0.87 0.87 0.86 0.73 0.88 0.88 0.67 0.66 0.89
8 0.86 0.87 0.84 0.75 0.87 0.87 0.69 0.70 0.92
9 0.63 0.74 0.61 0.66 0.50 0.63 0.56 0.61 0.74
均值 0.76 0.77 0.72 0.72 0.72 0.73 0.63 0.69 0.83
Tab.3 Structure-fidelity scores of different algorithms
图像 N
Drago et al. [4] Kimet al. [22] Reinhard et al. [23] Khan et al. [8] Krawcyzk et al. [24] Shan et al. [25] Liang et al. [11] Liet al. [26] 本研究算法
1 0.05 0.01 0.00 0.33 0.01 0.01 0.41 0.85 0.44
2 0.01 0.01 0.00 0.22 0.02 0.00 0.26 0.86 0.33
3 0.05 0.08 0.01 0.62 0.00 0.01 0.48 0.64 0.45
4 0.04 0.07 0.00 0.56 0.02 0.01 0.45 0.52 0.44
5 0.46 0.23 0.06 0.66 0.24 0.41 0.60 0.03 0.97
6 0.55 0.45 0.05 0.72 0.22 0.15 0.84 0.38 0.74
7 0.61 0.44 0.06 0.36 0.41 0.38 0.52 0.00 0.85
8 0.36 0.38 0.05 0.60 0.17 0.25 0.57 0.01 0.93
9 0.04 0.34 0.03 0.30 0.13 0.06 0.74 0.45 0.45
均值 0.24 0.22 0.03 0.49 0.14 0.14 0.54 0.42 0.62
Tab.4 Naturalness scores of different algorithms
Fig.6 Examples for comparison of tone mapping algorithms
算法 平均运行时间/s
Drago et al. [4] 5.61
Kimet al. [22] 2.08
Reinhard et al. [23] 2.73
Khan et al. [8] 8.95
Krawcyzk et al. [24] 11.90
Shan et al. [25] 169.01
Liang et al. [11] 68.07
Liet al. [26] 183.38
本研究算法 15.64
Tab.5 Average operation time of compared algorithms
[1]   REINHARD E, HEIDRICH W, DEBEVEC P, et al. High dynamic range imaging: acquisition, display, and image-based lighting [M]. Morgan: Kaufmann, 2010: 1-18.
[2]   方竞宇. 高动态范围彩色图像捕获与显示方法及技术研究[D]. 杭州: 浙江大学, 2016: 78-80.
FANG Jing-yu. Study on methodology and technology of acquisition and rendering for high dynamic range color images [D]. Hangzhou: Zhejiang University, 2016: 78-80.
[3]   WARD G A contrast-based scale factor for luminance display[J]. Graphics Gems, 1994, 4: 415- 421
[4]   DRAGO F, MYSZKOWSKI K, ANNEN T, et al. Adaptive logarithmic mapping for displaying high contrast scenes [C]// Computer Graphics Forum. Oxford, UK: Blackwell Publishing, Inc, 2003: 419-426.
[5]   TUMBLIN J, RUSHMEIER H Tone reproduction for realistic images[J]. IEEE Computer Graphics and Applications, 1993, 13 (6): 42- 48
doi: 10.1109/38.252554
[6]   SCHLICK C. Quantization techniques for visualization of high dynamic range pictures, photorealistic rendering techniques [M]. Berlin, Heidelberg: Springer, 1995: 7-20.
[7]   MA K, YEGANEH H, ZENG K, et al High dynamic range image compression by optimizing tone mapped image quality index[J]. IEEE Transactions on Image Processing, 2015, 24 (10): 3086- 3097
doi: 10.1109/TIP.2015.2436340
[8]   KHAN I R, AZIZ W, SHIM S O Tone-mapping using perceptual-quantizer and image histogram[J]. IEEE Access, 2020, 8: 31350- 31358
doi: 10.1109/ACCESS.2020.2973273
[9]   AMBALATHANKANDY P, IKEBE M, YOSHIDA T, et al An adaptive global and local tone mapping algorithm implemented on FPGA[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 30 (9): 3015- 3028
[10]   ZHANG X S, YANG K F, ZHOU J, et al Retina inspired tone mapping method for high dynamic range images[J]. Optics Express, 2020, 28 (5): 5953- 5964
doi: 10.1364/OE.380555
[11]   LIANG Z, XU J, ZHANG D, et al. A hybrid l1-l0 layer decomposition model for tone mapping [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City, Utah, 2018: 4758-4766.
[12]   FAHIM M A N I, JUNG H Y Fast single-image HDR tone-mapping by avoiding base layer extraction[J]. Multidisciplinary Digital Publishing Institute, 2020, 20 (16): 4378
[13]   NGUYEN N H, VAN V T, LEE C Human visual system model-based optimized tone mapping of high dynamic range images[J]. IEEE Access, 2021, 9: 127343- 127355
doi: 10.1109/ACCESS.2021.3112046
[14]   DUAN J, BRESSAN M, DANCE C, et al Tone-mapping high dynamic range images by novel histogram adjustment[J]. Pattern Recognition, 2010, 43 (5): 1847- 1862
doi: 10.1016/j.patcog.2009.12.006
[15]   OSKARSSON M Temporally consistent tone mapping of images and video using optimal k-means clustering[J]. Journal of Mathematical Imaging and Vision, 2017, 57 (2): 225- 238
doi: 10.1007/s10851-016-0677-1
[16]   LARSON G W, RUSHMEIER H, PIATKO C A visibility matching tone reproduction operator for high dynamic range scenes[J]. IEEE Transactions on Visualization and Computer Graphics, 1997, 3 (4): 291- 306
doi: 10.1109/2945.646233
[17]   WANG H, SONG M Ckmeans. 1d. dp: optimal k-means clustering in one dimension by dynamic programming[J]. The R Journal, 2011, 3 (2): 29
doi: 10.32614/RJ-2011-015
[18]   石雅婷, 顾晓娟 人眼亮度感知特性在高动态范围影像中的应用[J]. 现代电影技术, 2021, 9: 9- 16
SHI Ya-ting, GU Xiao-juan The application of human lightness perception in high dynamic range images[J]. Advanced Motion Picture Technology, 2021, 9: 9- 16
doi: 10.3969/j.issn.1673-3215.2021.09.002
[19]   徐海松. 颜色信息工程第2版 [M]. 杭州: 浙江大学出版社, 2015: 50-54.
XU Hai-song. Color information engineering and the second edition [M]. Hangzhou: Zhejiang University Press, 2015: 50-54.
[20]   PANETTA K, KEZEBOU L, OLUDARE V, et al TMO-Net: a parameter-free tone mapping operator using generative adversarial network, and performance benchmarking on large scale HDR dataset[J]. IEEE Access, 2021, 9: 39500- 39517
doi: 10.1109/ACCESS.2021.3064295
[21]   YEGANEH H, WANG Z Objective quality assessment of tone-mapped images[J]. IEEE Transactions on Image Processing, 2012, 22 (2): 657- 667
[22]   KIM M H, KAUTZ J. Consistent tone reproduction [C]// Proceedings of the Tenth IASTED International Conference on Computer Graphics and Imaging. ACTA Press Anaheim, 2008: 152-159.
[23]   REINHARD E, STARK M, SHIRLEY P, et al. Photographic tone reproduction for digital images [C]// Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques. San Antonio Texas, 2002: 267-276.
[24]   KRAWCZYK G, MYSZKOWSKI K, SEIDEL H P. Lightness perception in tone reproduction for high dynamic range images [C]// Computer Graphics Forum. North Holland, 2005: 635-646.
[25]   SHAN Q, JIA J, BROWN M S Globally optimized linear windowed tone mapping[J]. IEEE Transactions on Visualization and Computer Graphics, 2009, 16 (4): 663- 675
[26]   LI H, JIA X, ZHANG L Clustering based content and color adaptive tone mapping[J]. Computer Vision and Image Understanding, 2018, 168: 37- 49
doi: 10.1016/j.cviu.2017.11.001
[1] Pei-zhi WEN,Jun-mou CHEN,Yan-nan XIAO,Ya-yuan WEN,Wen-ming HUANG. Underwater image enhancement algorithm based on GAN and multi-level wavelet CNN[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(2): 213-224.
[2] Tong CHEN,Jian-feng GUO,Xin-zhong HAN,Xue-li XIE,Jian-xiang XI. Visible and infrared image matching method based on generative adversarial model[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(1): 63-74.
[3] Zhu-ye XU,Xiao-qiang ZHAO,Hong-mei JIANG. 3D model fitting method based on point distribution model[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(12): 2373-2381.
[4] Ying LI,Fang CHENG,Zhi-lin ZHAO. Machining precision online measurement of large span pin hole using structured light[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(3): 557-565.
[5] Wan-liang WANG,Xiao-han YANG,Yan-wei ZHAO,Nan GAO,Chuang LV,Zhao-juan ZHANG. Image enhancement algorithm with convolutional auto-encoder network[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(9): 1728-1740.
[6] Shuo-peng WANG,Peng YANG,Hao SUN,Mai LIU. Fingerprint-based sound source localization method using two-stage reference points matching[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(6): 1198-1204.
[7] ZHOU Hao, LI Ning, LI Yuan, ZHAO Meng-hao, CEN Ke-fa. Experimental study on ethanol spray combustion characteristics under oxy-fuel conditions[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(9): 1821-1827.
[8] ZHANG Cheng-zhi, FENG Hua-jun, XU Zhi-hai, LI Qi, CHEN Yue-ting. Piecewise noise variance estimation of images based on wavelet transform[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(9): 1804-1810.
[9] ZHOU Jia-li, CHEN Yi-jun, WU Min. Image acquisition and preprocessing method based on FPGA monitor[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(2): 398-405.
[10] HUANG Song, YI Ben-shun. Underwater image restoration algorithm based on adaptive transmission ratio[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(1): 166-173.
[11] ZHOU Hao, MA Wei chen, YANG Yu, CHEN Jian zhong. Investigation of flame characteristics of low NOx swirl burner[J]. Journal of ZheJiang University (Engineering Science), 2016, 50(4): 698-703.
[12] WU Yi,YIN Jun, ZHU Li, YUAN Yong ming. Two dimensional Arimoto gray entropy image thresholding based on bee colony optimization or decomposition[J]. Journal of ZheJiang University (Engineering Science), 2015, 49(9): 1625-1633.
[13] CHEN Kuo, FENG Hua-jun, XU Zhi-hai, LI Qi, CHEN Yue-ting. Fast detail-preserving exposure fusion[J]. Journal of ZheJiang University (Engineering Science), 2015, 49(6): 1048-1054.
[14] JIANG Shen-yu, CHEN Kuo, XU Zhi-hai, FENG Hua-jun, LI Qi, CHEN Yue-ting. Multi-exposure image fusion based on well-exposedness assessment[J]. Journal of ZheJiang University (Engineering Science), 2015, 49(3): 470-475.
[15] WANG Yuan-yuan, GUO Yan-en, SHI Guo-quan, WEI Jun-xia, XIA Shun-ren. Algorithm for seabed terrain boundary extraction based on acoustic images[J]. Journal of ZheJiang University (Engineering Science), 2015, 49(2): 376-383.