Document Type : Original/Review Paper
Authors
Electrical and Computer Engineering Faculty, Semnan University, Semnan, Iran.
Abstract
Image restoration and its different variations are important topics in low-level image processing. One of the main challenges in image restoration is dependency of current methods to the corruption characteristics. In this paper, we have proposed an image restoration architecture that enables us to address different types of corruption, regardless of type, amount and location. The main intuition behind our approach is restoring original images from abstracted perceptual features. Using an encoder-decoder architecture, image restoration can be defined as an image transformation task. Abstraction of perceptual features is done in the encoder part of the model and determines the sampling point within original images' Probability Density Function (PDF). The PDF of original images is learned in the decoder section by using a Generative Adversarial Network (GAN) that receives the sampling point from the encoder part. Concretely, sampling from the learned PDF restores original image from its corrupted version. Pretrained network extracts perceptual features and Restricted Boltzmann Machine (RBM) makes the abstraction over them in the encoder section. By developing a new algorithm for training the RBM, the features of the corrupted images have been refined. In the decoder, the Generator network restores original images from abstracted perceptual features while Discriminator determines how good the restoration result is. The proposed approach has been compared with both traditional approaches like BM3D and with modern deep models like IRCNN and NCSR. We have also considered three different categories of corruption including denoising, inpainting and deblurring. Experimental results confirm performance of the model.
Keywords
[1] Liu, D et al. (2020). Connecting image denoising and high-level vision tasks via deep learning. IEEE Transactions on Image Processing, vol. 29, pp. 3695-3706.
[2] Xie, J., Xu, L. & Chen, E. (2012). Image denoising and inpainting with deep neural networks. Advances in neural information processing systems, pp. 341-349.
[3] Tian, Ch., Xu, Y. Zuo, W. (2020). Image denoising using deep CNN with batch renormalization. Neural Networks, vol. 121, pp.461-473.
[4] Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, L. (2017). Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing, vol. 26, no. 7, pp. 3142-3155.
[5] Cai, W. & Wei., Zh. (2020). PiiGAN: Generative adversarial networks for pluralistic image inpainting. IEEE Access, vol. 8, pp. 48451-48463.
[6] Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T. & Efros, A.A. (2016). Context encoders: Feature learning by inpainting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, pp. 2536-2544, 2016.
[7] Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O. & Li, H. (2017). High-resolution image inpainting using multi-scale neural patch synthesis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, pp. 6721-6729, 2017.
[8] Cai, N., Su, Z., Lin, Z., Hang, H., Yang, Z., & Ling, B.W.K. (2017). Blind inpainting using the fully convolutional neural network. The Visual Computer, vol. 33, no. 2, pp. 249-261.
[9] Wang, Zh., Chen, J. & Hoi. S. (2020). Deep learning for image super-resolution: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 99, pp. 1-22.
[10] Kim, J., Lee, K. & Lee, M. (2016). Accurate image super-resolution using very deep convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, pp. 1646-1654, 2016.
[11] Zeng, K., Yu, J., Wang, R., Li, C., & Tao, D. (2017). Coupled deep auto-encoder for single image super-resolution. IEEE transactions on cybernetics, vol. 47, no. 1, pp. 27-37.
[12] Wan, Sh., Xia, Y., Qi, L., Yang, Y.H. & Atiquzzaman., M. (2020). Automated colorization of a grayscale image with seed points propagation. IEEE Transactions on Multimedia, pp. 1-10.
[13] Zhang, R., Isola, P., & Efros, A.A. (2016). Colorful image colorization. European Conference on Computer Vision, UK, pp. 649-666, 2016.
[14] Isola, P. Zhu, J.-Y., Zhou, T., & Efros, A.A. (2017). Image-to-image translation with conditional adversarial networks. arXiv preprint.
[15] Adam, T. & Paramesran., R. (2020). Hybrid non-convex second-order total variation with applications to non-blind image deblurring. Signal, Image and Video Processing, vol. 14, no. 1, pp. 115-123.
[16] Johnson, J., Alahi, A., & Fei-Fei, L. (2016). Perceptual losses for real-time style transfer and super-resolution. European Conference on Computer Vision, UK, pp. 694-711, 2016.
[17] Gao, R. & Grauman, K. (2017). On-demand learning for deep image restoration. Proc. IEEE Conf. Computer Vision and Pattern Recognition, Honolulu, HI, pp. 1086-1095.
[18] Burger, H.C., Schuler, C.J. & Harmeling, S. (2012). Image denoising: Can plain neural networks compete with bm3d? Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, IR, USA, pp. 2392-2399, 2012.
[19] Yang, J., Wright, J., Huang, T.S., & Ma, Y. (2010). Image super-resolution via sparse representation. IEEE transactions on image processing, vol. 19, no. 11, pp. 2861-2873.
[20] Dong, W., Zhang, L., Shi, G. & Wu, X. (2011). Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization. IEEE Transactions on Image Processing, vol. 20, no. 7, pp. 1838-1857.
[21] Chang, Y., Yan, L., Zhao, X.L., Fang, H., Zhang, Zh., & Zhong, Sh. (2020). Weighted low-rank tensor recovery for hyperspectral image restoration. IEEE Transactions on Cybernetics, pp. 1-15.
[22] Elad, M. & Aharon, M. (2006). Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image processing, vol. 15, no. 12, pp. 3736-3745.
[23] Sivakumar, K. & Desai, U.B. (1993). Image restoration using a multilayer perceptron with a multilevel sigmoidal function. IEEE transactions on signal processing, vol. 41, no. 5, pp. 2018-2022.
[24] Mao, X.-J., Shen, C., & Yang, Y.-B. (2016). Image restoration using convolutional auto-encoders with symmetric skip connections. arXiv preprint arXiv:1606.08921.
[25] Zhang, K., Zuo, W., Gu, S., & Zhang, L. (2017). Learning deep cnn denoiser prior for image restoration. IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, 2017.
[26] Zhang, Y., Tian, Y., Kong, Y., Zhong, B., & Fu., Y. (2020). Residual dense network for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence.
[27] Li, J., Skinner, K.A., Eustice, R.M., & Johnson-Roberson, M. (2018). Watergan: Unsupervised generative network to enable real-time color correction of monocular underwater images. IEEE Robotics and Automation Letters, vol. 3, no. 1, pp. 387-394.
[28] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. & Bengio, Y. (2014). Generative adversarial nets. Advances in neural information processing systems, pp. 2672-2680.
[29] Rudin, L.I., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena, vol. 60, no. 1-4, pp. 259-268.
[30] Chan, T., Esedoglu, S., Park, F., & Yip, A. (2005). Recent developments in total variation image restoration. Mathematical Models of Computer Vision, vol. 17, no. 2.
[31] Oliveira, J.P., Bioucas-Dias, J.M., & Figueiredo, M.A. (2009). Adaptive total variation image deblurring: a majorization-minimization approach. Signal processing, vol. 89, no. 9, pp. 1683-1693.
[32] Sahragard, E., Farsi, H., & Mohammadzadeh, S. (2018). Image restoration by variable splitting based on total variant regularizer. Journal of AI and Data Mining, vol. 6, no .1, pp. 13-33.
[33] Dabov, K., Foi, A., Katkovnik, V. & Egiazarian, K. (2007). Image denoising by sparse 3-d transform domain collaborative filtering. IEEE Transactions on image processing, vol. 16, no. 8, pp. 2080-2095.
[34] Charpiat, G., Bezrukov, I., Altun, Y., Hofmann, M. & SCH, B. (2009). Machine learning methods for automatic image colorization. Computational Photography: Methods and Applications, pp. 395-418.
[35] Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., & Manzagol, P.A. (2010). Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research, vol. 11, pp. 3371-3408.
[36] Jain V. & Seung, S. (2009). Natural image denoising with convolutional networks. Advances in Neural Information Processing Systems, pp. 769-776.
[37] Dong, C., Deng, Y., Change Loy, C., & Tang, X. (2015). Compression artifacts reduction by a deep convolutional network. Proceedings of the IEEE International Conference on Computer Vision, Las Condes, Chile, pp. 576-584, 2015.
[38] Pires, R.G., Santos, D.F., Pereira, L.A., De Souza, G.B., Levada, L.A., & Papa, J.P. (2017).
A robust restricted boltzmann machine for binary image denoising. Graphics, Patterns and Images (SIBGRAPI), 2017 30th SIBGRAPI Conference on, pp. 390-396.
A robust restricted boltzmann machine for binary image denoising. Graphics, Patterns and Images (SIBGRAPI), 2017 30th SIBGRAPI Conference on, pp. 390-396.
[39] Tang, Y., Salakhutdinov, R. & Hinton, G. (2012). Robust Boltzmann Machines for recognition and denoising. Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, RI, 2012, pp. 2264-2271, 2012.
[40] Ulyanov, D., Vedaldi, A. & Lempitsky, V. (2017). Deep image prior. arXiv preprint arXiv:1711.10925.
[41] Basioti, K., & Moustakides, G. V. (2020). Image Restoration from Parametric Transformations using Generative Models. arXiv preprint arXiv:2005.14036.
[42] Pan, X., Zhan, X., Dai, B., Lin, D., Loy, C. C., & Luo, P. (2020). Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation. arXiv preprint arXiv:2003.13659.
[43] Esteva, A., Kuprel, B., Novoa, R.A., Ko, J., Swetter, S.M., Blau, H.M., & Thrun., S. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, vol. 542, no. 7639, p. 115.
[44] Nogueira, K., Penatti, O.A. & dos Santos, J.A. (2017). Towards better exploiting convolutional neural networks for remote sensing scene classification. Pattern Recognition, vol. 61, pp. 539-556.
[45] Krizhevsky, A., Sutskever, I. & Hinton,G. (2012). Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, pp. 1097-1105.
[46] Hinton, G. (2012). A practical guide to training restricted boltzmann machines. Neural networks: Tricks of the trade, pp. 599-619.
[47] Arjovsky M. & Bottou, L. (2017). Towards principled methods for training generative adversarial networks. arXiv preprint arXiv:1701.04862.
[48] Theis, L., Oord, A., & Bethge, M. (2015). A note on the evaluation of generative models. arXiv preprint arXiv:1511.01844.
[49] Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2007). Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space. Image Processing, 2007. ICIP 2007. IEEE International Conference on, Texas, USA, pp. 313, 2007.
[50] Dong, W., Zhang, L., Shi, G., & Li, X. (2013). Nonlocally centralized sparse representation for image restoration. IEEE Transactions on Image Processing, vol. 22, no. 4, pp. 1620-1630.
[51] Dodge, S. & Karam, L. (2017). A study and comparison of human and deep learning recognition performance under visual distortions. Computer Communication and Networks (ICCCN), 2017 26th International Conference on, Canada, pp. 1-7, 2017.
[52] Levin, A., Weiss, Y., Durand, F., & Freeman, W.T. (2009). Understanding and evaluating blind deconvolution algorithms. 2009 IEEE Conference on Computer Vision and Pattern Recognition, FL, USA, pp. 1964-1971, 2009.
)