基于像素级生成对抗网络的复杂场景灰度图像彩色化
|
摘要
针对当前基于深度学习的彩色化模型在面对具有多个目标的复杂场景时存在的误着色问题,提出一种基于像素级生成对抗网络的彩色化模型.该模型在生成网络中采用全卷积网络模型处理不定尺度的输入灰度图像,并加入与真实彩色分量间的L1损失作为彩色化优化目标;在判别网络中,采用语义分割网络计算像素级Softmax损失,反向传递优化彩色化生成网络.在Pascal Segmentation及ILSVRC2012数据集上进行的彩色化图像质量比较,实验结果表明,与同类模型相比,本文模型在处理复杂场景灰度图像的彩色化任务中具有更高的着色准确率,并且对不同目标之间具有更好的区分度.
Traditional deep learning based colorization models may cause mistaken coloring in dealing with complex scenarios. For this problem, we proposed a pixel-wise generative adversarial network based colorization method. Firstly, we built a fully convolutional network for the generative model to deal with grayscale images of uncertainty scale. Moreover, the L1 loss between the output color maps and the real color components was calculated as the optimization goal. Secondly, we utilized a semantic segmentation network to build the discriminative model, of which a pixel-wise Softmax loss was calculated and propagated back to improve the performance of the colorization model for a better coloring output. Experimental results of color image quality comparison on Pascal Segmentation and ILSVRC2012 datasets show that the proposed colorization model achieves a higher accuracy and better discrimination between different objects compared with other colorization models.
Traditional deep learning based colorization models may cause mistaken coloring in dealing with complex scenarios. For this problem, we proposed a pixel-wise generative adversarial network based colorization method. Firstly, we built a fully convolutional network for the generative model to deal with grayscale images of uncertainty scale. Moreover, the L1 loss between the output color maps and the real color components was calculated as the optimization goal. Secondly, we utilized a semantic segmentation network to build the discriminative model, of which a pixel-wise Softmax loss was calculated and propagated back to improve the performance of the colorization model for a better coloring output. Experimental results of color image quality comparison on Pascal Segmentation and ILSVRC2012 datasets show that the proposed colorization model achieves a higher accuracy and better discrimination between different objects compared with other colorization models.
引文
[1]Horiuchi T, Hirano S. Colorization algorithm for grayscale image by propagating seed pixels[C]//Proceedings of the InternationalConferenceonImageProcessing.LosAlamitos:IEEE Computer Society Press, 2003, 1:I-457
[2]Levin A, Lischinski D, Weiss Y. Colorization using optimization[J]. ACM Transactions on Graphics, 2004, 23(3):689-694
[3]MarkleW,HuntB.Coloringablackandwhitesignalusing motion detection:Canadian, CA 1291260[P]. 1991-10-22
[4]LiJianming,YeFei,YuShouqiu,etal.Afastalgorithmof transferring color to greyscale images[J]. Journal of Image and Graphics, 2007, 12(3):536-540(in Chinese)(李建明,叶飞,于守秋,等.一种快速灰度图像彩色化算法[J].中国图象图形学报, 2018, 12(3):536-540)
[5]ChengZZ,YangQX,ShengB.Deepcolorization[C]//ProceedingsoftheIEEEInternationalConferenceonComputerVision.LosAlamitos:IEEEComputerSocietyPress,2015:415-423
[6]ZhangR,IsolaP,EfrosAA.Colorfulimagecolorization[C]//Proceedings of the European Conference on Computer Vision.Heidelberg:Springer, 2016:649-666
[7]Zhang R, Zhu J Y, Isola P, et al. Real-time user-guided image colorization with learned deep priors[J]. ACM Transactions on Graphics, 2017, 36(4):Article No.119
[8]Iizuka S, Simo-Serra E, Ishikawa H. Let there be color!:joint end-to-end learning of global and local image priors for automaticimagecolorizationwithsimultaneousclassification[J].ACM Transactions on Graphics, 2016, 35(4):Article No.110
[9]LarssonG,MaireM,ShakhnarovichG.Learningrepresentations for automatic colorization[C]//Proceedings of the EuropeanConferenceonComputerVision.Heidelberg:Springer,2016:577-593
[10]Shelhamer E, Long J, Darrell T. Fully convolutional networks forsemanticsegmentation[J].IEEETransactionsonPattern Analysis and Machine Intelligence, 2017, 39(4):640-651
[11]Isola P, Zhu J Y, Zhou T H, et al. Image-to-image translation with conditional adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press, 2017:5967-5976
[12]Luan T, Yin X, Liu X M. Disentangled representation learning GANforpose-invariantfacerecognition[C]//Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition.LosAlamitos:IEEEComputerSocietyPress,2017:1283-1292
[13]Huang R, Zhang S, Li T Y, et al. Beyond face rotation:global and local perception GAN for photorealistic and identity preservingfrontalviewsynthesis[C]//ProceedingsoftheIEEE InternationalConferenceonComputerVision.LosAlamitos:IEEE Computer Society Press, 2017:2458-2467
[14]KimT,ChaM,KimH,etal.Learningtodiscover cross-domainrelationswithgenerativeadversarialnetworks[OL].[2018-05-11]. https://arxiv.org/abs/1703.05192
[15]Liu Yujie, Dou Changhong, Zhao Qilu, et al. Sketch based image retrieval with conditional generative adversarial network[J].JournalofComputer-AidedDesign&ComputerGraphics,2017, 29(12):2336-2342(in Chinese)(刘玉杰,窦长红,赵其鲁,等.基于条件生成对抗网络的手绘图像检索[J].计算机辅助设计与图形学学报, 2017, 29(12):2336-2342)
[16]Li Ce, Zhao Xinyu, Xiao Limei, et al. Generative adversarial mapping nets with multi-layer perception for image dehazing[J].JournalofComputer-AidedDesign&ComputerGraphics,2017, 29(10):1835-1843(in Chinese)(李策,赵新宇,肖利梅,等.生成对抗映射网络下的图像多层感知去雾算法[J].计算机辅助设计与图形学学报, 2017,29(10):1835-1843)
[17]Goodfellow I. NIPS 2016 tutorial:generative adversarial networks[OL].[2018-05-11]. http://cn.arxiv.org/abs/1701.00160
[18]Ren S Q, He K M, Girshick R, et al. Faster R-CNN:towards real-timeobjectdetectionwithregionproposalnetworks[C]//Proceedingsofthe28thInternationalConferenceonNeural Information Processing Systems. Cambridge:MIT Press, 2015,1:91-99
[19]Goodfellow I J, Pouget-Abadie J, Mirza M, et al. Generative adversarialnets[C]//Proceedingsofthe27thInternational Conference on Neural Information Processing Systems. Cambridge:MIT Press, 2014, 2:2672-2680
[20]Everingham M, van Gool L, Williams C K I, et al. The Pascal visual object classes(VOC)challenge[J]. International Journal of Computer Vision, 2010, 88(2):303-338
[21]Russakovsky O, Deng J, Su H, et al. ImageNet large scale visual recognition challenge[J]. International Journal of Computer Vision, 2015, 115(3):211-252
[22]Wang Z, Bovik A C, Sheikh H R, et al. Image quality assessment:fromerrorvisibilitytostructuralsimilarity[J].IEEE Transactions on Image Processing, 2004, 13(4):600-612
[2]Levin A, Lischinski D, Weiss Y. Colorization using optimization[J]. ACM Transactions on Graphics, 2004, 23(3):689-694
[3]MarkleW,HuntB.Coloringablackandwhitesignalusing motion detection:Canadian, CA 1291260[P]. 1991-10-22
[4]LiJianming,YeFei,YuShouqiu,etal.Afastalgorithmof transferring color to greyscale images[J]. Journal of Image and Graphics, 2007, 12(3):536-540(in Chinese)(李建明,叶飞,于守秋,等.一种快速灰度图像彩色化算法[J].中国图象图形学报, 2018, 12(3):536-540)
[5]ChengZZ,YangQX,ShengB.Deepcolorization[C]//ProceedingsoftheIEEEInternationalConferenceonComputerVision.LosAlamitos:IEEEComputerSocietyPress,2015:415-423
[6]ZhangR,IsolaP,EfrosAA.Colorfulimagecolorization[C]//Proceedings of the European Conference on Computer Vision.Heidelberg:Springer, 2016:649-666
[7]Zhang R, Zhu J Y, Isola P, et al. Real-time user-guided image colorization with learned deep priors[J]. ACM Transactions on Graphics, 2017, 36(4):Article No.119
[8]Iizuka S, Simo-Serra E, Ishikawa H. Let there be color!:joint end-to-end learning of global and local image priors for automaticimagecolorizationwithsimultaneousclassification[J].ACM Transactions on Graphics, 2016, 35(4):Article No.110
[9]LarssonG,MaireM,ShakhnarovichG.Learningrepresentations for automatic colorization[C]//Proceedings of the EuropeanConferenceonComputerVision.Heidelberg:Springer,2016:577-593
[10]Shelhamer E, Long J, Darrell T. Fully convolutional networks forsemanticsegmentation[J].IEEETransactionsonPattern Analysis and Machine Intelligence, 2017, 39(4):640-651
[11]Isola P, Zhu J Y, Zhou T H, et al. Image-to-image translation with conditional adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press, 2017:5967-5976
[12]Luan T, Yin X, Liu X M. Disentangled representation learning GANforpose-invariantfacerecognition[C]//Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition.LosAlamitos:IEEEComputerSocietyPress,2017:1283-1292
[13]Huang R, Zhang S, Li T Y, et al. Beyond face rotation:global and local perception GAN for photorealistic and identity preservingfrontalviewsynthesis[C]//ProceedingsoftheIEEE InternationalConferenceonComputerVision.LosAlamitos:IEEE Computer Society Press, 2017:2458-2467
[14]KimT,ChaM,KimH,etal.Learningtodiscover cross-domainrelationswithgenerativeadversarialnetworks[OL].[2018-05-11]. https://arxiv.org/abs/1703.05192
[15]Liu Yujie, Dou Changhong, Zhao Qilu, et al. Sketch based image retrieval with conditional generative adversarial network[J].JournalofComputer-AidedDesign&ComputerGraphics,2017, 29(12):2336-2342(in Chinese)(刘玉杰,窦长红,赵其鲁,等.基于条件生成对抗网络的手绘图像检索[J].计算机辅助设计与图形学学报, 2017, 29(12):2336-2342)
[16]Li Ce, Zhao Xinyu, Xiao Limei, et al. Generative adversarial mapping nets with multi-layer perception for image dehazing[J].JournalofComputer-AidedDesign&ComputerGraphics,2017, 29(10):1835-1843(in Chinese)(李策,赵新宇,肖利梅,等.生成对抗映射网络下的图像多层感知去雾算法[J].计算机辅助设计与图形学学报, 2017,29(10):1835-1843)
[17]Goodfellow I. NIPS 2016 tutorial:generative adversarial networks[OL].[2018-05-11]. http://cn.arxiv.org/abs/1701.00160
[18]Ren S Q, He K M, Girshick R, et al. Faster R-CNN:towards real-timeobjectdetectionwithregionproposalnetworks[C]//Proceedingsofthe28thInternationalConferenceonNeural Information Processing Systems. Cambridge:MIT Press, 2015,1:91-99
[19]Goodfellow I J, Pouget-Abadie J, Mirza M, et al. Generative adversarialnets[C]//Proceedingsofthe27thInternational Conference on Neural Information Processing Systems. Cambridge:MIT Press, 2014, 2:2672-2680
[20]Everingham M, van Gool L, Williams C K I, et al. The Pascal visual object classes(VOC)challenge[J]. International Journal of Computer Vision, 2010, 88(2):303-338
[21]Russakovsky O, Deng J, Su H, et al. ImageNet large scale visual recognition challenge[J]. International Journal of Computer Vision, 2015, 115(3):211-252
[22]Wang Z, Bovik A C, Sheikh H R, et al. Image quality assessment:fromerrorvisibilitytostructuralsimilarity[J].IEEE Transactions on Image Processing, 2004, 13(4):600-612