An Architectural model for the sketch image colorization with deep learning

Abstract

Image colorization can be considered a vital task in several industries such as the comic, animation, and technical illustration industries. Because of that, it is a field of research with a considarable market demand. There are several fields of image coloring available such as grayscale image coloring and line sketch image coloring. Unlike grayscale image coloring, sketch image colorization is a challenging task because sketch images do not have any color-related detail such as the texture and color gradient. This research proposes the software architecture of a novel technique to automate sketch image colorization with deep learning concepts. The research aims to automate the sketch colorization process while maintaining a higher accuracy compared to existing image coloring methods. This also tries to reduce the identified color errors in the majority of the currently available sketch colorization methods including water-color blurring, color distortion and bleeding. The method which is proposed for sketch image coloring contains two steps. The first step is called the colorizing step which is responsible for initially coloring the sketch image. In this stage, the model assumes the color ranges and sprays a rich selection of colors on the sketch. The output of this stage is almost completely free from color faults such as color bleeding and watercolor blurring. But it may include mistakes such as unmatching differences in luminance or the unfitting colors that make an object distinguishable from other objects. To correct that issue, the output of the colorization stage is sent to the refinement stage. In that stage, the image is enhanced with a suitable contrast value and outputs the image with good quality and higher accuracy. To develop this model, two types of deep neural architectural models were adopted. For colorization model, the architecture of conditional Generative Adversarial Network (cGAN) has been used. This is a kind of Generative Adversarial Network architecture which additionally makes use of the input labels. For the refinement model, Convolutional Neural Network (CNN) architecture is used in this research. In this step, the model learns to predicts the most suitable contrast value for the given input image.With the new contrast value, it mostly fix the luminance issues and unfitting color issues that can be remaining in the output of the coloring step. The components in the two steps mentioned, are developed in a way that they can be used independently as well. For verifying the accuracy of the results, two types of evaluation methods have been used. They are peak-signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) methods which are widely used in many of the image colorization researches. With these evaluation methods the ground truth is compared with the output image and obtained a value to measure how similar these images are. Also in this research, outputs of a state-of-the-art methods over the several years are compared with the outputs of the proposed method in this research.