Inpainting large missing regions based on Seam Carving

Inpainting techniques are developed to recover missing image information. Existing inpainting approaches are: Partial Differential Equations Based Inpainting (PDE-BI) and Exemplar-Based Inpainting (EBI). PDE-BI methods used to fill in the missing information via information propagation from neighbouring areas. However, it can only reconstruct successfully small missing regions that are surrounded by limited texture. However, EBI methods are used to recover large regions with richly-textured/structured areas around them, moreover, artefacts are likely to occur. This paper proposes a technique to reduce the missing region size based on seam carving approach, which enables EBI and PDE-BI to recover the missing part. In our proposal, seam carving is used to reduce only the size of the missing region, to be subsequently recovered using EBI method. The added extra paths resulting from the added seams is repaired using PDE-BI. This method outperformed the state-of-art EBI methods.


Introduction
Nowadays, users of many photo sharing platforms may want to edit their photos before sharing them.Editings may include deletion of distracting scene elements and recovery of content in certain image regions.To perform such operations and many others, one might need to automatically fill-in specific regions in the image in an acceptable way.The art of modifying an image in a way which cannot be distinguished by the human eye, or an ordinary detector, is known as image inpainting.Roughly speaking, image inpainting is deployed in most of the cases where automatic region-filling is required, due to its inherent ambiguity and to the complexity of natural images.Image inpainting problem remains an active area of research and remains challenging in recovering large missing regions.The applications of the image inpainting are old image restoration, adding colours to white and black images (even movies), and removal of the unwanted objects from an image.This process will be carried out based on information captured from the other parts of the image.There are several image inpainting approaches which have been improved in recent years; they are roughly classified into two main types: non-Exemplar Based Inpainting (nEBI) and EBI methods.
The first type, nEBI methods uses Partial Differential Equations (PDE) to propagate the information from the surrounding (or neighbouring areas of the missing region) to the missing region.In the last two decades, researchers have proposed new methods based on PDE to restore old images with small cracks, or to remove unwanted information's, i.e. dates, texts, advertising signs, etc. as in [1], [2], [3], [4], and [5].Bertalmio et al. in [1], were the first to use the notion of digital image inpainting using 3rdorder PDE to recover damaged regions along the directions of isophotes.Total Variation (TV) and Curvature Driven Diffusion (CDD) models have been introduced by Chan and Shen in [2], and [3] respectively.Inpainting using TV will preserve edges and lines better than other techniques.CDD model which is an improved version of the TV model will enable the inpainting to recover bigger areas.The second order (linear and non-linear) PDEs are used to recover the missing regions because they are easy to calculate, such as isotropic, anisotropic and TV equations.However, second-order PDEs can only reconstruct small regions, and they will fail to deal adequately with edges and corners.The authors in [6] proposed a mathematical model based on a combination of both TV and CDD ideas, known as Mumford-Shah-Euler model (MSEm), which is a fourth order PDE.This model succeeded in recovering edges and corners, however, could not recover the large missing regions with the surrounding areas which are richly textured.Nevertheless, the fourth order models showed better visual results than other compared with second-order models for recovering corners or edges in the missing regions [7].Secondly, the EBI methods were proposed for the task of recovering large missing regions.In [8] Criminisi et al.
were the first to propose the idea of using a patch-basedexemplar method, with which they simultaneously reconstructed textures and structures in the missing region.This task depends mainly on the choice of filling order; an optimal choice will ensure that linear structures will be propagated before texture filling so that the connectivity and continuity of object boundaries will be preserved.Briefly, The EBI method works based on priority function (i.e. it uses data term and confidence term to locate better patch place to recover) and matching criteria.The data term encourages reconstruct the structure (the geometry) in the missing regions.This approach is not effective when one wants to remove a large object from an image whose surrounding area is richly textured.There are several studies concerning the optimisation of the method in [8], for which some improvements have been proposed.The improved methods may be categorised into two types.In the first category those which have been focused on improving texture reconstruction as in [9], [10], [11], [12], and [13].whereas those in the second category are a modification of the work in [8], the aim being to restore the structure from the surrounding areas into the missing region more accurately [14], [15], [16], [17], and [18].Other studies have worked on the efficiency of the method in [8].For instance, Waykule et al. [15] proposed a new method for eliminating big objects from images.This is done by redefining the data term by using a bi-dimensional Gaussian kernel filter [9] on the positions of the control points of contour into the damaged region.In a recent work, Z. Xu and S. Jian [16] proposed a gradient-based searchspace reduction.The spatial behaviour of selected regions to be inpainted is controlled by a gradient vector.In addition, Gaikar el al. [19] used three inpainting methods which are: the EBI method, the discrete cosine transformbased inpainting technique, and the fast-marching methodbased inpainting technique.These methods were applied together to remove unnecessary objects from the damaged image and extend the background with visually acceptable results.In this paper, we are proposing a novel inpainting technique to deal with large missing regions.The technique is built based on the seam-carving concept which enables the inpainting method to reconstruct a relatively large missing region without artefacts.Seam carving is a method for content-awareness of the image resizing and is used for both image size reduction and expansion, moreover seemcarving has been used for image enhancement and object removal, which will affect the image size relatively after the removal; more details can be found in section 4. The rest of the paper is organized as follows.Section 2 introduces a review of PDE-BI methods.A traditional EBI method is illustrated in section 3. Analysis of the seamcarving operator is presented in section 4. Our proposal is introduced in section 5.The results of the proposed technique are explained in section 6.The performance of the proposed technique is examined in section 7. Finally, the conclusion and future directions are presented in section 8.

PDE Based-Inpainting methods
The PDE-BI technique is one of the oldest inpainting techniques.It usually deals with small missing regions in an image with limited textured areas around them, while these techniques are also applied for reconstructing lines, edges and corners.PDE-BI techniques were proposed based on variational methods (energy methods).In the variational methods, the image-inpainting problem is described as the minimisation of a variational functional via the application of the Euler-Lagrange equation, which will lead to a PDE.Finally, the numerical solution of this PDE will reconstruct the missing information in an image.The performance of high order PDE inpainting models is more effective as compared to the second order PDE models with regard to the preservation and continuation of edges and lines during the reconstructions of the missing regions in an image [7].Hence, we will use high order inpainting models, such as MSEm.This model has the ability to deal with large-scale image inpainting problems, and it has been proposed in [6] based on the need to accommodate curvature so as to overcome the issue of large missing regions, and also to accurately recover borders of the missing region in an image.MSEm is proposed to improve the shortcomings of the Mumford-Shah model by enhancing its embedded curve model with Euler's Elastica curve model, for more details about this model see [6].MSEm model successfully recovered the lines, and corners in the small missing regions of the treated image.The EBI method is applied to reconstruct large missing region with simple texture and structure areas around them as explained in the next section.method is based on what is known as a "priority function" with which to select a patch ψ p (template patch) that needs to be recovered first, and then finding outside of the missing region the nearest patch (target patch) that has similar information based on sum-of-squared distances (for example ψ q′ or ψ q′′ ) to be selected.As in [8], the patch size has been fixed to 9-by-9 pixels.Generally, the EBI method can only be applied to images which have simple texture and structure, as otherwise visually unacceptable artefacts will be produced while filling a missing region [8].[8].Given the patch   ,   is the normal to the contour Ω of the target region Ω and   ⊥ is the isophote (direction and intensity) at point .The entire image is denoted with .

Figure 1. EBI notation diagram in
To overcome the drawbacks facing the EBI process, we proposed an improvement to the method regarding the size of patch propagation, thereby improving the priority function and matching criteria and reducing the artefact problem to an unnoticeable level.To accomplish this task, we applied the emerging Topological Data Analysis (TDA) scheme to adaptively select a patch size when applying EBI; more details can be found in [20].This Topological based EBI (TEBI) method succeeded in recovering large missing regions and in removing unwanted objects but could not recover the missing regions with the high texture areas surrounding them when the size of the missing region is above 20% of the total area of the image.Nevertheless, the TEBI method showed better visual results than other compared with the state-of-the-art EBI methods for the case of the missing region with high texture areas surrounding it.However, to handle a larger missing region, in this work, we try to reduce the size of the missing region without affecting the other parts of the image, and this allows the original TEBI method to work effectively.This resizing will be based on the seam carving method described in the section below.

Seam carving approach
The seam carving method is a method used for image resizing which can be used for both reduction and expansion of an image without affecting the contents of the image.The basic idea of seam carving is to remove the redundant seams which are not noticeable visually.A seam is an optimal 8-connected path of pixels on an image from left to right and/or from top to bottom of the image, where an image energy function defines the optimality of the seam.This approach works by removing optimal seams [21].Roughly speaking, the seam carving procedure will preserve pixels that have high energies in comparison with their surrounding pixels.Mathematically speaking, the energy of a pixel with respect to the  and  axes may be computed through the magnitudes of the derivatives with respect to  and  determined by the pixel's neighbourhood, which is known as gradient operator.There are different definitions of energy function which have been proposed in these papers [22], [23], [24], [25], and [26].A Full explanation of the seam carving procedure can be seen in [21].Below, we only present the mathematical equation for the energy function; let  be an  ×  image: It is worth noting that the type of energy function used to remove seams from a missing region is based on calculating energy function from I, once by gradient and then by entropy operators, then adding both of them together as clarified in Figure 2 The pixels of the path of seam  , for instance, a vertical seam {S i } will be Observe that, after the removal of row or column from an image, the missing seam will be compensated by shifting all the pixels of the image left or up.The energy of a seam is defined as follows: The optimal seam (path)  * , which minimises total seam energy of each pixel in the path is defined by the next formula: The general aim of using the seam carving operator on an image is to resize the whole image for extension or shrinking.In our work, we will use seam carving operator to reduce the size of the occluded region to serve the inpainting method.Figure 2 shows the application of seam carving to the occluded region in the image.Many inpainting methods have limitations with large missing regions.Based on our observations from the literature, the largest missing region that can be recovered using the TEBI method is of an area corresponding to 20% of the total area of the original image.Therefore, we are proposing to reduce the size of the missing region using the seam carving method, after which the TEBI method is applied to reconstruct the reduced size missing region.To recover the original size of the image, the old seams will be added back; this will leave missing thin lines that can be recovered using PDE-BI method.The method of our technique will be explained in the next section.

The proposed technique
The steps of our technique are represented as follows:  Read the input image with a marked region to be removed. The seam carving approach is applied to shrink the missing region size. The TEBI method is used to recover the shrunk missing region in the image. Add back the old seams to recover the original size of the image.However, the original information of the added seams is missing. MSEm is applied to reconstruct the missing paths (seams).
Whether the shrinking of the missing region occurs vertically or horizontally is based on the nature of the shape of the missing region in an image.The steps of the proposed technique are described in Figure 3.The proposed technique can be used for large object removal and for recovering large missing region in an image.The experimental results are introduced to check the effectiveness of this technique and comparing it with the results described in [8] and [20].

Experimental results
In this section, we tested the proposed technique on a variety of 100 natural images, selected from Berkeley Segmentation Dataset and Benchmarks 500 (BSDS500), see [27], and compared the experimental results of our approach with that of the classical EBI method [8] and TEBI method [20].The proposed technique is used to remove large unwanted objects, and to reconstruct missing regions of various large sizes.The above results show that the proposed technique outperforms EBI and TEBI methods.However, the number of seams to be removed is limited as it is directly related to the performance of the MSEm.In the case of using a large number of seams, MSEm produces artefacts when the seams are condensed next to each other.We found by experiment that the number of seams is better to be limited between 20% and 30% of the size of the missing region.Table 1 presents the size of the missing regions to the whole image in f Figure 4, before and after using the seam carving method.
Table 1.The size of the missing regions to the whole images in Figure 4 before and after reduction using seam carving approach.column (c) inpainted images using EBI method, column (d) inpainted images using TEBI method, column (e) inpainted images using the proposed technique.
Table 2 introduces the size of missing regions to the whole image before and after using the seam carving approach on the natural images that are described in figure 5.
Table 2.The size of the missing regions to the whole images in figure 5 before and after reduced using seam carving approach.From the several experiential works we had conducted in image inpainting, we observed that there is a clear logical link between the geometry directions of the surrounding areas of the missing region and the direction of the removed seams used in seam carving method.More precisely, the direction of seams can be determined based on the structure directions of the surrounding areas of missing regions of the image; this helps the patch selection to be propagated better.As an, e.g. the size of the missing region in row 2 and 4 in figure 4 is reduced vertically by removing the vertical seams.While we can see in row 3 and 4 in figure 4 and 5, respectively, the size of the missing region has been reduced horizontally.Also, we reduced the size of the missing regions in both directions as seen in row1 figure 4. Therefore, the proposed technique outperforms the original EBI and TEBI methods, especially when the size of the missing region is relatively large and the surrounding area of the missing region has high texture and structure as seen in figures 4 and 5.
More precisely, as it can be seen in the first and fourth rows images in f Figure 4 and the second and third rows images in figure 5, the proposed method successfully reconstructed the missing regions in column (d), but there is a need to continue the edges from the outside of the missing region to the inside.This shows that our proposed technique can successfully reconstruct sharp edges sequentially even when the missing region is relatively large due to using seam carving approach which enhances good patch-size propagation selection using topological invariants.As a result, the priority function determines the best location in which to propagate the information steadily.The missing region can now be reconstructed fully based on the information in the surrounding area.Next, it is necessary to check the quality of the restored image so that one can check the suitability of the inpainting method as well as whether the produced image is visually

Inpainting Quality Assessment
An accurate evaluation method which simultaneously assesses inpainted images qualitatively and quantitatively is not an easy task and has not been fully solved yet [28].Therefore, we depended on visual analysis to assess inpainted images qualitatively.However, for quantitative evaluation, we used different statistical measurements to evaluate the quality of the inpainted images.These inpainted images have been evaluated based on the presence and the absence of the reference image.Firstly, the Similarity Inpainting Measurement (SIM) [29], Entropy [30] and Coherence Structure Quality Measurement (CSQM) [29] have been used to check the quantitatively quality of these inpainted images in the absence of the reference images.Secondly, the Peak Signal-to-Noise Ratio (PSNR), and Structural Similarity (SSIM) have used to evaluate the quality of the inpainted image when having reference images.Table 3 below summarises the measurements of Entropy, SIM, and CSQM to compare our technique with the methods in [8] and [20].[30].The proposed technique obtained lower entropy value than the methods described in [8] and [20].Also, the SIM is used to study the coherence extent of the inpainted region with the rest of the image.While CSQM has used characteristics of the visual coherence of the inpainted regions and the visual saliency describing the visual importance of the inpainted region.
The high values of SIM and CSQM represent better results [29].The proposed technique obtained higher values of SIM and CSQM than the methods in [8] and [20], this means the inpainted regions obtained by our technique is more coherence with the rest of the images.clearly show that the proposed technique is outperforming the methods in [8] and [20].However, the proposed technique takes a bit more time due to the amount of calculation during the matching stage.During the testing, it was found that while some images could look visually pleasing and alike, although they have different PSNR values.
The performance of the proposed technique has dramatically improved in reconstructing the edges and corners in the large missing regions.The reduced size of the missing regions introduces massive assistance and allows good patch propagation selection.We directed the seam carving approach to reduce the size of the missing region vertically if we want to reconstruct it horizontally and that helps the patch selection to propagate better as seen in figure 5.However, the seam carving approach has been applied to reduce the size of the missing regions horizontally, if we want to reconstruct the missing regions vertically.

Conclusion and Future work
We proposed a novel technique to reconstruct large missing regions in the natural images using seam carving.This technique can be used to recover large missing regions with high texture contents around them.Since most of the existing methods cannot recover large missing regions, the size of the missing region is reduced by using seam carving approach.Next TEBI method is used to recover the missing region.Additionally, MSEm is used to recover the seam lines after adding them back to the inpainted image.This technique has been tested on 100 natural images with visually acceptable results.Furthermore, the proposed method shows better performance than using the EBI method without the resizing methods as in [8 ] and [20].In the future, better energy functions can be used to define the seams in an image without changing its contents.In order to develop a seamless application, it is better to have an automatic technique for reducing the size of the missing areas using seam carving method based on the relation between the direction of seams and the geometry direction in the surrounding areas of the missing region.The gradient magnitude can be used to study the geometry direction of surrounding areas which help to choose a good direction for the seams.Also, this work can be enhanced by using other kinds of PDE methods.Furthermore, we aim to use TDA approach to check the quality of different inpainting methods instead of using only statistical measurements.Finally, this technique can be extended to recover bigger missing regions.
Many image inpainting schemes restore the patches in images by propagating geometrical structures into the missing region through diffusion, which is inspired by the partial differential equations of physical heat flow.The EBI EAI Endorsed Transactions on Industrial Networks and Intelligent Systems 09 2018 -11 2018 | Volume 5 | Issue 16| e4

Figure 2 .
Figure 2. Seam carving steps.(a) Original image.(b) The energy of the original image.(c) Determine 20 vertical seams.(d) Determine 20 horizontal seams.e) Remove 20 horizontal seams.(f) Remove 20 vertical seams.Many inpainting methods have limitations with large missing regions.Based on our observations from the literature, the largest missing region that can be recovered using the TEBI method is of an area corresponding to 20% of the total area of the original image.Therefore, we are proposing to reduce the size of the missing region using the seam carving method, after which the TEBI method is applied to reconstruct the reduced size missing region.To recover the original size of the image, the old seams will be added back; this will leave missing thin lines that can be recovered using PDE-BI method.The method of our technique will be explained in the next section.

Figure 3 .
Figure 3. Inpainting-based seam carving technique, (a) Original image.(b) Masked image (image with the occluded region).(c) Shrinking occluded region by removing 30 seams.(d) Inpainted image of the shrunk region using TEBI method.(e) Returning the seams to the inpainted region.(f) Inpainting of image seams using MSEm.The proposed technique can be used for large object removal and for recovering large missing region in an image.The experimental results are introduced to check the effectiveness of this technique and comparing it with the results described in[8] and[20].

Figure 4
Figure 4 presents some results using the proposed technique, EBI and TEBI methods for large object removal.It is considered as a visual comparison of their performances.

Table 4 .
Inpainted image quality assessment comparison using PSNR, SSIM, and CSQM.Table 3 proves that our proposed technique is capable of effective region filling and gives high CSQM values with low Entropy values, and the SIM values are close to 1.The entropy represents the amount of disorder in the inpainted image.Therefore, generally speaking, lower entropy values are better than higher entropy