ERC Advanced Grant INCOVID:
Inpainting-based Compression of Visual Data

Generating huge amounts of visual data, be it images or videos, has never been easier than today. This creates a growing demand for lossy codecs (coders and decoders) that produce visually convincing results also for very high compression rates. Popular transform-based codecs such as JPEG and JPEG 2000 have reached a state where one cannot expect significant improvements anymore. To go beyond their limitations, fundamentally different ideas are needed.

Inpainting-based codecs can change this situation. They store only a small, carefully optimised part of the data. In the decoding step, the missing information is filled in with a suitable inpainting mechanism. A successful realisation of inpainting-based codecs can offer decisive advantages over transform-based codecs: The stored information is more intuitive and closer to the mechanisms of human perception. Moreover, the concept is very flexible: It allows to integrate a number of different features and can be tailored towards dedicated applications. Most importantly, the higher the compression rate, the larger are the qualitative advantages over transform-based codecs. However, the potential of these codecs was widely unexplored, since difficult fundamental problems had to be solved first.

The INCOVID project has addressed these challenges in an integrated approach that covers a wide spectrum of aspects. We have established important theoretical foundations, introduced better data selection strategies, developed dedicated codecs for specific applications, implemented highly efficient numerical algorithms, and created a demonstrator that offers real-time performance in 4K resolution. These achievements show that inpainting methods are far more than a visually pleasant image editing tool: They have become the key component of a well-founded and powerful alternative paradigm in coding. It can play an important role in future codecs that we will use in our daily life.

This project was funded by the EU through the ERC Advanced Grant INCOVID for Joachim Weickert (Grant Agreement No. 741215). Its budget for research was 2.46 million Euro, to be spent over a period of five years (from October 1, 2017 until September 30, 2022).

The INCOVID project has accomplished its envisioned goals in all four key areas: It has led to fundamental theoretical insights, novel approaches with substantial quality improvements, extensions to new application areas, and highly efficient algorithms that allow a real-time demonstrator. Here is a list of the highlights:

I. Establishing the Foundations

• We have introduced a general framework for pseudodifferential inpainting. This new inpainting paradigm connects inpainting using linear shift invariant differential operators with radial basis function interpolation (Augustin et al. 2019).

• Among the nonlinear inpainting operators, edge-enhancing anisotropic diffusion (EED) is known to performs particularly well. We have established a rigorous existence theory for this nonlinear integrodifferential equation (Bildhauer et al. 2021).

• We have laid the theoretical foundations of inpainting-based codecs in terms of sparsification and quantisation scale-spaces (Cárdenas et al. 2019, Peter 2021). This has strengthened ties between two hitherto fairly unconnected scientific communities: the coding community and the scale-space community

• We have extended the Mumford-Shah model to image compression. The resulting energy functional serves as an important step towards a full optimisation model that also incorporates the coding cost (Jost et al. 2020).

II. Pushing the Quality to the Limit

• The aforementioned Mumford-Shah based codec of Jost et al. (2020) is particularly well-suited for piecewise smooth data, e.g. depth maps. Here it significantly outperformed state-of-the-art codecs such as HEVC by up to 3 dB.

• We have pioneered data optimisation for exemplar-based inpainting. It introduces densification by dithering of error maps, and it allows an inpainting-based sparse representaton of highly textured data in unprecented quality (Karos et al. 2018).

• Deep learning methods for selecting inpainting data have been (Alt et al. 2022, Peter et al. 2022, Peter et al. 2023, Schrader et al. 2023). They offer a similar quality as model-based approaches at a substantially reduced runtime.

• We have achieved a breakthrough in the simultaneous incorporation of different feature types for inpainting-based image representations (Jost et al. 2023). This approach is simple, fairly general, and provides both spatial and tonal optimisation. Moreover, it allowed us to find a novel promising feature type based on local averages.

III. Extending the Spectrum

• Various alternative data feature types have been evaluated in student theses, e.g. isophotes (Nebel, 2019), local binary patterns (Sirazitdinov, 2019), and junctions or corners (Gusenburger, 2020). They have merits for specific imagery.

• Inpainting-based compression has been extended to audio signals where it offers competitive performance in terms of PSNR (Peter et al. 2019). Moreover, we have established novel algorithms for piecewise constant signal approximations with superior quality (Bergerhoff et al. 2019).

• As a component for inpainting-based video compression, we have proposed a novel edge-aware inpainting technique that allows to represent motion fields in high quality (Jost et al. 2020). Moreover, we have introduced a video codec that outperforms previous inpainting-based ones in both quality and speed (Andris et al., SSVM 2021).

IV. Algorithms, Prototypes, and Demonstrators

• We have performed a systematic evaluation of coding strategies for sparse inpainting masks (Mohideen et al. 2021), in order to find and adapt the most powerful ones.

• We have developed an neural surrogate algorithm for inpainting with Euler's elastica that is simpler than classical ones, but offers competitive quality (Schrader et al. 2022). Moreover, we have also introduced a novel hybrid codec that combines the advantages of inpainting- and transform-based compression methods (Andris et al., PCS 2021).

• Exemplar-based inpainting offers good quality for highly textured images, but suffers from slow nonlocal algorithms. As a remedy, we have introduced space-filling curves as acceleration structures (Dahmen et al. 2020).

• Highly efficient codecs have been developed that combine Shepard interpolation with joint inpainting and prediction (Peter 2019).

• For homogeneous diffusion inpainting, we have developed fast finite element algorithms that exploit adaptive triangulations (Chizhov/Weickert 2021). Moreover, we have used domain decomposition methods on GPUs (Kämper/Weickert 2022). They allow us to inpaint almost 40 colour images in 4K resolution (3840 * 2160 pixels) in one second on a contemporary GPU. This serves as the envisioned 4K real-time demonstrator which concludes the INCOVID project.

1. T. Alt, P. Peter, J. Weickert:
   Learning sparse masks for diffusion-based image inpainting.
   In A. J. Pinho, P. Georgieva, L. F. Teixeira, J. A. Sánchez (Eds.): Pattern Recognition and Image Analysis. Lecture Notes in Computer Science, Vol. 13256, Springer, Cham, 528-539, 2022.
   Also available as arXiv:2110.02636 [eess.IV], revised March 2022.
2. S. Andris, P. Peter, R. M. K. Mohideen, J. Weickert, S. Hoffmann:
   Inpainting-based video compression in FullHD.
   In A. Elmoataz, J. Fadili, Y. Quéau, J. Rabin, L. Simon (Eds.): Scale Space and Variational Methods in Computer Vision. Lecture Notes in Computer Science, Vol. 12679, Springer, Cham, 425-436, 2021.
   Also available as arXiv:2008.10273 [eess.IV], revised May 2021.
3. S. Andris, J. Weickert, T. Alt, P. Peter:
   JPEG meets PDE-based image compression.
   Proc. 35th Picture Coding Symposium (PCS 2021, Bristol, UK, June 2021), IEEE, 2021.
   Also available as arXiv:2102.01138 [eess.IV], revised May 2021.
4. M. Augustin, J. Weickert, S. Andris:
   Pseudodifferential inpainting: The missing link between PDE- and RBF-based interpolation.
   In J. Lellmann, M. Burger, J. Modersitzki (Eds.): Scale Space and Variational Methods in Computer Vision. Lecture Notes in Computer Science, Vol. 11603, 67-78, Springer, Cham, 2019.
5. L. Bergerhoff, J. Weickert, Y. Dar:
   Algorithms for piecewise constant signal approximations.
   Proc. 27th European Signal Processing Conference (EUSIPCO 2019, A Coruña, Spain, Sept. 2-6, 2019), IEEE, 2019.
   Also available as arXiv:1903.01320v3 [eess.SP], June 2019.
6. M. Bildhauer, M. Cárdenas, M. Fuchs, J. Weickert:
   Existence theory for the EED inpainting problem.
   St. Petersburg Mathematical Journal, Vol. 32, No. 3, 481-497, May 2021.
   Invited Paper.
   Also available as arXiv:1906.04628v2 [math.AP], September 2019.
7. M. Cárdenas, P. Peter, J. Weickert:
   Sparsification scale-spaces.
   In J. Lellmann, M. Burger, J. Modersitzki (Eds.): Scale Space and Variational Methods in Computer Vision. Lecture Notes in Computer Science, Vol. 11603, 303-314, Springer, Cham, 2019.
8. V. Chizhov, J. Weickert:
   Efficient data optimisation for harmonic inpainting with finite elements.
   In N. Tsapatsoulis, A. Panayides, T. Theocharides, A. Lanitis, C.S. Pattichis, M. Vento (Eds.): Computer Analysis of Images and Patterns. Part 2. Lecture Notes in Computer Science, Vol. 13053, Springer, Cham, 432-441, 2021.
   Also available as arXiv:2105.01586 [eess.IV], revised July 2021.
9. T. Dahmen, P. Trampert, P. Peter, P. Bheed, J. Weickert, P. Slusallek:
   Space-Filling Curve Indices as Acceleration Structure for Exemplar-Based Inpainting.
   arXiv:1712.06326 [cs.CV], January 2020.
10. F. Jost, V. Chizhov, J. Weickert:
    Optimising different feature types for inpainting-based image representations.
    Proc. 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023, Rhodes, Greece, June 2023), 2023.
    Also available as arXiv:2210.14949 [eess.IV], October 2022.
11. F. Jost, P. Peter, J. Weickert:
    Compressing piecewise smooth images with the Mumford-Shah cartoon model.
    Proc. 28th European Signal Processing Conference (EUSIPCO 2020, Amsterdam, Netherlands, January 2021), 511-515, 2021.
    Also available as arXiv:2003.05206 [eess.IV], March 2020.
12. N. Kämper, J. Weickert:
    Domain decomposition algorithms for real-time homogeneous diffusion inpainting in 4K.
    Proc. 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2022, Singapore, May 2022), 1680-1684, 2022.
    Also available as arXiv:2110.03946 [eess.IV], revised February 2022.
13. L. Karos, P. Bheed, P. Peter, J. Weickert:
    Optimising data for exemplar-based inpainting.
    In J. Blanc-Talon, D. Helbert, W. Philips, D. Popescu, P. Scheunders (Eds.): Advanced Concepts for Intelligent Vision Systems. Lecture Notes in Computer Science, Vol. 11182, 547-558, Springer, Cham, 2018.
14. R. M. K. Mohideen, P. Peter, J. Weickert:
    A systematic evaluation of coding strategies for sparse binary images.
    Signal Processing: Image Communication, Vol. 99, Article 116424, November 2021.
    Also available as arXiv:2010.13634 [eess.IV], revised July 2021.
15. P. Peter:
    Fast inpainting-based compression: Combining Shepard interpolation with joint inpainting and prediction.
    Proc. 2019 IEEE International Conference on Image Processing (ICIP 2019, Taipei, Taiwan, Sept. 2019), 3557-3561, 2019.
16. P. Peter:
    Quantisation scale-spaces.
    In A. Elmoataz, J. Fadili, Y. Quéau, J. Rabin, L. Simon (Eds.): Scale Space and Variational Methods in Computer Vision. Lecture Notes in Computer Science, Vol. 12679, Springer, Cham, 15-26, 2021.
    Also available as arXiv:2103.10491 [eess.IV], March 2021.
17. P. Peter:
    A Wasserstein GAN for joint learning of inpainting and its spatial optimisation.
    In H. Wang, W. Lin, P. Manoranjan, G. Xiao, K.L. Chan, X. Wang, G. Ping, H. Jiang: Image and Video Technology. Lecture Notes in Computer Science, Vol. 13763, Springer, Cham, 132-145, 2023.
    Also available as arXiv:2202.05623 [eess.IV], February 2022.
18. P. Peter, J. Contelly, J. Weickert:
    Compressing audio signals with inpainting-based sparsification.
    In J. Lellmann, M. Burger, J. Modersitzki (Eds.): Scale Space and Variational Methods in Computer Vision. Lecture Notes in Computer Science, Vol. 11603, 92-103, Springer, Cham, 2019.
19. P. Peter, K. Schrader, T. Alt, J. Weickert:
    Deep spatial and tonal optimisation for homogeneous diffusion inpainting.
    Pattern Analysis and Applications, 2023.
    Invited Paper.
    Also available as arXiv:2208.14371 [eess.IV], revised March 2023.
20. K. Schaefer, J. Weickert:
    Diffusion-shock inpainting.
    In L. Calatroni, M. Donatelli, S. Morigi, M. Prato, M. Santavesaria (Eds.): Scale Space and Variational Methods in Computer Vision. Lecture Notes in Computer Science, Vol. 14009, Springer, Cham, 588-600, 2023.
    Also available as arXiv:2303.09450 [eess.IV], March 2023.
21. K. Schrader, T. Alt, J. Weickert, M. Ertel:
    CNN-based Euler's elastica inpainting with deep energy and deep image prior.
    Proc. 10th European Workshop on Visual Information Processing (EUVIP 2022, Lisbon, Portugal, Sept. 2022), IEEE, 2022.
    Also available as arXiv:2207.07921 [cs.CV], July 2022.
22. K. Schrader, P. Peter, N. Kämper, J. Weickert:
    Efficient neural generation of 4K masks for homogeneous diffusion inpainting.
    In L. Calatroni, M. Donatelli, S. Morigi, M. Prato, M. Santavesaria (Eds.): Scale Space and Variational Methods in Computer Vision. Lecture Notes in Computer Science, Vol. 14009, Springer, Cham, 16-28, 2023.
    Also available as arXiv:2303.10096 [eess.IV], March 2023.

MIA Group
©2001-2023
The author is not
responsible for
the content of
external pages.

Imprint - Data protection