Multi-Frame Super-Resolution Toolbox

Introduction

The goal of multi-frame super-resolution algorithms is to reconstruct a high-resolution image from a set of low-resolution frames taken from the same scene. For this purpose, reconstruction algorithms exploit complementary information across different frames to fuse them into an image of higher spatial resolution. In a common paradigm, subpixel motion between low-resolution frames is employed as a cue for super-resolution. Over the past years, this methodology has become an emerging field of research within the field of signal and image processing with various applications, e.g. in surveillance imaging, remote sensing or medical imaging.

In this toolbox, we provide the implementations of several state-of-the-art algorithms as well as novel methods developed in our projects on image super-resolution. The algorithms available in the toolbox cover general-purpose reconstruction algorithms that exploit subpixel motion to gain super-resolved data and tailor-made solutions for specific applications (with focus on applications in medical imaging). The software is developed in MATLAB and is partly accelerated using C++ software integrated by a MEX interface.

The use of this software is free for research purposes. Please cite the papers associated with the different algorithms, if you use them in your own work. The toolbox is provided for noncommercial purposes only, without any warranty of merchantability or fitness for a particular purpose.

Features

The multi-frame super-resolution toolbox implements several state-of-the-art algorithms with a common user interface. It is designed in a modular way and extendable by new algorithms in future works. In its current version, the following setups and algorithms are covered:

Super-Resolution of a single modality

Maximum a-posteriori (MAP) based super-resolution reconstruction employing different observation models and image priors [1]
Robust MAP super-resolution using M-estimator based observation and prior models as proposed in [2,3]
MAP super-resolution with image quality self-assessment for automatic regularization parameter selection as proposed in [4] for application in retinal imaging

Multi-sensor and multi-channel super-resolution
- Multi-sensor super-resolution to super-resolve one modality under the guidance of high-resolution data (acquired with a complementary modality) as investigated in our work on hybrid range imaging published in [5, 6]
- Guided super-resolution to simultaneously super-resolve a pair of complementary modalities as proposed for photo-geometric resolution enhancement in [7]
- Multi-sensor super-resolution for multi-channel images using the locally linear regression model presented in [8]

News & Version Information

March 2014 (version 1.0): Initial version of super-resolution toolbox released
September 2014 (version 1.1):
- Guided super-resolution (presented at GCPR '14) included in the toolbox
- Added demo scripts for different algorithms (see 'examples')
December 2014 (version 1.2):
- Super-resolution with quality self-assessment (presented at MICCAI '14) included in the toolbox
- Changed setup script of toolbox to simplify the setup of the Matlab paths
August 2015 (version 1.3): Multi-sensor super-resolution (presented in Medical Image Analysis) included in the toolbox

September 2015 (version 1.4): "Super-resolution for multi-channel images using locally linear regression" (presented at BMVC '15) included in the toolbox.
November 2015 (version 1.5): The toolbox has been extended by implementations of the adaptive edge preserving bilateral TV algorithm (BEP) of Zeng and Yan [8] as well as re-descending M-estimators (e.g. Lorentzian function) for robust MAP estimation as proposed by Patanavijit and Jitapunkul [2]. The algorithms are included in the new 'Robust' module.
January 2016 (version 1.6): "Robust multiframe super-resolution employing iteratively reweighted minimization" (accepted for IEEE TCI) included in the toolbox.
September 2016 (version 1.6.1): Fixed bugs in the multi-sensor super-resolution example scripts
January 2018 (version 2.0): The toolbox was updated with the algorithms described in my PhD Thesis "Multi-frame super-resolution reconstruction with applications to medical imaging." arXiv preprint arXiv:1812.09375.

Downloads

The current version (version 2.0, 77 MB) of the toolbox including example images for demo scripts is available Initiates file download here. A basic version (3 MB) containing only the algorithm source code without example scripts and data is available here.

Please note that our toolbox contains the following dependencies to third-party libraries:

For numerical optimization, we use the Netlab Toolbox (version 3.3) available here
Please include Netlab to /common/3rdparty/netlab3_3
For motion estimation, we use the code of C. Liu to estimate optical flow (available here) and the Image Alignment Toolbox for parametric motion estimation (available here). We plan to include our own motion estimation software in a future version of the toolbox.

Related Projects

CUDA Super-Resolution implements a limited functionality of this toolbox but is accelerated on the graphics cards for interactive applications using NVIDIAs CUDA platform.
In our data archive, we provide evaluation datasets used in our work on multi-sensor super-resolution.

Contact

For questions or comments please feel free to contact Opens window for sending email Thomas Köhler

References

[1] Elad, M., & Feuer, A. (1997). Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Transactions on Image Processing, 6(12), 1646–1658.

[2] Patanavijit, V., & Jitapunkul, S. (2007). A Lorentzian Stochastic Estimation for a Robust Iterative Multiframe Super-Resolution Reconstruction with Lorentzian-Tikhonov Regularization. EURASIP Journal on Advances in Signal Processing, 2007(1), 034821.

[3] Köhler, T., Schebesch, F., Aichert, A., Maier, A., & Hornegger, J. (2015). Robust Multi-Frame Super-Resolution Employing Iteratively Re-Weighted Minimization. IEEE Transactions on Computational Imaging, 2(1), 42 - 58, 2016

[4] Köhler, T., Brost, A., Mogalle, K., Zhang, Q., Köhler, C., Michelson, G., Hornegger, J. & Tornow, R. P. (2014). Multi-frame Super-resolution with Quality Self-assessment for Retinal Fundus Videos. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2014 (pp. 650–657).

[5] Köhler, T., Haase, S., Bauer, S., Wasza, J., Kilgus, T., Maier-Hein, L., Feußner, H., Hornegger, J. (2013). ToF Meets RGB: Novel Multi-Sensor Super-Resolution for Hybrid 3-D Endoscopy. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2013 (pp. 139–146).

[6] Köhler, T., Haase, S., Bauer, S., Wasza, J., Kilgus, T., Maier-Hein, L., Hornegger, J. & Feußner, H. (2015). Multi-sensor super-resolution for hybrid range imaging with application to 3D endoscopy and open surgery. Medical Image Analysis, 24(1), 220–234.

[7] Ghesu, F. C., Köhler, T., Haase, S., & Hornegger, J. (2014). Guided Image Super-Resolution: A New Technique for Photogeometric Super-Resolution in Hybrid 3-D Range Imaging. In Pattern Recognition (pp. 227–238).

[8] Köhler, T., Jordan, J., Maier, A., Hornegger, J. (2015) A Unified Bayesian Approach to Multi-Frame Super-Resolution and Single-Image Upsampling in Multi-Sensor Imaging. In Proc. 26th British Machine Vision Conference (BMVC 2015).

[9] Zeng, X., Yang, L. (2013) A robust multiframe super-resolution algorithm based on half-quadratic estimation with modified BTV regularization. Digital Signal Processing, 23(1), 98-109.