Image Super-Resolution by Neural Texture Transfer



Bicubic (4x)

SRCNN

SRGAN

SRNTT (ours)

Reference

Bicubic (4x)

SRGAN

CrossNet

SRNTT (ours)
Zoom-in for better visualization. CrossNet and SRNTT use the same reference image.
Abstract

Due to the significant information loss in low-resolution (LR) images, it has become extremely challenging to further advance the state-of-the-art of single image super-resolu-tion (SISR). Reference-based super-resolution (RefSR), on the other hand, has proven to be promising in recovering high-resolution (HR) details when a reference (Ref) image with similar content as that of the LR input is given. However, the quality of RefSR can degrade severely when Ref is less similar. This paper aims to unleash the potential of RefSR by leveraging more texture details from Ref images with stronger robustness even when irrelevant Ref images are provided. Inspired by the recent work on image stylization, we formulate the RefSR problem as neural texture transfer. We design an end-to-end deep model which enriches HR details by adaptively transferring the texture from Ref images according to their textural similarity. Instead of matching content in the raw pixel space as done by previous methods, our key contribution is a multi-level matching conducted in the neural space. This matching scheme facilitates multi-scale neural transfer that allows the model to benefit more from those semantically related Ref patches, and gracefully degrade to SISR performance on the least relevant Ref inputs. We build a benchmark dataset for the general research of RefSR, which contains Ref images paired with LR inputs with varying levels of similarity. Both quantitative and qualitative evaluations demonstrate the superiority of our method over state-of-the-art.
Network Architecture

 
Overview of SRNTT
 

 
Texture Transfer
 
Paper

Citation

Zhifei Zhang, Zhaowen Wang, Zhe Lin, and Hairong Qi, "Image Super-Resolution by Neural Texture Transfer", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

Bibtex
@inproceedings{ZhangSRNTT2019,
  author = {Zhang, Zhifei and Wang, Zhaowen and Lin, Zhe and Qi, Hairong}, 
  title = {Image Super-Resolution by Neural Texture Transfer},
  booktitle = {arXiv:1903.00834v1},
  year = {2019}
}
        
Results

Quantitative Evaluation

Qualitative Evaluation

Percentage of votes of SRNTT as compared to each of other methods in the user study
Texture Transfer with Different References (bottom-right)

Investigation on Extreme References

Testing Dataset --- CUFED5

CUFED5 Dataset --- 126 Testing Images (click on the thumbnails to change the sample image)
Original
Bicubic SRCNN SCN
DRCN LapSRN EDSR
SelfEx Landmark ENet
SRGAN SRNTT- (ours) SRNTT (ours)