Wireless and Digital Services

Magic using super resolution neural networks

By Monty Barlow - Last updated: Thursday, August 24, 2017

There’s a scene in Ridley Scott’s 1982 film Blade Runner where the protagonist zooms deeper and deeper into a photograph of a crime scene, revealing a seemingly impossible level of detail. Just how fanciful this scene really is has been debated by sci-fi fans for decades, but the ability for photographers to significantly enhance a soft image remains just a dream. Or at least that’s what I thought, until last week…

In our Digital Greenhouse AI laboratory, one of our Machine Learning teams was experimenting with super resolution neural networks. They trained such a network on high resolution and corresponding low resolution (blurred) images of tens of thousands of photos of works of art. This ‘super-res’ network therefore learned what brushstrokes and other details on artwork look like, when blurred.

The team then presented previously unseen artwork to the super-res network and asked it to produce an enhanced version. Take a look at the two paintings below. In each case we started with a fairly low resolution internet image published by the artist, and generated a new image with much more detail in brushstrokes and edges. For each painting you can see a zoom in on a particular region: The left side showing conventional upscaling, the right showing the output of our super-res network. It even correctly detects the difference between dust on the original painting and the artist’s small brush marks – the image with red rings shows an example of this. I think the results are nothing short of astonishing.

 

‘Apple meets crock’, Catherine Twomey, 2014

 

Apple meets crock, by Catherine Twomey, 2014.

Original image

 

Conventional upscaling versus super-res

Conventional upscaling versus super-res

 

‘Roses and reflections’, Diana Probst, 2015

 

Original image

Original image

 

Conventional upscaling versus super-res

Conventional upscaling versus super-res

 

Distinguishing dust from tiny brush marks

Let’s be clear: We cannot create information from nothing. These enhanced images contain no more original information than the low-res images we started with. Such techniques may not be appropriate for enhancing poor quality security camera footage at a crime scene. However there are serious uses in enhancing data sets for developing and testing machine learning systems – hence our interest in the first place. The scarcity and cost of high quality data sets mean that tools that can convert and process data to extract the most use from it are an important part of a Machine Learning team’s arsenal.

Regardless, the enhanced images are more attractive and more convincing than the originals, and have surprised all that have seen them – particularly artists and photographers who are well aware of the limits of the sharpening tools built into image manipulation software.

This feeling of having previously held convictions over-turned by the new results staring you in the face is one of the most enjoyable aspects of working with Machine Learning. Finding our beliefs in the limits of the possible shattered with increasing regularity is one of the clearest indicators of the growing potency of Machine Learning technology.

 

Paintings credits
Apple meets crock, Catherine Twomey: https://www.catherinetwomey.com/
Roses and reflections, Diana Probst: http://dianaprobst.com/