Wireless and Digital Services

Augmented Festivity – and the search for more reality

By Monty Barlow - Last updated: Thursday, December 22, 2016

‘Augmented Festivity’ – A term which yields only one hit on Google, and that’s from 1891. Let’s see if we at Cambridge Consultants can make it real…

Imagine a future where your smart retina implants start overlaying festive imagery every time you enter a public space during the holiday season.

Our Machine Learning team has brought a glimpse of that future to 2016. Having run some state-of-the-art ‘Generative Adversarial Networks’ in our Digital Greenhouse, we can automatically enhance the “christmasiness” of any scene, seeded only with some greetings card images of baubles, tinsel, and snowy scenes.

No, we don’t think it will catch on either. At least we really hope not…

There’s a serious side to this, though. Modern Machine Learning approaches – particularly deep learning variants – are hugely hungry for data. Data is not just needed when training systems, but also in testing them prior to deployment. Recording sufficient real-world data to support progressively more advanced Machine Learning systems is getting difficult. And until recently, the machine generation of data was far less convincing than real-world data.

Let’s imagine that we are developing a system to automatically detect unusual traffic patterns – perhaps using only basic video camera feeds. The system would filter out factors such as lighting, weather, vehicle types, daily patterns (such as rush hour) which don’t matter, whilst reliably spotting accidents or other unusual events. Such a system can easily be conceived of today, but would require vast amounts of training data – potentially millions or billions of examples of smooth traffic flow, congestion, accidents and so on. Recording this training set could take a century. What we need is a machine that can generate as much “reality” as we want.

The new kid on the data generation block is the Generative Adversarial Network (GAN). The details are somewhat complex, but the idea is that a Generator attempts to “forge” data, whilst a Discriminator attempts to spot the forgeries from real data (hence ‘adversarial’ – we pit them against each other). Both Generator and Discriminator improve over many iterations of the GAN. If we get this right, we end up with a Generator which can produce very realistic variations of the data we originally provided it. We could use a GAN to produce millions of hours of video footage with extra cars, variation in sunshine and so forth.

So we trained a GAN to produce and detect “christmasiness” in images. You can see in our film above that we then used it to make arbitrary images and videos more like typical Christmas imagery. One source video was shot in our Cambridge UK reception area; notice how the tree has been left relatively unchanged by the GAN, whereas the roof has been removed and replaced with the blue glow of the North Pole sky in many Christmas scenes. Some snow and sparkle has been added too. A few years ago the idea of a machine making these changes automatically based on just a few source greetings card images would have been unthinkable.

GANs are a rapidly evolving area of Machine Learning, and there is a lot more for the Machine Learning community to discover. But GANs already hint at a promising future where Machine Learning is harnessed to support the development of other Machine Learning systems. This will be particularly important in artificial intelligence (AI) applications such as self-driving vehicles. GANs could enable us to subject a vehicle AI to conditions far more challenging than it would ever encounter in the real world.

Fears are often (very reasonably) expressed about the dangers of widespread use of Machine Learning, especially where human beings could be hurt. However I believe that Machine Learning offers huge potential to improve and prove the safety of such systems, as well as to realise them in the first place. As for ‘Augmented Festivity’, let’s retire it immediately…