Hashing Faces: How students made hash validation subconscious with machine learning

‌In the past people struggled to manage long numbers, identifying information and configurations in their working memory. A team of students worked to create CheckFace, a developer service that leverages preattentive processing to move the mundane to the subconscious. They identified the need to make tasks like checksum verification, troubleshooting, build versioning, comparing files and others easier using Generative Adversarial Networks and Open Source Software to generate unique, human faces.

Grand Possibilities of Machine Learning

I was fortunate enough to be a part of this team, originally working with a classmate and close friend late into the night on new Generative Adversarial Networks from Nvidia Labs. Possibilities of applications were in the back of our minds - we knew what we were working with was a combination of a very innovative architecture and could have some really cool applications. Reading through the paper from Nvidia really focuses on the advancements of StyleGAN, which details the ways in which the Generator and Discriminator networks were constructed to capture varying levels of detail in the source images from Flickr. As we watched the space, projects like https://thispersondoesnotexist.com built upon StyleGAN which garnered huge amounts of media attention, reaching the front page of Hacker News and The Verge. We knew this kind of GAN was getting some serious attention, embroiling a fervent interest in the potential applications.

Facebook Hack Melbourne 2019 drew a crowd of 80 vetted coders to a large hotel dining room to hack away for 24 hours. Arriving early to scout out some talent, a friend and I became progressively dismayed when team after team of students arrived. We had some prepared ideas - maybe something to help developers with checksums, or to enable real time collaboration while livestreaming on services like twitch. Standing in the foyer around some buzzword vultures, scouting around for anyone with experience in Machine Learning, AI, Blockchain, Smart Cities, IoT or Big Data. We had accepted our fate as the speakers started the event.

But the two of us coders were relieved, when as the event two latecomers arrived as the event was opened with solid experience in design and user experience arrived and were ushered towards our table after hearing a second-hand pitch about our developer experience idea and it had taken hold. ‌‌Problems became clear after surveying the room. People were lazy, and there were some real needs to help validate file integrity, compare versions and assist troubleshooting. We recalled StyleGAN and it's uncanny ability to create faces from a number - called a seed. While not really the goal of the Nvidia Labs researchers, using this seed, a number from which a long sequence of seemingly random numbers can then be created from we could generate unique looking faces.

We realised how large the problem was when we asked how many of the extremely tech-savvy participants had used a checksum in the past. We found that everyone knew what they were, why they were important and also admitted to not using them at all. We were energised, nervous at the scale of our prototype plan, but eager to create something using machine learning to solve something in a really unique way.

How might we make comparing hashes so easy for humans that it's done by our subconscious?

GAN's

GAN's were born out of an idea in Game Theory, a branch of mathematics which is focussed on the study of competitive scenarios, where the actions of some player effects and is affected by another player's actions. Game Theory has been applied to strategies of nuclear deterrence, economics and law among other disciplines.

In 2014 Ian Goodfellow published a paper using the idea of two competing players. Goodfellow suggests the algorithm behaves like a criminal counterfeiting money and police, checking notes from both real and fake. After several months of operation, the criminal will be very good at creating realistic notes, and the police excellent at determining fake notes.

What we wanted to do with our application is pull out and repurpose the criminal to produce some very realistic notes for us. Give a big number in, and out comes a high quality image.

The Day

We brainstormed and begun planning our solution - something that would be extremely absurd but had the allure of elegance and impact. Working alongside we all staked out parts of the architecture, mapping out an experience. A user would be presented with a file download online - to which a face would be generated. After a moment of memorisation the user would then be able to right click the file, in any operating system and view another face generated by the downloaded file. The simplicity of the idea was tangible and struck a chord when getting early feedback Each of the team focussed on one element of the flow, backend, website, chrome integration and right-click menu.

Code was being written and features were appearing. A chrome app appeared, some right clicking happening. Learning was a huge part of the initial phase. We needed some GPU resources to run Nvidia's code to do the actual generation of the faces. Problems began to arise, when our right click menu was failing to work. Right before the presentation we had nothing working. Hours of coding had already gone into the build only the right click in explorer wasn't working. So with only a few hours to go we decided to dump all the right click code and write a windows only version for the demo.

Numbers are actually faces

The pretrained model we used from Nvlabs allows us to generate faces using 512 latent factors, which can be moved independently. While they have no inherent meaning, we can show how we're able to generate a wide range of different faces using my dragging them around.

This interpolation between different points in the latent space means that we can create gruesome artwork like this. Of course, the Checkface team is greater than the sum of its parts, and far far greater than the average face in the middle. Quite disconcerting, definitely somewhere in the uncanny valley.

Demo[lition]

After a full 24 hours of building we presented what we had. By no means a polished solution we stepped up for our demo. Our slightly weary and jittery team assembled on the stage in front of the whole event.

I can't believe how many times I've presented but still -- I completely froze in the presentation. Gawking through the presentation, we made it through the demo. The right click menu also failed silently and we had a screen already as a backup. At this point you could see it on the team's face - we'd put in a lot of work to get towards our user's experience and the demo had more or less failed. A little shriveled, we returned to our seats to guess at which project might place on the podium.

So the winners had been announced, and we were betting on either some machine learning on a Facebook stock market integration or an AR based mapping tool. To the utter surprise of all who had poured hours into the project already - Checkface placed 1st at the hackathon. It was a relief and slightly unbelievable, several mentoring opportunities were extended and interest for the next steps of fully building out the prototype were tabled. We had some buy in and excitement from our techy early adopters. Time to develop the prototype some more.

Opening the Source

We realised that, after releasing a very underdeveloped prototype to the world, we got some excellent feedback. People weren't using checkface for what we thought they would, verifying their file downloads.

We had people using them for, whatever they were hacking on at the time

I've just been finding a CheckFace API super useful to pull into all kinds of projects

In UI mockups for fake users and also as a part of the design process, the StyleGAN generated faces were being used to avoid legal issues and asking for permission. Faces were generated to create photorealistic "Amanda" or "Graeme" personas to target a design process towards.

I've spent a lot of time here, just looking at these faces - it's so weird to think that they're not real

Others with less practical applications took to stockpiling the images. We don't know why. More generated pictures of their friends, and just entering strange data. Please do not enter your passwords into the text fields - that's not a good idea though we're now handling them better than we were before.

Next Steps

The most valuable thing isn't actually the UX or the problem of generating nice faces for you to verify hashes - it's actually the backend API and being able to hook it into a number of other things. This realisation really surprised us, but now we're focussed on building a platform to allow fast iteration and development on top of a photorealist face generation back end.

We are working on building an API to make consumer facing products easy to build and highly irritable. Have a play with Checkface and let us know what you're using it for, and reach out to us using the chat (bottom right) GitHub, via email, LinkedIn or Twitter. We're also open to mentoring new members of the open source community to contribute to Checkface on GitHub.

AWS Community Day Presentation

During the AWS Community Day Melbourne, we were invited to present what we had achieved along our CheckFace journey. In preparation, we decided to run some stress tests and see if we could handle the load of the 300+ expected attendees of the talk, plus a few more online.

So we hacked together a bit of a bash script using curl to generate 100 simultaneous users all requesting unique images

seq 1 10000 | xargs -n1 -P100 bash -c 'i=$0; url="https://example.com/?value=ac${i}"; curl -k -s -O $url'

During our testing, we put all that through a loadbalancer and spun up a number of GPU instances on AWS to see which were the most economical to run. We found we were GPU bottlenecked and additional or faster CPU didn't have much of an effect. Connected to the instances is our Grafana dashboard which shows how the load was handled during one point in our testing.

FAQs

What about doppelgangers? - A security feature that is used, is much better than a completely secure solution that isn't used at all.

It's hard to quantify how many distinct, identifiable faces a human would be able to perceive, and further whether or not all of these possible faces could bee generated by the StyleGAN model

Is running the inference models expensive?

Per image, it's not expensive to run - however due to the startup time of GPU instances and ongoing cost of running a GPU the cost over time is relatively expensive. Currently we run the architecture across AWS and on premise, to provide baseline image delivery on premise and any bursts are handled by a scaling group on AWS

Slide deck of the full presentation I delivered at AWS Community Day Melbourne

Check out the GitHub repo and download the chrome extension, and the context menu integration

‌Icons made by Freepik from www.flaticon.com licenced by CC 3.0 BY Icons made by Smashicons from www.flaticon.com licenced by CC 3.0 BY Icons made by monkik from www.flaticon.com licenced by CC 3.0 BY Icons made by smalllikeart from www.flaticon.com licenced by CC 3.0 BY                                                                                                                             ‌