How Snapchat’s filters work

You must have heard some phrases like- “If you break the streak, I will break you” or “checkout the new filter of snapchat” or “let’s take a selfie, open your snapchat” or you might have said this yourself! But have you ever wondered how the filters of snapchat work? Read to find out.

While you call them filters, snapchat calls it lenses. Now they may look very funny and easy but the engineering behind them is not. The technology used for the ‘filters’ initially came from a startup from Ukraine called ‘Looksery’ which was later acquired by snapchat for $150 million dollars. It was reported to be the most talked about and biggest acquisitions Ukraine ever witnessed. Snapchat acquired the startup tech company back in September 2015.

How Does Snapchat Work?

Snapchat uses the pixel data from the camera of the mobile and identify objects to identify the 3D space. This technology is known as ‘Computer Vision’. That’s how Facebook identifies the people in the group photos or self-driving cars identify objects and not run on them and that’s how snapchat identifies where’s your nose and where are your lips in order to put virtual makeup on you face (if that’s the filter you are using)

So How Does The Filters Work?

There are various steps in face detection find out-

The first step is DETECTION. How do they know which part is face from a whole image, that too when its moving continuously? A human brain can easily detect objects and faces but if you think that’s how the computer detects face, you are wrong.There is a color value of each individual pixel in terms of numerals.so how do up find a face? Well, it looks at the areas with light and dark spots which is nothing but the contrast. The face detection tool used is called the ‘Voila-Jones’ algorithm. It repeatedly scans the image data and calculate the grey scale pixel boxes which are underneath the white boxes. Our face has various contrasts in various areas, for instance the cheek bones are lighter than the eye sockets or the point of the nose is lighter than the side areas. And just like that it scans the whole image continuously to find these areas. If it finds these features, it concludes that there is a face. Now imagine how fast this whole process takes place. When you open snapchat, within seconds your face is detected and also of the friend of yours who is standing right next to you!

But the algorithm is constricted to a front face only. So, if you tilt your face sideways or upward, it will not detect your face.This technology is used by the camera for years. That’s how they put boxes around the faces.

But how do they put that virtual lipstick or the deep dark Halloween eyes? It works through the “active shape model”. It’s a model of face change that has been created by people manually through putting the facial features and borders by marking it through thousands of sample images. Now the algorithm takes that data to align with the face image from the camera of the phone by making changes according to the angle and rotating it to fix. Now its not that easy, sometimes it has to relocate the points by looking at contrast levels. It completely works on the pattern that has been set. Once all the points are being located, they are used to create a 3D face mask which can still work after the face is being moved or rotated. Once the face mask is detected, you can do anything with these filters such as broaden your face or change the eye color, and animations to trigger for example you open your mouth or raise your eyebrow. It works really well, as you can swap your face with your friend. How much fun is that! Although the technology is not new, what’s new is that the algorithm works really fast.

WHY?

Why is snapchat putting in so much efforts and technology for just a selfie?

Snapchat found real business and revenue in it. remember when snapchat introduced a Bahubali filter and people went crazy about it? well, that was just an advertisement stunt and you know evidently how much revenue Bahubali earned. This is the era of advertisement. And the best way to get people to look at your advertisement is to put it on our faces!

The Dark Side

Since the algorithm identifies faces easilyand each face has different points, so it can be used to identify your name. various tech giants like Facebook and google and even FBI are databasing the faces and there’s no particular law restraining or regulating it. they know how to detect your name by your face.so, there are various tools that camouflage your faces in such a way that it cannot be detected by these face detecting algorithms. This initiative is taken by some privacy advocates. So, there are certain places where you can wear face masks legally to hide your face from these algorithms. Technology is working and coming up with tools that can be used in order to not let the computers detect your faces at all by some makeup or props or the masks.