My Journey on how I built a product using Deep Fakes which ended up as a runner up at SigmaHacks3
The Context
Bored of the normal job like every other newly joined SDE, me and two other fellow human friends decided that let us give a hackathon. We came across SigmaHacks 3. It was a one day hackathon and seemed quite interesting. The Education Path tbh seemed exciting to me quite a lot. But like every friend ever, they backed out at the last moment :D Still since for the sole reason I had nothing else to do and was a tad bit excited I decided let us give it. It was a short one day hackathon so I was sure that my excitement would not fade away that soon!
The Ideation phase
The initial idea was to work in the domain of making sure students attended the online classes by means of some interesting activities. Asked a few of my friends around on firstly how to detect that one is actually attending the lecture. For the one’s where the video camera was supposed to be kept on all the time, it was easy, use temporal FFTs to check for repeating patterns. The hard part was to come up with something where it was not possible for students to keep their web cameras on. I was thinking something in line with random attendance for each student or random questions based on the field, but with my friends dropping out, it seemed a bit of a stretch to do it all by myself.
With so much social media presence and students getting used to it, it is sometimes hard for students to get through boring lengthy texts or lecture notes. Also the need for entertaining teaching methods for toddlers and younger kids has been on a rise. In some rural areas, online lectures are not possible which in the end results in students losing a lot. So I thought, why not use Deep Fakes for some good. For those who don’t know what deep fakes are, in layman terms they are fake videos created based on an audio sample and anything ranging from an image to a video. With the Hackathon not being according to my timezone, I decided to proceed with this and got some sleep at night.
The Building phase
It was Sunday morning and like every other Sunday Morning in my life, I woke up Late. My parents were travelling to Ahmedabad to meet Masi(Maternal Aunt) and I had heavy heartedly given up on going to attend this hackathon, simply because I needed some excitement back in life. So lazily waking up, the game plan was clear: There is no time to make and train my own Deep Fake, so go through all available online Deep Fake generators and choose a good lightweight one. Build upon it and then think on what to do next.
After looking at a lot of codes available online, making them work on my system, trying and testing with my own video and audio, I came across one from my Institute itself. It was kind of state of the art work and seemed pretty good. But there was a catch, it was heavy. I could not make it run on my system locally. After successfully letting it crash my PC 2-3 times, I gave up. That is when I decided I will host it on colab. But there was something I needed to figure out, I was planning to make a full fledged UI to make it easy to use. How would I trigger my colab code from my Flask app? The solution was simple, I won’t :D I would rather host my Flask App in colab. So all set to go and some rigorous coding later, I was ready with making the Deep Fake work.
This is when I decided that since text is the most boring entity out there, I also need to use TTS to make text interesting. After experimenting a bit I decided not to look for that great of a TTS and simply go with Google TTS for now. Meanwhile, I discussed this idea with some of my friends and took their feedback. Some liked it, some were like Meh! Anyways I was too deep into it to overthink. With the base system in place, it was time to experiment. I needed some good examples to show in the presentation and hence I started searching.
A sample here combines fortnite character with an excerpt from audio book about earthquakes:
But I was bored, so in the middle of all the fuss, I went out with my friends, had a bit of fun, talked to them, brought shakes back home and sat with them and holy cow, it was 10 PM already. I had loads of work left to do. After they went back home because of the lockdown, Like every other lazy kid’s inner voice I thought : Kushal see you have to complete it and you need to pull off an all nighter, better watch a movie and then start :facepalm. Call it dumb but it probably helped with keeping my mind fresh.
After experimenting a lot from audio lectures to extracting audio from youtube videos and finding inspiration in animated characters of Fortnite, I found quite some good examples. Tbh I even took my friend’s profile picture from WhatsApp for one of the experiments just for fun :P It was now time to make the UI component and link it. Flask with ngrok was a viable option and I was ready with it. It sounds simple but there was a lot of UI theme selection, building, making appropriate examples from Deep Fakes. It was around 5 AM and I was done. All I had to do was record a video and submit. With 3-4 takes, making a submission on Devpost it was 6AM and I was done. I was literally done, didn’t care anymore about the hackathon, overthinking that probably should have gone to Ahmedabad, all I saw was my bed :D
The final presentation looked something like this (watch in 0.75x ples):
The Results
Tbh I don’t even remember what time the results were being declared. It was like 12:30 AM I think and there was a live stream by the organizers to declare the results. I was not sure it would have been the best idea of the lot, since I was working alone on it, but they said FAKEducation and my heart melted. It was the Eureka Eureka moment, except there was no one around. So I remember texting my junior since she was the only one awake and she was probably confused why I was so excited xD. Super excited, I took a photo that I won and sent it to my friends. I was excited and proud and happy. It was a good detour from my daily software life and maybe was a day back to my college days… Won 50$, gave it to my cousins because Paypal in India sucks. All in all it was the most fun experience I had in a long long while!
My Takeaway
I learnt that it is okay to do whatever you want by yourself, because it will make you a proud human. Also having such supportive friends around always helps and it sucks when they sleep early. And the most important one : Hackathons are alwayssss fun!
Scope of Improvement
There is a lot of scope for improvement. With new and improved deep fake methods coming in, one with gestures can be used to make sure it seems more real. TTS can be improved simply by using a paid service which is near human voice. Also watermark should be added to prevent misuse.
Credits where credits are due
Ravi Sawlani and Mishal Shah for cancelling on me to take part -.-
Mishal Shah and Harnish Rajput for listening to my idea
Ravi Sawlani, Mishal Shah and Harnish Rajput for going out that night (obv joined by above 2 fellow humans)
P.S Call it a coincidence, I just took my vaccine the day before. It seems vaccine has something to do with boosting my brain xD
