Neural networks running on the HP Z8 Workstation create more authentic connections with priceless bygone moments. PDF-version is availible here.

Time for a RE-think

What visuals come to mind when you hear AI and machine learning? Engineers cranking away in a lab, solving complicated sci-fi type problems?

Time for a re-think.

A tiny machine learning start-up in Gdańsk, Poland is allowing audiences to travel back in time in unprecedented ways. Chances are, you have already seen their work.

Offering a more vibrant view of the past

neural.love is the brainchild of CEO Denis Shiryaev, CTO Ilya Chekalsky, Machine Learning Engineer Artem Legotin, and Client Relations Manager Elizabeth Peck. It provides the wizardry behind the enhancement of a variety of historical clips, including the Lumière brothers’ 1895 French short of a train pulling into a station.

The AI-enhanced result was a labor of love on the part of Shiryaev, who couldn’t help but merge his love for history with his product management and machine learning skills to enliven the famous short film. In early 2020, he experimented with open- source software and AI models to upscale it to 4K resolution and smooth the jerky motion from 15 frames per second (fps) to 60 fps. “I posted it one night, and when I woke up the next day, I had a million views and was on the front page of Reddit. My inbox was exploding,” Shiryaev says.

He knew immediately that he had tapped into something big. “Due to the limitations of technology at that time and degradation of materials, old films have significant barriers to a full experience. They are grainy, the movement is jerky, they are black and white, and the speed is off. We can use AI to break down these barriers, to better preserve and enhance this history, and in doing so create more empathy with today’s audiences for these experiences.”

With B2B and B2C requests coming in from around the world, the team decided to combine their collective expertise and try launching a company together. Peck says, “Coming from a rather eclectic background in communication, education, and the arts, what neural.love was doing really caught my attention. The notion of telling stories from an earlier time in a more robust and meaningful way—to make them more accessible to a modern audience, along with the interdisciplinary combination of data science and art—that really called to me.”

Suddenly, this little group of ex-pats living in Poland had both B2C and B2B audiences clamoring for more.

“WE CAN TRAIN COLORIZATION MODELS AND RUN EXPERIMENTS THAT USED TO TAKE US FIVE DAYS IN JUST 36 HOURS. MORE IMPORTANTLY, WITH THE GIANT GPU MEMORY ON THE Z8, WE CAN TUNE HYPERPARAMETERS TO RUN BIGGER BATCH SIZES, WHICH TRANSLATES TO IMPROVED RESULTS.” Artem Legotin, Machine Learning Engineer, neural.love

“AS A RESULTOF Z8’S CAPABILITIES, NEURAL.LOVE IS ABLE TO DO MORE, AND DO IT FASTER AND BETTER, TO MEET THE NEEDS OF VARIOUS CLIENTS.” Elizabeth Peck, Client Relations Manager, neural.love

Deploying computer science to enrich art and history

AI. Machine learning. Data science. No longer the sole realm of mathematicians, engineers, and scientists, these disciplines are now being used across various industries. Very soon, no business will be without it, whether it’s deployed to create intelligent products and services or to design smarter business processes. That the tangible impact of AI can be seen in something as visible as video and film is one of the things that makes this a near-obsession for Shiryaev. The work is enabling today’s audiences to stroll through the streets of New York City in 1911 or watch Apollo 16 astronauts drive the lunar rover on the moon in 1972—all with the clarity and detail we expect from a modern smartphone video.

Peck concurs: “Even if your background is not in tech, you can experiment and collaborate in this space. Machine learning is enabling people from all walks of life to do new, innovative things with technology and do traditional tasks, like editing text, even faster and better than before. I liked that neural.love was at the forefront of shaping the language around some of these concepts and introducing new ideas to the world.”

A look back with a new lens

So, what is it, exactly, that neural.love is doing? The company is training neural networks—computer systems designed to mimic the way biological neurons in the human brain and nervous system signal one another—to fill in approximations of data in old film. It’s different from traditional restoration, which eliminates imperfections. Instead, neural.love uses algorithms to identify patterns in order to add color, remove noise, restore damaged frames, upscale image resolution, provide movement control, refine facial details and boost frames per second.

Legotin explains, “This feels like a real frontier. We are doing things that no one has done before. Machine learning is like a magic box: we put in the models but you are never quite sure exactly what will come out. Every time is a bit of a lottery.”

Case in point: colorization. The DeOldify algorithm learns to recognize certain objects, such as a bridge or grass, and which colors correspond to each object. But it can only rely on what is already known; it can’t make educated or informed guesses. Peck explains, “If most bridges are usually grey, then the algorithm doesn’t know that the Golden Gate Bridge is an exception and should be orange- red.” It’s a big problem with objects that look alike but have colors with very different meanings, such as flags and sport jerseys. Over the past few months, neural.love has been exploring how to build custom colorization algorithms and has already presented them to several clients.

Another example is upscaling, an algorithm that learns from a neural network trained on pairs of images, working to make the low-quality version of the image look exactly like the high- quality version from the dataset. After it learns the patterns, the algorithm can then be applied to lower-resolution footage to generate a higher resolution copy, improving the overall quality of the visual experience.

Then there is the issue of AI bias. Most datasets are consciously built to be representative but may still exhibit a form of inherent bias in their creation. Hence, the outcomes also need to be checked for accurate representation. “Facial recognition is an important area to check; we generated thousands of images of different people and fed them into our model to make sure that it produces 100% bias-free output. This is helping us to enhance film accurately, reflecting accurate facial features, skin color, regions, age, and other details that make images more authentic,” Legotin says.

Sharing their passion with the world is important, too. Using the B2B funnel to support their R&D and software development efforts, neural.love was able to make their bespoke tools available for a wide variety of B2C clients through the launch of their custom-built, online software- as-a-service product. Peck says, “Everyone has an old wedding video, a clip from an important event, or visual mementos of a loved one who has passed. This is an accessible and affordable way for them to more fully enjoy and relive those memories. Even independent filmmakers and documentarians who need to upscale or use tools like cinematic slow-down are able to utilize our cloud service to enhance and expand the reach of their work.”

“THIS IS A BEAUTIFUL THING WE ARE DOING: TRANSFORMING HOW WE ALL CAN EXPERIENCE THE PAST. TECHNOLOGY AND THE Z BY HP ARE MAKING IT POSSIBLE.” Denis Shiryaev, CEO, neural.love

Power and speed enable business growth

When neural.love first learned of the HP Z8 Desktop Workstation’s capabilities, they weren’t quite sure they needed that much firepower. After all, they had been making it work on Shiryaev’s old souped-up gaming computer. But the promise of GPUs with upwards of 5,000 cores to perform multiple tasks at the same time; low latency for direct capture and real-time analysis of data; GPU- acceleration for industry-leading processing speed of massive amounts of data; and rapid cleaning for faster, higher accuracy insights persuaded them to give it a try.

Immediate benefits followed. Legotin says, “We can train color models and run experiments that used to take us five days in just 36 hours. More importantly, with the giant GPU memory on the Z8, we can tune hyperparameters to run bigger batch sizes, which translates to improved results.”

The development team is also using the Z8 to re-size images to boost image quality. In the past, they would feed 300-pixel images into the neural network and resize them progressively larger, a task that requires massive GPU memory. Legotin says now they can feed in images that are two times larger, for highly accurate details. Shiryaev adds, “Plus, because the processing speed is so fast, we can re-size and crop in at least half the time that it took us before.”

All of these performance improvements have enabled neural.love to accommodate the growing demand for its services. Peck says, “As a result of the Z8’s capabilities, neural.love is able to do more, and do it faster and better, to meet the needs of various clients."

Filling in gaps in history

The power to bring history more vividly to life is both thrilling and satisfying. During their initial B2B phase, neural.love helped a Dutch documentary company colorize scenes from WWII, a Polish filmmaker restore and enhance never-before-seen archival footage, and PBS’ American Experience celebrate the triumph of women’s suffrage as part of the interactive website “She Resisted."

The development team is now focused on improving the quality and functionality of their existing SaaS offering, and expanding their software development capabilities to include API access for professional and commercial clients.

Here, at the intersection of machine learning and art, neural.love is creating richer, more captivating memories. Shiryaev sums it up this way: “This is a beautiful thing we are doing: transforming how we all can experience the past. Technology and the Z by HP are making it possible.”

About neural.love

neural.love helps users transform historical content by providing an easily accessible, cloud-based option for custom video restoration and enhancement. Through the magic of machine learning and a tailored pipeline of neural networks, clients from film studios to YouTube aficionados can automatically colorize, remove noise, upscale image resolution, and boost the number of frames per second for a wide variety of source materials.

© Copyright 2022 HP Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.