What is Synthetic Media: The Ultimate Guide
Media expressed in the purely synthetic form will radically accelerate the process of content creation and delivery. Its accessibility and interactivity will usher in an exciting new era of digital media, one in which creativity, insight, and imagination determine content dissemination instead of the limitations imposed by physical space.
Novel forms of synthetic media blur the distinction between physical and digital environments. As a futurist, I use synthetic media in my keynotes and this new creative expression category will unleash powerful user experiences built on a new dynamic relationship between media and human perception.
What is Synthetic Media?
Just think about it. When we are talking about this new world of AI-generated synthetic media, we are talking about a space that combines some of the most potent forces in our world: live video, visual content, and audio, along with the most advanced technology platform to drive them.
Synthetic media is a new form of virtual media produced with the help of artificial intelligence (AI). It is characterized by a high degree of realism and immersiveness. Furthermore, synthetic media tends to be indistinguishable from other real-world media, making it very difficult for the user to tell apart from its artificial nature. It is possible to generate faces and places that don't exist and even create a digital voice avatar that mimics human speech.
Research in Synthetic Media has a long history. Experiments in this field date back to the 1950s and 1960s, when algorithmic and generative experiments were conducted. It went through a hiatus till the late 1980s and early 1990s, when computational power started to grow and reached a critical point with the advent of the World Wide Web.
In 1997, the Video Rewrite: Driving Visual Speech with Audio paper was written by Bregler, Covell, and Slaney. It developed a Video Rewrite program, the first to combine all of these technologies (previous research had developed increasingly convincing synthetic faces and speech). Analysis from this study was used in Hollywood blockbusters such as Star Wars Episode II: Attack of the Clones (2002) and Spider-Man 2 (2004).
How Synthetic Media Works
Artificial intelligence is applied to every industry and field of knowledge. But of all the various forms of machine learning, one stands out. Deep learning empowers many AI applications today by teaching computers how to think like humans and make intelligent decisions. While the technology has many applications in different industries, synthetic media is gearing up to be one of its most significant transformations yet.
Deep neural networks are more powerful than ever before. Generative Adversarial Networks (GANs) have helped to make this possible by learning from existing images while also being able to produce entirely new ones. Since GAN outputs look natural and indistinguishable from the original photos, they enable synthetic media that is difficult to distinguish from real media, particularly in computer vision and image processing applications.
GANs are a machine learning technique that enables computers to create realistic content. It is powered by two neural networks: one that makes fake images based on real photos and another that acts as a judge to determine whether an image is real.
The field of artificial intelligence is moving at a breakneck pace. Training computers to learn from data seemed like science fiction a few years ago. Now it's a reality, and researchers are making huge strides in developing systems that can learn to perform tasks that were once thought impossible for machines.
At the same time, advances in machine learning and deep learning have made it possible to train computer vision algorithms on large datasets of images. As a result, today's neural networks can see things in photos that humans can't even detect with our own eyes.
These capabilities have led to impressive new applications for AI-powered systems in recent years — including video games, autonomous cars and drones, facial recognition, and more. But they've also raised concerns about potential privacy breaches and ethical dilemmas.
One of them is the emergence of deepfakes, which I’ll cover in more depth later in this article. First, let's see how synthetic media can be useful to us.
Advantages of Synthetic Media
Synthetic media tools are reinventing our work with more intelligent, efficient methods that produce quality media experiences as never thought possible.
The main advantages of synthetic media include:
- Its products are created quickly, with minimal human involvement. They can cover various topics and be modified to fit any audience, anywhere in the world.
- They are often more convenient because it's accessible 24/7, and you can attach more interactive elements to them. Because they are dynamic, they are also less likely to get stale.
- Synthetic media is broad in its output. Generally, the medium can incorporate writing, music, drawings, paintings, voice, or visuals. This flexibility allows for diverse ways of storytelling through media. The process can be more creative and fulfilling than other forms of expression because it will enable the artist to explore their creativity in multiple ways.
- With its high flexibility, synthetic media can be implemented across various platforms. In addition to games, companies apply it to apps and websites, VR/AR experiences, and many more digital channels. This makes it a highly adaptable form that can be used widely across many industries, such as marketing, education, journalism, entertainment, and the arts.
- It can create an illusion of authenticity. This media type allows companies to connect with their audiences without paying actors or hiring professional photographers or videographers.
Synthetic Media Examples and Applications
The field of synthetic media is complex, intricate, and ever-changing. It is also very versatile and can be applied to numerous domains such as optics, energy storage, transmission, communication, or even online businesses. The following real-life examples will help you understand the reach of synthetic media applications.
Virtual celebrities/influencers
Virtual celebrities are "real" people who have created a unique persona that audiences can relate to, even though it is not real. These influencers have an audience, usually numbering thousands of followers on social media platforms like Instagram and Facebook. The virtual celebrity may have talent in singing or acting but does not sing or act; instead, they create accounts for people to follow and see their life online through photos and videos.
Lil Miquela
With social media increasingly focusing on the use of virtual influencers, there is a handful that stands out from the rest. Lil Miquela is one of them. She is the world's most popular virtual influencer on Instagram, with 3 million followers.
In addition to shooting ads for brands like Calvin Klein and Loreal, she has also appeared in videos with celebrities like Bella Hadid and J Balvin. She, however, isn't real. Brud's virtual effects team created Lil Miquela as a 3D model. She (it?) and other virtual influencers are becoming increasingly popular and will continue to do so.
MetaHuman
MetaHuman is a robust character generator that enables you to create any realistic human from scratch, whether in-game character design and development, animation and cinematic content, advertising, or entertainment.
MetaHuman Creator enables you to create fully rigged photorealistic digital humans in minutes, in real-time, for use in video games, virtual reality and augmented reality content, architectural visualizations, and more. As you may know, I started working with an avatar several years ago, and I am now transitioning to a MetaHuman character; a digital twin of myself!
Vtubers
Another excellent example of this trend is the rise of Vtubers like Blu. He is a space-themed Vtuber with a simple goal: Take over the entire galaxy on his spaceship, Xanadu. While he's orbiting the earth on his spaceship, Blu is getting to know earthlings, preparing himself for possible combat against all kinds of aliens that might challenge him, and exploring the latest in virtual reality technology. This YouTube channel has 75k subscribers and shows one of the many personas created with synthetic media today.
Synthetic Videos
Synthetic Videos allow you to combine the worlds of photography and videography. They have taken on many forms, but one of the most popular types is deepfakes. These are Face Swaps, where one person's face replaces another's (like this clip featuring Joe Biden and Donald Trump face swap in "Avengers: Endgame").
Face reenactment is another form of synthetic video in which the source actor controls the face of the target actor. For example, this technology allowed us to hear different world leaders singing Imagine by John Lennon and David Beckham speaking nine other languages for a campaign to end malaria in 2019.
Besides, one of the major innovations we will witness more frequently is the generation of text to video. The latest is a text-to-video AI called CogVideo that allows computers to generate short, coherent video clips from text descriptions alone. This is a significant step from our more recent, high-quality text-to-picture models.
Synthetic Images
Today Artificial Intelligence can recreate images more realistically than ever before. As a result, synthetic images have made big waves in the past year, being used for everything from creating NFT art to generating realistic stock photos.
Synthetic imaging is creating two-dimensional optical images using mathematical modeling computations of compiled data rather than the more traditional photographic process of using light waves focused through cameras or other optical instruments.
As an example of synthetic images, we have Thispersondoesnotexist.com, a website that uses AI to render photographic images of fictional people with a realistic result, launched some time ago.
Synthetic Art
Synthetic art is a category of artwork that combines digital images and computer graphics, typically with virtual 3D models and textures, to create a convincing simulation. The term is broad in scope, covering any form of media designed to visually appear as the result of synthetic means rather than naturally occurring phenomena.
One of the exponents of this new movement that has received a lot of notoriety recently is the Russian-French virtual reality artist Anna Zhilyaeva. You will see a completely different perspective by looking at paintings that Zhilyaeva has created in VR. She uses VR as an extension of her art studio to create new worlds and to push her painting medium past its limits.
Most recently, Dall-E 2, an advanced AI trained on 250 million images named after the surrealist artist Salvador Dalí and Pixar's Wall-E—now available in its beta version—creates unique, synthesized art by combining words with specific image features.
LinkedIn founder Reid Hoffman turned his Dall-E AI art into NFTs in Solana, the largest NFT marketplace. He launched his first collection of NFTs on July 21st. This is just a small sample of what this AI has to offer.
Reid Hoffman — is pioneering a new blockchain-powered art format that lets collectors purchase original artwork generated by artificial intelligence.
The predecessor of Dall-E 2 is named Dall-E. This framework is a GPT-3 (Generative Precomputed Tomography-3) trained to generate images from text descriptions. As well as combining and creating anthropomorphized versions of animals and objects, it can render text and apply transformations to existing images.
Many synthetic media startups are creating important artistic demonstrations and raising awareness about these new ways of making art through AI, including AIArtists.org, Wombo, Art AI, and Hotpot.
Synthetic Audios
More and more people are using audio tech to build their businesses, such as podcasts, social media influencers, online radio stations, and advertising campaigns. The process of recording anything, however, requires time, money, and effort (including voice artists, studios, equipment, etc.). Thus, artificial voice technology, such as text-to-speech (TTS) and voice cloning, has become very popular. For example, Resemble.ai is a popular company that allows you to clone your voice to create digital avatars and use them in movies.
From intros to outros, Respeecher's voice cloning technology makes it easy to create high-quality audio with almost no effort. Perfect for filmmakers and other content creators.
Another platform that has been a leader in this format is Voiseed. Its technology differs from others because it makes audio content more human by creating a voice interface that communicates in authentic, natural language using emotion and intellect.
Last but not least, Deepdub is an Israeli postproduction company specializing in international, multi-lingual audio and video media localization. They offer an innovative solution to the challenges of producing content for global markets.
Synthetic Music
With a meteoric rise in popularity over the past couple of years, it's no surprise that artificial intelligence-generated music creation software has the potential to change the way people make music drastically.
Synthetic music has the potential to generate sounds that are indistinguishable from a human-produced track. AI generates ever-shifting soundscapes for relaxation and focus, powers recommendation systems in streaming services, facilitates audio mixing and mastering, and creates rights-free music.
You can find AI-generated and copyright-free music using platforms like Icons8 and Evoke Music.
Synthetic Media at the Workplace: An Ethical Approach
Synthetic media tools provide benefits that are difficult to quantify. They could influence how we perceive performance, increase employee productivity, improve the quality of work, and foster a culture of innovation—in short, they have the potential to make organizations more competitive.
Synthetic media tools allow for creating complex data visualizations, or even videos, using only a spreadsheet. Analysts and researchers often use these to present findings to a broader audience. Art directors also use it to mock up ideas before they bring them to life in development.
In addition, when communicating with clients who speak different languages, synthetic media tools can help. For example, a customer who speaks German is calling in to ask about a product that can be served by an English-speaking employee using an artificial media tool that sounds like they are speaking German. As a result, the technology allows brands to provide the best service possible, regardless of language barriers. This technology is being perfected, increasingly being one of the pioneers in this regard Translatotron, an AI software launched in 2019 powered by Google.
Other uses in the workplace include creating training videos for employees and customers; creating personalized marketing campaigns for your most valuable prospects by having them say what you want them to say; and providing a unique selling point or quote to your business as a case study when pitching new clients/customers. Several platforms can create these projects, such as Synthesia, which offers a variety of solutions from employee training to marketing services, all generated with their AI software.
Despite many people's concerns about the potential risks of AI, many businesses are still committed to using it in one way or another. In fact, according to IBM, 66% of companies are executing or planning to apply AI to become more sustainable. AI innovations demonstrated in synthetic media breakthroughs are an excellent opportunity for companies to bring positive societal change. But the adoption of synthetic media can be scary, too. For ethical practices in AI development, technological innovations must work in tandem with various efforts aimed at regulating the use of AI by enterprises and individuals alike.
We must have rules of the road to guide their deployment—especially regarding ethics. As mentioned in previous articles, we should bring ethics to the code. As AI becomes ever more sophisticated, so do the ethical challenges AI faces. The biggest concern involves how to instill ethical behavior into AI. A significant element is ensuring that the algorithm will not engage in abusive or unethical practices towards humans and vice versa.
Synthetic Media and Deepfakes
We are entering a new age where more people will be exposed to synthetic media. It's a mass social experiment, and we have no idea what the consequences of this medium might be. If we cannot predict or study its impact accurately, there is little hope of protecting ourselves against its dangers.
While synthetic media can be compelling, it does come with risks. Since the system is in charge of creating meaningful and appropriate content for users, there is less control over what is created.
One of the most common uses of AI-driven media synthesis is to generate text that looks plausible but contains misleading, false, or non-existent information, popularly known as fake news. This tactic is known to be used in spam campaigns and malicious advertising practices.
The best-known use of AI media synthesis is generating fake audio and video. For example, people can use this technology to create a movie of someone saying things they never spoke, also known as "deepfake."
Ninety percent of Americans believe deepfakes could lead to more harm than good. And rightly so. The rapid growth in deepfake videos from 2019 to 2020 began penetrating mainstream internet platforms. If you've been paying attention to TikTok recently, you may have seen deepfakes. Specifically, famous faces became victims of deepfakes.
You might want to watch out for one account: @deeptomcruise on TikTok. With 3.6 million followers to date, Miles Fisher (the creator behind this account) posts deepfakes of the actor—and its videos have racked up millions of views. The account makes videos of him talking, laughing, and making faces realistically and convincingly. It is so impressive that you cannot even imagine it is not the real actor.
Sometimes people use deepfakes to advance political viewpoints they support, and they make it clear that the video is intended as commentary and not as a tool for harassment.
When it comes to deepfake issues, journalism cannot escape the fact that its old forms of reporting are under pressure due to the rise of digital information. Therefore, we need media literacy and verification to report on these deepfake videos and regarding worldwide disinformation or propaganda.
On the legal side, personal rights and intellectual property laws also affect counterfeiting scenarios. The legality of AI-generated counterfeit content is often unclear, making it difficult to know where your rights lie. Copyright law protects original intellectual property from copying; however, in an era of exponential growth, it will soon be unable to distinguish between "real" and "fake" text. Moreover, whether we should allow people who have not created a text or image to profit from it remains.
Synthetic media will need to be regulated by law and policy, so we'll need new rules to determine ownership and licensing.
Common questions are: Who should own the rights to a synthetic movie where all the actors are created digitally? The studio or the creators of the algorithm that generated the characters?
These issues need to be addressed now before they become real problems down the road.
The Future of Synthetic Media at Work
Do algorithms make the decisions morally correct?
This question has been at the center of much debate in recent years. This is because algorithms are not only models of a particular aspect of the world but also, increasingly, models of complex social interactions.
Ethical issues are not only computational problems but also sociocultural ones. For example, they may result from algorithms (such as biases and discriminatory outcomes) or data collection and use. As a result, they require a much deeper understanding of the social impact of AI-based technologies, past experiences with similar innovations, and their ethical implications, especially when it comes to synthetic media.
For example, a working definition of harm would need to be established if an AI machine was asked to make ethical decisions based on whether it could cause potential harm or benefit to another entity in its immediate environment.
While ethically responsible behavior is an essential aspect of modeling, it can be challenging to model. Data governance can be used to address the easier-to-research elements of ethical behavior, like bias in data. Data governance is an approach that is implemented to support data quality. This process enables firms to stay compliant, audit data, and use it for reporting and investigating purposes.
One of the key ethical questions is: Can you ensure that AI-influenced content is of high quality? If we hand over the power of content creation to AI, we need to understand what that entails from a human perspective.
AI has already proven its worth in various industries—from automotive manufacturing to voice recognition. But what if AI becomes so good at creating content that humans become obsolete?
There are two areas I see this affecting the workplace. Generally, using AI for content creation will reduce some jobs and create new ones. The areas where I see this happening are in moderation and curation. I think that as AI becomes better at writing or designing than humans, we could use it in moderation or curation of content.
Organizations will develop a new type of job that integrates tasks typically performed by humans with machine learning capabilities that are better suited for these tasks than humans. As a result, new roles will emerge where the primary focus is interacting with AI to help it become more intelligent and capable.
As companies realize that AI can reduce costs and increase profits, it will be increasingly difficult for organizations to continue doing business without it. But the skill of those who work with AI is equally important. If employees do not stay updated on technological advancements and improve their knowledge, they could be forced out of their jobs—no matter how hard they try to avoid automation.
Final Thoughts
The key takeaway is that synthetic media is a new kind of media that is getting more realistic and easier to handle. Moreover, they combine traditional media and digital means, making the whole thing capable of fantastic results. That's how synthetic media will bring the movie experience we all crave–but better.
Over the next few years, we expect that customers will continue to embrace technological advances most innovatively and practically possible. Companies will need to continue to develop new technologies and keep up with the pace of change to maintain profitable operations.
The global industry will continue to be dominated by several major players who can innovate and be flexible. Innovation has been the key driving force behind business growth over the past years and will likely remain so for years.
Cover image: A set of AI-generated variations on a self-portrait by Salvador Dalí