VibeVO – Advertisers and creators in one ecosystem

Tag: ai avatar

  • AI Avatar: How to Add a Face to a Voice Assistant and Avoid Pitfalls

    AI Avatar: How to Add a Face to a Voice Assistant and Avoid Pitfalls

    Integrating an AI avatar into the existing voice AI administrator Aika is not just about adding a face; it involves a whole chain of technical challenges. We aimed to preserve the current architecture on Yandex Realtime so that the new service would only animate lips to already generated sound. During testing of Simli and Anam, nuances emerged with pauses before responding, video playback on iPhone, and mobile internet stability. And when creating the second character, Eva, it became clear that the persuasiveness of the neural network depends not only on the face itself but also on the context.

    Our voice engine was already functional, so the task was solely to animate the face. Aika communicates with website visitors directly in the browser via WebSocket, which is connected to our FastAPI server, and that, in turn, maintains a session with Yandex Realtime. This ensures a “speech-to-speech” mode, eliminating the need for separate recognition chains, text models, and synthesis.

    The server transmits uncompressed sound in PCM16 format at 24 kHz in chunks (pcm_chunk) to the browser, and it was precisely to this stream that the avatar’s facial expressions needed to be synchronized.

    Choosing a platform for the AI Avatar: Simli vs Anam

    Both tested services — Simli and Anam — offer a mode where the user provides their audio stream, and the service generates video with an animated face synchronized with the sound, returning media via WebRTC. Thus, Yandex continues to generate speech, and the avatar service is responsible for the visual component.

    Technical aspects of sound integration

    • Sampling rate: Yandex outputs sound at 24 kHz. Simli requires 16 kHz, Anam was also configured for 16 kHz, although it supports other formats.
    • Processing audio chunks: Sound arrives in chunks. To avoid artifacts at the junctions, the fractional position and the last sample are carried over to the next chunk. This ensures smooth animation, although the quality of frequency recalculation without a full-fledged filter still requires improvement.

    Security and session management

    The service key does not reach the browser. The page requests a short-term session token from our server via POST /voice/avatar-session. The server checks the origin and limits token issuance: 6 per hour per IP and 30 per hour total.

    If the token is not received or the video does not start, the page switches to normal voice mode. 8 seconds are allotted for the token request, and 15 for SDK launch. It is important that a successful SDK launch does not guarantee the immediate appearance of the first frame; the face here is an addition, not a critical function.

    Managing Avatar State in Anam

    Anam provides two key commands:

    • endSequence(): called after sending the last audio segment of a replica.
    • interruptPersona(): stops the avatar’s current speech when interrupted, after which endSequence() is also called.

    Comparison of Simli (old generation) and Anam Cara-4

    We compared the versions available to us: Simli of the previous generation and Anam Cara-4. This is not an exhaustive comparison of all features, but a selection for a specific task.

    Parameter Simli (previous generation) Anam Cara-4
    Mimicry Lips move, other mimicry is limited Blinks, smiles, teeth visible, tilts head
    Face creation From photo via previous generation API From a single photo, trial video available

    Subjectively, Anam showed more lively and natural mimicry. For us, this was a decisive factor. Next, we encountered a number of specific problems.

    Media Playback Features

    • Sound: In Simli, sound was played through a separate audio element; in Anam, it was played through the same video element. This means that the Anam video element cannot be left muted, otherwise the avatar will “speak” silently.
    • Resolution: Video often started at a low resolution (384×576) and only switched to high resolution (768×1152) after a few seconds. The sessionOptions.videoQuality: "high" option helped, but did not guarantee a high-quality start.
    • Frame generation delay: Anam needs to receive 0.8 seconds of audio before starting frame generation. This is not always equal to 0.8 seconds of real-time waiting, as audio can arrive faster than it is played.

    Microphone and Session Management

    • Echo suppression: On the phone, while Aika is speaking, the microphone is muted to avoid picking up her voice from the speaker. On the PC, the microphone is active, and when the user starts speaking (barge_in event from Yandex Realtime), the avatar stops its current speech.
    • Busy session slot: On the free plan, one concurrent session is available. If the browser crashed, the slot could remain occupied. The GET /v1/sessions/concurrency request helps diagnose slot occupancy.
    • Connection cancellation: If the user clicks “stop” during video connection, it is necessary to correctly complete asynchronous operations so as not to leave an unnecessary session. For this, a generation counter videoGen was introduced.

    Problems with Safari on iPhone

    On iPhone in Safari, the video did not start. The hypothesis is related to autoplay rules. A preliminary call to play() on the video element with temporary stream feeding from canvas and unmuting helped in a test run, but is not a universal solution.

    Optimizing the Response Pause

    A long pause before an avatar’s response, where it silently stares at the user, feels much heavier than without it. Initially, the Voice Activity Detection (VAD) threshold was set to 2.2 seconds to prevent Aika from interrupting users dictating information with pauses.

    We transferred the logic of the telephone bridge to the website: in a normal conversation, the threshold is 1500 ms, when dictating — 2200 ms. This allows dynamic adjustment of VAD sensitivity. Settings are applied via session.update without reconnection.

    AI-аватар: как добавить лицо голосовому помощнику и избежать подводных камней — illustration 2

    Creating a Second Character: Eva

    For the studio administrator Eva, we faced the problem of a “stock blonde” against an empty background. The turning point came when an interior appeared behind her: warm lamps, shelves, plants, a blurred background. This made the image much more convincing, transforming a “catalog” into a “person at their workplace.”

    Details of Background and Portrait Creation

    • Background: We used a ready-made interior in our style, removing people with a generator.
    • Character Cutout: Eva was cut out using macOS Vision (VNGeneratePersonSegmentationRequest with .accurate level). The mask edge was narrowed and softened to eliminate the halo.
    • Scale: It is important to maintain the scale of objects in the background to avoid comical situations (e.g., a giant mug behind her). The background was taken only above the countertop and blurred for a portrait lens effect.
    • Resolution: For Cara-4, a square source of 1152×1152 or higher is recommended. Pure Real-ESRGAN ×4 upscaling made the skin too smooth. A combination of neural network upscaling with regular scaling, sharpening, and grain addition looked better.

    Eva’s Voice Acting

    For Eva, we chose a voice from Yandex’s LiveTTS family. In SpeechKit API v3, the model is selected via "model": "livetts", and the voice and role are passed via hints. For long texts, the text was broken down into sentences and synthesized separately.

    Out of seven options for voicing one phrase, we chose the vera voice in the casual role. We had to rewrite the phrase because the synthesis consistently made a mistake with the stress.

    Recording Videos with the Avatar

    Eva’s video was recorded using the same animation engine as a live call, using MediaRecorder directly from video.srcObject (video and audio together, WebM with VP9 and Opus). For correct recording start, the stream was pre-“woken up” with a second of silence so that the video would not start with a black screen.

    Frequently Asked Questions

    What is an AI avatar and how does it work?

    An AI avatar is a digital character capable of animating facial expressions and lip movements in accordance with provided audio. It uses neural networks to create a realistic visual image that synchronizes with speech, making interaction more natural and engaging.

    Can I use my own voice for an AI avatar?

    Yes, most platforms, such as Anam and Simli, allow you to use your own voice engine or pre-recorded audio. The avatar will animate its face to this audio stream, maintaining your brand or persona’s recognition.

    What are the main challenges when integrating an AI avatar?

    The main challenges include audio and video synchronization, processing audio chunks, optimizing latency, ensuring stable operation on various devices (especially mobile), and creating a convincing visual image that matches the context and brand.

    How does the background affect the perception of an AI avatar?

    The background plays a critical role in creating a convincing image. An empty or inappropriate background can make the avatar appear unnatural. An interior that matches the character’s or brand’s style significantly increases the avatar’s realism and trustworthiness, creating the feeling of a “person at work.”

    How much does it cost to create an AI avatar?

    The cost of creating an AI avatar can vary greatly depending on the chosen platform, the level of customization, the detail of facial expressions, and the need for unique functions. Some services offer subscription models, while others may charge per minute of generated video or for the development of a unique character.

    Conclusion

    Integrating an AI avatar is a powerful tool for increasing user engagement, but it requires deep technical development. From platform selection to fine-tuning backgrounds and pauses, every step affects the final result. We have seen that neural networks can breathe new life into voice interfaces, but only with careful attention to detail.

    If you want to scale your content and create AI video for advertising, consider the possibilities of AI production by subscription. This allows you to generate hundreds of videos, whether it’s AI UGC for a crypto project or an AI video factory for e-commerce. We are ready to help you with this! Contact us to find out how AI content production can transform your business.

  • How to Create Professional AI Videos with Gemini Omni: A Complete Guide

    How to Create Professional AI Videos with Gemini Omni: A Complete Guide

    Want to create professional videos without filming or hiring a team? Wondering how to use AI-generated video intros to grab attention on social media? In this article, you will learn how to create an AI avatar, produce catchy video hooks, edit existing material, and assemble long videos from 10-second clips using Google Gemini Omni. No more racking your brain over choosing a contractor or wasting nerves searching for specialists – all this can be done in one place, reducing costs and time.

    Introduction: Revolutionizing Video Production with Google Gemini Omni

    In today’s world, where visual content reigns supreme, creating high-quality videos is becoming a key success factor for brands and individual content creators. However, traditional video production often involves high costs and complex processes. This is where Google Gemini Omni comes in, offering a revolutionary solution. This tool allows you to create professional videos using advanced artificial intelligence technologies, making the process accessible, fast, and cost-effective.

    What is Google Gemini Omni (OmniFlash)?

    Google Gemini Omni, officially known as OmniFlash, is the successor to Veo3 – Google’s first AI video generator. Omni is built on Google’s “world models”, which were trained on an extensive YouTube video library. This training gave it a unique understanding of movement, environment, people, and speech intonations, which sets it apart from other AI video tools.

    Thanks to this, you get unique and high-quality videos that are difficult to distinguish from those shot by professionals. OmniFlash opens new horizons for content creation, allowing users to embody the boldest ideas without the need for expensive equipment or specialized skills.

    Access to Gemini Omni: Levels and Plans

    Omni is available on several platforms, making AI video creation accessible at different levels. The easiest way is through the Gemini app, available on desktop and mobile devices. Within the Gemini chat interface, by clicking the “plus” button, you will see the “Video” option, which launches Omni in the background.

    • Gemini App: Ideal for beginners. Subscription starts at $20/month.
    • Google Labs: For advanced editing and modification features.
    • Third-party aggregators: Open Art and Higgs Field, which combine several AI models.

    According to Eva Whitaker, a basic Gemini subscription for $20 a month is sufficient to start. The more expensive “ultra” tier at $200 a month includes access to all Google tools, but most users won’t need it right away. Credits are consumed for each generation, with higher resolution and longer clips requiring more credits.

    Starting with the lowest tier and purchasing additional credits as needed will allow you to control costs.

    “For those serious about AI video production, Eva recommends combining a direct Google Labs subscription with a third-party aggregator. The aggregator provides access to multiple AI models from a single dashboard, while Google Labs unlocks the full Omni editing capabilities.”

    Creating an AI Avatar: Your Digital Twin

    Unlike clones created by HeyGen, which generate talking-head videos from a long script, an avatar is an “AI twin”: a digital representation of a real person that can be placed into any scene or scenario using text prompts. This allows for unique content and makes it as personalized as possible.

    Avatar Creation Process

    The avatar creation process takes about five minutes. Open the Gemini app on your phone, tap the plus button, and scroll to “avatar.” The app will guide you through a capture sequence similar to Face ID: look left, right, up, down. Then it will display nonsensical sentences for you to read aloud to record your voice.

    The intentionally strange text is designed to capture natural speech patterns without overthinking pronunciation.

    • Lighting: Natural light works best.
    • Audio: Minimize background noise for a clean voice recording.
    • Clothing: Choose a versatile outfit, as it will become the default outfit.
    • Hats and glasses: Avoid them during recording so that the AI does not distort the face.
    • Tone of voice: Speak naturally to maintain flexibility in future generations.

    Once created and named, the avatar syncs across platforms. This means that a user who created their avatar on their phone can also summon it in Google Labs on the desktop by typing the “@” symbol followed by the avatar’s name.

    Effective Prompts for AI Video: The Four-Element Formula

    Effective prompts for AI video follow what Eva Whitaker calls the “four-element prompt formula”: subject, action, environment, and camera. Omni responds well to natural language, meaning the days of formatting prompts in JSON or coded syntax are over.

    1. Subject: Who or what appears in the video (e.g., @avatar_name).
    2. Action: What the subject is doing (dancing, talking to the camera).
    3. Environment: The setting (a beach in Mexico, a kitchen).
    4. Camera: The camera’s position and movement relative to the subject.

    Eva advises starting simple and iterating. For example, if the first generation places the subject too far from the camera, the next prompt adds: “she stands close to the camera” or “she approaches the camera.” Each generation shows how Omni handles different types of instructions.

    Including a script in the prompt is crucial. Without it, Omni will generate its own dialogue, which sometimes leads to interesting coincidences, but more often to unusable audio.

    How to Create Professional AI Videos with Gemini Omni: A Complete Guide — illustration 2

    Creating Engaging Video Hooks for Social Media

    The primary use case Eva recommends for Gemini Omni is creating attention-grabbing hooks for short videos on TikTok, Instagram, YouTube Shorts, and other platforms. The concept is simple: shoot a standard talking-head video the traditional way, then use Omni to create a visually striking 2-3 second hook to open the video. This allows for maximum audience engagement as quickly and effectively as possible.

    Hook Examples and Creation Process

    Eva created hooks where her avatar is slimed Nickelodeon-style, has a cake shoved in her face while wearing an evening gown, stands in a park where tennis balls fall from the sky, rides a buffalo in front of a house for sale sign, skydives onto a lawn next to a “for sale” sign, and walks across a kitchen counter as a six-inch avatar, saying: “The only small thing in this kitchen is me.”

    Speed of creation is part of the appeal. Eva created the buffalo-riding hook while waiting in line for coffee: she opened the Gemini app on her phone, summoned her avatar, asked it to ride a buffalo in front of a house for sale sign, using a Zillow photo as a reference image, and had the finished clip by the time she picked up her order. This mobile workflow allows for polished hooks to be created anywhere, without a desktop computer or editing software.

    The “Captain Obvious” Method for Creative Ideas

    Eva uses LLMs to generate AI video hook concepts rather than brainstorming them alone. The initial process starts with describing a topic and asking for creative visual hooks. The initial suggestions from any LLM are typically predictable, and this is where Eva’s “Captain Obvious” technique comes in.

    This method allows for guiding the LLM away from its default, predictable suggestions to more creative and distinctive ideas. When the LLM suggests a predictable visual, such as a person sitting at a computer with too many tabs open to represent “AI overload,” Eva tells it:

    “That’s too obvious. That’s something anyone could come up with. Give me something smarter, more creative, something that makes people stop and think, ‘What was that?’”

    This redirection consistently leads to more creative and distinctive concepts. When combining an AI-generated hook with a traditionally filmed talking head video, visual continuity is important. If the avatar appears in different clothing than in the video, the transition feels jarring. Eva’s solution is to take a screenshot from the video and upload it as a reference image, then ask Omni to dress the avatar in the same clothing. The hook clips and talking head video are then edited together in CapCut, Instagram Edits, or a similar tool.

    Editing Existing Videos with Gemini Omni

    Gemini Omni can alter existing videos using natural language prompts, making editing that once required a professional editor accessible to anyone. These capabilities, available through Google Labs and aggregator platforms, go far beyond simple generation. Real footage, shot on a phone or AI-generated, can be uploaded and transformed with a text description of the desired change. This allows for budget savings on post-production and quick results.

    Editing Capabilities

    • Changing the environment: A summer park can be transformed into a winter one with a single prompt.
    • Adding motion graphics: Animated text overlays with ingredient names can be added to a cooking video.
    • Changing the subject’s clothing: A character can change outfits at your request.
    • Removing objects from a scene: Remove unwanted elements from the frame.
    • Transforming a person into a cartoon version: Create a unique visual style.
    • Complete background change: A talking head video can be transported to a beach or a news studio.

    Tip: Videos uploaded for editing should be trimmed to 10 seconds or less. Omni provides a built-in trimming tool, so longer footage can be cut to the appropriate segment before applying modifications.

    Creating Long Videos from 10-Second Clips

    The 10-second generation limit does not prevent the creation of longer AI video content. This requires prior planning and a third-party editor. For a 90-second video, the approach is to generate nine 10-second clips and edit them together. The key is to plan what the last frame of each clip looks like and how it transitions into the first frame of the next clip.

    Eva suggests using an LLM for this planning, giving it the full concept and specifying the limitations: “create me a 90-second video consisting of 10-second segments, and give me transition recommendations”. The LLM will suggest transition strategies and help plan each segment.

    Frequently Asked Questions

    Here we have collected answers to the most frequent questions about Google Gemini Omni.

    How to get started with Gemini Omni?
    Start with the Gemini app, available on desktop and mobile devices. A subscription starting from $20/month will allow you to explore the basic features. For more advanced capabilities, consider Google Labs or third-party aggregators.
    What are the limitations of Gemini Omni?
    The main limitation in the Gemini app is the generation limit (about 5-6 videos per hour). Google Labs and aggregators offer more generous limits. Avatars are limited to 10-second clips, which need to be edited to create longer videos.
    Can I use my avatar on different platforms?
    Yes, once created, the avatar is synchronized across platforms. You can create it on your phone and use it in Google Labs on your computer.
    How to achieve high-quality avatar?
    The recording environment plays an important role: natural lighting, minimal background noise, versatile clothing. Avoid hats and glasses during recording. Speak naturally to maintain flexibility in future generations.
    How does Gemini Omni help save budget?
    Using AI tools like OmniFlash allows you to replace a team with a service, reducing video production costs. You can order a full cycle of video production in one place, from avatar creation to editing and posting, which is significantly cheaper than hiring individual specialists.

    Conclusion: Full-Cycle Turnkey Video Production

    Google Gemini Omni (OmniFlash) is a powerful tool for creating professional videos using AI. It allows not only to generate unique avatars and captivating hooks but also to edit existing videos and assemble long videos from short segments. This is an ideal solution for those who want to get a full cycle of short video production, comprehensive promotion through videos, and content outsourcing for social networks, replacing an SMM team with one service.

    Start using Gemini Omni today to save budget and nerves by creating high-quality video content that truly captures the audience’s attention. Order everything in one place and see how effective this approach is!