Vocal AI Studio

VocaAI Studio is an AI-powered real-time voice interaction and video voiceover platform. It allows users to communicate naturally with an AI assistant through their microphone while also providing a separate feature for adding AI-generated voiceovers to uploaded videos. The main objective of the project is to combine real-time conversational AI with AI-based voice generation in a single, user-friendly platform.

The real-time conversation feature is built using Agora Conversational AI, which provides the real-time communication infrastructure between the user and the AI agent. When the user starts a conversation, their voice is captured through the microphone and transmitted through an Agora channel. Deepgram Speech-to-Text (STT) converts the user's speech into text, which is then processed by the OpenAI Large Language Model (LLM) to understand the user's request and generate an appropriate response. The generated response is then converted back into natural speech using MiniMax Text-to-Speech (TTS) and delivered to the user in real time. This creates a two-way conversational experience where the user can speak to the AI naturally and receive an immediate voice response.

The application also provides a customized conversation dashboard where users can view the ongoing conversation and transcript, monitor the AI interaction, control their microphone, and end the conversation. The interface is designed specifically for the project rather than simply presenting the default Agora demonstration interface. This allows Agora to function as the underlying real-time conversational technology while the project provides its own customized user experience and functionality.

In addition to real-time conversations, VocaAI Studio provides an AI video voiceover feature. Users can upload a video through the application and preview it before processing. They can provide a voiceover script and select an AI voice. The system uses Edge TTS to convert the provided script into an AI-generated voice. The generated audio is then combined with the uploaded video using FFmpeg, producing a new video containing the original visual content along with the generated AI voiceover.

Overall, the project combines real-time communication, speech recognition, large language models, speech synthesis, and video processing into one platform. It can be used for applications such as AI assistants, educational content, video narration, tutorials, presentations, accessibility solutions, and AI-powered content creation. The main innovation of the project is the integration of a real-time conversational AI experience with an additional AI video voiceover workflow, making the platform useful beyond a conventional voice-assistant demonstration.