The problem
Organizations often have useful knowledge locked inside long training videos. A manager may remember that a five-minute explanation exists somewhere in hundreds of hours of recordings, but the exact file and timestamp are difficult to find with traditional search.
The disclosed system turns that video library into a searchable question-and-answer experience. Instead of asking users to scrub through folders of MP4 files, it lets them ask a natural-language question and receive an answer grounded in the relevant video moment.
The core idea
The efficient move is to convert video into aligned text before retrieval. Speech-to-text captures what was said, computer vision describes what appeared on screen, and timestamps connect both streams to the exact second of the source video.
Once those audio and visual descriptions are merged into one structured text record, the system can use a conventional vector database and RAG pipeline. The retrieval layer searches text, while the answer layer uses an LLM to produce a response that points back to the relevant evidence.
Pipeline
- Prepare the video files and upload them to cloud object storage.
- Extract frames at fixed intervals so visual moments can be described.
- Generate captions for those frames and preserve their timestamps.
- Convert video audio into text using speech-to-text.
- Use an LLM to clean the transcript into readable sentences.
- Merge transcript and visual captions into a timestamped dataset.
- Load the resulting text into a vector database for semantic search.
- Pass top matching chunks to an LLM-backed assistant for answers.
Architecture
This is the original architecture diagram from the disclosure, showing how document data, video data, vector search, tools, agents, and the LLM work together.
Why it is efficient
The architecture avoids storing image embeddings as the primary retrieval asset. By representing both speech and visuals as text, the system keeps the searchable layer simpler, easier to inspect, and compatible with standard RAG infrastructure.
The same assistant can also search supporting documents. In the diagrammed architecture, document text and video-derived text live in vector databases, while an agent chooses the right retrieval tool, collects the top matching chunks, and asks the LLM to synthesize the final response.