Overview
A Chrome extension that transcribes the current YouTube video, embeds the transcript and answers context-aware questions about it from a popup interface.
The problem
Long videos hide the answer you need somewhere in an hour of audio. Viewers had no way to ask a video a question and jump to the relevant part.
Context
A browser extension with tight latency and memory constraints, running against arbitrary videos with no pre-processing.
Constraints & challenges
- 01Transcripts must be generated and indexed on demand for videos never seen before.
- 02Embeddings need to be multilingual and robust to noisy speech-to-text.
- 03Answers must reference the relevant segments, not summarise the whole video.
- 04Everything has to feel instant inside a popup UI.
My role
Solo developer: extension front end, transcription pipeline, embedding and vector indexing, retrieval and LLM answer generation.
The solution
The extension extracts the video ID, transcribes the audio through a transcription API, splits the transcript into timestamped segments and embeds them with BGE-M3 into Pinecone. Questions from the popup are embedded, matched against the most relevant segments and answered by Gemini with segment context.
- One-click transcription of the current video
- Semantic search over the transcript
- Context-aware Q&A in a popup
- Segment-level references in answers
System architecture
System architecture
- Source
- API / Service
- Processing
- Model / LLM
- Storage
- Interface
- Output
Indexing happens once per video and is keyed by video ID so repeat questions are answered from the existing vectors.
Key engineering decisions
- 01
BGE-M3 for embeddings
Strong multilingual, long-context embeddings that hold up on noisy transcripts.
- 02
Pinecone for retrieval
Managed vector search kept the extension stateless and fast to iterate on.
- 03
Segment-level context
Passing timestamped segments to Gemini lets answers point to where in the video the information appears.
Technology
- JavaScript
- Python
- LangChain
- Pinecone
- BGE-M3
- Gemini
- Transcription API
- Chrome Extension
Results
Delivered a working RAG assistant for any YouTube video, demonstrating a complete on-demand ingestion → embedding → retrieval → generation loop inside a browser extension.
Lessons learned
- On-demand indexing changes the cost model — cache aggressively per resource.
- Embedding quality on noisy text matters more than prompt tweaks.
Building something similar?