Building a RAG System — My First AI Side Project
Building a RAG System — My First AI Side Project
Over the winter break, I dove into building a Retrieval-Augmented Generation (RAG) system from scratch. What started as a learning exercise turned into a proper side project that's now open source on GitHub.
What is RAG?
RAG is a technique that combines the power of large language models with external knowledge retrieval. Instead of relying solely on what the LLM "memorized" during training, you give it relevant documents at query time so it can answer based on actual source material.
This is especially useful for domain-specific questions where the LLM's training data might be outdated or insufficient.
The Project: rag-research
I built rag-research — a complete RAG pipeline using Rust's official documentation as the knowledge base. The idea was simple: ask questions about Rust in natural language, and get answers grounded in the official docs.
Tech stack:
- LangChain for orchestrating the RAG pipeline
- ChromaDB as the vector database for storing document embeddings
- OpenAI API for embeddings (text-embedding-3-small) and generation (GPT-4o-mini)
- FastAPI for the REST API with SSE streaming
- Chainlit for an interactive web UI
- Docker for containerized deployment
How it works:
- Download and process Rust documentation into text chunks
- Generate embeddings and store them in ChromaDB
- At query time, find the most similar chunks to the user's question
- Feed those chunks as context to the LLM for answer generation
- Stream the response back in real-time via SSE
What I Learned
Building this taught me a lot about the practical challenges of RAG:
- Chunking strategy matters — how you split documents affects retrieval quality significantly
- Embedding quality is crucial — garbage in, garbage out applies to vector search
- Streaming improves UX — users don't want to wait 10 seconds for a wall of text
- Multilingual RAG is hard — supporting EN/JA/ZH required careful handling of language-specific queries
What's Next
This project became the foundation for Kokoron, the AI assistant on this blog. I'm planning to evolve it from a Rust docs Q&A tool into a more general-purpose blog content assistant.
Check out the project on GitHub if you're interested. Feel free to reach out via email if you have questions or want to discuss RAG architectures.