🧠 Local RAG Chatbot with Ollama on Mac

Lightweight, private, and customizable retrieval-augmented chatbot running entirely on your Mac.

Based on the excellent work by pruthvirajcyn and his Medium article.

⚙️ About This Project

This is my personal implementation of a local RAG (Retrieval-Augmented Generation) chatbot using:

  • Ollama for running open-source LLMs and embedding models locally.
  • Streamlit for a clean and interactive chat UI.
  • ChromaDB for storing and querying vector embeddings.

As of 2025-07-17, I’m using:

  • 🔍 Embedding model: nomic-embed-text-v2-moe
  • 🧠 LLM: gemma3n

💡 Why Run a RAG Locally?

  • 🔒 Privacy: No data is sent to the cloud. Upload and query your documents entirely offline.
  • 💸 Cost-effective: No API tokens or cloud GPU costs. You only pay electricity.
  • 📚 Better than summarizing: With long PDFs or multiple documents, even summaries may not contain the context you need. A RAG chatbot can drill deeper and provide contextual answers.

✅ Recommended: At least 16GB of RAM on your Mac. Preferably 24GB+ for smoother experience.

🛠️ 1. Installation

1. Clone the Repository

git clone https://github.com/eplt/RAG_Ollama_Mac.git
cd RAG_Ollama_Mac

2. Create a Virtual Environment

python3 -m venv venv
source venv/bin/activate

3. Install Dependencies

pip install -r ./src/requirements.txt

🚀 2. Usage

1. Start Ollama and Pull the Models

ollama serve
ollama pull gemma3n
ollama pull toshk0/nomic-embed-text-v2-moe:Q6_K

2. Load Documents

Place your .pdf files in the data/ directory.

python ./src/load_docs.py

To reset and reload the vector database:

python ./src/load_docs.py --reset

3. Launch the Chatbot Interface

streamlit run ./src/UI.py

4. Start Chatting

Ask questions and the chatbot will respond using relevant context retrieved from your documents.

🧩 3. Customization

  • ✏️ Modify Prompts
    Update prompt templates in UI.py to guide the chatbot’s tone or behavior.
  • 🔄 Try Different Models
    Ollama supports various LLMs and embedding models. Run ollama list to see what’s available or try pulling new ones.
  • ⚙️ Tune Retrieval Parameters
    Adjust chunk size, overlaps, or top-K retrieval values in load_docs.py for improved performance.
  • 🚀 Extend the Interface
    Add features like file upload, chat history, user authentication, or export options using Streamlit’s powerful features.

🧯 4. Troubleshooting

  • Ollama not running?
    Make sure ollama serve is active in a terminal tab.
  • Missing models?
    Run ollama list to verify models are downloaded correctly.
  • Dependency issues?
    Double-check your Python version (3.7+) and re-create the virtual environment.
  • Streamlit errors?
    Ensure you’re running the app from the correct path and activate your virtual environment.

📌 Notes & Future Plans

  • Planning to support non-PDF formats (Markdown, .txt, maybe HTML).
  • Will experiment with additional LLMs like phi-3, mistral, and llama3.
  • Might integrate chat history persistence and better document management.

👋 Final Thoughts

Local RAG is now more accessible than ever. With powerful small models and tools like Ollama, anyone can build a private, intelligent assistant — no cloud needed.

If you found this useful or have ideas to improve it, feel free to open a PR or drop a star ⭐️