My Local AI Search Tool Using a 7B Model, details on how it was made below

[u/Impressive_Sentence7]

Stack:

LLM: Qwen2.5-7B-Instruct (Q5_K_M) (~5.4 GB) via llama.cpp (CUDA with Flash Attention & q8_0 KV cache) or Ollama fallback, Qwen is very good at tool use, Gemma may also work; Deepseek likes to <think> too hard and breaks things

Re-Ranker: FlashRank(ms-marco-MiniLM-L-12-v2).

Backend / GUI: FastAPI + WebSockets + PyWebView (Native Desktop window, zero Electron overhead).

VRAM Footprint: ~6–7 GB on an RTX 3080 / 3060 / 4070 (also supports CPU fallback).

The code is 100% open-source and free to check out, run, or fork: GitHub: https://github.com/BloopieBlair/LocAi-Search

Would love to hear any thoughts, feedback, or ideas for extra features!