r/agi • u/HotEstablishment7184 • 4d ago

Under 3 Seconds

After a lot of iteration, I finally got Christine’s latency consistently down to under 3 seconds using Warranted Retrieval.

That matters because Christine is not a cloud wrapper. She is laptop-bound, runs with no internet access, and has to operate within the actual limits of local hardware. Getting the response path down into a consistently usable range was a major milestone for me.

Now that the latency fight is finally in a much better place, it’s time to focus much harder on Christine’s training.

The next phase for me is less about shaving milliseconds and more about improving: - domain depth - retrieval quality - abstraction across domains - reasoning consistency - task usefulness under strict local constraints

Current laptop: - CPU: Intel Core Ultra 9 285H - RAM: 33.8 GB total physical memory - GPU 1: NVIDIA GeForce RTX 5050 Laptop GPU - GPU 2: Intel Arc 140T GPU - NPU: Intel AI Boost

I’m especially interested in what other people are doing with NPUs.

Are any of you actually using the NPU in a meaningful way for local/offline AI right now? If so: - what workloads are you pushing onto it - is it helping with latency, power efficiency, or always-on assistant behavior - are you using it for STT, routing, embeddings, background inference, or something else - and is it genuinely useful, or mostly just there in theory

Would like to hear from people building real local systems, especially laptop-bound ones.

0 Upvotes

50% Upvoted

3 comments sorted by

5

u/Minute-Plastic157 4d ago

Ah Christine, haven’t heard of her in a while (ever).

1

u/ibstudios 4d ago

My ai is not a llm but I move python logic into C and cuda if I can.

1

u/jacobpederson 3d ago

3 seconds - very impressive. I was happy to get to real time hiding behind a 30 second buffer https://www.youtube.com/watch?v=eZJVeI0GfAs&t=3s (100% local).