Hi,

I’m looking to run a local AI model on a laptop. The business use case involves managing highly complex building files, with several thousand documents forming a knowledge base.

I’m considering an on-premises laptop setup with an Intel i7, 64 GB of RAM, and an RTX 4090 with 16 GB of VRAM.

It should theoretically be able to run a 30B-parameter model, but I’m still looking for the best model to use.

Does this sound like a good approach?
Welcome.

So, I have a lot of ram, 1.5 TB ddr4 ecc rdimm in 32gb sticks in old servers running a proxmox cluster, they don't support any decent gpus. Thinking maybe making an AI server for MoE models like glm5.3 flash UD-Q4_K_XL, and adding a gpu. Ideally I'd want a board with 16 ram slots and support for pciex16 3.0 or 4.0.

Or would it be better to sell all or most of it and try to get a more modern 512gb unified memory system?

Hey everyone,

I’m one of the co-founders of axem. We recently open sourced shaide, the infrastructure we’ve been building for running multiple LLMs across GPU servers.

It’s probably not very interesting if your setup is one model on one GPU. We started building it when our setups grew beyond that and we needed to run several models at the same time, scale them independently across GPU nodes, route traffic between replicas, and keep everything running inside infrastructure we controlled.

Instead of configuring all of those pieces separately for every cluster, we ended up packaging them into one Kubernetes-native platform.

Current setup:

• vLLM for inference
• llm-d for multi-instance orchestration
• multiple models running and scaling independently
• KV-cache-aware scheduling
• internal OCI registry for container images + model weights
• OpenAI-compatible API
• the entire platform is managed as infrastructure as code
• interactive installer that runs from Docker against an existing Kubernetes cluster
• can operate fully air-gapped with no cluster egress

We mainly use it with on-prem RKE2 clusters, but it also works with EKS/GKE/AKS.

It’s Apache 2.0 and still early, so we’re at the point where feedback from people actually running multi-GPU servers is especially useful.

GitHub:
https://github.com/axem-solutions/shaide

I'm curious how people here handle this once a setup grows beyond a single machine.

If you're running several models in parallel, what are you using for routing/orchestration?

**•   Fixed the biggest issue**: high-VRAM systems (24GB+) now actually get 14B–35B model suggestions instead of tiny outdated ones  
**•   Physics-based speed estimation**: throughput is now calculated from real GPU/RAM memory bandwidth instead of rough size buckets  
**•   Multi-GPU support**: VRAM across multiple cards is now aggregated instead of only reading the primary GPU  
**•   MoE-aware**: proper handling for MoE models (like 35B-A3B) using active parameter count for speed estimates  
**•   RAM speed/channel detection**: DDR4/DDR5 speed now factored into hybrid CPU+GPU estimates  
**•   Fixed the layout bug**: disk cards and featured recommendations are now responsive, no more manual window stretching  
**•   Live sync improvements**: auto-syncs from Hugging Face on startup, shows “last synced X min ago”, filters for uncensored models and release date range  
**•   SHA256 + antivirus false-positive explanation** added to the README for anyone who got a scary flag

GitHub: https://github.com/keplerTR/LocalAI-Advisor

Thanks again to everyone who commented last time — several of these came directly from your suggestions (multi-GPU, MoE offload estimation, the layout fix). Keep the feedback coming.

Special thanks to u/Pika357, u/tbbtbbt, u/SnooOwls412, u/Jstratos9, u/Equivalent_Bass_879, u/QuarterDistinct857, u/arthax33, and u/RevolutionarySeven7 — several of the fixes above came directly from your comments on the original post.

I’m researching how developers set up vendor AI toolchains on Linux, such as ROCm/Ryzen AI, OpenVINO/oneAPI, CUDA, and others. I’d rather hear firsthand accounts than make assumptions.

1.Which stack, hardware, and distribution did you use?
2.How long did it take from a fresh installation to a model running on the accelerator?
3.What caused the issue? If you remember, provide specific details, such as a package, path, driver, or compiler version.
4. How did you verify that the GPU or NPU was being used and not silently falling back to the CPU?
5. Did you set up multiple vendor stacks? Was the second one easier to configure, or did most of the knowledge transfer not occur?
6. Did you document the process or create scripts to avoid repeating it?
7. If someone gave you a laptop with different silicon tomorrow and asked for the same setup, how would you feel about it?

I’m happy to share a summary of my findings with the thread.

Im not sure if this belongs in here. But im trying to figure out the best LLMs to use on my 12gb Rtx 2060. I'm not looking for something that excels in one area or another but something general use that can handle extended conversations, light to medium co pilot coding, and general everyday use.