r/LocalLLaMA • u/SavunOski • Jul 27 '26

News Kimi K3 weights now released.

Post image

Kimi K3 weights are finally released!

3.3k Upvotes

97% Upvoted

661 comments sorted by

View all comments

Show parent comments

74

u/throw123awaie Jul 27 '26

They released it in MXFP4 so with around 55GB RAM you could!

3

u/tokomaunited Jul 27 '26

Can you share the link brother?

12

u/throw123awaie Jul 27 '26

literally their huggingface site https://huggingface.co/moonshotai/Kimi-K3 the model is MXFP4 on inference.

3

u/Protheu5 Jul 28 '26

I'm sorry, but I don't understand where it is. I'm new and only clicked on Finetunes/Quantisations to find a download, and I clicked through them and didn't find anything that small, the smallest was 500+ gigabytes. I have no idea what "MXFP4 on inference" means despite trying to read up on the subject, sorry.

2

u/TotallyToxicToast Jul 28 '26

You can't download a single expert for a mixture of experts model.

The downloads always contain all experts. If you take the 500GB download which contains the full 2.8T parameters and only take 104B of the parameters, that would actually give you only roughly 20GB per expert.

Now running a single expert in a mixture of experts model is not very useful.
But Technichally with 500GB total model size fitting one Expert is possible.
The "I could fit one expert" was a joke.

1

u/Protheu5 Jul 28 '26

Thanks for the explanation. My naïveté got me.

Oh well, a person can dream, right? Back to trying to stop Qwen from getting stuck in an infinite loop, I guess.

1

u/tokomaunited Jul 28 '26

Thanks dude

1

u/kernelwedge Jul 28 '26 edited Jul 28 '26

Interesting! I was looking at it and wasn't sure if could fit. I recently setup a 128GB rig, wanting to stretch its legs. This will be a great test.

---
After doing further research no way its fitting anywhere https://recipes.vllm.ai/moonshotai/Kimi-K3?hardware=mi300x 8x192G....