GPU bandwidth limits the amount of tokens/sec that you get.
I 've mostly been running QWEN-3.6-35B-A3B at Q8 and QWEN-3.5-122B-A10B at Q4 with Q8 kv-cache on llama.cpp (use vulkan version). Also use MTP GGUF's you get ~ 50% speed up when predicting 2 or 3 tokens.
Haven't tested the new laguna S 2.1 yet.
Another thing I liked was that I had enough ram to also run an embedding open model for LLM Wiki apps.
Overall I think it's cheaper and better (intelligence wise) to get a $20 codex sub and always run gpt-5.6-luna than getting a desktop for local inference. Unless you have explicit needs for it (whether it is just experimentation or you have personal things you 'd rather keep on your computers or w/e). OpenCode go at $10 is also a great option.
Thanks for the detailed response. My goal is to deploy small to medium-sized LLMs on it for things that I'd rather keep private and local, i.e. automation around smart home, but also tracking finances and portfolio, aggregating & summarizing news feeds and blog posts and so on. For these purposes, medium intelligence at low TPS seems good enough to me.
But a large part of it is also wanting to learn more about serving these models, and potentially some small-scale experiments with training and fine tuning.
Own a framework desktop 128gb.
GPU bandwidth limits the amount of tokens/sec that you get.
I 've mostly been running QWEN-3.6-35B-A3B at Q8 and QWEN-3.5-122B-A10B at Q4 with Q8 kv-cache on llama.cpp (use vulkan version). Also use MTP GGUF's you get ~ 50% speed up when predicting 2 or 3 tokens.
Haven't tested the new laguna S 2.1 yet.
Another thing I liked was that I had enough ram to also run an embedding open model for LLM Wiki apps.
Overall I think it's cheaper and better (intelligence wise) to get a $20 codex sub and always run gpt-5.6-luna than getting a desktop for local inference. Unless you have explicit needs for it (whether it is just experimentation or you have personal things you 'd rather keep on your computers or w/e). OpenCode go at $10 is also a great option.
Thanks for the detailed response. My goal is to deploy small to medium-sized LLMs on it for things that I'd rather keep private and local, i.e. automation around smart home, but also tracking finances and portfolio, aggregating & summarizing news feeds and blog posts and so on. For these purposes, medium intelligence at low TPS seems good enough to me.
But a large part of it is also wanting to learn more about serving these models, and potentially some small-scale experiments with training and fine tuning.