Unfortunately Nvidia is, by fair, the best choice for local LLM coder hosting, and there are basically two tiers:

• Buy a used 3090, limit the clocks to like 1400 Mhz, and then host Qwen 2.5 coder 32B.

• Buy a used 3060, host Arcee Medius 14B.

Both these will expose an OpenAI endpoint.

Run tabbyAPI instead of ollama, as it’s far faster and more vram efficient.

You can use AMD, but the setup is more involved. The kernel has to be compatible with the rocm package, and you need a 7000 card and some extra hoops for TabbyAPI compatibility.

Aside from that, an Arc B570 is not a terrible option for 14B coder models.

It’s not functional yet.

Maybe we’re overthinking this.

What if it was a front-end for, like, Google or Apple Pay, PayPal, and other centralized financial services?

So basically, the “Fediverse” part is the account, UI, integration with other Fediverse apps, but ultimately it does not hold any financial information or perform any transactions. All it does is conveniently connect donors to creators better, and more flexibly, than a bare “here’s my PayPal’ link.

Yep. I didn’t mean to process shame you or anything, just trying to point out obscure but potentially useful projects most don’t know about :P

https://en.m.wikipedia.org/wiki/External_memory_algorithm

Unfortunately that’s not really relevant to LLMs beyond inserting things into the text you feed them. For every single word they predict, they make a pass through the multi-gigabyte weights. Its largely memory bound, and not integrated with any kind of sane external memory algorithm.

There are some techniques that muddy this a bit, like MoE and dynamic lora loading, but the principle is the same.

A1111

Eh, this is a problem because the “engine” is messy and unoptimized. You could at least try to switch to the “reforged” version, which might preserve extension compatibility and let you run features like torch.compile.

Oh you should be able to batch the heck out of that on a 4080. Are you not using HF diffusers or something?

I’d check out stable-fast if you haven’t already:

https://github.com/chengzeyi/stable-fast

VoltaML is also old at this point, but it has really fast AITemplate implementation for SD 1.5: https://github.com/VoltaML/voltaML-fast-stable-diffusion

Oh, 16GB should be plenty for SDXL.

For flux, I actually use a script that quantizes it down to 8 bit (not FP8, but true quantization with huggingface quanto), but I would also highly recommend checking this project out. It should fit everything in vram and be dramatically faster: https://github.com/mit-han-lab/nunchaku

You don’t want it to anyway, as “automatic” spillover with an LLM painfully slow.

The RAM/VRAM split is manually configurable in llama.cpp, but if you have at least 10GB VRAM, generally you want to keep the whole model within that.

Try a new quantization as well! Like an IQ4-M depending on the size of your GPU, or even better, an 4.5bpw exl2 with Q6 cache if you can manage to set up TabbyAPI.

Depends which 14B. Arcee’s 14B SuperNova Medius model (which is a Qwen 2.5 with some training distilled from larger models) is really incrtedible, but old Llama 2-based 13B models are awful.

No, all the weights, all the “data” essentially has to be in RAM. If you “talk to” a LLM on your GPU, it is not making any calls to the internet, but making a pass through all the weights every time a word is generated.

There are system to augment the prompt with external data (RAG is one word for this), but fundamentally the system is closed.

Oh I didn’t mean “should cost $4000” just “would cost $4000”

Ah, yeah. Absolutely. The situation sucks though.

I wish that the vram on video cards was modular, there’s so much ewaste generated by these bottlenecks.

Not possible, the speeds are so high that GDDR physically has to be soldered. Future CPUs will be that way too, unfortunately. SO-DIMMs have already topped out at 5600, with tons of wasted power/voltage, and I believe desktop DIMMs are bumping against their limits too.

But look into CAMM modules and LPCAMMS. My hope is that we will get modular LPDDR5X-8533 on AMD Strix Halo boards.

GDDR is actually super cheap! I think it would only be like another $75 on paper to double the 4090’s VRAM to 48GB (like they do for pro cards already).

Nvidia just doesn’t do it for market segmentation. AMD doesn’t do it for… honestly I have no idea why? They basically have no pro market to lose, the only explanation I can come up with is that their CEOs are colluding because they are cousins. And Intel doesn’t do it because they didn’t make a (consumer) GPU that was eally worth it until the B580.

The issue with Macs is that Apple does price gouge for memory, your software stack is effectively limited to llama.cpp or MLX, and 70B class LLMs do start to chug, especially at high context.

Diffusion is kinda a different duck. It’s more compute heavy, yes, but the “generally accessible” software stack is also much less optimized for Macs than it is for transformers LLMs.

I view AMD Strix Halo as a solution to this, as its a big IGP with a wide memory bus like a Mac, but it can use the same CUDA software stacks that discrete GPUs use for that speed/feature advantage… albeit with some quirks. But I’m willing to put up with that if AMD doesn’t price gouge it.

second-hand TPU

From where? I keep a look out for used Gaudi/TPU setups, but they’re like impossible to find, and usually in huge full-server configs. I can’t find Xeon Max GPUs or CPUs either.

Also, Google’s software stack isn’t really accessible. TPUs are made for internal use at Google, not for resale.

You can find used AMD MI100s or MI210s, sometimes, but the go-to used server card is still the venerable Tesla P40.

You can’t let it overflow if you’re using LLMs on windows. There’s a toggle for it in the Nvidia settings, and get llama.cpp to offload though its settings (or better yet, use exllama instead).

But…. Yeah. Qwen 32B fits in 24GB perfectly, and it’s great, but 72B really feels like the intelligence tipping point where I can dump so many API models, and that won’t fit in 24GB.

I’m self hosting LLMs for family use (cause screw OpenAI and corporate, closed AI), and I am dying for more VRAM and RAM now. Even if I had a 4090, it wouldn’t be nearly enough.

My 3090 is sitting at 23.9GB/24GB because I keep Qwen 32B QwQ loaded and use it all the time. I even have my display hooked up to my IGP to save VRAM.

Seriously looking at replacing my 7800X3D with Strix Halo when it comes out, maybe a 128GB board if they sell one. Or a 48GB Intel Arc if Intel is smart enough to sell that. And I would use every last megabyte, even if I had a 512GB board (which is the bare minimum to host Deepseek V3).

“Check the documentation” should absolutely be a retort though.

One of my least favorite things about the fediverse (and especially Discord and Reddit) is members asking the same simple question hundreds of times because they didn’t bother to do a simple search and didn’t bother to check obvious documentation.

They didn’t know the documentation exists? OK, I will happily show you, and show you how to find it in general. Question only partially novel? Great, I will link an old answer and explain the rest… But I am kinda fed up with how “ephemeral” social media is, which is by design, as that repetitiveness increases engagement dramatically. Many forums should be structured more like a wiki, and its users should reflect that.

To actually answer this, you could look into free APIs of open source models, which have daily limits but are otherwise largely catch-free. You could even mirror endpoints on your VPS if you need to, or host “middleware” like prompt formatters and enhancers.

I say this because, as others said, you cannot actually host AI on a VPS…