MudMan

A quick look at US Amazon spits out that the only 24Gb card in stock is a 3090 for 1500 USD. A look at the European storefront shows 2400EUR for a 4090. Looking at other assorted stores shows a bunch of out of stock notices.

It’s quite competitive, I’m afraid. Things are very stupid at this point and for obvious reasons seem poised to get even dumber.

Yeah, for sure. That I was aware of.

We were focusing on the Mini instead because… well, if the OP is fretting about going for a big GPU I’m assuming we’re talking user-level costs here. The Mini’s reputation comes from starting at 600 bucks for 16 gigs of fast shared RAM, which is competitive with consumer GPUs as a standalone system. I wanted to correct the record about the 24Gig starter speccing up to 64 because the 64 gig one is still in the 2K range, which is lower than the realistic market prices of 4090s and 5090s, so if my priority was running LLMs there would be some thinking to do about which option makes most sense in the 500-2K price range.

I am much less aware of larger options and their relative cost to performance because… well, I may not hate LLMs as much as is popular around the Internet, but I’m no roaming cryptobro, either, and I assume neither is anybody else in this conversation.

You didn’t, I did. The starting models cap at 24, but you can spec up the biggest one up to 64GB. I should have clicked through to the customization page before reporting what was available.

That is still cheaper than a 5090, so it’s not that clear cut. I think it depends on what you’re trying to set up and how much money you’re willing to burn. Sometimes literally, the Mac will also be more power efficient than a honker of an Nvidia 90 class card.

Honestly, all I have for recommendations is that I’d rather scale up than down. I mean, unless you also want to play kickass games at insane framerates with path tracing or something. Then go nuts with your big boy GPUs, who cares.

But for LLM stuff strictly I’d start by repurposing what I have around, hitting a speed limit and then scaling up to maybe something with a lot of shared RAM (including a Mac Mini if you’re into those) and keep rinsing and repeating. I don’t know that I personally am in the market for AI-specific muti-thousand APUs with a hundred plus gigs of RAM yet.

Thing is, you can trade off speed for quality. For coding support you can settle for Llama 3.2 or a smaller deepseek-r1 and still get most of what you need on a smaller GPU, then scale up to a bigger model that will run slower if you need something cleaner. I’ve had a small laptop with 16 GB of total memory and a 4060 mobile serving as a makeshift home server with a LLM and a few other things and… well, it’s not instant, but I can get the sort of thing you need out of it.

Sure, if I’m digging in and want something faster I can run something else in my bigger PC GPU, but a lot of the time I don’t have to.

Like I said below, though, I’m in the process of trying to move that to an Arc A770 with 16 GB of VRAM that I had just lying around because I saw it on sale for a couple hundred bucks and I needed a temporary GPU replacement for a smaller PC. I’ve tried running LLMs on it before and it’s not… super fast, but it’ll do what you want for 14B models just fine. That’s going to be your sweet spot on home GPUs anyway, anything larger than 16GB and you’re talking 3090, 4090 or 5090, pretty much exclusively.

This is… mostly right, but I have to say, macs with 16 gigs of shared memory aren’t all that, you can get many other alternatives with similar memory distributions, although not as fast.

A bunch of vendors are starting to lean on this by providing small, weaker PCs with a BIG cache of shared RAM. That new Framework desktop with an AMD APU specs up to 128 GB of shared memory, while the mac minis everybody is hyping up for this cap at 24 GB instead.

I’d strongly recommend starting with a mid-sized GPU on a desktop PC. Intel ships the A770 with 16GB of RAM and the B580 with 12 and they’re both dirt cheap. You can still get a 3060 with 12 GB for similar prices, too. I’m not sure how they benchmark relative to each other on LLM tasks, but I’m sure one can look it up. Cheap as the entry level mac mini is, all of those are cheaper if you already have a PC up and running, and the total amount of dedicated RAM you get is very comparable.

Cool.

So?

I mean, you are assuming “decentralized” is good, but it’s only as good as what it gets you. On paper, and until proven otherwise, I may choose less decentralized and more “capable of proper, effective moderation” instead. Especially if “less decentralized” is actually “somewhat decentralized”. I haven’t seen a case that fundamental decentralization trumps all so far.

See, but as I was saying above about the privacy stuff, the perception is supposed to be that this is somehow “the alogrithm’s fault” or caused on purpose by corporate media to boost engagement.

Even your take is letting Fedi design off the hook, IMO. The answer here isn’t “oh, well, what can you do?” it’s designing proper moderation tools.

I know people get mad when you praise Bluesky around these parts, but they have an actually good block system, compared to Masto, Lemmy and Fedi in general. It really helps cut this crap short.

Well, where are you all when the Fedi cheerleading squad keeps posting about how bad it is that this or that competitor stores this or that information and how secure and private and great it is in Fedi servers because they don’t store anything?

Because I’ve spent years chiming in to explain these things in those and it normally just gets people angry and complaining that you’re shilling for corporate social media or whatever. The image being projected, both accidentally and on purpose is that no centralized data collection means your data on Fedi is private when it is extremely not.

Well, for one thing it only works asymmetrically. It’s fine if you have a very specific source of issues that you can isolate and cut off, but it’s not really useful if what you have is hostile users across the network. And it only protects the larger space. For smaller instances it’s a choice between functioning as social media or not existing at all.

It’s extremely far from a magic bullet, it is not resilient to large scale, systemic issues and the only reason its limitations haven’t been apparent is that the AP ecosystem is too small to suffer most of the issues of larger social media.

Aaaaand it’s designed to function via the petty squabbles of FOSS developer arguments, which I hate anyway. But that’s a me thing.

Again, doesn’t matter. There’s data on logged in users and it’s also many orders of magnitude larger than Fedi.

By most independent metrics Reddit has more visits than Netflix. Than Pornhub, while we’re at it. It’s one of the top ten most visited sites on the Internet, and by most accounts it’s actually grown since the “exodus”.

I don’t use it and I do like it here, but the idea that Lemmy is somehow encroaching on it is absurd. And self-defeating, too. Lemmy and its satellites are very worthwhile for what they are… but just a gnat in the wind as a Reddit alternative. Better to measure them on their own merits.

It may or may not be.

It is definitely not inflating its numbers by the orders of magnitude it’d take to make a dent on this particular takeaway.

Sure. I mean, having a single log-in for all of that is definitely useful, as is being able to chat with others. Defederation as a moderation tool is… overrated, but it is there.

Yep. Which ends up being why old forums were such tight-knit communities. You ended up hanging out with a handful of people. I’m mostly fine with that. If anything, it requires starting something yourself for your niche interests and being fine with it being dormant most of the time.

It doesn’t really matter. For one thing, MAU and unique users are different metrics and they’re both valid, so if Lemmy is counting verified uniques they can just call it that.

For another, I looked at the data for logged in users and Fedi’s MAU is 0.125% of their daily logged in users, so the point stands regardless.

So by my math and some googling, that’s about 0.00005% of Reddit’s MAU.

On the one hand, cool, growth is growth.

On the other hand maybe it’s… healthy to stop looking at Lemmy as an “alternative” to anything and start thinking about it as this small forum you like to use sometimes. Worked for me in the 90s, works for me now.

Yeeeah, if you have to write a network of interconnected arguments in multiple languages the answer you’re transmitting may not be the one you want.

Federated apps and communities may be better off spending more effort working to be appealing and less explaining why they think they already are.

I feel like this conversation does a very good job of explaining why FOSS alternatives so often have terrible usability. “Not how most people would do it in a selfhost environment” is effectively “not how a tiny, teensy, borderline irrelevant proportion of users would do it”.

Selfhosting is moving towards being accessible to the average user in some areas. Not coincidentally, I suspect, mostly in areas where someone is trying to make money on the side (see Home Assistant increasingly trying to upsell you into their cloud subscription and branded hardware, for instance). This idea that structuring the software for the average phone user as opposed to the average home server admin is “bad” or “complicated” is baffling to me.

Oh, and for the record, no, that’s not the line for legality when it comes to watching the media I own. I am perfectly within my rights to access the files in my hard drive in any way I want. At least where I live. I make no promises for whatever dystopian crap is legal in the US. If anything there is a gray area on my using a specific type of drive to be able to rip commercial optical media that is theoretically DRMd in ways that my drive just happens to ignore. But remotely accessing my legal backups in my local storage? Nah, even if I was more worried about piracy than I am I’d feel fine on those grounds.

But also, copyright as currently designed is broken and not fit for purpose, and I suspect you don’t disagree and your pearl clutching here may have more to do with disliking Plex and not wanting to acknowledge an actually useful feature they provide than anything else. Maybe I’m reading too much into that.

I am very confused here. You seem to have slipped from arguing that it was difficult and complicated to arguing that it’s bad to be able to share content remotely because it’s a felony, which seems like a pretty big leap.

For one thing, it’s not illegal and I do rip my own media. I will access it from my phone or my laptop remotely whenever I want, thank you very much.

For another, and this has been my question all along, how is it possibly more difficult and complicated to have remote access ready to go than being “a DNS record away”? Most end users don’t even know what a DNS is.

And yes, not having (obvious) server configurations up front is transparent. That’s what I’m saying. It does mix at least two sources (their unavoidable, rather intrusive free streaming TV stuff and your library), but it doesn’t demand that you set it up. The entire idea is to not have to worry about whether it’s local content. Like I said, there are edge cases where that can lead to a subpar experience (mainly when it’s downsampling your stuff to route it the long way around without telling you), but from a UX perspective I do get prioritizing serving you the content over warning you of networking issues.

I don’t know, man, I’m not saying you shouldn’t prefer Jellyfin. I wouldn’t know, I never used it long enough to have a particularly strong opinion. I just don’t get this approach where having the thing NOT surface a bunch of technical stuff up front reads as “complicated and difficult”. I just get hung up on that.

Okay, but… how is it confusing from the front end if what you’re doing is going through the same steps of creating an account? You punch in a login and password in both.

Sure, Plex is doing this extra thing where it’s also bringing in centralized content along with your library and it will default to its remote access system if you log in from outside your network. But again, from the front-end that is transparent. You log in and you have your library. If anything they’re being a bit too transparent, I’ve had times where networking stuff got in the way and it took me a minute to notice that Plex was routing my library through their remote access system instead.

I can see objections to it working that way, you trade a (frankly super convenient) way to share content remotely and access content from outside your network without too much hassle for… well, going through someone else’s server and having their content sitting alongside yours. But “confusing and difficult” isn’t how I’d describe it. It seems to work like any other service, self-hosted or not, as far as the user-facing portions are concerned. I guess I just don’t see the confusing part there.

Wait, isn’t Jellyfin the same way? Pretty much every self-hosted app I run uses some web interface you log into so you can use it anywhere on the network. Sure, Plex also has some pre-set remote connection thing, but from the end user perspective it’s the same set of steps. I also had to make a login for all the stuff I fully self-host.

Is there no account management on Jellyfin? I would probably want that as a feature.