Hiya,
Recently upgraded my server to an i5-12400 CPU, and have neen wanting to push my server a bit. Been looking to host my own LLM tasks and workloads, such as building pipelines to scan open-source projects for vulnerabilities and insecure code, to mention one of the things I want to start doing. Inspiration for this started after reading the recent scannings of the Curl project.
Sidenote: I have no intention of swamping devs with AI bugreports, i will simply want to scan projects that i personally use to be aware of its current state and future changes, before i blindly update apps i host.
What budget friendly GPU should i be looking for? Afaik VRAM is quite important, higher the better. What other features do i need to be on the look out for?
It’s all VRAM, that’s the bottleneck for even the best GPUs. AMD support is spotty so you should stay in Nvidia’s claws unless you know what you’re doing. Figure out what kind of money you’re willing to part with, and then get whatever Nvidia GPU gets you the most VRAM.
Yeah, for a budget friendly AI GPU I would look for a 5060 Ti 16GB.
The size of the LLM should be less than the amount of VRAM available.
Learning about quant level was helpful as well