#nobridge

  • 0 Posts
  • 130 Comments
Joined 2 years ago
Cake day: March 14th, 2025
  • anamethatisnt@sopuli.xyztoSelfhosted@lemmy.world•[AIT] Local LLM for Server Setup Guidance?English
    1·
    2 months ago

    Depending on the use and the size of your graphics card you might be better of trying Gemma 4 12B too.
    On my Nvidia 5060 Ti 16GB I can use either Gemma 4 A4B IQ3_M (larger model but highly compressed) or Gemma 4 12B Q8_0 (smaller model but much less compression) and after a night of testing with the Gemma 4 12B Q8_0 I’d say it’s a better helper when it comes to coding but a worse gm for my dnd inspired fantasy world roleplaying adventure.

    I imagine your “setup a selfhosted environment” is closer to the first.

  • anamethatisnt@sopuli.xyztoSelfhosted@lemmy.world•[AIT] Local LLM for Server Setup Guidance?English
    3·
    2 months ago

    Easy setup would be to use koboldcpp + SillyTavern + Gemma 4 26B A4B GGUF of the largest quant your graphics card can fit together with your context.
    Remember to setup SillyTavern to allow network connections and create a user and password, default installation is localhost only.
    Set the temperature to 0.3 if using IQ3_M, higher quants allow higher temperature, and make sure all the formatting templates inside Advanced Formatting are set to Gemma 4.

    Then use either Gemma 4 itself or the free google.com AI to create some “W++ Denze with Horizontal Lines summaries for Gemma 4” of the relevant documentation of the latest version of the home server apps your friends are gonna install.
    You should ensure the resulting lorebook entries are no more than 1k tokens each to leave some context for your friends chats, add it to a character card in SillyTavern and set some keywords to allow them to load dynamically and not stay in context memory all the time.

    Regarding the character card you can ask Gemma 4 to write that for you too, I find “Write a character card in W++ Denze with Horizontal Lines style for Gemma 4 with this name, personality, attitude and skillset” works well for that.
    Then ask it to write a “First message prompt that starts with X, continues with Y and ends with Z for that character card” and you get a first draft to rewrite and paste into the “First Message” of the character card. The first message works as a template that Gemma 4 will imitate when you chat with it. Then simply try the chatbot out before letting others use it.

    I find my own Gemma 4 26B A4B IQ3_M works well for practicing hiragana and katakana, discussing programming or troubleshoot existing code or writing a small function but it can’t be expected to write a correct DatabaseService.cs from scratch and stuff like that.

    Oh and forget about finding good cards and lorebooks for SillyTavern use online, most users use it for NSFW Roleplaying chats. It is a very easy UI to use to create harnesses for your local LLM though.

  • anamethatisnt@sopuli.xyztoSelfhosted@lemmy.world•Why allow "everyone" to have read write permission?English
    3·
    4 months ago

    The main thing I see you can avoid with locking down the docker images into a separate low permission user that can only access what they really need is if someone successfully attacks a project and you get infected with some shit when your Synology pulls image:latest.
    It could limit the traversal of a ransomware that successfully breaks free of the container but ends up having no permissions outside as an example.
    I would probably purge the whole NAS and setup from my backup for my own peace of mind even with the user separation though.

    edit: updating “low user” to “low permission user”, amazing how the brain can fill in words for you when reading your own texts.

  • anamethatisnt@sopuli.xyztoSelfhosted@lemmy.world•Is there a way to auto backup from iPhones?English
    2·
    5 months ago

    The only thing I can think of then is to get your family members to start curating the photos into different albums using the Immich app. That way the sync gets to work and you get the usage statistics of the app up higher allowing the background task to run. If you create some shared albums and ask them to contribute photos to them f.e.
    Or simply telling them about your shared album and getting them to check it out using the app.

  • anamethatisnt@sopuli.xyztoSelfhosted@lemmy.world•Establishing proper VLAN'sEnglish
    9·
    5 months ago

    I usually end up doing it very simple with huge /24 ipv4 networks, f.e.
    10.100.10.0/24 = VLAN 10 = User devices and purely internal servers
    10.100.20.0/24 = VLAN 20 = IoT
    10.100.30.0/24 = VLAN 30 = Servers that are reachable from outside
    10.100.40.0/24 = VLAN 40 = Guests

    The main thing for me is to ensure that traffic that wants to pass between VLANs go through my firewall/router and allow Suricata to do its IPS work.