Hi all,

If you’re just now signing in for the first time in 12+ hours, you may just now be finding out that Lemmy World and other instances where hijacked. The hijackers had the full abilities of hijacked user, mod, and admin accounts. At this time, I am only aware of instance defacing and URL redirections to have been done by the hijackers.

If you were not forced to sign back in this morning, contact your instance admin to verify mitigations were completed on your instance.

How?

This occurred due to an XSS attack in the recently added custom emojis. Instance admins should follow the issue tracker on the LemmyNet GitHub, as well as the Matrix Chat. Post-Incident Activity is still on-going.

Currently, it is likely that just your session cookie was stolen, with instance admins being targeted specifically by checking for navAdmin, an HTML element only instance admins had. I do not believe this to affect users across instances, but I have yet to confirm this.

What happens next?

As I am not the developers or affected instance admins, I cannot make any guarantees. However, here is what you’ll likely see:

  1. Post Incident investigation continues. This will include inspecting code, posts, websites, and more used by the hijackers. An official incident writeup may occur. You should expect the following from that report:
  • Exactly what happened, when.
  • The incident response that occurred from instance admins
  • Information that might have helped resolve the issue sooner
  • Any issues that prevented successful resolution
  • What should have been done differently by admins
  • What should be improved by developers
  • What can be used to identify the next attack
  • What tools are needed to identify that information
  1. A CVE is created. This is an official alert of the issue, and notifies security experts (and enthusiasts), even those not using lemmy, about the issue.

  2. A code security audit is done. This will likely just be casual reviews by technical lemmy users. However, I will be reaching out to the Mozilla Foundation and Cure53 as they recently did an audit of Mastodon. If there is interest in an external audit of lemmy and the costs are affordable, I’ll look into crowdfunding this cost.

  • redcalcium
    link
    fedilink
    English
    198
    edit-2
    1 year ago

    This incident made me realize not to use an admin account for my primary lemmy account in my personal instance. I setup another account for instance admin purpose (with 2FA enabled) and keep it logged out, then remove my primary account from the instance admin list.

    • @Renacles@lemmy.world
      link
      fedilink
      English
      751 year ago

      This is a good mindset in general, when working in AWS you are not supposed to use your root account unless it’s absolutely necessary even if you are the only user. Hosting a Lemmy instance should be no different.

      • @kakes@sh.itjust.works
        link
        fedilink
        English
        111 year ago

        Yeah, even in Windows or what-have-you, you should always keep your admin account separate from your daily driver account for exactly this reason.

      • @Trainguyrom@reddthat.com
        link
        fedilink
        English
        2
        edit-2
        1 year ago

        I just setup a VPS for a Minecraft server for some friends and did exactly that. I was under a bit of a time crunch, but still took the time to think through those challenges in access for everything. Created an unprivileged user to run the server as, created a seperate unprivileged user for another service. Disabled password-based SSH login, etc.

        I should probably setup a dedicated non-root admin account for administrative functions but that’s a problem for after work

    • @Crackhappy@lemmy.world
      link
      fedilink
      English
      171 year ago

      I work with 2 factor, Oath, SAML, etc. all the time for work, and for the life of me I can’t get it working properly with Lemmy.

      • Notorious
        link
        fedilink
        English
        211 year ago

        Lemmy decided to go with SHA256 for TOTP seed. This is a very odd move since many 2FA apps don’t support SHA256. I actually had to write a quick python script to spit out my 2FA code since Bitwarden doesn’t support it. Hopefully either Lemmy will change to SHA-1 or Bitwarden will start to support SHA256 seeds.

        • redcalcium
          link
          fedilink
          English
          17
          edit-2
          1 year ago

          Wait, I use bitwarden and it works just fine. And yes, it’s using SHA256.

          Screenshot:

          • @subtext@lemmy.world
            link
            fedilink
            English
            31 year ago

            Bitwarden definitely works with SHA256 and even SHA512, I believe. And with codes longer than 6 digits also. Oh and even Steam codes if you really want.

            • redcalcium
              link
              fedilink
              English
              21 year ago

              You’ll need to either subscribe to bitwarden premium ($10 / year), or deploy vaultwarden in your own server and have your bitwarden extension/app use that vaultwarden server instead of the official server.

              • @Trapping5341@lemmy.world
                link
                fedilink
                English
                11 year ago

                Yeah I have premium and every account I have that offers 2fa is set up but Lemmy will only let me set it up in authy or Google auth. It won’t give bitwarden as an option or give me the code to manually add it to bitwarden.

                • redcalcium
                  link
                  fedilink
                  English
                  21 year ago

                  Just right click at the 2fa button and select “copy link”, then paste the link into bitwarden TOTP field. The link you copied should already be in the format otpauth://totp/xxxx which will be recognized by bitwarden.

          • @subtext@lemmy.world
            link
            fedilink
            English
            21 year ago

            Doesn’t matter too too much for a TOTP though. It just needs to be pseudorandom enough for a relatively difficult to guess 6 digit (or longer) code that changes every 30 seconds. Much more likely for someone to get phished for their TOTP than for someone to reverse engineer someone’s TOTP seed. Plus how would that even happen? The attacker would need to have the device and / or track the user’s 6 digit codes for quite a while at which point the game’s already up.

      • redcalcium
        link
        fedilink
        English
        71 year ago

        I use bitwarden, so it’s pretty simple: just copy the link from the 2fa button in Lemmy into the TOTP field in your account editor in bitwarden, and it’ll automatically recognize the format.

    • BarterClub
      link
      fedilink
      English
      61 year ago

      Yup. Basics of running a server for anything. Never use your admin account and make a default backup with 2 factor.

  • Uriel-238
    link
    fedilink
    English
    981 year ago

    Having recently migrated from Reddit (and kept up with commercial social media hacks) I’m used to Nothing To See Here! We totally didn’t store your personal information in plaintext for hackers to snatch. Oh and maybe please change your passwords. All Part Of The Show!

    So, by comparison, the response here is downright heartwarming.

  • @0x4E4F@sh.itjust.works
    link
    fedilink
    English
    711 year ago

    If there is interest in an external audit of lemmy and the costs are affordable, I’ll look into crowdfunding this cost.

    It could get VERY, VERY expensive… depends on code complexity.

  • @CifrareVerba@lemmy.world
    link
    fedilink
    English
    45
    edit-2
    1 year ago

    The way the hack was utilized is honest very creative and interesting; either way, if all big communities could crowdsource money for security audits, I believe that it could help prevent something like this in the future.

    Edit for clarification: I am NOT talking about how the exploit itself was created or achieved, I am talking about the act of having the power they did and just doing shock crap.

    • @nikt@lemmy.ca
      link
      fedilink
      English
      41 year ago

      HTML injection / XSS vulnerabilities tend to be a sign of amateur hour to be honest. Made me a bit worried that I’m hitching my reddit escape wagon to the wrong technology. But sounds like this was due to instance-specific customizations rather than the core Lemmy tech, so hopefully we’re still on solid ground.

    • @graphite@lemmy.world
      link
      fedilink
      English
      41 year ago

      The way the hack was utilized is honest very creative and interesting;

      That’s often the case with exploits.

      • @CifrareVerba@lemmy.world
        link
        fedilink
        English
        61 year ago

        Your not wrong, however, most of the time when these events happen it’s meant for something more insidious than lemon party.

        • @graphite@lemmy.world
          link
          fedilink
          English
          0
          edit-2
          1 year ago

          Sure. Not sure how that’s relevant though?

          In general, finding an exploit requires looking for little tiny details that could exist in, really, any area of a given system; looking for a bug, and then exploiting that bug by understanding how input data can be used to create a deterministic chain of events.

          This almost always requires thinking outside of the box.

          There are people who are also paid to find these before malicious actors do.

          It’s always going to be creative in some way, at least in the beginning.

          It’s like when people first discover Quake’s fast inverse square root. Sure, the first time around it seems genius. In reality, code like that is actually everywhere, and there is a somewhat trivial aspect to optimizing those kinds of problems.

  • P03 Locke
    link
    fedilink
    English
    281 year ago

    A code security audit is done. This will likely just be casual reviews by technical lemmy users. However, I will be reaching out to the Mozilla Foundation and Cure53 as they recently did an audit of Mastodon. If there is interest in an external audit of lemmy and the costs are affordable, I’ll look into crowdfunding this cost.

    You don’t need to pay money. You just need to listen to the recommendations already made by free tools.

    Here, fix this shit first and then worry about a professional audit later.

  • @where_am_i@sh.itjust.works
    link
    fedilink
    English
    261 year ago

    also, I cannot properly login into my lemmy.world account anymore. username/password work, but when I try to upvote it tells me i gotta be logged in. Tried apps and web.

  • @PagingDoctorLove@lemmy.world
    link
    fedilink
    English
    241 year ago

    How do I contact my instance admin?

    I went ahead and logged out and back in on my own, but I was not prompted to do so by Lemmy and I don’t know enough about this stuff to say whether or not doing it on my own would’ve helped. I’m still pretty overwhelmed trying to figure this site out, so I apologize if this is a dumb question, just want to make sure I’m doing what I can to protect my own privacy.

    I don’t want to fall for the millennial version of a Nigerian prince scam, lol.

    • @BadAtNames@lemmy.fmhy.ml
      link
      fedilink
      English
      3
      edit-2
      1 year ago

      You’re on the lemmy.world instance, so you can reach the admins by emailing info@lemmy.world, or posting in the support forum !support@lemmy.world

      Now to answer whether there’s a difference between being promoted and doing it yourself - In this case, it’s suspected that session tokens were compromised. You know how when you enter some events, they vet you/your ticket once at the door and then put a stamp on your hand? If you go out and want to get back in, you don’t have to do the whole verification song and dance again, just show them your stamp? Well, that’s pretty much what a session token is - Lemmy vets your password once when you log in, and gives an unique session token to whatever browser or app you used to log in. That way, when you reopen Lemmy, you don’t have to enter your password again.

      Now that token is compromised, you have to assume a hacker has your unique token. When you logged yourself back in, Lemmy did the whole validation process again and gave your browser/app a new, unique session token - that’s just how logging in works. But the important question is, did it invalidate the old session token when you logged out? Otherwise the hacker can still show the old token and pretend to be you.

      Now if your browser/app prompted you to log-in today, you can be sure that your browser/app tried to get into Lemmy and was denied access. That means you can be sure your old stamp/token is now invalid. Logging out and in yourself doesn’t give you the same guarantee - you will have to check Lemmy code (or run some experiments) to know if logout does actually invalidate the old token. I haven’t validated Lemmy’s code, but I will say most half decent software will invalidate your token when you log out. If you want an extra layer of protection, change your password as well - even the software devs that forget to invalidate tokens on logout usually remember to invalidate them on password changes.

    • @TimewornTraveler@lemm.ee
      link
      fedilink
      English
      3
      edit-2
      1 year ago

      I wonder which instances required logins. I had to relog into my .world account, but my .ee account was still logged in.

      • @gila@lemmy.world
        link
        fedilink
        English
        21 year ago

        I heard lemm.ee wasn’t vulnerable, so logging users out shouldn’t be necessary. To be vulnerable there would need to be custom emotes defined on the local instance by the admins, so I’m guessing they had none.

  • Samæ
    link
    fedilink
    English
    231 year ago

    Good reminder that: small instances = small impact. Both true for the positive and the negative.

    • Notorious
      link
      fedilink
      English
      101 year ago

      Not sure that’s entirely true. Thankfully this attack vector required custom emojis, so it was limited to those specific Lemmy instances. Other attack vectors we may not be so lucky and it could spread through federation.

  • @monerobull@monero.town
    link
    fedilink
    English
    181 year ago

    Good thing we don’t have custom emojis on monero.town and the admin account isn’t used for things outside of the local community :D

    • @hackitfast@lemmy.world
      link
      fedilink
      English
      391 year ago

      Honestly, I see it as a win.

      The people that did this didn’t really act out in a coordinated attack. They were just kind of playing around, redirecting to lemonparty, changing page elements.

      It could have been a lot worse. The site could have been redirecting to malicious websites, downloading trojans, doing a lot of bad things. Instead, we got direct attention to the security vulnerabilities in question, and they’re being worked on and patched out relatively quickly. Helps that a lot of those on these communities are focused in programming and cybersecurity.

      • drdaemanA
        link
        English
        141 year ago

        The fundamental issue is not that emoji XSS (that’s just a vector), but how JWTs are implemented and [not] secured. I’ve read that it was reported at least this January (https://akkoma.nrd.li/notice/AXXhAVF7N5ZH1V972W).

        So, developers were already aware, yet - as I’m checking 0.18.1 - they have not fixed the unsafe-inline and unsafe-eval CSP, haven’t made jwt cookie HttpOnly, and haven’t done anything about exp and jti in the JWTs. I hope the recent events will make them do to so, and not just patch this particular XSS.

        • P03 Locke
          link
          fedilink
          English
          21 year ago

          Kegan may have been better off submitting a GitHub issue. And if issues haven’t been open for these critical security flaws, you should, too.

      • @sneakyninjapants@sh.itjust.works
        link
        fedilink
        English
        81 year ago

        Agree completely. In the grand scheme of things the damage that appears to have happened here is small potatoes, but it brought attention to the vulnerability so it was patched quickly. Going forward now, the authors and contributors to the project might be a bit more focused on hardening the software against these types of vulnerabilities. Pen testing is invaluable on wide user-base internet accessible platforms like this because it makes better, more secure software. Unfortunately this breech wasn’t under the “ethical pen testing” umbrella but it sure as hell brought the vulnerability to the mindshare of everyone with a stake in it, so I view it as a net win.

    • RCMaehl [Any]OP
      link
      fedilink
      English
      7
      edit-2
      1 year ago

      To add context to this. What I’ve been told is that a community running on a lemmy fork with 5 digit users had used this code for a while and backported(?) the code upstream when they federated back. I guessing there was an assumption of safety as they had been using the custom emojis code for quite a while without it being exploited.

    • @FightMilk@lemmy.world
      link
      fedilink
      English
      -71 year ago

      Yeah it’ll be hard to regain my trust after this one. I mean I’ll still use Lemmy but for now I’ll assume mine or any other account could be hacked at any time and act accordingly. This is a really amateur mistake even by FOSS standards.

      • @Mantiz_Shrimp@lemmy.world
        link
        fedilink
        English
        51 year ago

        Timception you should contact the admin for your lemmy server. If you didn’t need to log back in, your server likely was not fixed. This means someone could hack it and steal the session cookies to impersonate server members.

        • TheSaneWriter
          link
          fedilink
          English
          31 year ago

          It could also be that they never implemented custom emoticons, which are what caused the security issue. Even still, I would recommend confirming with your server admin.

        • Timception
          link
          fedilink
          English
          11 year ago

          I was able to scroll through content, but had to relogin to upvote. However, today I was forced to re-login since I entered the Memmy app. So I guess, all good? Thx for the tips.

  • @kep@lemmy.world
    link
    fedilink
    English
    121 year ago

    This post is weird. You’re typing like you’re in charge of things, but you’re apparently not.

    It’s one thing to show some initiative, but you’re literally demanding a full report like the Lemmy devs work for you. You sound like someone who does this kind of thing for a living and felt the need to flex. Because otherwise, what the hell are you even doing?

    Setting neurotically-specific demands for the developers makes sense if you represent a big instance or something, but you’re literally just a dude. You could have framed this entire post in a different way and gotten away with it. Right now, it’s creepy to anybody who actually reads the entire thing.

    • RCMaehl [Any]OP
      link
      fedilink
      English
      2
      edit-2
      1 year ago

      These aren’t demands, but I can definitely see how they can come off that way. These are industry standard post cybersecurity incident review questions by defined by NIST (NIST SP 800-61 Rev 2 Section 3.4.1) slightly rephrased.