You must log in or register to comment.
Why do we place so much reliance on one mega company? This level of importance. It should be seized by the government.
God no, not the government!
They couldn’t organise a paper bag party
Large corporations and oligarchs are better? I’ll take the government. At least we can vote on them.
Government is also the entity that will be prosecuting/persecuting you when they don’t like what you have to say.
Sorry but this is a ridiculous argument. What entity has dropped nukes on an entire population? Who is the current president of the US? Insane take.
An idiot bought by corporations.
Do you literally hear yourself? You think large corporate and oligarchs run insurance, tech, etc., is a better route than a public option? 💀 Jeff Bezos, Musk, Thiel, and Ellison for everything?
When was the last time you heard about a large government computer outage? (I don’t count the VA because that’s broken on purpose.)
according to that page the issue stemmed from an underlying system responsible for health checks in load balancing servers.
how the hell do you fuck up a health check config that bad? that’s like messing up smartd.conf and taking your system offline somehow
Well, you see, the mistake you are making is believing a single thing the stupid AWS status board says. It is always fucking lying, sometimes in new and creative ways.
I mean if your OS was “smart” as not to send IO to devices that indicate critical failure (e.g. by marking them read-only in the array?), and then thinks all devices have failed critically, wouldn’t this happen in that kind of system as well…
that is an understatement 😂
I hate how Signal went down because of this… Wish it wasn’t so centralised.
My friend messaged me on Signal asking if Instructure (runs on AWS) was down. I got the message. That being said, it’s scary that Signal’s backbone depends on AWS
I hope they consider other ways of doing things after this incident.
Signal’s love affair with big tech is deeply disturbing.
It’s wild that these cloud providers were seen as a one-way stop to ensure reliability, only to make them a universal single point of failure.
Well companies use not for relibibut to outsource responsibility. Even a medium sized company treated Windows like a subscription for many many years. People have been emailing files to themself since the start of email.
For companies moving everything to msa or aws just was the next step and didn’t change day to operations
universal single point of failure.
If it’s not a region failure, it’s someone pushing untested slop into the devops pipeline and vaping a network config. So very fired.
Apparently it was DNS. It’s always DNS…
It’s mostly a skill issue for services that go down when USE-1 has issues in AWS - if you actually know your shit, then you don’t get these kinds of issues.
Case in point: Netflix runs on AWS and experienced no issues during this thing.
And yes, it’s scary that so many high-profile companies are this bad at the thing they spend all day doing
Case in point: Netflix runs on AWS and experienced no issues during this thing.
But Netflix did encounter issues. For example the account cancel page did not work.
Yeah, if you’re a major business and don’t have geographic redundancy for your service, you need to rework your BCDR plan.
But… that costs money.
So does an outage, but I get that the C-suite can only think one quarter at a time
Absolutely this. We are based out of one region, but also have a second region as a quick disaster recovery option, and we have people 24/7 who can manage the DR process. We’re not big enough to have live redundancy, but big enough that an hour of downtime would be a big deal.
I love the “git gud” response. Sacred cashcows?
What’s the general plan of action when a company’s base region shits the bed?
Keep dormant mirrored resources in other regions?
I presumed the draw of us-east-1 was its lower cost, so if any solutions involve spending slightly more money, I’m not surprised high profile companies put all their eggs in one basket.
I presumed the draw of us-east-1 was its lower cost
At no time is pub-cloud cheaper than priv-cloud.
The draw is versatility, as change didn’t require spinning up hardware. No one knew how much the data costs would kill the budget, but now they do.
But if everyone else is down too, you don’t look so bad 🧠
One of our client support people told an angry client to open a Jira with urgent priority and we’d get right on it.
… the client support person knew full well that Jira was down too : D
At least, I think they knew. Either way, not shit we could do about it for that particular region until AWS fixed things.
No one ever got fired for buying IBM.
Yes but now it is nobody ever got fired for buying Cisco.
I wouldn’t be so sure about that. The state government of Queensland, Australia just lifted a 12 year ban on IBM getting government contracts after a colossal fuck up.
It’s an old joke from back when IBM was the dominant player in IT infrastructure. The idea was that IBM was such a known quantity that even non-technical executives knew what it was and knew that other companies also used IBM equipment. If you decide to buy from a lesser known vendor and something breaks, you might be blamed for going off the beaten track and fired (regardless of where the fault actually lay), whereas if you bought IBM gear and it broke, it was simply considered the cost of doing business, so buying IBM became a CYA tactic for sysadmins even if it went against their better technical judgement. AWS is the modern IBM.
AWS is the modern IBM.
That’s basically why we use it at work. I hate it, but that’s how things are.
Such a monstrous clusterfuck, and you’ll be hard pressed to find anyone having been sacked, let alone facing actual charges over the whole debacle.
If anything, I’d say that’s the single best case for buying IBM - if you’re incompetent and/or corrupt, just go with them and even if shit hits the fan, you’ll be OK.
It is still a logical argument, especially for smaller shops. I mean, you can (as self-hosters know) set up automatic backups, failover systems, and all that, but it takes significant time & resources. Redundant internet connectivity? Redundant power delivery? Spare capacity to handle a 10x demand spike? Those are big expenses for small, even mid-sized business. No one really cares if your dentist’s office is offline for a day, even if they have to cancel appointments because they can’t process payments or records.
Meanwhile, theoretically, reliability is such a core function of cloud providers that they should pay for experts’ experts and platinum standard infrastructure. It makes any problem they do have newsworthy.
I mean,it seems silly for orgs as big and internet-centric as Fortnite, Zoom, or forturne-500 bank to outsource their internet, and maybe this will be a lesson for them.
It’s also silly for the orgs to not have geographic redundancy.
No it’s not. It’s very expensive to run and there are a lot of edge cases. It’s much easier to have regional redundancy for a fraction of the cost.
They zigged when we all zagged.
Decentralisation has always been the answer.
yeah, so many things now use AWS in some way. So when AWS has a cold, the internet shivers
sidekicks in '09. had so many users here affected.
never again.
A single point of failure you pay them for.
Good
That explains why my Matrix <-> Signal bridge was complaining about being disconnected.
Does this break all the various security features that are the reason to use Signal in the first place?
The Matrix server is a normal Signal client that can encrypt/decrypt messages from your account.
Assuming you trust your server, no. I would not use it on a third party Matrix server.
You can connect Matrix with Signal?
Sure, I got all my Signal/Telegram chats synced to my Matrix server.
I think I’ll be heavily looking into this then! Thanks
The one that hits us in self hosted is https://auth.docker.io/
mirror.gcr.io is google’s public mirror of dockerhub.
You guys don’t selfhost a registry?
I know this is selfhosted so most people here are hobbyists, but it’s a ton of work to selfhost in enterprise setting. I’d wager 90%+ of people using image registries are using Docker Hub, GHCR, or AWS ECR.
For your personal use, you don’t need an enterprise setting. It’s just a simple compose file that you run.
You can host a registry in pull through mode, so you still have all the images you use locally, but if it’s not in your registry yet, it pulls it from docker hub or whatever.
The only pain point is that a single registry can’t do both. So if you want to push your own docker images AND have a “cache” of stuff from docker hub, you need to run two registries in two different modes.
Pretty sure you could run Pulp in pull-through mode and add your local Forgejo/whatever registry as a remote, which would at least give you a unified “pull” URL. Then just use Forgejo actions to handle the actual build/publish for your local images whenever you push to main (or tag a release, or whatever).
Pulp might actually be able to handle both on its own, I haven’t ever tried though.
I hadn’t actually considered that before. What’s your preferred way to do that?
I have just this (which ironically won’t work now cause docker hub is down)
services: registry: restart: always image: registry:2 ports: - 5000:5000 dns: - 9.9.9.9 - 1.1.1.1 volumes: - ../files/auth/registry.password:/auth/registry.password - registry-data:/var/lib/registry environment: REGISTRY_STORAGE_DELETE_ENABLED: true REGISTRY_HEALTH_STORAGEDRIVER_ENABLED: false REGISTRY_HTTP_SECRET: ${REGISTRY_HTTP_SECRET} REGISTRY_AUTH: htpasswd REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm REGISTRY_AUTH_HTPASSWD_PATH: /auth/registry.password # REGISTRY_PROXY_REMOTEURL: "https://registry-1.docker.io/" volumes: registry-data:I don’t even remember how and when I set it up. I think it might be this: https://github.com/distribution/distribution/releases/tag/v2.0.0
Recently somebody has created a frontend, which I bookmarked but didn’t bother to set up: https://github.com/Joxit/docker-registry-ui
Harbor
Yeah I ran into this as well. Wondered why it needs a call to auth for public container images in the first place.
Oh god, that just 404s for me
Who wants to bet Amazon gave AI full access to their prod config and it screwed it up.
Or some engineer decide today would be a great day to play with BGP
That’s a good theory haha
This gif is audible
A bad day for Jeff Bezos is a good day for all of us
Yeah, was reading about it here too
Ring doorbells, Alexa, ahh… the joys of selfhosting.
Is there no way to check the doorbell video locally?
An Amazon employee misconfigures something and now your doorbell doesn’t work
Obligatory
Oh wow their front page doesn’t mention at all that their products run locally and don’t require subscriptions.
It mentions push notifications and emails, so I guess they must require an account, or can you configure them to use SMTP directly, as with the Amcrest Pro cameras?
I would be very surprised if there was
I don’t have one (because of that point), so I don’t know…
Presumably the app and doorbell are hardcoded to go to an AWS URL (so it’s “easier” for consumers), but in theory the data’s all on your wifi.
And I’m having a very good day now :3
Are you an IT contractor or something?
In some way, I am, but mainly I feel my need to only use selfhosteable stuff, and selfhost 90% of those services, confirmed.
For some reason I hear Gilfoyle pontificating about what he does
It makes me wish I was selfhosting more services, music & chat in particular. It wasn’t important enough to set up yet
Can recommend Jellyfin, I use it for both music and tv/movies. Not sure on the chat bit, there are so many option it could get a long list
I have Jellyfin, but I haven’t tried it with music. How does it compare to Navidrome?
For chat, I was thinking something super simple for the weird situations like this. Alternatively, Briar if you’re near the person you want to contact
I’d provide a plug for LMS! If you don’t give much of a damn for music video type stuff, it’s pretty solid and exposes more metadata through the Subsonic API than Navidrome does. My use case required Composer tags in addition to the usual smorgasbord. Bonus is that it combines SUPER well with Symfonium and is compatible with Audiomuse AI.
All that said, I would switch over to Jellyfin for music if they upped their music metadata game and made genre exploration a bit easier (assuming you have hundreds of distinct genre tags like I do).
Finamp as a music specialized client is really awesome. Just get the beta version as they are reworking it deeply and the stable one is not really updated (also app password make it easier to use OIDC sso plugin on jellyfin)
I moved from subsonic to jellyfin years ago, cuz subsonic didnt do video very well.
Jellyfin looks to do all the stuff Navidrome does, plus video in the same way
