Rendered at 03:23:36 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
dajonker 3 days ago [-]
> Making Kubernetes good is inherently impossible, a project in putting (admittedly high quality) lipstick on a pig.
So well put, my good sir, this describes exactly my feelings with k8s. It always starts off all good with just managing a couple of containers to run your web app. Then before you know it, the devops folks have decided that they need to put a gazillion other services and an entire software-defined networking layer on top of it.
After spending a lot of time "optimizing" or "hardening" the cluster, cloud spend has doubled or tripled. Incidents have also doubled or tripled, as has downtime. Debugging effort has doubled or tripled as well.
I ended up saying goodbye to those devops folks, nuking the cluster, booted up a single VM with debian, enabled the firewall and used Kamal to deploy the app with docker. Despite having only a single VM rather than a cluster, things have never been more stable and reliable from an infrastructure point of view. Costs have plummeted as well, it's so much cheaper to run. It's also so much easier and more fun to debug.
And yes, a single VM really is fine, you can get REALLY big VMs which is fine for most business applications like we run. Most business applications only have hundreds to thousands of users. The cloud provider (Google in our case) manages hardware failures. In case we need to upgrade with downtime, we spin up a second VM next to it, provision it, and update the IP address in Cloudflare. Not even any need for a load balancer.
adamtulinius 3 days ago [-]
If you spin up Kubernetes for "a couple of containers to run your web app", I think you're doing something wrong in the first place, also coupled with your comment about adding SDN to Kubernetes.
People use Kubernetes for way too small things, and it sounds like you don't have the scale for actually running Kubernetes.
ownagefool 3 days ago [-]
It depends what you're doing it.
My app is fairly simple node process with some side car worker processes. k8s enables me to deploy it 30 times for 30 PRs, trivially, in a standard way, with standard cleanup.
Can I do that without k8s? Yes. To the same standard with the same amount of effort? Probably not. Here, I'd argue the k8s APIs and interfaces are better than trying to do this on AWS ( or your preferred cloud provider ).
Where things get complicated is k8s itself is borderline cloud provider software. So teams who were previously good using a managed service are now owning more of the stack, and these random devops heros aren't necessarily making good decisions everywhere.
So you really have three obvious use cases:
a) You're doing something interesting with the k8s APIs, that aren't easy to do on a cloud provider. Essentially, you're a power user.
b) You want a cloud abstraction layer because you're multi-cloud or you want a lock-in bargaining chip.
c) You want cloud semantics without being on a cloud provider.
However, if you're a single developer with a single machine, or a very small team and you're happy working through contended static environments, you can pretty much just put a process on a box and call it done. k8s is overkill here, though not as much as people claim until the devops heros start their work.
shimman 3 days ago [-]
Call me old fashion but I prefer tools like Dokploy that make deployment across different VPS extremely easy. Dokploy allows me to utilize my home media server, using local instances of forgejo to deploy code, to great effect.
k8s appears to be a corporate welfare jobs program where trillion dollar multinational monopolistic companies are the only ones who can collectively spend 100s of millions sustaining. Since most companies aren't trillion dollar monopolies, adopting such measures seems extremely poor.
All it signals to me is that we have to stop letting SV + VC dictate the direction of tech in our industry, because their solutions are unsustainable and borderline useless for the vast majority of use cases.
I'll never forget the insurance companies I worked at that orchestrated every single repo with a k8s deployment whose cloud spend was easily in the high six figures a month to handle a work load of 100k/MAU where the concurrent peak never went more than 5,000 users, something the company did know with 40 years of records. Literally had a 20 person team whose entire existence was managing the companies k8s setup. Only reason the company could sustain this was that it's an insurance company (insurance companies are highly profitable, don't let them convince you otherwise; so profitable that the government has to regulate how much profit they're legally allowed to make).
Absolute insanity, unsustainable, and a tremendous waste of limited human resources.
Glad you like it for your node app tho, happy for you.
wredcoll 2 days ago [-]
K8s is just a standardized api for running "programs" on hardware, which is a really difficult problem it solves fairly well.
Is it complex? Yes, but so is the problem it's trying to solve. Is its complexity still nicer and easier to use than the previous generation of multimachine deployment systems? Also yes.
ownagefool 2 days ago [-]
I wrote a scheduler for VMs a long time ago. k8s is basically just the same thing but for containers.
It really confuses me how someone can argue for cloud providers over a decent open solution without realising their argument is simply they don't want to be managing the thing.
And that's fine, most teams shouldn't be neck deep in managing a platform. But that doesn't make the solution bad.
reissbaker 2 days ago [-]
K8s isn't even hard! My team of three manages everything on K8s and we spend ~0 minutes per week on it. Write a script to generate some YAML files, stick it in a CI pipeline, and it's basically fire-and-forget.
You're going to want most of what K8s has anyway: blue-green deployments, some way to specify how many replicas you want, health checks, etc.
The initial setup cost is annoying if you've never done it before, but in terms of maintenance it's very very easy.
icedchai 2 days ago [-]
Are you using a service like GKE, or running your own cluster? I've set up clusters "from scratch" in dev environments to get familiar with it.
reissbaker 2 days ago [-]
We do both: managed Kubernetes when it's available (AWS, Nebius, others), but for some hardware vendors they just give us raw machines and we self-host K3s on their nodes. We're an open-source LLM inference company so we're basically always scrambling for GPUs wherever we can get them, which means we need to be fairly scrappy with what we support while still having a semi-sane interface for ourselves internally. Kubernetes makes that pretty easy: onboarding a new vendor takes ~minutes, and then everything Just Works and we can interact with the pool of compute the same way we do every other pool since the K8s API is standard, with all of our built-in prod monitoring tools immediately set up and running.
That being said I love exe.dev and have been a happy customer since launch. It's a different use case but they do an amazing job at it. Very, very easy personal cloud dev box. But K8s is very very good too, just for production workloads rather than personal ones!
wredcoll 1 days ago [-]
This makes a lot of sense and is like the perfect use case for programming towards a "standard api".
I run it at home and at work, and while I do hate installing it, once that part is done I've never run into these problems that people claim requires a 20 person(!) team to baby sit it. Maybe my scale is too small or whatever, but its hard not to think that maybe they are just "holding it wrong"...
johnmaguire 3 days ago [-]
Just as a quick aside, I tried Coolify, Dokploy, Dockge, and Komodo, and if you're trying to do a Heroku-style PaaS, Dokploy is really good. Hands down the best UX for delivering apps & databases. It's too bad about the licensing. (e.g. OIDC + audit logs behind a paid enterprise license.)
Coolify is full of features, but the UX suffers and they had a nasty breaking bug at one point (related to Traefik if you want to search it.) Dockge is just a simple interface into your running Docker containers and Komodo is a bit harder to understand/come up with a viable deployment model, and has no built-in support for things like databases.
evanphx 2 days ago [-]
If you're open, love to get your thoughts on https://miren.dev. We've doing similar things, but leaning into the small team aspects of these systems, along with giving folks an optional cloud tie in to help with auth, etc.
neocron 2 days ago [-]
How is it monetized?
I read something about open core and paying for additional services I think.
What are those and where can I find them?
indigodaddy 3 days ago [-]
I use Cosmos Cloud on a free 24g oracle VM. Nice UI, solid system
johnmaguire 2 days ago [-]
Cosmos Cloud looks neat! At a first glance from looking at the web page, it looks more focused on delivering a "personal cloud" or "1-click deploy apps."
Dokploy is more Heroku-styled: while you can deploy third-party apps (it's just Docker after all), it seems really geared towards and intended for you to be deploying your own apps that you developed, alongside a "managed" database (meaning, the DB is exposed in the UI, includes backup functionality, and can even be temporarily exposed publicly on the internet for debugging.)
Coolify feels a bit like a mix of the two deployment models, while Dockge is "bring your own deployment" and Komodo offers to replace Terraform/Ansible/docker-compose through its own declarative GitOps-style file-based config but lacks features like managed databases, or built-in subdomain provisioning.
fud101 2 days ago [-]
Isn't Dokku a worthy mention anymore?
josegonzalez 2 days ago [-]
For better or worse, folks _really_ like a free UI. Dokku doesn't offer that (Dokku Pro is paid). With AI increasingly making that sort of thing easier to build - and Dokku being very easy to integrate via MCP but also good for building tools on top of - I'm not actually sure how to proceed with Dokku Pro.
Whether it's a worthy mention or not, I'm not sure. I'd like to think its worthy :)
Disclaimer: I am the maintainer.
ownagefool 2 days ago [-]
I took over tech for a POS company some years ago. They were a .net shop with about 80 developers, less than 200 concurrent connections, 6 figures spend cloud, and 0 nines uptime with a super traditional setup.
Point being, it's not the tools the causes the probem.
blobbers 2 days ago [-]
Read this as a Piece of sh... company. Then I saw response of someone saying they're a POS developer and was like oh I think he means point of sale.
Or that guy is just a really bad programmer.
icedchai 1 days ago [-]
I worked at a company that developed a niche POS as part of a larger system. It was, by far, the worst part of the code base. Just imagine a bunch of late 90's era Java 1.2 code, complete with a Swing UI, tons of concurrency issues, singleton objects and synchronized blocks all over the place, custom binary protocols...
ownagefool 2 days ago [-]
Both work (:
But the point was it was in a comparble situations without the microservices / k8s / whatever pet tech you want to hate on.
robotnikman 2 days ago [-]
Just curious, are you still looking for developers? Asking as someone who is a developer that works with POS systems.
ownagefool 2 days ago [-]
I no longer work in that industry.
electroly 2 days ago [-]
> I'd argue the k8s APIs and interfaces are better than trying to do this on AWS
I think Amazon ECS is within striking distance, at least. It does less than K8S, but if it fits your needs, I find it an easier deployment target than K8S. There's just a lot less going on.
ownagefool 2 days ago [-]
I ran renderapp in ECS before I ran it in k8s.
The deployment files / structure were mostly equivalent with the main differences being I can't shell into ECS and I lose kubectl in favour of looking at the AWS GUI ( which for me is a loss, for others maybe not ).
The main difference is k8s has a lot of optionality, and folks get analysis paralysis with all the potential there. You quickly hit this in k8s when you have to actually need the addon to get cloudwatch logs.
This is also where k8s has sharp edges. Since amazon takes care of the rest of the infrastructure for you in ECS, you don't really need to worry about contention and starving node resources resulting in killing your logging daemon, which you could technically do in k8s.
However, you'll note that this is a vendor choice. EKS Auto Mode does away with most of the addons you need to run yourself, simplifying k8s, moving it significantly closer to a vendor supported solution.
laserlight 2 days ago [-]
> I can't shell into ECS
Is there a specific reason why you can't shell into ECS? IIRC, I was able to do so by following the guide [0].
No, I was simply wrong. Thanks for pointing that out.
jitl 2 days ago [-]
if you use Kubernetes, shelling into an instance from k9s cli is pressing "s" with the instance highlighted in the TUI. it's great. haven't found a shrink-wrapped tool like that for ECS thats as good/easy as k9s for Kubernetes.
tayo42 2 days ago [-]
That or fargate if your just running a few containers.
evanphx 3 days ago [-]
Totally, it's all about the primitives. I'm curious where exe.dev is gonna build on the the base, or just leave it up to folks to add all their own bespoke stuff to do containers, logs, etc.
The last 20 years has given us a lot of great primitives for folks to plug in, I think that lots of people don't want to wrangle those primitives, they just want to use them.
wredcoll 2 days ago [-]
> a) You're doing something interesting with the k8s APIs, that aren't easy to do on a cloud provider. Essentially, you're a power user. b) You want a cloud abstraction layer because you're multi-cloud or you want a lock-in bargaining chip. c) You want cloud semantics without being on a cloud provider.
This is well put and it's very similar to the arguments made when comparing programming languages. At the end of the day you can accomplish the same tasks no matter which interface you choose.
Personally I've never found kubernetes that difficult to use[1]. It has some weird, unpredictable bits, but so does sysvinit or docker, that just ends up being whatever you're used to.
[1] except for having to install your own network mesh plugin. That part sucked.
bharat1010 2 days ago [-]
[dead]
sdevonoes 3 days ago [-]
Depends. For personal projects, yeah definitely. But at work? Typically the “Platform” team can only afford to support 1 (maybe 2) ways of deployment, and k8s is quite versatile, so even if you need 1 small service, you’ll go with the self-service-k8s approach your Platform team offers. Because the alternative is for you (or your team) to own the whole infrastructure stack for your new deloyment model (ecs? lambda? Whatever): so you need to setup service accounts, secret paths, firewalls, security, pipelines, registries, and a large etc. And most likely, no one will give you access rights for all of that , and your PM won’t accept the overhead either.
So having everyone use the same deployment model (and that’s typically k8s) saves effort. I don’t like it for sure
limaho 3 days ago [-]
This is where I'm at. Using Podman daily to run Python scripts and apps and it's been going great! However trying to build things like monitoring, secure secret injection, centralized inventory, remote logging, etc. has fallen on us. Has lead to some shadow IT (running our own container image registry, hashicorp vault instance, etc.) which makes me hesitant to share with others in the company how we're operating.
I like to think if we had a K8s environment a lot of this would be built out within it. Having that functionality abstracted away from the developer would be a huge win in my opinion.
evanphx 3 days ago [-]
Are you doing that across a fleet of machines or just one?
limaho 2 days ago [-]
We have 4 servers we run containers on. Calling that a fleet feels too generous. Not much rhyme or reason as to what containers run on which server
dajonker 3 days ago [-]
I totally agree, but that's not what happens in reality: the average devops knows k8s and will slap it onto anything they see (if only so they can put in on their resume). The average manager hears about k8s, gets convinced they need and hires beforementioned devops to build it.
goombaskoop 3 days ago [-]
> the average devops knows k8s and will slap it onto anything they see
This is certainly the case from all the third person accounts I hear. Online. I never actually met a single one that is like that, if anything, those same people are the ones that are first to tell me about their Hetzner setups.
hkt 3 days ago [-]
DevOps here.
The trouble is that we are literally expected to do this everywhere we go. I've personally advocated for approaches which use say, a pair of dedicated servers, or VMs as in GPs example. If you want it outside of AWS/GCP/Azure, you're regarded as a crazy person. If you don't adopt "best practices" (as defined by vendors) then management are scared. Management very often trust the sales and marketing departments of big vendors more than their own staff. Many of us have given up fighting this, because what it comes down to is a massive asymmetry of information and trust.
regularfry 3 days ago [-]
There is a kernel of validity lurking in the heart of all this, which is that immutable images you have the ability to throw away and refresh regularly are genuinely better than long-running VMs with an OS you've got to maintain, with the scope for vulnerabilities unrelated to the app you actually want to run. Management has absorbed this one good thing and slapped layer after layer of pointless rubbish on it, like a sort of inverse pearl. Being able to say "we've minimised our attack surface with a scratch image" (or alpine, or something from one of the secure image vendors) is a genuinely valuable thing. It's just the all of the everything that goes along with it...
dijit 3 days ago [-]
Sure.
The challenge is convincing people that "golden images" and containers share a history, and that kubernetes didn't invent containers: they just solved load balancing and storage abstraction for stateless message architectures in a nice way.
If you're doing something highly stateful, or that requires a heavy deployment (game servers are typically 10's of GB and have rich dynamic configuration in my experience) then kubernetes starts to become round-peg-square-hole. But people buy into it because the surrounding tooling is just so nice; and like GP says: those cloud sales guys are really good at their jobs, and kubernetes is so difficult to run reliably yourself that it gets you hooked on cloud.
There's a literal army of highly charismatic, charming people who are economically incentivised to push this technology and it can be made to work so- the odds, as they say, are against you.
vladvasiliu 3 days ago [-]
> If you want it outside of AWS/GCP/Azure, you're regarded as a crazy person. If you don't adopt "best practices" (as defined by vendors) then management are scared. Management very often trust the sales and marketing departments of big vendors more than their own staff. Many of us have given up fighting this, because what it comes down to is a massive asymmetry of information and trust.
I think this is the crux of the matter. Also, "everybody is doing it, so they must be right" is also a very common way of thinking amongst this population.
nz 3 days ago [-]
The following happened to a friend.
Around the time of the pandemic, a company wanted to make some Javascript code do a kind of transformation over large number of web-pages (a billion or so, fetched as WARC files from the web archive). Their engineers suggested setting up SmartOS VMs and deploying Manta (which would have allowed the use of the Javascript code in a totally unmodified way -- map-reduce from the command-line, that scales with the number storage/processing nodes) which should have taken a few weeks at most.
After a bit of googling and meeting, the higher ups decided to use AWS Lambdas and Google Cloud Functions, because that's what everyone else was doing, and they figured that this was a sensible business move because the job-market must be full of people who know how to modify/maintain Lambda/GCF code.
Needless to say, Lambda/GCF were not built for this kind of workload, and they could not scale. In fact, the workload was so out-of-distribution, that the GCP folks moved the instances (if you can call them that) to a completely different data-center, because the workload was causing performance problems, for _other_ customers in the original data-center.
Once it became clear that this approach cannot scale to a billion or so web-pages, it was decided to -- no, not to deploy Manta or an equivalent -- but to build a custom "pipeline" from scratch, that would do this. This system was in development for 6 months or so, and never really worked correctly/reliably.
This is the kind of thing that happens when non-engineers can override or veto engineering decisions -- and the only reason they can do that, is because the non-engineers sign the paychecks (it does not matter how big the paycheck is, because market will find a way to extract all of it).
One of the fallacies of the tech-industry (I do not mean to paint with too broad a brush, there are obviously companies out there that know what they are doing) is that there are trade-offs to be made between business-decisions and engineering-decisions. I think this is more a kind of psychological distortion or a false-choice (forcing an engineering decision on the basis of what the job market will be like some day in the future -- during a pandemic no less -- is practically delusional). Also, if such trade-offs are true trade-offs, then maybe the company is not really an engineering company (which is fine, but that is kind of like a shoe-store having a few podiatrists on staff -- it is wasteful, but they can now walk around in white lab-coats, and pretend to be a healthcare institution instead of a shoe-store).
Personally, I believe that the tech industry sustains itself via technical debt, much like the real economy sustains itself on real debt. In some sense, everyone is trying to gaslight everyone else into incurring as much technical debt as possible, so that a way to service the debt can be sold. Most of the technical debt is not necessary, and if people were empowered to just not incur it, I suspect it would orient tech companies towards making things that actually push the state of the art forward.
jcgrillo 3 days ago [-]
There was a moment ca. 2020 when everyone was losing their minds over Lambda and other cloud services like SQS and S3 because they're "so cheap!!11". Innumeracy is a hell of a drug.
p_l 3 days ago [-]
Still is, just details change.
A lot of criticism of k8s is always centered about some imagined perfect PaaS, or related to being in very narrow goldilocks zone where the costs of "serverless" are easier to bear...
3 days ago [-]
WorldMaker 2 days ago [-]
> Personally, I believe that the tech industry sustains itself via technical debt, much like the real economy sustains itself on real debt. In some sense, everyone is trying to gaslight everyone else into incurring as much technical debt as possible, so that a way to service the debt can be sold.
This feels like a reminder that everything "Cloud" is still basically the same as IBM's ancient business model. We've always just been renting time on someone else's computers, and those someone else people are always trying to rent more time. The landlords shift, but the game stays the same.
jimbokun 2 days ago [-]
“Nobody ever got fired for buying…
…IBM
…Microsoft
…AWS.”
jcgrillo 3 days ago [-]
> Management very often trust the sales and marketing departments of big vendors more than their own staff.
They're getting kickbacks from cloud vendors. Prove me wrong.
r_lee 2 days ago [-]
not sure if this is a thing with Cloud vendors, but e.g. in Finance, you'll definitely get the opportunity to call your rep over for free fancy dinners or whatever you want, because those are "customer meetings"
better than nothing, I don't blame em.
jcgrillo 2 days ago [-]
> better than nothing, I don't blame em.
Money ain't got no owners, only spenders.
ownagefool 3 days ago [-]
To be fair, I have k8s on my hetzner :p
darkwater 3 days ago [-]
And the average developer doesn't even know where to start to deploy things in prod. When the feature product asks passes QA... to the next sprint! we are done!
chrisweekly 3 days ago [-]
Whose responsibility is it to establish the prerequisite CICD pipelines, HITL workflows, and Observability infr in order for devs to shepherd changes to prod (and track their impact)? Hint: it's not the developer's.
philipallstar 3 days ago [-]
This was the point of "devops" (the concept, not the job title): the team should be responsible for development and operations, so one isn't prioritised hugely over the other.
liveoneggs 3 days ago [-]
But those things all require more pods on the cluster! We've looped back around to the beginning.
darkwater 3 days ago [-]
Exactly my point.
But then developers: "I just want to go to my Heroku days again!" but then with a sufficient big company there are maaany developers doing things their slightly different way, and then other effects start compounding, and then costs go up because 15 different teams are using 27 different solutions and and and...
But yeah, let's just spin-up a shadow IT VM with Debian like GP said, it's easy!
throwup238 3 days ago [-]
> But yeah, let's just spin-up a shadow IT VM with Debian like GP said, it's easy!
That’s literally how they sold AWS in the beginning.
Cloud won not because of costs or flexibility but because it allowed teams to provision their own machines from their budget instead of going through all the red tape with their IT departments creating… a bunch of shadow IT VMs!
Everything old is new again, except it works on an accelerated ten year cycle in the IT industry.
darkwater 2 days ago [-]
Indeed. And it stems from the illusion that what works in solo/small teams/scrappy startup works the same when you are bigger, and that a developer can take over all the corollary work to the actual product development.
And yes, a dev that's able to do that properly (stress on properly) is indeed a signal of a better overall developer but they are a minority and anyway as orgs scale up there is just too much of "side salad" that it becomes a separated dish.
tete 3 days ago [-]
> the average devops knows k8s
If you'd know Kubernetes, you know not to use it. I say that as someone who used to do consulting for it.
The reality is that yet again "making money" completely collides with efficient, quality, sane productive work.
For me one of the main reasons to leave that space is that I couldn't really deal with the fact that my work collides with a client's success. That said I have helped to get off that stuff and other things that they thought they needed, that just wasted time and money. It just feels odd going into a company that hired you to consult on a topic only to end up telling them "The best approach for you is not doing that at all". Often never. Like some people thought "Well, if we have hundreds of thousands or even millions of users" and the reality was that even in these scenarios if you went away from that abstract thought and discussed a hypothetical based on their product they realized that they'd still be better off without it. Besides the fact that this hypothetical often was in a future that made it likely that they said they'd likely have completely different setup so preparing for that didn't even make sense.
I think a big thing related to that was/is the microservice craze where people end up moving to a complex architecture for not many good reasons and then they increase complexity way faster than what they actually deliver in terms of the product, because it somehow feels good. I know it does, I've been there. When in reality the outcome often is just a complex mess with what could have been a relatively simple monolith. And these monoliths do work. And in the vast majority of cases they are easy to scale, because your problem switches from "how do we best allocate that huge amount of very different services across our infrastructure" to (for the most part) "how do we spin up our monolith on one more server" which tends to be a way easier to tackle service.
And nothing stops you from still using everything else if you want. Just because it's a monolith doesn't mean you need to skip on any of the cloud offerings, etc. For some reason there seems to be that idea that if you write a monolith you are somehow barred from using modern tooling, infrastructure, services, etc. Not sure where that comes from.
r_lee 2 days ago [-]
I think one big problem is that using microservice architecture doesn't mean that literally everything has to be a "microservice". if you don't truly need granual scaling (i.e. your "app" doesn't get a bunch of asymmetric loads across different paths), then you can just have more monolithic "microservices" until they need to be split up
imo this should achieve a nice balance?
necovek 2 days ago [-]
Before microservices became trendy, there was something called SOA (service oriented architecture). Microservices is but one application of that pattern where each service is minimal.
SOAs have most utility in scaling teams, not software: creating independent services allows autonomy to independent teams if they apply a few simple patterns for good SOA.
tjarjoura 3 days ago [-]
In some sense, Kubernetes is just a portable platform for running Linux services, even on a single node using something like K3s. I almost see it as being an extension of the Linux OS layer.
acedTrex 3 days ago [-]
This is what I do for small stuff, debian vm, k3s on it for a nicer http based deployment api.
throwaway894345 2 days ago [-]
Yep, this is the way. Linux is just a platform for running services on one or more computers without needing to know about those computers individually, and even if your scale is 1, it's often easier to install k3s and manage your services with it rather than memorizing a bunch of disparate tools with their own configuration languages, filepath conventions, etc. It's just a lot easier to use k3s than it is to cobble together stuff with traditional linux tools. It's a standard, scalable pane of glass and as much as I may dislike kubectl, it's worlds better than systemctl and journalctl and the like.
sgt 3 days ago [-]
Then why can't we put a wrapper onto systemd and make that into a light weight k8s?
tjarjoura 3 days ago [-]
This may be familiarity bias, but I often find `kubectl` and related tools like `k9s` more ergonomic than `systemctl`/`journalctl`, even for managing simple single-replica processes that are bound to the host network.
jcgl 3 days ago [-]
See Podman quadlets.
3 days ago [-]
enos_feedler 3 days ago [-]
Remember fleet?
marcosdumay 3 days ago [-]
Systemd is on the wrong layer here. You need something that can set your machine up, like docker.
jasonjayr 2 days ago [-]
Systemd seems to be moving in that direction, the features are coming together to actually enable this.
Though imagining the unholy existence of an init system who's only job is to spin up containers, that can contain other inits, OS images, or whatever ..... turtles all the way down.
sgt 2 days ago [-]
I don't see why not. Maybe it should be on the same layer going forward - for true cloud compute (including on-premise cloud)
necovek 2 days ago [-]
Docker does not set your machine up: it sets your process up.
Tech like Flatpak or Snap is closer to Docker than "machines" are — except that Docker has local, virtualized networking built-in as the IPC layer.
sgt 3 days ago [-]
Okay it sets the machine up, but not the underlying host machine though.
Thanemate 3 days ago [-]
I know that "resume-driven development" exists, where the tradeoffs between approaches aren't about the technical fit of the solution but the career trajectory. I've seen people making plain workstation preparation scripts using Rust, only to have something to flex about in interviews.
I'm not surprised even in the slightest that DevOps workers will slap k8s on everything, to show "real industry experience" in a job market where the resume matches the tools.
capitol_ 3 days ago [-]
Your first example sound very sensible to me?
Using new technology in something small and unimportant like a setup script is a perfect way to experiment and learn. It would be irresponsible to build something important as the first thing you do in a new language.
bananamogul 2 days ago [-]
For your own use, yes.
But if you're working with others, you should default to using standard industry tools (absent a compelling reason not to) because your work will be handed off to others and passed on to new team members. It's unreasonable to expect that a new Windows or Linux sysadmin or desktop support tech must learn Rust to maintain a workstation setup workflow.
r_lee 2 days ago [-]
agreed. I think if we all went with this HN mindset of "html4 and PHP work just fine" we wouldn't have gone anywhere with regards to all the technical advancements we enjoy today in the software space
JALTU 2 days ago [-]
We are building a religion, we are building it bigger
We are widening the corridors and adding more lanes
We are building a religion, a limited edition
We are now accepting coders linking new AI brains
(Apologies to Cake. And coders.)
ororoo 3 days ago [-]
there are alsp people with devops title that do not know anything else than the hammer, and then everything is a hammer problem.
I mean, I worked with people who were suprised that you can run more applications inside ec2 vm than just 1 app.
tete 3 days ago [-]
> there are alsp people with devops title that do not know anything else than the hammer, and then everything is a hammer problem.
To be fair though, that's true for every profession or skill.
> I mean, I worked with people who were suprised that you can run more applications inside ec2 vm than just 1 app.
I've seen something similar where people were surprised that you can use an object storage (so effectively "make HTTP requests") from every server.
jauntywundrkind 2 days ago [-]
Conversely, we had millions of server huggers before, who each knew their company's stuff in a way that wasn't really applicable if they went somewhere else.
Every company used to have a bespoke collection of build, deployment, monitoring, scaling, etc concerns. Everyone had their own practices, their own wikis to try to make sense of what they had.
I think we critically under-appreciate that k8s is a social technology that is broadly applicable. Not just for hosting containers, but as a cloud-native form of thinking, where it becomes much easier to ask: what do we have here, and is it running well, and to have systems that are helping you keep that all on track (autonomic behavior/control loops).
I see such rebellion & disdain for where we are now, but so few people who seem able to recognize and grapple with what absolute muck we so recently have crawled out of.
moooo99 2 days ago [-]
> People use Kubernetes for way too small things, and it sounds like you don't have the scale for actually running Kubernetes.
This is a problem I've run into enterprise deployments. K8s is often the lowest common denominator semi small platform engineering teams arrive on. At my current employer, a platform managed K8s namespace is the only thing we got in terms of PaaS offering, so it is what we use. Is it overpowered? Yes. Is it overly complex for our usecase? Definitely. Could we basically get by hosting our services on a few cheap mini computers with no performance penalty? Also yes.
geodel 3 days ago [-]
Doing Kubernetes like doing Agile is mandatory nowadays. I've been asked to package a 20 line worth of bash script as docker image so it can be delivered via CI/CD pipeline via Kubernetes pods in cloud.
Value is not that I got job done at a day's notice. It is black mark that I couldn't package it as per industry best practices.
Not doing would mean out of job/work. Whether it is happening correctly is not something decision makers care as long it is getting done anyhow.
jauntywundrkind 2 days ago [-]
I don't think there are any other industry best practices you could have followed.
That's basically why k8s is so compelling. It's tech is fine but it's a social technology that is known and can be rallied behind, that has consistent patterns that apply to anything you might dream of making "cloud native". What you did to get this script available for use will closely mirror how anyone else would also get any piece of software available.
Meanwhile conventional sys-op stuff was cobbling together "right sized" solutions that work well for the company, maybe. These threads are overrun with "you might not need k8s" and "use the solution that fits your needs", but man, I pity the companies doing their own frontiers-ing to explore their own bespoke "simple" paths.
I do think you are on to something with there not being food taste making, with not good oversight always.
necovek 2 days ago [-]
In my 20+ years in the industry, I've been at one company which really did Agile, and that was the one I started with.
Everyone else is communicating they are doing Agile while being very far away from it ;)
bdangubic 2 days ago [-]
if anyone knew what agile is maybe more would have a chance if making it work (it won’t). in my 30* the only “process” that worked and works is “hire the right people and get the F out of the way.”
necovek 2 days ago [-]
That's pretty much what agile is :)
chickensong 2 days ago [-]
It depends on your situation of course, but there are a lot of good reasons to package up that bash script and run it through the pipeline. If everyone does some backdoor deployment of their snowflake shell script that's not great. It doesn't matter if it's 20 lines or 2 lines.
johnmaguire 3 days ago [-]
There are many organizations which still ship software without Kubernetes. Perhaps even the vast majority.
geodel 3 days ago [-]
Of course. I used to think I am working for one such organization for long time. Until leadership decided "modernization" as top priority for IT teams as we are lagging far.
firesteelrain 2 days ago [-]
We have a hobby web based app that consists of multiple containers. It runs in docker compose. Serves 1000 users right now (runs 24/7). Single VM.
No Kubernetes whatsoever.
I agree with you.
giobox 2 days ago [-]
Docker compose is brilliant while your stack remains on a single box, and will scale quite nicely for some time this way for most applications with minimum maintenance overhead.
My personal strategy has always been to start off in docker compose, and break out to a k8s configuration later if I have to start scaling beyond single box.
2 days ago [-]
mbesto 2 days ago [-]
> it sounds like you don't have the scale for actually running Kubernetes.
You don't set up k8s because your current load can't be handled, you do for future growth. Sometimes that growth doesn't pan out and now you're left with a complex infrastructure that is expensive to maintain and not getting any of the benefit.
rvz 3 days ago [-]
They use it for inflating their resume for career progression rather than actually evaluating if they need it in the first place.
This is why you get many folks over-thinking the solution and picking the most hyped technologies and using them to solve the wrong problems without thinking about what they are selling.
You don't need K8s + AWS EC2 + S3 just to host a web app. That tells me they like lighting money on fire and bankrupting the company and moving to the next one.
p_l 3 days ago [-]
Often the alternatives presented as cheaper to me in discussions are actually burning money.
But given how I always see "you don't need k8s because you're not going to scale so fast" I am feel like even professional k8s operators have missed the fundamental design goals of it :/ (maximizing utilization of finite compute)
throwaway894345 3 days ago [-]
Even if using just one VM, I'll probably slap k3s on it and manage my application using manifests. It's just so much easier than dealing with puppet or chef or vanilla cloud-init. Docker compose works too, but at that point it's just easier to stick with k3s and then I can have nice things like background jobs, a straightforward path to HA, access to an ecosystem of existing software, and a nicer CLI.
tayo42 2 days ago [-]
Thats what I don't get when people bring up this idea k8s is complicated.
All of those other tools are complicated and fragile
throwaway894345 2 days ago [-]
I think the things that trip people up are:
1. People expect k8s to be an opinionated platform and it's very happy to let you make a mess
2. People think k8s is supposed to be a cross platform portability layer and ... it maybe can be if you're very careful, but it's mostly not that
3. People compare k8s/cloud/etc to some monolithic application with admin permissions to everything and they compare that to the "difficulty" of dealing with RBAC/IAM/networking/secrets management
4. People don't realize how much more complicated vanilla Linux tooling and how much more accidental complexity is involved
altmanaltman 3 days ago [-]
yeah it's like wanting to drive to the mall in the Space Shuttle and then complaining how its too complicated
hombre_fatal 3 days ago [-]
k8s is useful when you have services that must spin up and down together, and you want to swap out services and deploy all/some/one.
and then also package this so that you and other developers can get the infrastructure running locally or on other machines.
jen20 2 days ago [-]
The problem with Kubernetes is that it doesn't scale down to small deployments very well, but it sure as shit doesn't scale up to large ones either. Large shared multi-tenant clusters have massive problems even when running parts of the same application with the same incentives, it falls apart completely when the tenants are diverse.
Nomad has neither of these problems.
littlestymaar 3 days ago [-]
I have nom doubt that there are legit use cases for something like k8s at Google or other multi-billion companies.
But if its use was confined to this use case, pretty much nobody would be using it (unless as a customer of the organization's infra) and barely would be talking about it (like how there isn't too much talk about Borg).
The reason k8s is a thing in the first place is because it's being used by way too many people for their own goods. (Most people having worked in startups have met too many architecture astronauts in our lives).
If I had to bet, I'd wager that 99% of k8s users are in the “spin a few containers to run your web app” category (for the simple reason that for one billion-dollar tech business using it for legit reasons, there's many thousands early startups who do not).
rantanplan 3 days ago [-]
The legit use case for companies like Google/Amazon etc is only to sell it to customers. None of these companies use K8s internally for real critical workloads.
oblio 3 days ago [-]
Google uses Kubernetes' grandpa, called Borg, for everything.
But to quote someone: "you are not Google".
bitexploder 3 days ago [-]
Ehm, that is simply not true. Google built it for themselves first. It is essentially the open source version of the internal architecture. It gets used.
zaphar 3 days ago [-]
I worked at google. k8s does not really look at all like what they used internally when I was there, aside from sharing some similar looking building blocks.
oblio 3 days ago [-]
Yeah, but is the internal tool simpler? I'd be surprised.
zaphar 3 days ago [-]
Simpler to use? yes. Simpler under the hood? No.
akdev1l 3 days ago [-]
Also Amazon definitely uses k8s for stuff.
Teams are free to use EKS internally.
littlestymaar 3 days ago [-]
I said “something like k8s” above, and Google for sure uses something like k8s called Borg.
rantanplan 3 days ago [-]
Yes, it is Borg. Not k8s. Granted it is similar
jen20 2 days ago [-]
Nomad is substantially more like Borg than Kubernetes is. And, funnily enough, scales better in both directions than Kubernetes does!
eddythompson80 3 days ago [-]
And those devops folks just let your single debian VM be? It sounds like you have, like many of us, an organizational/people problem, not a k8s problem.
Maybe those devops folks only pay attention to k8s clusters and you're flying under their radar with your single debian VM + Kamal. But the same thinking that results in an overtly complex, impossible to debug, expensive to run k8s cluster can absolutely result in the same using regular VMs unless, again, you are just left to your own devices because their policies don't apply to VMs, yet.
The problem usually is you're one mistake away from someone shoving their nose in it. "What are you doing again? What about HA and redundancy? slow rollout and rollback? You must have at least 3 VMs (ideally 5) and can't expose all VMs to the internet of course. You must define a virtual network with policies that we can control and no wireguard isn't approved. You must split the internet facing load balancer from the backend resources and assign different identities with proper scoping to them. Install these 4 different security scanners, these 2 log processors, this watchdog and this network monitor. Are you doing mtls between the VMs on the private network? what if there is an attacker that gains access to your network? What if your proxy is compromised? do you have visibility into all traffic on the network? everything must flow throw this appliance"
onlybosshaskeys 3 days ago [-]
I mean, it's pretty clear the only reason they even got to swap to a single VM and take the glory is because they fired the devops in question. As in, they're the actual boss of a small operation. That's what saying goodbye and nuking the cluster implies here.
psviderski 3 days ago [-]
A single VM is indeed the most pragmatic setup that most apps really need. However I still prefer to have at least two for little redundancy and peace of mind. It’s just less stressful to do any upgrades or changes knowing there is another replica in case of a failure.
And I’m building and happily using Uncloud (https://github.com/psviderski/uncloud) for this (inspired by Kamal). It makes multi-machine setups as simple as a single VM. Creates a zero-config WireGuard overlay network and uses the standard Docker Compose spec to deploy to multiple VMs. There is no orchestrator or control plane complexity. Start with one VM, then add another when needed, can even mix cloud VMs and on-prem.
sf1tzp 17 hours ago [-]
Wow, that's very similar to my set up. Their diagram even shows off this staging/production pattern which is nearly identical to my own projects!
I use ansible to set up my hosts, and from there each app just brings it's own caddy & compose file. A bit of scripting and SOPS wraps the lint / build / deploy steps. Compared to my past life which was all k8s infra projects in helm... it is SO NICE to use these lightweight files instead.
ozim 3 days ago [-]
People have it backwards.
If you have an app and you want to run a single app yeah silly to look for K8s.
If you have a beefy server or two you want to utilize fully and put as many apps on it without clashing dependencies you want to use K8s or docker or other containers. Where K8s enables you to go further.
psviderski 2 days ago [-]
Why would you want to use K8s for one or two beefy servers? It's designed for solving a different problem at a large scale.
ozim 2 days ago [-]
I think automatic scaling is useful to utilize server fully - apps that don't need resources automatically scale down, apps that need resources can auto scale up.
I bet you can do it in some other way but that's built in feature of k8s.
psviderski 2 days ago [-]
There is very little reason to need auto-scaling when you run on pre-purchased VMs/servers. You've paid for all the compute so you can run as many replicas as it can fit and you need to handle the projected amount of traffic.
There are no benefits to scaling down in this case. And scaling up won't help handle more load if you've already allocated all resources to running replicas. You need more machines, not more replicas on the existing machine(s).
It all comes down to simple, boring capacity planning and static resource allocation. Fewer moving parts results in fewer failure modes, hence more robust infra and less ops and maintenance work.
ozim 2 days ago [-]
Your response seems like you are talking about a single product / single application.
You have apps A, B and C (you have N teams and N products) each developed by different teams - that you want to run on that one server, when app A doesn't have much traffic apps B and C can use more of compute. Then having deployment management aligned for all teams/products.
sgt 3 days ago [-]
That looks pretty interesting. Is it being used in production yet (I mean serious installs) ?
psviderski 3 days ago [-]
Yes but at small scale. Myself and a handful of others from our Discord run it in production. The core build/push/deploy workflows are stable and most of the heavy lifting at runtime is done by battle-tested projects: Docker, Caddy, WireGuard, Corrosion from Fly.io.
Radboud University recently announced they're rolling it out for managing containers across the faculty which is the most "serious install" I know about, but there could be other: https://cncz.science.ru.nl/en/news/2026-04-15_uncloud/
sgt 3 days ago [-]
Did you improve the security concerns? E.g the way it executes in a `curl | bash` level. I was a bit concerned about that.
projektfu 3 days ago [-]
TBF, the documentation says you can download and review the script, then run it. Or use other methods like a homebrew or (unofficial) Debian package, or you can just install the binary where you want it, which is all the install.sh script (107 lines, 407 words) does.
I mean how commands are run on the servers - indirectly or indirectly. It's likely a code quality issue?
psviderski 2 days ago [-]
Yep, the latest release embeds the install script in the CLI binary instead of using curl | sh, so it's versioned together with the CLI.
dzonga 3 days ago [-]
this is dope work.
yard2010 3 days ago [-]
I don't get it, I think that k8s is the best software written since win95. It redefines computing in the same way IMHO. I have some experience in working with k8s on prod and I loved every moment of it. I'm definitely missing something.
Hasz 2 days ago [-]
Took a while to find this. K8s is great, IMO most of the people with alternative setups are just rebuilding (usually worse) or compressing (specific to their use case) k8s features that have been GA for a long time.
Spend some time learning it, using it to deploy simple apps, and you won't go back to deploying in a VM again imo.
This only gets better with ai-assisted development, any model is going to produce much better results for k8s given the huge training set vs someone's bespoke build rube-goldberg machine.
ipsento606 2 days ago [-]
I deploy prod by running a shell script I wrote that rsyncs the latest version of the codebase to my server, then sshs into the server and restarts the relevant services
how could k8s improve my deployment process?
Hasz 2 days ago [-]
You know your app better than me, but here are some practical reasons for the typical B2C app:
split deployments -- perhaps you want to see how an update impacts something: if error rates change, if conversion rates change, w/e. K8s makes this pretty easy to do via something like a canary or blue green deployment. Likewise, if you need to rollback, you can do this easily as well from a known good image.
Perhaps you need multiple servers -- not for scale -- but to be closer to your users geographically. 1 server in each of -5-10 AZs makes the updates a bit more complicated, especially if you need to do something like a db schema update.
Perhaps your traffic is lumpy and peaks during specific times of the year. Instead of provisioning a bigger VM during these times, your would prefer to scale horizontally automatically. Likewise, depending on the predictable-ness of the distribution of traffic, running a larger machine all the time might be very expensive for only the occasional burst of traffic.
To be very clear, you can do all of this without k8s. The question is, is it easier to do it with or without? IMO, it is a personal decision, and k8s makes a lot of sense to me. If it doesn't make a ton of sense for your app, don't use it.
ianburrell 2 days ago [-]
What happens when your new version is broken? Kubernetes would rollback to old version. You have to rerun the deployment script and hope you have the old version available. Kubernetes will even deploy new version to some copies, test it, and then roll out the whole thing when it works.
Also, Kubernetes uses immutable images and containers so you don't have to worry about dependencies or partial deploys.
r_lee 2 days ago [-]
I think it's just that k8s allows you to shoot yourself in the foot, thus it gets all the blame.
when in reality, you can go very bare-bones with k8s, but people pretend like only the most extreme complexity is what's possible because it's not easy to admit that k8s is actually quite practical in a lot of ways, especially for avoiding drift and automation
that's my take on it
dionian 3 days ago [-]
I noticed in his article he said something like 'and then devops team puts a ton of complexity...' which doesnt seem like a k8s problem.
RyanHamilton 3 days ago [-]
Can you expand how it redefined computing for you personally?
jen20 2 days ago [-]
> the best software written since win95
This feels like what us Brits would call "damning with faint praise".
Windows 95 was terrible. Really bad. If you really mean to say that Kubernetes is revolutionary and well-engineered, Windows 2000 would be a much better example.
dilyevsky 2 days ago [-]
it sold like 7 mil copies in a month. yes 98 was much more polished overall but 95 revolutionized personal computing as it was much more accessible than NeXT stuff
linuxftw 2 days ago [-]
You're not missing anything. There's legions of amateurs that dislike k8s because they don't understand the value.
c0rruptbytes 2 days ago [-]
it's always a skill issue when it comes to people complaining with k8s
knowing when and when not to use k8s, is also a skill
dilyevsky 2 days ago [-]
Missing some hn snobbery
bfivyvysj 3 days ago [-]
I thought we collectively learned this with stack overflows engineering blog years ago.
Scale vertically until you can't because you're unlikely to hit a limit and if you do you'll have enough money to pay someone else to solve it.
Docker is amazing development tooling but it makes for horrible production infrastructure.
KronisLV 3 days ago [-]
Docker is great development tooling (still some rough edges, of course).
Docker Compose is good for running things on a single server as well.
Docker Swarm and Hashicorp Nomad are good for multi-server setups.
Kubernetes is... enterprise and I guess there's a scale where it makes sense. K3s and similar sort of fill the gap, but I guess it's a matter of what you know and prefer at that point.
Throw on Portainer on a server and the DX is pretty casual (when it works and doesn't have weird networking issues).
Of course, there's also other options for OCI containers, like Podman.
KaiserPro 2 days ago [-]
> Docker Swarm
IS that a thing still?
> Kubernetes is... enterprise
I would contest that. Its complex, but not enterprise.
Nomad is a great tool for running processes on things. The problem is attaching loadbalancers/reverse proxies to those processes requires engineering. It comes for "free" with k8s with ingress controllers.
KronisLV 2 days ago [-]
> IS that a thing still?
Yeah, using it in production. If you don't need the equivalent of CRDs or other complex stuff like network meshes, it's stable and pretty okay! My ingress is just a regular web server image, for example.
> It comes for "free" with k8s with ingress controllers.
Ingress Controllers will keep working but the API is frozen, I think nowadays you're supposed to use Gateway instead: https://gateway-api.sigs.k8s.io/
habinero 2 days ago [-]
I tried it out last year when I wanted to ditch our compose stuff and wanted to like it, but yeah, it seemed like it was mostly a zombie project. Plus it had a lot of sharp edges, IIRC. I forget what, exactly. Secrets? Ingress? Something like that.
bfivyvysj 2 days ago [-]
Docker swarm is getting development again (caught something on their slack a few weeks back).
I'd also contest the k8s is enterprise. Unless by enterprise you just mean over engineered in which case I agree.
necovek 2 days ago [-]
> Docker is great development tooling (still some rough edges, of course).
Show me a Docker in use where build caching was solved optimally for development builds (like eg. make did for C 40 or 50 years ago)?
Perhaps you consider Docker layers one of the "rough edges", but I believe instant, iterative development builds are a minimum required for "great development tooling".
I did have great fun optimizing Docker build times, but more in the "it's a great engineering challenge to make this shitty thing build fast" sense.
KronisLV 2 days ago [-]
A multi-stage Docker build where you separate pulling in dependencies from building the thing you want is as close as you're going to get.
Something like the following works well in practice:
1) pinned base image (e.g. Ubuntu LTS)
2) your own custom base image in a registry rebuilt whenever you want (e.g. with tools you need for debugging or available across all of your images)
3) your own runtime-specific base image, like a JDK one, can be used later both as a basis for development images with additional tooling, as well as for runtime images of your app
4) your own runtime-specific development images, like one that's based on the JDK image above + Maven, alongside any other development tooling you need
5) your multi-stage application image, where the first stage uses the development image to COPY in the dependency description files you need and then pull the dependencies, then does the build (layer cache takes care of reusing things where possible), and then the second stage is based on the runtime image (e.g. JDK) where you just copy your finished artifact (e.g. .jar file)
If you don't need or want to build your own images, you can fold steps 1-4 into just using upstream images off of Docker Hub or whatever you prefer, but in practice it works pretty okay across numerous stacks. Of course, it's also possible to easily have very high standards in regards to what you mean as "optimal", so Docker probably won't live up to that.
necovek 1 days ago [-]
Start thinking about things like:
- I changed one source file, I need to rebuild the whole app
- I upped one dependency by a minor version
...
How long is the rebuild time? As I said, compare it to simple, standard make-based setup.
The problem is that layers are really — in the end — sequential, and even a simple change in an earlier one invalidates the latter cached layers.
I do not consider up-to-few-seconds for cases like above "very high standards", but it is a lot of work and care with Docker — especially as you need it not to mess up development or production environments as people skip versions and similar (if you use tricks like reading some config from a file).
staticassertion 3 days ago [-]
This is why there's an endless cycle of shitty SaaS with slow APIs and high downtime. People keep thinking that scale is something you can just add later.
dodu_ 2 days ago [-]
What's a more reasonable general approach then?
Let's say you're a team of 1-3 technical people building something as an MVP, but don't necessarily want to throw everything away and rewrite or re-architect if it gets traction.
What are your day 1 decisions that let you scale later without over-engineering early?
I'm not disagreeing with you btw. I genuinely don't know a "right" answer here.
staticassertion 2 days ago [-]
I don't think there's a right answer, you need to sit down and try to think about these problems upfront. What will scaling look like? What decisions will you regret? Make the guesses you can, but don't ignore scale or performance.
elktown 2 days ago [-]
I'd argue on the contrary that it's the last decades' over-engineering bender that's coming home to roost. Now too many things have too many moving parts to keep stable.
sibellavia 3 days ago [-]
Clearly, Kubernetes wasn’t the right solution for your case, and I also agree that using it for smaller architectures is overkill. That said, it’s the standard for large-scale production platforms that need reproducibility and high availability. As of today I don’t see many *truly* viable alternatives and honestly I haven't even seen them.
abdjdoeke 3 days ago [-]
I dunno the more people dig into this approach they will probably end up just reinventing Kubernetes.
I use k3s/Rancher with Ansible and use dedicated VMs on various providers. Using Flannel with wireguard connects them all together.
This I think is reasonable solution as the main problem with cloud providers is they are just price gouging.
valzam 3 days ago [-]
I always feel like I am taking crazy pills when I read these threads. The k8s API and manifests config feels like a create standardardized way to deploy containers. I wouldn't want to run a k8s cluster from scratch but EKS has been pretty straightforward to work with. Being able to use kind locally for testing is amazing and k9s is my new favourite infra monitoring tool.
Even if you just run on 2 nodes with k3s it seems worth it to me for the standardized tooling. Yes, it is not a $5 a month setup but frankly if what you host can be served by a single $5 a month VM I don't particularly care about your insights, they are irrelevant in a work context.
serbrech 3 days ago [-]
Yes, I mean, I’m an engineer on a cloud Kubernetes service, and I don’t run Kubernetes for my home services. I just run podman quadlets (systems units).
But that is entirely different from an enterprise scale setup with monitoring, alerting, and scale in mind…
mindcrime 3 days ago [-]
Similar deal here. My $dayjob title is "Cloud Engineer" and I spend a lot of my time working with AKS and Istio. But for some recent personal projects at home, I've just been running Docker Swarm on a single server. It's just lighter and less complicated, and for what I'm doing it more than satisfies my needs. Now if this was going to production at mass scale, I might consider switching to K8S, but for experimentation and initial development, it would be way overkill.
limaho 3 days ago [-]
> But that is entirely different from an enterprise scale setup with monitoring, alerting, and scale in mind
Do you have experience with Kubernetes solving these issues? Would love to hear more if so.
Currently running podman containers at work and trying to figure out better solutions for monitoring, alerting, etc. Not so worried about scale (my simple python scripts don't need it) but abstracting away the monitoring, alerting, secure secret injection, etc. seems like it'd be a huge win.
a34729t 2 days ago [-]
As the strongest engineer I ever worked with commented: "Across multiple FAANG-adjacent companies, I've never seen a k8s migration go well and not require a complete reimplementation of k8s behind the APIs."
dodu_ 2 days ago [-]
Is that because kubernetes was the right fit from the beginning, or because the initial implementation was designed around kubernetes, which caused the migration to eventually end up taking that same shape?
ferngodfather 3 days ago [-]
Cloud providers have put a lot of time and effort into making you believe every web app needs 99.9999% availability. Making you pay for auto scaled compute, load balancers, shared storage, HA databases, etc, etc.
All of this just adds so much extra complexity. If I'm running Amazon.com then sure, but your average app is just fine on a single VM.
IsTom 3 days ago [-]
And funnily recently many of the Big Serious Cloud Websites are shitting the bed of availability aggressively.
AlexeyBelov 22 hours ago [-]
Well, they don't need to be highly available anymore. They've already cornered their respective markets.
gloomyday 3 days ago [-]
Marketing has such a gigantic influence in our field. It is absolutely insane. It feels unavoidable, since IT is (was?) constantly filled with new blood that picks up where people left off.
flyinglizard 3 days ago [-]
That thought crossed my mind recently as well. Not to mention the huge software stacks and the potential supply chain vulnerabilities that entails.
ajayvk 2 days ago [-]
Kubernetes offers powerful low-level primitives that can support virtually any deployment architecture. However, working with these primitives directly requires significant YAML wrangling. It makes sense to build specialized solutions on top of Kubernetes that simplify common deployment patterns. Knative is one such solution. Any solution that tries to expose all underlying primitives will inevitably become as complex as Kubernetes itself.
I have been building https://github.com/openrundev/openrun, which provides a declarative solution to deploy internal web apps for teams (with SAML/OAuth and RBAC). OpenRun runs on a single-machine with Docker or it can deploy apps to Kubernetes.
PunchyHamster 3 days ago [-]
Well, you used a tank to plow a field then complained about maintenance and fuel usage.
If you have actual need to deploy few dozen services all talking with eachother k8s isn't bad way to do it, it has its problems but it allows your devs to mostly self-service their infrastructure needs vs having to process ticket for each vm and firewall rules they need. That is saying from perspective of migrating from "old way" to 14 node actual hardware k8s cluster.
It does make debugging harder as you pretty much need central logging solution, but at that scale you want central logging solution anyway so it isn't big jump, and developers like it.
Main problem with k8s is frankly nothing technical, just the "ooh shiny" problem developers have where they see tech and want to use tech regardless of anything
bedobi 3 days ago [-]
At a previous job, our build pipeline
* Built the app (into a self contained .jar, it was a JVM shop)
* Put the app into a Ubuntu Docker image. This step was arguably unnecessary, but the same way Maven is used to isolate JVM dependencies ("it works on my machine"), the purpose of the Docker image was to isolate dependencies on the OS environment.
* Put the Docker image onto an AWS .ami that only had Docker on it, and the sole purpose of which was to run the Docker image.
* Combined the AWS .ami with an appropriately sized EC2.
* Spun up the EC2s and flipped the AWS ELBs to point to the new ones, blue green style.
The beauty of this was the stupidly simple process and complete isolation of all the apps. No cluster that ran multiple diverse CPU and memory requirement apps simultaneously. No K8s complexity. Still had all the horizontal scaling benefits etc.
germandiago 2 days ago [-]
I have designed a backend with exactly the same underlying philosophy as you ended up: load balancer? Oh, a problem. So better client-side hashing and get rid of a discovery service via a couple dns tricks already handled elsewhere robustly.
I took it to its maximum: every service is a piece that can break ---> fewer pieces, fewer potential breakages.
When I can (which is 95% of the time, I add certain other services inside the processed themselves inside the own server exes and make them activatable at startup (though I want all my infra not to drift so I use the same set of subservices in each).
But the idea is -- the fewer services, the fewer problems. I just think, even with the trade-offs, it is operationally much more manageable and robust in the end.
elAhmo 3 days ago [-]
Not advocating for complexity or k8s, but if your workflow can be served by a single VM, then you are magnitudes away from the volume and complexity that would push you to have k8s setup and there is even no debate of it.
There are situations where a single VM, no matter how powerful is, can do the job.
goosejuice 2 days ago [-]
I started using GKE at a seed stage company in 2017. It's still going fine today. I had zero ops experience and I found it rather intuitive. We brought in istio for mtls and outbound traffic policies and that worked pretty well too. I can only remember one fairly stressful outage caused by the control plane but it ended up remedying itself. I would certainly only use a managed k8s.
So I guess I'm a fan. I use a monolith for most of my stuff if I have the choice, but if I'm working somewhere or on something where I have to manage a bunch of services I'm most certainly going to reach for k8s.
stasge 2 days ago [-]
The problem is not Kubernetes but how it's treated. From its inception I've been seeing two anti-patterns: treating it as a platform (and being frustrated for Kubernetes not meeting expectations) and treating it as a product or part of a product (investing heavily into its customization and making it a dependency). Neither is practical unless you are building a platform and it is your product. Otherwise it should be viewed as an OS and treated as a commodity. You create a single big VM with MicroK8s per project (zero-ops vanilla Kubernetes) and make no dependency on how exactly Kubernetes is setup. This way you can run the same setup locally and in a data center. If ever needed your app could be moved to any cloud as long as that cloud meets basic prerequisites (like presence of persistent storage or load balancer). The best part is Kubernetes (unlike traditional OS) is API driven and your apps could be nicely packaged and managed using Terraform/OpenTofu or similar tooling.
throwaway894345 3 days ago [-]
This feels like the microservices versus monolith problem. You can use cloud services or not, and that's orthogonal to running your app in Kubernetes or in a VM.
Similarly, I suspect (based on your "hardening" grievance) that a lot of your tedium is just that cloud APIs generally push you toward least-privileges with IAM, which is tedious but more secure. And if you implement a comparably secure system on your single VM (isolating different processes and ensuring they each have minimal permissions, firewall rules, etc) then you will probably have strictly more incidents and debugging effort. But you could go the other way and make a god role for all of your services to share and you will spend much less time debugging or dealing with incidents.
Even with a single VM, you could throw k3s on it and get many of the benefits of Kubernetes (a single, unified, standardized, extensible control plane that lots of software already supports) rather than having to memorize dozens of different CLI utilities, their configuration file formats, their path preferences, their logging locations, etc. And as a nice bonus, you have a pretty easy path toward high availability if you decide you ever want your software to run when Google decides to upgrade the underlying hardware.
m4ck_ 3 days ago [-]
And if you need a cluster, Hashicorp Nomad seems like a more reasonable option than full blown kubernetes. I've never actually used it in prod, only a lab, but I enjoyed it.
ghthor 3 days ago [-]
We run nomad at work. I’m very happy with it from an administrative standpoint.
3 days ago [-]
whalesalad 3 days ago [-]
Your use case is very small and simple. Of course a single VM works. You’re changing a literal A record at CF to deploy confirms this.
That is not what kube is designed for.
zerkten 3 days ago [-]
>> Then before you know it, the devops folks have decided that they need to put a gazillion other services and an entire software-defined networking layer on top of it.
I don't work that closely with k8s, but have toyed with a cluster in my homelab, etc. Way back before it really got going, I observed some OpenStack folks make the jump to k8s.
Knowing what I knew about OpenStack, that gave me an inkling that what you describe would happen and we'd end up in this place where a reasonable thing exists but it has all of this crud layered on top. There are places where k8s makes sense and works well, but the people surrounding any project are the most important factor in the end result.
Today we have an industry around k8s. It keeps a lot of people busy and employed. These same folks will repeat k8s the next time, so the best thing people that who feel they have superior taste is to press forward with their own ideas as the behavior won't change.
dobreandl 3 days ago [-]
We've reduced our costs on Hetzner to about 10% on what we've paid on Heroku, for 10x performance. Kamal really kicks ass, and you can have a pretty complicated infrastructure up in no time. We're using terraform, ansible + kamal for deploys, no issues whatsoever.
TheTaytay 3 days ago [-]
Can you elaborate a bit on what terraform and mandible are doing for you in your setup?
dobreandl 3 days ago [-]
We've configured our Hetzner servers with terraform, so we can easily spin up a new one in case we notice that we need another slave to handle extra work (1-2 mins). Ansible is responsible for configuring the server, installing all the required packages and software (not all our infrastructure is deployed with Kamal, for instance we have clickhouse instances, DBs, redis etc and normal app slaves). TLDR; it helps us have a new instance up an runing in minutes, or recreating our infrastructure for a new client environment
TheTaytay 2 days ago [-]
Ah very cool. Thank you!
dgb23 3 days ago [-]
> Then before you know it, the devops folks have decided that they need to put a gazillion other services and an entire software-defined networking layer on top of it.
I'm not familiar with kubernetes, but doesn't it already do SDN out of the box?
mystifyingpoi 3 days ago [-]
> doesn't it already do SDN out of the box
Yes and no. Kubernetes defines specification about network behavior (in form of CNI), but it contains no actual implementation. You have to install the network plugin basically as the first setup step.
jkukul 3 days ago [-]
> I ended up saying goodbye to those devops folks,
The irony is that "DevOps" was supposed to be a culture and a set of practices, not a job title. The tools that came with it (=Kubernetes) turned out to be so complex that most developers didn't want to deal with them and the DevOps became a siloed role that the movement was trying to eliminate.
That's why I have an ick when someone uses devops as a job title. Just say "System Admin" or "Infrastrcutre Engineer". Admit that you failed to eliminate the siloes.
icedchai 3 days ago [-]
Yep, "Cloud Infrastructure Engineer" is what I prefer.
I am primarily a backend developer but I do a lot of ops / infra work because nobody else wants to do it. I stay as far away from k8s as possible.
lazyant 11 hours ago [-]
if your app runs on a single VM, then run it on a VM (two VMs behind LB for extra reliability)
simonebrunozzi 2 days ago [-]
> nuking the cluster, booted up a single VM with debian, enabled the firewall and used Kamal to deploy the app with docker.
Absolutely brilliant. Love it.
1dom 3 days ago [-]
I think this comment and replies capture the problem with Kubernetes. Nobody gets fired for choosing Kubernetes now.
It's obvious to you, me and the other 2 presumably techie people who've responded within 15 mins that you shouldn't have been using Kubernetes. But you probably work in a company of full of techie people, who ended up using Kubernetes.
We have HN, an environment full of techie people here who immediately recognise not to use k8s in 99% of cases, yet in actually paid professional environments, in 99% of cases, the same techie people will tolerate, support and converge on the idea they should use k8s.
I feel like there's an element of the emperors new clothes here.
dajonker 1 days ago [-]
You hit the nail on the head here. Saying and doing are two very different things. It's also especially tempting to find an excuse to use some shiny new thing that everyone is always talking about. Both for personal learning and curiosity but also for future job prospects. The reality is that it's easier to get a k8s job if you have k8s experience.
dnnddidiej 3 days ago [-]
That is good but at bigger orgs with massive workloads and the teams to build it out k8s makes sense. It is a standard and brilliant tech.
robshep 3 days ago [-]
If you replaced k8s with a single app on a single VM then you’ve taken a hype fuelled circuitous route to where you should have been anyway.
huijzer 2 days ago [-]
I always looked into k8s and then realized it solves YouTube-scale problems, which I don’t have.
BirAdam 3 days ago [-]
So... if you're at the point where you're using a single VM, I have to ask why bother with docker at all? You're paying a context switch overhead, memory overhead, and disk overhead that you do not need to. Just make an image of the VM in case you need to drop it behind an LB.
imtringued 3 days ago [-]
If you've ever had the displeasure of seeing the sorry state of VM tooling you would have known that building custom VM images is a very complicated endeavour compared to podman build or docker build.
I once tried to build a simple setup using VM images and the complexity exploded to the point where I'm not sure why anyone should bother.
When building a container you can just throw everything into it and keep the mess isolated from other containers. If you use a VM, you can't use the OCI format, you need to build custom packages for the OS in question. The easiest way to build a custom package is to use docker. After that you need to build the VM images which requires a convoluted QEMU and libvirt setup and a distro specific script and a way to integrate your custom packages. Then after all of this is done you still need to test it, which means you need to have a VM and you need to make it set itself up upon booting, meaning you need to learn how to use cloud-init.
Just because something is "mature" doesn't mean it is usable.
The overhead of docker is basically insignificant and imperceptible (especially if you use host networking) compared to the day to day annoyances you've invited into your life by using VM images. Starting a a VM for testing purposes is much slower than starting a container.
tayo42 2 days ago [-]
This comment chain is probably talking about like aws images, amis, which is just an api call and it snapshots the vm for you. Or use packer
staticassertion 3 days ago [-]
There's one extra process that takes up a tiny bit of CPU and memory. For that, you get an immutable host, simple configuration, a minimal SBOM, a distributable set of your dependencies, x-platform for dev, etc.
amusingimpala75 3 days ago [-]
Yes but NixOS does all of these things already, without the process overhead
umvi 3 days ago [-]
Even the minimal SBOM part? It's hard to be more minimal than a busybox binary.
amusingimpala75 3 days ago [-]
That’s fair, NixOS avoids the direct stuff from Docker itself but if you’re basing on an Alpine image or something that would probably be more minimal / smaller
staticassertion 2 days ago [-]
Nix wraps your process in namespaces and seccomp?
amusingimpala75 2 days ago [-]
Not by default but tools like agent-sandbox.nix (bwrap, seccomp) or other nixpak (just bwrap but more popular) can provide those capabilities if you want in a fairly simple interface
mkj 3 days ago [-]
How is docker a context switch overhead? It's the same processes running on the same kernel.
BirAdam 3 days ago [-]
You're adding all of the other supporting processes within the container that needn't be replicated.
akdev1l 3 days ago [-]
It depends, you could have an application with something like
FROM scratch
COPY my-static-binary /my-static-binary
ENTRYPOINT “/my-static-binary”
Having multiple processes inside one container is a bit of an anti-pattern imo
dnnddidiej 3 days ago [-]
Sidecars? Not in a simple app.
marcosscriven 3 days ago [-]
First time I’ve heard of Kamal. Looks ideal!
Do you pair it with some orchestration (to spin up the necessary VM)?
wernerb 3 days ago [-]
DevOps lost the plot with the Operator model. When it was being widely introduced as THE pattern I was dismayed. These operators abstract entirely complex services like databases behind yaml and custom go services. When going to kubecon i had one guy tell me he collects operators like candy. Answers on Lifecycle management, and inevitable large architectural changes in an ever changing operator landscape was handwaved away with series of staging and development clusters. This adds so much cost..
Fundamentally the issue is the abstractions being too much and entirely on the DevOps side of the "shared responsibility model".
Taking an RDBMS from AWS of Azure is so vastly superior to taking all that responsibility yourself in the cluster..
Meanwhile (being a bit of an infrastructure snob) I run Nixos with systemd oci containers at home. With AI this is the easiest to maintain ever.
lifty 3 days ago [-]
Those managed databases from the big cloud providers have even more machinery and operator patterns behind them to keep them up and running. The fact that it's hidden away is what you like. So the comparison makes no sense.
ricardo_lien 3 days ago [-]
Yes, I've had similar experiences. My life has been much easier since I migrated to ECS Fargate - the service just works great. No more 2AM calls (at least not because of infra incidents), no more cost concerns from my boss.
j45 3 days ago [-]
There exists a sweet spot between docker swarm and docker, not quite portainer, but a bit more.
The tools in this space can really help get a few containers in dev/staging/production much more manageable.
mynameisash 2 days ago [-]
My first and really only experience with Kubernetes was a project I did about six years ago. I was tasked with building a thing that did some lightly distributed compute using Python + Dask. I was able to cobble together a functioning (internal) product, and we went to production.
Not long after, I found that the pods were CONSTANTLY getting into some weird state where K8s couldn't rebuild, so I had to forcibly delete the pods and rebuild. I blamed myself, not knowing much about K8s, but it also was extremely frustrating because, as I understood/understand it, the entire purpose of Kubernetes is to ensure a reliable deployment of some combination of pods. If it couldn't do that and instead I had to manually rebuild my cluster, then what was the point?
In the end, I ended up nuking the entire project -- K8s, Docker containers, Python, and Dask -- and instead went with a single Rust binary deployed to an Azure Function. The result was faster (by probably an order of magnitude), less memory, cheaper (maybe -80% cost), and much more reliable (I think around four nines).
mrhottakes 2 days ago [-]
Why on earth would you use kubernetes for that application in the first place?
tdrz 2 days ago [-]
I'm very happy with my k8s setup for my small startup. I believe it would have been much harder for me to get it off the ground, manage it etc. without it.
mrhottakes 2 days ago [-]
Yep, when you use the wrong tool for the job it doesn't work very well
BowBun 3 days ago [-]
What scale is this story operating at? My experience managing a fleet of services is that my job would take 10x as long without k8s. It's hard, not bad.
collimarco 3 days ago [-]
Kubernetes is not bad, it's just low level. Most applications share the exact same needs (proof: you could run any web app on a simple platform like Heroku). That's why some years ago I built an open source tool (with 0 dependencies) that simplify Kubernetes deployments with a compact syntax which works well for 99% of web apps (instead of allowing any configuration, it makes many "opinionated" choices): https://github.com/cuber-cloud/cuber-gem I have been using it for all the company web apps and web services for years and everything works nicely. It can also auto scale easily and that allows us to manage huge spikes of traffic for web push (Pushpad) at a reasonable price (good luck if you used a VM - no scaling - or if you used a PaaS - very high costs).
wutwutwat 3 days ago [-]
It's not just low level, in most cases, it's also overkill.
Most companies aren't "web scale" ™ and don't need an orchestrator built for google level elasticity, they need a vm autoscaling group if anything.
Most apps don't need such granular control over fs access, network policies, root access, etc, they need `ufw allow 80 && ufw enable`
Most apps don't need a 15 stage, docker layer caching optimized, archive promotion build pipeline that takes 30 minutes to get a copy change shipped to prod, they need a `git clone me@github.com:me/mine.git release_01 && ln -s release_01 /var/www/me/mine/current`
This is coming from someone who has had roles both as a backend product engineer and as a devops/platform engineer, who has been around long enough to remember "deploy" to prod was eclipse ftping php files straight to the prod server on file save. I manage clusters for a living for companies that went full k8s and never should have gone full k8s. ECS would have worked for 99% of these apps, if they even needed that.
Just like the js ecosystem went bat shit insane until things started to swing back towards sanity and people started to trim the needless bloat, the same is coming or due for the overcomplexity of devops/backend deployments
valzam 3 days ago [-]
If this works `git clone me@github.com:me/mine.git release_01 && ln -s release_01 /var/www/me/mine/current` then your Docker builds should also be extremely quick. Where I have seen extremely slow docker builds is with Python services using ML libraries. But those I reallly don't want to be building on the production servers.
"ECS would have worked for 99% of these apps, if they even needed that."
I used to agree with that but is EKS really that much more complicated? Yes you pay for the k8s control plane but you gain tooling that is imho much easier to work with than IaC.
foobarian 2 days ago [-]
Wait a minute, if this is on AWS then what are we talking about? On-prem k8s sounds fine to me but you don't have the ECS option.
elktown 2 days ago [-]
I'm afraid the bloat is too lucrative for everyone involved.
gregdelhon 3 days ago [-]
Not so surprised that the architecture approach pushed by cloud vendors are... increasing cloud spend!
Melatonic 2 days ago [-]
microVM's are going to make all of these redundant
znpy 3 days ago [-]
> It always starts off all good with just managing a couple of containers to run your web app. Then before you know it, the devops folks have decided that they need to put a gazillion other services and an entire software-defined networking layer on top of it.
As a devops/cloud engineer coming from a pure sysadmin background (you've got a cluster of n machines running RHEL and that's it) i feel this.
The issues i see however are of different nature:
1. resumeè-driven development (people get higher-paying job if you have the buzzwords in your cv)
2. a general lack of core-linux skills. people don't actually understand how linux and kubernetes work, so they can't build the things they need, so they install off-the-shelf products that do 1000 things including the single one they need.
3. marketing, trendy stuff and FOMO... that tell you that you absolutely can't live without product X or that you must absolutely be doing Y
to give you an example of 3: fluxcd/argocd. they're large and clunky, and we're getting pushed to adopt that for managing the services that we run inside the cluster (not developer workloads, but mostly-static stuff like the LGTM stack and a few more things - core services, basically). they're messy, they add another layer of complexity, other software to run and troubleshoot, more cognitive load.
i'm pushing back on that, and frankly for our needs i'm fairly sure we're better off using terraform to manage kubernetes stuff via the kubernetes and helm provider. i've done some tests and frankly it works beautifully.
it's also the same tool we use to manage infrastructure, so we get to reuse a lot of skills we already have.
also it's fairly easy to inspect... I'm doing some tests using https://pkg.go.dev/github.com/hashicorp/hcl/v2/hclparse and i'm building some internal tooling to do static analysis of our terraform code and automated refactoring.
i still think kubernetes is worth the hassle, though (i mostly run EKS, which by the way has been working very good for me)
KaiserPro 2 days ago [-]
> and an entire software-defined networking layer on top of it.
This is one of the main fuckups of k8s, the networking is batshit.
The other problems is that secrets management is still an afterthought.
The thing that really winds me up is that it doesn't even scale up that much. 2k nodes and it starts to really fall apart.
holoduke 2 days ago [-]
And nowadays with Claude you can spin up clusters of vps machines in a few hours. All bare Debian without anything except nginx and the apps. Mass configuring without any tools using only Claude. Works perfectly. The costs saved without all the overhead is massive.
daxesh_tech 2 days ago [-]
[dead]
draw_down 3 days ago [-]
[dead]
stingraycharles 3 days ago [-]
Potentially useful context: OP is one of the cofounders of Tailscale.
> Traditional Cloud 1.0 companies sell you a VM with a default of 3000 IOPS, while your laptop has 500k. Getting the defaults right (and the cost of those defaults right) requires careful thinking through the stack.
I wish them a lot of luck! I admire the vision and am definitely a target customer, I'm just afraid this goes the way things always go: start with great ideals, but as success grows, so must profit.
Cloud vendor pricing often isn't based on cost. Some services they lose money on, others they profit heavily from. These things are often carefully chosen: the type of costs that only go up when customers are heavily committed—bandwidth, NAT gateway, etc.
But I'm fairly certain OP knows this.
t_sawyer 2 days ago [-]
I've run an Openstack cloud. Local to the host NVME's directly attached to VMs is unbeatable. All clouds offer this. But that storage is ephemeral and it was when I implemented it in Openstack too.
There's not enough redundancy. You could raid1 those NVME's when before they get attached to a VM and that helps with hardware failures, but you get less of them to attach. Even if you RAID them, there's not a good way to move that VM to another host if there's a RAM or CPU or other hardware issue on that host.
These VM's with NVME's directly attached have to basically be treated as bare metal servers and you have to do redundancy at the application layer (like database replication).
But again, all of the major cloud services offer these types of machines if you NEED NVME IO speed. There are quirks though. For example, in Azure it seems like you have to be able to expect the VM to be moved whenever Azure feels like it and expect that ephemeral data to be wiped. Whereas in Openstack, we would do local block level migrations if we HAD to move the VM to another host. That block level migration required the VM to be turned off but it did copy the local NVME data to another host. If this happened it was all planned and the particular application had app level redundancy built in so it was not a problem. If the host crashed, that particular VM would just be down till the host was fixed and came back online.
yencabulator 11 hours ago [-]
> There's not enough redundancy. You could raid1 those NVME's when before they get attached to a VM and that helps with hardware failures, but you get less of them to attach. Even if you RAID them, there's not a good way to move that VM to another host if there's a RAM or CPU or other hardware issue on that host.
The trick is building a block storage system that treats the local disk as write-back cache with async replication to networked storage. Like the blog post says they'll be doing.
The async replication has some integrity/recovery concerns for sure, but it the trick that enables local speeds. And people have been happy with async replication for their database for a very long time. Just need good observability for the durability delay.
Once you have that, you can do live VM migration if you're careful enough about dirty data. The new node just starts out with an empty cache.
It's not exactly trivial, but it's also probably not the biggest challenge if you're genuinely building a brand new cloud and going to compete against the hyperscalers. (Hell, hire me and I can write it for you. It'll take time and CPU hours to get stable, but the magic required is only mildly arcane.)
> Even if you RAID them, there's not a good way to move that VM to another host if there's a RAM or CPU or other hardware issue on that host.
This is the critical point. All hardware fails eventually. The CPU and RAM are, in a real sense, also ephemeral. The only relevant question is what the risk tolerance of the use-case is. If restoring from async backup is sufficient, then embrace ephemerality and keep backups. If you need round-the-clock availability, pick an architecture that lets you fall over gracefully to another machine, and embrace the ephemerality when you inevitably need to do so.
chatmasta 2 days ago [-]
> There's not enough redundancy
So build resiliency into your application layer.
sshine 1 days ago [-]
You don't always choose the application layer.
When you're an OpenStack cloud provider, your customers choose.
When you're a customer using Open Source software, your vendors choose.
Using a mixture of directly attached NVMe and network-attached volumes with backup is the sweet spot for me.
I don't need to maintain my own network filesystem (Ceph), and I can put applications that mirrors its database natively on NVMe and everything I don't have much control over on network-attached volumes.
I feel like there's something better not yet made.
faangguyindia 3 days ago [-]
i was just curious so i tested this actually.
Using fio
Hetzner (cx23, 2vCPU, 4 GB)
~3900 IOPS (read/write)
~15.3 MB/s
avg latency ~2.1 ms
99.9th percentile ≈ ~5 ms
max ≈ ~7 ms
DigitalOcean (SFO1 / 2 GB RAM / 30 GB Disk)
~3900 IOPS (same!)
~15.7 MB/s (same!)
avg latency ~2.1 ms (same!)
99.9th percentile ≈ ~18 ms
max ≈ ~85 ms (!!)
using sequential dd
Hetzner: 1.9 GB/s
DO: 850 MB/s
Using low end plan on both but this Hetzner is 4 euro and DO instance is $18.
yard2010 3 days ago [-]
I love Hetzner so much. I'm not affiliated I'm a really happy customer these guys just do everything right.
ratg13 3 days ago [-]
As long as you never have to interact with them. If you run into issues they have caused themselves, you'll find yourself dealing with a unique mix of arrogance and incompetence.
drcongo 3 days ago [-]
I've been using Hetzner for ~20 years and every single support interaction I've ever had with them has been top tier. Never AI bots, always humans who are helpful, courteous and prompt. I can't think of a single company, let alone hosting company, whose customer service has been so consistently good.
Aeolun 3 days ago [-]
It certainly helps the service never does anything wonky that requires a support interaction in the first place.
debo_ 2 days ago [-]
Are you sure they're not just being German and you are misinterpreting what they are saying?
zuhsetaqi 3 days ago [-]
Just for comparison I use the cheapest netcup root server:
IOPS
Read: 70.1k IOPS
Write: 30.1k IOPS
~100k IOPS total
Throughput
Read: 274 MiB/s
Write: 117 MiB/s
Latency
Read avg: 1.66 ms, P99.9: 2.61 ms, max 5.644 ms
Write avg: 0.39 ms, P99.9: 2.97 ms, max 15.307 ms
yread 3 days ago [-]
Nice, on Hetzner AX41-nvme (~50 eur, from 2020) non-raid I get:
IOPS: read 325k, write 139k
Throughput: read 1271MB/s, write 545MB/s
Latency: read avg 0.3ms, P99.9 2.7ms, max 20ms; write: 0.14ms, P99.9 0.35ms max 3.3ms
so roughly 100 times iops and throughput of the cloud VMs
Medowar 3 days ago [-]
That is a bit of a unfair comparison. The Hetzner and DO instances are shared hosting, you are using dedicated ressources.
Using a Netcup VPS 1000 G12 is more comparable.
read: IOPS=18.7k, BW=73.1MiB/s
write: IOPS=8053, BW=31.5MiB/s
Latency Read avg: 5.39 ms, P99.9: 85.4 ms, max 482.6 ms
Write avg: 3.36 ms, P99.9: 86.5 ms, max 488.7 ms
Nnnes 3 days ago [-]
Hetzner has dedicated resources too, but they also have 2 levels of shared resources, "Cost-Optimized" and "Regular Performance". The 3900 IOPS CX23 above is "Cost-Optimized".
Here are some "Regular Performance" shared resource stats
Hetzner CPX11 (Ashburn, 2 CPUs, 2GB, 5.49€ or $6.99/month before VAT)
Storage performance is practically always a shared resource, and that's what y'all are talking about here...
3 days ago [-]
fragmede 3 days ago [-]
> Cloud vendor pricing often isn't based on cost.
Business 101 teaches us that pricing isn't based on cost. Call it top down vs bottom up pricing, but the first principles "it costs me $X to make a widget, so 1.y * $X = sell the product for $Y is not how pricing works in practice.
jeffrallen 3 days ago [-]
Just to spell this out more clearly for the back row.of the classroom:
The price is what the customer will pay, regardless of your costs.
barrkel 3 days ago [-]
Economics teaches us that a big difference between cost and price attracts competition which should make the price trend towards the cost.
dns_snek 3 days ago [-]
Practice taught me that that "should" is doing a lot of heavy lifting here and it's often not the case, even across long time periods (years) that should allow competitors to emerge.
For example I calculated the cost of a solar install to be approximately: Material + Labour + Generous overhead + Very tidy profit = 10,000€
In practice I keep getting offers for ~14,000€, which will be reduced to 10,000€ with a government subsidy and my request for an itemized invoice is always met with radio silence.
ncruces 3 days ago [-]
Only if the barrier of entry is low.
Which it won't be, if at every turn you choose the hyperscaler.
stingraycharles 3 days ago [-]
If this is the case, cheap bandwidth for AWS, when?
conductr 3 days ago [-]
Economics has a lot of other lessons teaching us why prices of major clouds have remained somewhat expensive relative to cost
_bohm 3 days ago [-]
A big difference between cost and price is often won at the expense of many years of concerted R&D, though
_el1s7 3 days ago [-]
Exactly.
jwpapi 2 days ago [-]
The price is what maximizes profit (long term)
_el1s7 3 days ago [-]
That's not a business 101.
lelanthran 3 days ago [-]
> That's not a business 101.
It kinda is, but obscured by GP's formula.
More simply; if it costs you $X to produce a product and the market is willing to pay $Y (which has no relation to $X), why would you price it as a function of $X?
If it costs me $10 to make a widget and the market is happy to pay $100, why would I base my pricing on $10 * 1.$MARGIN?
carefree-bob 3 days ago [-]
Exactly. The mechanism by which the price ends up as X plus margin is just competition. Others enter the market and compete with you until the returns are driven down to the rental rate of capital. Any barriers to entry result in higher margins.
But that is an equilibrium result, and famously does not apply to monopolies, where elasticity of substitution will determine the premium over the rental rate of capital.
sroussey 3 days ago [-]
Many cloud vendors have you pay through the nose for IOPS and bandwidth.
Edit: I posted this before reading, and these two are the same he points out.
stingraycharles 3 days ago [-]
Yes, but you can’t directly compare SAN-style storage with a local NVMe. But I agree that it’s too expensive, but not nearly as insane as the bandwidth pricing. If you go to a vendor and ask for a petabyte of storage, and it needs to be fully redundant, and you need the ability to take PIT-consistent multi-volume snapshots, be ready to pay up. And this is what’s being offered here.
And yes, IO typically happens in 4kb blocks, so you need a decent amount of IOPS to get the full bandwidth.
amluto 3 days ago [-]
Sure, but a petabyte of block storage with redundancy and PIT backups is a poor abstraction to build on, in large part because it’s not a thing that can be built without either paying an wild amount of money or taking a huge performance hit or both. If you do your PIT recovery at a higher layer, you have to work a bit harder but you get far better cost, perf and recovery.
That latter part is a big deal, too. If I buy 1PB of block storage, I’m decently likely to be running a fancy journaled or WAL-ed or rollback-logged thing on top, and that thing might be completely unable to read from a read only snapshot. So actually reading from a PIT snapshot is a pain regardless of what I paid for it. Even using EBS or similar snapshots is far from being an amazing experience.
torginus 3 days ago [-]
>3000 IOPS
If that's true, I wonder if this is a deliberate decision by cloud providers to push users towards microservice architectures with proprietary cloud storage like S3, so you can't do on-machine dbs even for simple servers.
AnthonyMouse 3 days ago [-]
It's probably a combination of high density storage nodes getting I/O bound and SSDs having finite write endurance. Anything that improves the first problem costs them money to improve it and then makes the second problem worse, and the second one costs them money again, so why would they want to make the default something that costs then more twice if most people don't need it?
Instead they make the default "meager IOPS" and then charge more to the people who need more.
zamadatix 2 days ago [-]
How often is the storage in cloud providers even local vs how often are laptops doing anything other than raw access to a single local disk with a basic FS?
I remember my worked laptop's IOPS beating a single VM on the first SSD based SAN I deployed as well. Of course, the SAN scaled well beyond it with 1,000 VMs.
torginus 2 days ago [-]
I'm not sure about this but I remember that a lot of servers at my old company stuck with hard disks as late as 2018 - exactly for the same reason - HDDs for all their faults dont have write endurance issues. This was quite surprising to me back then.
aliasxneo 3 days ago [-]
There's a common conversation that goes on around AI: some people swear its a complete waste of time and total boondoggle, some that its a good tool when used correctly, and others that its the future and nothing else matters.
I see the same thing happen with Kubernetes. I've run clusters from various sizes for about half a decade now. I've never once had an incident that wasn't caused by the product itself. I recall one particular incident where we had a complete blackout for about an hour. The people predisposed to hating Kubernetes did everything they could to blame it all on that "shitty k8s system." Turns out the service in question simply DOS'd by opening up tens of thousands of ports in a matter of seconds when a particular scenario occurred.
I'm neither in the k8s is the future nor k8s is total trash. It's a good system for when you genuinely need it. I've never understand the other two sides of the equation.
johnmaguire 3 days ago [-]
The complaints I see about Kubernetes are typically more about one of two things: (a) this looks complex to learn, and I don't have a need for it - existing deployment patterns solve my use case, or (b) Kubernetes is much less inefficient than running software on bare-metal (energy or cost.)
Usually they go hand in hand.
aliasxneo 3 days ago [-]
Which is an interesting perspective, considering I've led a platform based on Kubernetes running on company-owned bare-metal. I was actually hired because developers were basically revolting at leaving the cloud because of all the "niceties" they add (in exchange for that hefty cloud tax) which essentially go away on bare-metal. The existing DevOps team was baffled why the developers didn't like when they were handed a plain Ubuntu VM and told to deploy their stack on it.
By the time I left, the developers didn't really know anything about how the underlying infrastructure worked. They wrote their Dockerfiles, a tiny little file to declare their deployment needs, and then they opened a platform webpage to watch the full lifecycle.
If you're a single service shop, then yeah, put Docker Compose on it and run an Ansible playbook via GitHub Actions. Done. But for a larger org moving off cloud to bare-metal, I really couldn't see not having k8s there to help buffer some of the pain.
johnmaguire 2 days ago [-]
For many shops, even Docker Compose is not necessary. It is still possible to deploy software directly on a VM/LXC container.
I agree that Kubernetes can help simplify the deployment model for large organizations with a mature DevOps team. It is also a model that many organizations share, and so you can hire for talent already familiar with it. But it's not the only viable deployment model, and it's very possible to build a deployment system that behaves similarly without bringing in Kubernetes. Yes, including automatic preview deployments. This doesn't mean I'm provided a VM and told to figure it out. There are still paved-path deployment patterns.
As a developer, I do need to understand the environment my code runs in, whether it is bare-metal, Kubernetes, Docker Swarm, or a single-node Docker host. It impacts how config is deployed and how services communicate with each other. The fact that developers wrote Dockerfiles is proof that they needed to understand the environment. This is purely a tradeoff (abstracting one system, but now you need to learn a new one.)
bloppe 3 days ago [-]
It can be inefficient because controllers (typically ~40 per cluster) can maintain big caches of resource metadata, and kubelet and kube-proxy usually operate pretty tight while-loops. But such things can be tuned and I don't really consider those issues. The main issue I've actually encountered is that etcd doesn't scale
p_l 3 days ago [-]
The funniest thing is that kubernetes was designed for bare metal running, not cloud...
jeffbee 2 days ago [-]
Yeah if someone says that k8s is costing them energy they are either using it very, very incorrectly, or they just don't know what they are talking about.
johnmaguire 2 days ago [-]
Running a Kubernetes deployment requires running many additional orchestration services that bare-metal deployments (whether running on-prem or in the cloud) do not.
p_l 2 days ago [-]
"bare metal" "cloud" - pick one.
Also, those simpler deployments usually burn more money per utilized compute, or involve reinventing 80% of kids, often badly
Everything is about trading convenience for knowledge/know how.
It's up to the individual to choose how much knowledge they want to trade away for convenience. All the containers are just forms of that trade.
simonebrunozzi 2 days ago [-]
> (b) Kubernetes is much less inefficient than running software on bare-metal (energy or cost.)
You surely meant "much less efficient than"
johnmaguire 2 days ago [-]
I did, thanks for the correction.
There also seems to be confusion about what I meant by "bare-metal." I wasn't intending to refer to the server ownership model, but rather the deployment model where you deploy software directly onto an operating system.
conductr 3 days ago [-]
Seems like this can be applied to an increasingly large pool of subjects, where things are polarized by default and having a moderate/indifferent opinion is unusual. For example, I thought of US politics while reading your comment
mohamedkoubaa 2 days ago [-]
I thought of how people view hot pepper sauce
abustamam 3 days ago [-]
Good insight. It's always easy to blame that which you don't understand. I know nothing about k8s, and my eyes kinda glaze over when our staff engineer talks about pods and clusters. But it works for our team, even if not everyone understands it.
When all you have is a hammer, every problem starts to look like a nail. And the people with axes are wondering how (or indeed even why) so many people are trying to chop wood with a hammer. Further, some axewielders are wondering why they are losing their jobs to people with hammers when an axe is the right tool for the job. Easy to hate the hammer in this case.
aliasxneo 3 days ago [-]
Yeah, I would attribute that to tribalism. There's an intense amount of dogma in the Kubernetes community, likely stemming from the billions of dollars that get fed into the ecosystem by Big Tech. I genuinely think people adopt it as part of their identity and then become hostile to anyone who "doesn't understand the excellence of Kubernetes." I only say this because I've had many lunch time conversations with random strangers at the various KubeCon conferences I've attended - and let's just say some were pretty eye opening.
p_l 3 days ago [-]
I would also say that a lot of people, even people who are professional k8s operators, don't understand enough of the "theory" behind it. The "why and how", to put it shortly.
And the end result is often that you have two tribes that have totally incorrect idea of even what tools they are using themselves and how, and it's like you swapped them an intentionally wrong dictionary like in a Monthy Python sketch.
rockostrich 3 days ago [-]
At the end of the day it's all different levels of abstractions and whether or not you're using the abstraction correctly. With k8s, the best practices are mostly set in a lot of use cases. For LLMs, we still have no idea what the best practices are.
3 days ago [-]
p_l 3 days ago [-]
Funnily enough the post isn't shitting on k8s, it's shitting on cloud and that k8s (lipstick) can't fix the pig (cloud)
physicles 2 days ago [-]
That part was really surprising to me because for the kind of compute lake he’s talking about building, k8s seems like a pretty good fit for the layer that sits just above it.
We run k8s with several VMs in a couple different cloud providers. I’d love it if I could forget about the VMs entirely.
Is there a simpler thing than k8s that gets you all that? Probably. But if you don’t use k8s, aren’t you doomed to reimplement half of it?
Like these things:
- Service discovery or ingress/routing (“what port was the auth service deployed on again?”)
- Declarative configuration across the board, including for scale-out
- Each service gets its own service account for interacting with external systems
- Blue/green deployments, readiness checks, health checks
- Strong auditing of what was deployed and mutated, when, and by whom
aliasxneo 3 days ago [-]
Yeah, I meant to respond to another thread (the top one currently) that was talking more specifically about k8s-hate.
sahil-shubham 3 days ago [-]
The point about VMs being the wrong shape because they’re tied to CPU/memory resonates hard. The abstraction forces you to pay for time, not work.
I ended up buying a cheap auctioned Hetzner server and using my self-hostable Firecracker orchestrator on top of it (https://github.com/sahil-shubham/bhatti, https://bhatti.sh) specifically because I wanted the thing he’s describing — buy some hardware, carve it into as many VMs as I want, and not think about provisioning or their lifecycle. Idle VMs snapshot to disk and free all RAM automatically. The hardware is mine, the VMs are disposable, and idle costs nothing.
The thing that, although obvious, surprised me most is that once you have memory-state snapshots, everything becomes resumable. I make a browser sandbox, get Chromium to a logged-in state, snapshot it, and resume copies of that session on demand. My agents work inside sandboxes, I run docker compose in them for preview environments, and when nothing’s active the server is basically idle. One $100/month box does all of it.
Love the website and how you have implemented payments! Also giving each box an email is a nice touch.
Thank you for sharing!
stavros 2 days ago [-]
This is pretty cool, I turned a NUC at home into this, and would probably rather use you guys instead. However, is there a way for me to keep a session open without being connected? Sometimes I want the session to be there so I can connect/disconnect to check up on it, so I want "just disconnecting for a bit" to be different from "I don't care about this any more, destroy it".
At home, I've done that with a Zellij session (everything is tied to the session, and quitting Zellij completely means "I'm done with this". Merely disconnecting keeps it running).
messh 2 days ago [-]
Yes, keep a box running regardless of connections using `ssh shellbox.dev keepalive mybox`
otterley 3 days ago [-]
This looks interesting at first blush.
My only feedback so far is that a lot of the documentation, though thorough and useful, looks clearly AI-written. That's not bad in and of itself, but it could be more concise. I especially love the "design decisions" section as I learned something new already.
Have you posted it on "Show HN" already? If not, you should.
sahil-shubham 3 days ago [-]
Thank you for the feedback! I appreciate it, looking forward to you trying it out and logging any issues.
I am aware of the documentation, it’s what I have been focusing on before I can post on HN. I want to make it a delight to read for other people!
As for the design decisions, I have tried keeping all the plans I made in the repo too. I wouldn’t have been able to make bhatti in a month without LLMs.
martypitt 3 days ago [-]
OT - but Bhatti looks really cool! Well done!
sahil-shubham 3 days ago [-]
Thank you :)
codethief 3 days ago [-]
> My agents work inside sandboxes
Out of interest, what sandboxing solution do you use?
sahil-shubham 3 days ago [-]
Not sure what you mean. I use the above linked personal project, bhatti, which internally uses Firecracker microVMs.
codethief 3 days ago [-]
Ah, for some reason I didn't make the connection between your VM setup and your agent sandbox setup and thought those were two separate things. Sorry about that!
sahil-shubham 3 days ago [-]
No worries! I too have been using machines/sandboxes/VMs/microVMs very interchangeably these days. Former is understood by a broader group, but latter is more precise. Trying to find a balance.
varun_chopra 3 days ago [-]
Bhatti is a great name!
clktmr 3 days ago [-]
> Agents, by making it easiest to write code, means there will be a lot more software. Economists would call this an instance of Jevons paradox. Each of us will write more programs, for fun and for work.
There is already so much software out there, which isn't used by anyone. Just take a look at any appstore. I don't understand why we are so obsessed with cranking out even more, whereas the obvious usecase for LLMs should be to write better software. Let's hope the focus shifts from code generation to something else. There are many ways LLMs can assist in writing better code.
delbronski 3 days ago [-]
I think we, as engineers, are a bit stuck on what “software” has traditionally been. We think of systems that we carefully build, maintain, and update. Deterministic systems for interacting with computers. I think these “traditional” systems will still be around. But AI has already changed the way users interact with computers. This new interaction will give rise to another type of software. A more disposable type of software.
I believe right now we are still in the phase of “how can AI help engineers write better software”, but are slowly shifting to “how can engineers help AI write better software.” This will bring in a new herd of engineers with completely different views on what software is, and how to best go about building computer interactions.
pvtmert 2 days ago [-]
the disposeware is here. maybe we can call it spongeware (similar to software) which is like a kitchen sponge, one uses it several times until it starts to break, just to be replaced with a similar but different one...
skybrian 3 days ago [-]
Sometimes “better” means “customized for my specific use case.” I expect that there will be a lot of custom software that never appears in any app store.
stingraycharles 3 days ago [-]
The amount of single purpose scripts in my ~/playground/ folder has increased dramatically over the past year. Super useful, wouldn’t have had the time for it otherwise, but not in any way shareable. Eg “parse this excel sheet I got from my obscure bank and upload it to my budgeting app’s REST API”. Wouldn’t have had the time or energy to do this before, now I have it and it scratches an itch.
AussieWog93 3 days ago [-]
This. Just today I added a full on shopping list system to our internal dashboard at work (small business) simply because it was slightly annoying and could be solved in 3 prompts and 15 minutes.
Gareth321 3 days ago [-]
If we take it a step further, in a few years, why would anyone purchase generic software anymore? If we can perfectly customise software for our needs and preferences for almost free, why would anyone purchase generic software from an App Store? I genuinely think Apple's business model is in jeopardy.
skybrian 2 days ago [-]
Most apps aren’t standalone and the services they depend on are nontrivial to build. For example, maybe you could vibe code a guitar tuner app, but not a ride share app.
Gareth321 2 days ago [-]
I agree. The services which will be left standing will be those with a competitive moat: critical mass (Tinder, Facebook), content (YouTube, AppleTV), and scale (frontier AI models requiring expensive hardware), etc.
That said, if you look at the apps on your phone, I wager a large proportion don't have these moats. Translation, passwords, budget, reminders, email, to do, project management, messaging, browser, calendar, fitness, games, game tracking, etc.
moooo99 2 days ago [-]
Customization often turns out to be a long term liability. Funnily enough, my employer learned this 20 years ago with our ERP and we are still paying the price.
croemer 3 days ago [-]
That's not what Jevons paradox means though. He's just name dropping some concept.
Jevons paradox would be if despite software becoming cheaper to produce the total spend on producing software would increase because the increase in production outruns the savings
Jevons paradox applies when demand is very elastic, i.e. small changes in price cause large changes in quantity demanded. It's a property of the market.
ffin 3 days ago [-]
> Agents, by making it easiest to write code, means there will be a lot more software.
He's saying that agents make code much cheaper, therefore there will be a large increase in demand for code. This appears to be exactly what you're describing.
cush 3 days ago [-]
> I don't understand why we are so obsessed with cranking out even more... the obvious usecase for LLMs should be to write better software
I honestly think this is ideal. Video games aside, I think one day we'll look back and realize just how insane it was that we built software for millions or even billions of users to use. People can now finally build the software that does exactly what they've wanted their software to do without competing priorities and misaligned revenue models working against them. One could argue this kind of software, by definition, is higher quality.
edot 3 days ago [-]
I don't think this will be true for average consumers. Perhaps for nerds like us, who enjoy a bit of tinkering and can put up with weird behaviors. I mean, are you envisioning that everyone would have their own custom messaging app, for example? Or email? Or banking app? I mean, I think most people's demands for those things are all extremely homogenous. I want messages to arrive, I want emails to get spam filtered a little but not too much, and I want my bank to only allow me to log in and see my balances, etc.
I could see maybe more customization of said software, but not totally fresh. I do agree that people will invent more one-off throwaway software, though.
cush 2 days ago [-]
> Perhaps for nerds like us, who enjoy a bit of tinkering
Tinkering? Even today, people don’t need to understand software. They just need to be able to describe their problems and goals to create an app.
> I mean, are you envisioning that everyone would have their own custom messaging app, for example? Or email?
Well first I think there’s a good chance that most apps as we know them today won’t even exist, and most “apps” will be tool use on APIs. But even then, shopping apps, for example, could be so highly personalized that no two people have the same one.
> I mean, I think most people's demands for those things are all extremely homogenous.
They aren’t, as evidenced by the fact there are many dozens of popular messaging apps with millions of users. Despite the network effects for a messaging app to even be viable.
Also, I’m not talking one-off throwaway apps… these are living, breathing pieces of production-grade software users will mold to fit their needs and evolve with them for years.
I’m not sure what “totally fresh” means
woeirua 3 days ago [-]
I think you’re glossing over a lot of use cases. For example, I want my email’s spam controls much tighter.
kelvinjps10 3 days ago [-]
maybe it will be something like excel where people have their custom workflows
esjeon 3 days ago [-]
> Let's hope the focus shifts from code generation to something else. There are many ways LLMs can assist in writing better code.
My view is actually the opposite. Software now belongs to cattle, not pet. We should use one-offs. We should use micro-scale snippets. Speaking language should be equivalent to programming. (I know, it's a bit of pipe dream)
In that sense, exe.dev (and tailscale) is a bit like pet-driven projects.
Gareth321 3 days ago [-]
The most recent software paradigm has been SaaS - software as a service. Capex is distributed among all customers and opex is paid for through the subscription. This avoids the large upfront capex and provides easy cost and revenue projections for both sides of the transaction. The key to SaaS is that the software is maximally generic. Meaning is works well for the largest number of people. This necessitates making tough cuts on UX and functionality when they only benefit small parts of the userbase.
Vibe coding or LLM accelerated development is going to turn this on its head. Everyone will be able to afford custom software to fit their specific needs and preferences. Where Salesforce currently has 150,000 customers, imagine 150,000 customers all using their own customised CRM. The scope for software expansion is unbelievably large right now.
bananamogul 2 days ago [-]
SaaS is not a new idea and has been renamed multiple times.
In the 70s, it was called "time-sharing". Instead of buying a mainframe, you got a CICS application instance on a mainframe and used that. (tangentially, spare time on these built-out nation-wide dialup-supported networks is what gave birth to CompuServe and GEnie).
In the dot-com era, it was called "application service providers". Salesforce and actually started in this era (1999). So did NetSuite. This was the first attempt to be browser-based but bandwidth and browsers sucked then.
I think PaaS is a more recent software paradigm, albeit a far less successful one.
rvz 3 days ago [-]
There will be only 1 Microsoft® Excel, 1 Google Sheets and 1 LibreOffice and the rest are billions of dead vibe-coded "Excel killers" that no-one uses.
mmarian 3 days ago [-]
Democratization of software through SaaS & new engineers brought Airtable, Smartsheet, Baserow, Monday, and many more that I can't remember though.
fragmede 3 days ago [-]
Except that list originally had one item, and that item was Visicalc. Times change, but that list is going to stop being relevant before Excel gets knocked off the list.
If you're doing anything complicated, Excel just doesn't make sense anymore. it'll still the be data exchange format (at least, something more advanced than csv), but it's no longer the only frontend.
"No one uses" is no longer the insult it once was. I don't need or want to make software for every last person on the world to use. I have a very very small list of users (aka me) that I serve very well with most of the software that I generate these days outside of work.
rvz 3 days ago [-]
> "No one uses" is no longer the insult it once was.
It certainly is for lots of businesses, otherwise they go out of business.
There is something called 'revenue' which they need to make from customers which are their 'users', and that revenue pays for the 'operating costs' which includes payroll, office rent, infrastructure etc.
This just means that it is important than ever to know what to build just as how it is built. It is unrealistic for a business to disregard that and to build anything they want and end up with zero users.
No users, No revenue. No revenue, No business.
fragmede 2 days ago [-]
Oh is that how it works? This "user" concept. Is that like a "customer"? Do they enjoy being talked down to? Are they excited to give money to people who treat them like that? I suspect you're not as good as you think you are at this business thing.
dgb23 3 days ago [-]
Both will likely happen to some degree.
As for the average quality: it’s unclear.
My intuition is that agents lift up the floor to some degree, but at the same time will lead to more software being produced that’s of mediocre quality, with outliers of higher quality emerging at a higher rate than before.
3 days ago [-]
andai 3 days ago [-]
Alas, we shifted from quality to quantity somewhere in the mid 19th century.
appreciatorBus 3 days ago [-]
Humans have been making quality versus quantity decisions since the time we first grew these big giant brains of ours a million or two years ago, maybe longer.
If you wanted to, you could make an argument about the principal-agent problem - that as hunter-gatherers or subsistence, farmers, our quality versus quantity decisions only affected us, whereas in a market economy, you could argue that one person’s quality versus quantity decision affects someone else.
But dismantling capitalism will not solve this problem. It just moves the decision-making to a different group of people. Those people will face the same trade-offs and the same incentives. After the Revolution, even the most loyal comrade will have to contend with the fact that they can choose to provide the honourable working class with more of a thing if they drop the quality.
What does that have to do with the mid 19th century?
kyle-rb 3 days ago [-]
In the California gold rush, the people who got rich were the ones selling shovelware.
fragmede 2 days ago [-]
One upvote is not enough for that one.
amelius 3 days ago [-]
Yes, and most applications still have GUIs, where we could be just talking to an LLM instead.
culi 2 days ago [-]
Big agree. I would love the focus to be on contributing, improving, and consolidating around existing open-source solutions. Unfortunately, most AI-enabled contributions have been slop and the maintenance burden of open source has increased
farfatched 3 days ago [-]
Nice post. exe.dev is a cool service that I enjoyed.
I agree there is opportunity in making LLM development flows smooth, paired with the flexibility of root-on-a-Linux-machine.
> Time and again I have said “this is the one” only to be betrayed by some half-assed, half-implemented, or half-thought-through abstraction. No thank you.
The irony is that this is my experience of Tailscale.
Finally, networking made easy. Oh god, why is my battery doing so poorly. Oh god, it's modified my firewall rules in a way that's incompatible with some other tool, and the bug tracker is silent. Now I have to understand their implementation, oh dear.
No thank you.
LoganDark 3 days ago [-]
I find it difficult to configure Tailscale for my use case because they seem to completely not support making ACL rules based on the identity of the device rather than a part of the address space. I'm not configuring a router here, I'm configuring a peer-to-peer networking layer... or at least I'm supposed to be...
spockz 3 days ago [-]
I remember from the docs you can use node names. At the very least you can use tags for sure. Assign tags to nodes and define the ACL based on those.
LoganDark 3 days ago [-]
Last I read the docs while troubleshooting this very problem, you cannot specify node names as the source or destination of a grant. You can specify direct IP address ranges, node groups (including autogenerated ones) or tags, but not names.
Tags permanently erase the user identity from a device, and disable things like Taildrop. When I tried to assign a tag for ACLs, I found that I then could not remove it and had to endure a very laborous process to re-register a Tailscale device that I added to Tailscale for the express purpose of remotely accessing
ghthor 3 days ago [-]
You can ack based on groups, and you can out users into groups. So if you auth a node, it’s now your node and the ACL for your user / group will apply.
But yes I don’t think you can ACL based o the hostname
andrew-d 3 days ago [-]
Hi there, I work at Tailscale.
Part of the reason that we don't (currently) let you do this is that a hostname is a user-reported field, and can change over time; it's not a durable form of identity that you can write ACLs on. One could imagine, for example:
1. Creating an ACL rule that allows hostname "webserver" to hostname "db".
2. (time passes)
3. Hostname "webserver" is deleted/changed to "web"/etc.
4. Someone can now register a user device with the system hostname set to "webserver"
Should they be allowed to inherit the pre-existing ACL rule?
However, you can accomplish something very close to what you're asking for, I think, by defining a "host" in the policy file (https://tailscale.com/docs/reference/syntax/policy-file#host...) that points to a single Tailscale IP. Since we don't allow non-admins to change their Tailscale IP, this uniquely identifies a single device even if the hostname changes, and thus you can write a policy similar to:
That's why my next instinct was to try specifying a node key, which does not change unless the device is re-registered, but that does not work either.
codethief 3 days ago [-]
> because they seem to completely not support making ACL rules based on the identity of the device rather than a part of the address space
Could you rephrase that / elaborate on that? Isn't Tailscale's selling point precisely that they do identity-based networking?
EDIT: Never mind, now I see the sibling comment to which you also responded – I should have reloaded the page. Let's continue there!
farfatched 3 days ago [-]
> No thank you.
I hope this wasn't interpreted towards exe.dev. That really is a cool service!
mancerayder 3 days ago [-]
Numerous people are denigrating DevOps people - resume padding, over-complexity, etc.
I think that's startup-thinking, at least in my experience. Maybe in a small company the DevOps guy does all infra.
In my experience, especially in financial services, who runs the show are platform engineering MDs - these people want maximum control for their software engineers, who they split up into a thousand little groups who all want to manage their own repos, their own deployments, their own everything. It's believed that microservices gives them that power.
I guarantee you devops people hate complexity, they're the ones getting called at night and on the weekend, because it's supposedly always an "infrastructure issue" until proven otherwise.
Also the deployment logs end up in a log aggregation system, and god forbid software developers troubleshoot their own deployments by checking logs. It's an Incident.
Are microservices a past fad yet?
faangguyindia 3 days ago [-]
i just use Hetzner.
Everything which cloud companies provide just cost so much, my own postgres running with HA setup and backup cost me 1/10th the price of RDS or CloudSQL service running in production over 10 years with no downtime.
i directly autoscales instances off of the Metrics harvested from graphana it works fine for us, we've autoscaler configured via webhooks. Very simple and never failed us.
i don't know why would i even ever use GCP or AWS anymore.
All my services are fully HA and backup works like charm everyday.
mattbee 3 days ago [-]
I founded a hosting company 25 years ago when User-Mode Linux was the hot new virtualisation tech. We aspired to just replicate the dedicated server experience because that was obviously how you deploy services with the most flexibility, and UML made it so cheap! Through the 2010s I (extremely wrongly) assumed that being metered on each little part of their stack was not something most developers would choose, for the sake of a little convenience.
Does a regular 20-something software engineer still know how to turn some eBay servers & routers into a platform for hosting a high-traffic web application? Because that is still a thing you can do! (I've done it last year to make a 50PiB+ data store). I'm genuinely curious how popular it is for medium-to-big projects.
And Hetzner gives you almost all of that economic upside while taking away much of the physical hassle! Why are they not kings of the hosting world, rather than turning over a modest €367M (2021).
I find it hard to believe that the knowledge to manage a bunch of dedicated servers is that arcane that people wouldn't choose it for this kind of gigantic saving.
jasongi 3 days ago [-]
> I find it hard to believe that the knowledge to manage a bunch of dedicated servers is that arcane that people wouldn't choose it for this kind of gigantic saving.
Managing servers is fine. Managing servers well is hard for the average person. Many hand-rolled hosting setups I've encountered includes fun gems such as:
- undocumented config drift.
- one unit of availability (downtime required for offline upgrades, resizing or maintenance)
- very out of date OS/libraries (usually due to the first two issues)
- generally awful security configurations. The easiest configuration being open ports for SSH and/or database connections, which probably have passwords (if they didn't you'd immediately be pwned)
Cloud architecture might be annoying and complex for many use-cases, but if you've ever been the person who had to pick up someone else's "pet" and start making changes or just maintaining it you'll know why the it can be nice to have cloud arch put some of their constraints on how infra is provisioned and be willing to pay for it.
apublicfrog 2 days ago [-]
For the record, I have seen every one of those in cloud based hosting multiple times. None of those issues require special work any more than they do than in traditional hosting.
moooo99 2 days ago [-]
> And Hetzner gives you almost all of that economic upside while taking away much of the physical hassle! Why are they not kings of the hosting world, rather than turning over a modest €367M (2021).
Hetzner is an oldschool German company, it is not surprising to see them act this way. They are very profitable (165M Euro in 2024) and have very little debt. They also seem to be mostly bootstrapped and are not VC funded
Companies buy cloud services because they want to reduce in-house server management and operations, for them it's a trade-off with hiring the right people. But you are right, when you can find the right people doing it yourself can be a lot cheaper.
mrweasel 3 days ago [-]
In some sense I'm starting to think it has more to do with accounting. Hardware, datacenters and software licenses (unless it's a subscription, which is probably is these days) are capital expenses, cloud is an operation expense. Management in a lot of companies hates capital expenditures, presumable because it forces long term thinking, i.e. three to five years for server hardware. Better to go the cloud route and have "room for manoeuvrability". I worked for a company that would hire consultants, because "you can fire those at two weeks notice, with no severance". Sure, but they've been here for five years now, at twice the cost of actual staff. Companies like that also loves the cloud.
Whether or not cloud is viable for a company is very individual. It's very hard to pin point a size or a use case that will always make cloud the "correct" choice.
whyagaindavid 3 days ago [-]
Another point (but my common observation) is the responsibility. By going SaaS or using cloud - any kind of data protection, rules/responsibility etc is moved away. and in many ways it is better - Google, dropbox or Onedrive will have better PR to take the pain if something goes crazy. Tickbox compliance is easy.
duped 3 days ago [-]
Something I know nothing about is whether the depreciation on server hardware outpaces the value it creates for a business, creating a tax incentive to own your own metal.
fnoef 3 days ago [-]
Right... That's why the hire "AWS Certified specialist ninja"
Tepix 3 days ago [-]
I get the feeling that with LLMs in the mix, in-house server management can do a lot more than it used to.
tgv 3 days ago [-]
Perhaps it saves some time looking through the docs, but do you really trust an LLM to do the actual work?
windex 3 days ago [-]
Yes and an LLM checks it as well. I am yet to find a sysadmin task that an LLM couldn't solve neatly.
jdkoeck 3 days ago [-]
A nice bonus is that sysadmin tasks tend to be light in terms of token usage, that’s very convenient given the increasingly strict usage limits these days.
andoando 3 days ago [-]
Yes, with a lot of reviewing what its doing/asking questions, 100%
fragmede 3 days ago [-]
By this point? Absolutely. They still get stuck in rabbit holes and go down the wrong path sometimes, so it's not fully fire and forget, but if you aren't taking advantage of LLMs to perform generic sysadmin drudgery, you're wasting your time that could be better spent elsewhere.
mattbee 3 days ago [-]
The internet of 20 years ago was awash with info for running dedicated servers, fragmented and badly-written in places but it was all there. I can absolutely believe LLMs would enable more people to find that knowledge more easily.
huijzer 3 days ago [-]
Agree, I used to always use Heroku or Render style platforms for my own software, but nowadays I just have a Linux server with Docker Compose and a Cron job. The cron job every minute runs docker pull (downloads latest image) and docker up -d (switches to new version only if there is a new version). And put caddy in front for the HTTPS. This has been very cheap and reliable for years now.
saltmate 3 days ago [-]
What images are you running that you'd need the latest version up after just a minute?
burner420042 3 days ago [-]
I'm not the OP but I'd clarify the cron check for new versions is done every minute. So when new images are pushed they're picked up quickly.
OP is not saying they push new versions at such a high frequency they need checks every one minute.
The choice of one minute vs 15 minute is implementation detail and when architected like this costs nothing.
I hope that helps. Again this is my own take.
huijzer 3 days ago [-]
When I push new images via CI, I want it to go in production immediately. Like Heroku/Render/Dokku
jstanley 2 days ago [-]
So why do you only poll once per minute? You could be sitting around for 59 seconds while nothing happens.
Maybe you meant to say "automatically" instead of "immediately"? Because if you really mean "immediately" then there is still plenty of low-hanging fruit to be had.
RandomBK 3 days ago [-]
One annoyance (I don't know if they've since fixed it) was that Docker Hub would count pulls that don't contain an update towards the rate limit. That ultimately prompted me to switch to alternate repositories.
faangguyindia 3 days ago [-]
one way is to host a manifest file (can host one on r2) and update it on each deploy and when manifest changes, new container image is pulled.
Jn2G3Np8 3 days ago [-]
Also using Hetzner.
But I came across Mythic Beasts (https://www.mythic-beasts.com/) yesterday, similar idea, UK based. Not used them yet but made the account for the next VPS.
tubs 3 days ago [-]
This is way way more expensive than hetzner. Not even comparable?
pants2 3 days ago [-]
Especially these days you can SSH to a baremetal server and just tell Claude to set up Postgres. Job done. You don't need autoscaling because you can afford a server that's 5X faster from the start.
i5heu 3 days ago [-]
You just use docker.
It is like 4 lines of config for Postgres, the only line you need to change is on which path Postgres should store the data.
spockz 3 days ago [-]
You also probably want the Postgres storage on a different (set) of disks.
Maybe change the filesystem?
kippinsula 3 days ago [-]
we've done both. Hetzner dedicated was genuinely fine, until a disk started throwing SMART warnings on a Sunday morning and we remembered why we pay 10x elsewhere for some things. probably less about the raw cost and more about which weekends you want back.
omnimus 3 days ago [-]
Isn't this nature of every dedicated server? You also take on the hardware management burden - that's why they can be insanely cheap.
But there is middleground in form of VPS, where hardware is managed by the provider. It's still way way cheaper than some cloud magic service.
RandomBK 3 days ago [-]
VPS comes at the cost of potential for oversubscription - even from more reputable vendors. You never really know if you're actually getting what you're paying for.
faangguyindia 3 days ago [-]
They also offer dedicated VPS with guaranteed resource allocation.
faangguyindia 3 days ago [-]
Well, you gotta take all that into consideration before your build out.
You can use block storage if data matters to you.
Many services do not need to care about data reliability or can use multiple nodes, network storage or many other HA setups.
alishayk 3 days ago [-]
I find it interesting that Hetzner was never a consideration, until... LLMs started recommending them.
alternatex 3 days ago [-]
Hetzner was raved about before AI was cool. I know since based on those good reviews I moved half of my apps from DigitalOcean to Hetzner. My DigitalOcean droplet was lacking in RAM and it was more expensive for me to grow it than move some stuff to another small VPS on Hetzner.
3 days ago [-]
MagicMoonlight 3 days ago [-]
Because if I have a government service with millions of users, I don’t want the cheap shitter servers to crap out on me.
An employee is going to cost anywhere between 8k and 50k per month. Hiring an employee to save 200/month on servers by using a shitty VPS provider is not saving you any money.
kennywinker 3 days ago [-]
If you have millions of users, you absolutely need to have someone whose whole job is managing infrastructure. Expecting servers or cloud services to not crap out on you without someone with the skills and time to keep things running seems foolish.
swingboy 3 days ago [-]
Do you run containers? What orchestrator or deploy tool do you use?
TiccyRobby 3 days ago [-]
Honestly I like Hetzner a lot but lately it has been very unstable for us. https://status.hetzner.com/ this page always has couple of incidents happening at the same time. I really appreciate the services they provide but i wish they were more stable.
lifty 3 days ago [-]
There are several things going on even now, 1 hour after your comment. But I appreciate that they list them. That hopefully means that they have a good culture of honesty, and they can improve.
omnimus 3 days ago [-]
I looked through the issues and basically only ongoing thing is that backup power is not working in one of the data centers (could be a problem). The rest are warnings about planned shutdown of some services and speed limitation of object storage in one location.
I am sure it's luck but we have few hetzner VPSes in both German locations and in last 5 years afaik they've never been down. On our http monitor service they have 100s of days uptime only because we restarted them ourselves.
kubb 3 days ago [-]
[dead]
socketcluster 3 days ago [-]
Virtual machines are the wrong abstraction. Anyone who has worked with startups knows that average developers cannot produce secure code. If average developers are incapable of producing secure code, why would average non-technical vibe-coders be able to? They don't know what questions to ask. There's no way vibe coders can produce secure backend software with or without AI. The average software that AI is trained on is insecure. If the LLM sees a massive pile of fugly vibe-coded spaghetti and you tell it "Make it secure please", it will turn into a game of Whac-a-Mole. Patch a vulnerability and two new ones appear. IMO, the right solution is to not allow vibe-coders to access the backend. It is beyond their capabilities to keep it secure, reliable and scalable, so don't make it their responsibility. I refuse to operate a platform where a non-technical user is "empowered" to build their own backend from scratch. It's too easy to blame the user for building insecure software. But IMO, as a platform provider, if you know that your target users don't have the capability to produce secure software, it's your fault; you're selling them footguns.
messh 3 days ago [-]
There are plenty of alternatives out there. I built https://shellbox.dev, which gives you instant vms via ssh where unlike exe you pay only for what you use-- scale to zero. It is also regular linux, supporting vscode and zed remote, Nested virtualization, etc.
If you're looking to invest im fine with only $5M :)
pbronez 2 days ago [-]
Neat service. Website doesn't provide enough information for me to trust any workloads to it. Not clear where the underlying infrastructure is, what security guarantees I get, etc.
I quickly grokked the product and pricing from Exe's website. You need a page with less text, less scrolling and more specifics. Can VMs not autostop? Will an API call to a service on a stopped VM fail at the first attempt? Your pricing formula is simple but I don't want to do the math on my phone. How do payments work exactly (processor)?
Running Shellbox 24/7 is ~25% cheaper than Exe, with 2x storage but 50% of RAM. Exe seems to provide additional features (which I don't need). Not presenting this information upfront and in an easily digestible format makes me suspicious.
I dig the overall aesthetic and may give Shellbox v2 a try.
SeriousM 2 days ago [-]
The other day I vibed a very stable codeserver (vscode in browser) instance with zellij browser mode (console in browser), syncthing (filesyncing), ssh, pi agent and wireguard. No exposed ports, every web frontend is password secured.
I don't want to make that public, it's my way of an isolated dev environment and it runs on my private raspberry behind my tv. Costs me nothing.
`ssh you/repo/branch@box.clawk.work` → jump directly into Claude Code (or Codex) with your repo cloned and credentials injected.
Firecracker VMs, 19€/mo.
POC, please be kind.
hmokiguess 3 days ago [-]
I’m curious about it do you have a page with more details on specs configs and what else goes on in there?
aayushdutt 3 days ago [-]
This looks nice, when did you launch this? Do you have validation / paying users?
celrenheit 3 days ago [-]
Launched today, this HN thread is the first public mention. Zero paying users yet, just finished building it over the past few weeks as a solo project.
If you want to try it: code `HNPRELAUNCH` on checkout, first month free, then 19€/mo (cancel anytime from your Stripe receipt). Limited to the first 20 redemptions, expires in a week.
Honest feedback on what breaks would mean a lot.
chimpanzee2 3 days ago [-]
honestly sounds interesting
at 19€/mo are you subsidizing it given the sharp rise of LLM costs lately?
or are you heavily restricting model access. surely there is no Opus?
celrenheit 3 days ago [-]
The 19€/mo is infra only. Claude Code inside the VM signs in via OAuth to the user's own Anthropic account.
I'd love to explore bundling open models (Qwen, etc..) into the subscription down the line, but that needs product validation first, not going to ship something I'm not sure people actually want.
3 days ago [-]
zackify 3 days ago [-]
That's insane funding so congrats.
Just shows I'm the Dropbox commentator. I have what exe provides on my own and am shocked by the value these abstractions provide everyone else!! One off containers on my own hardware spin up spin down run async agents, etc, tailscale auth, team can share or connect easily by name.
sixhobbits 3 days ago [-]
Investment is done by relationships, belief in a future vision and team, and growth metrics like number of paying customers.
The technology itself in its current form is not valuable
isoprophlex 3 days ago [-]
Sobering comment for all the little people like myself who dream of owning a business based on a vision of cool tech that just does what it promises (as opposed to all the corporate shovelware out there)
crawshaw 3 days ago [-]
Author here.
Almost every VC rejected us when we went to get seed funding for Tailscale, we knew none of them. Friends of friends of acquaintances got us meetings. Fundraising is very possible for you if you are committed to building a business. Most important thing is don't think of fundraising as the goal, it is just a tool for building a business. (And some businesses don't need VC funding to work. Some do.)
The biggest challenge is personal: do you want to build a business or do you want to work with cool tech? Sometimes those goals are aligned, but usually they are not. Threading the needle and doing both is difficult, and you always have to prioritize the business because you have to make payroll.
momojo 2 days ago [-]
How did you eventually get funding after those initial rejections? What changed?
crawshaw 2 days ago [-]
One VC (well, two) understood it. Despite what you hear, there is a lot of variation. Speak to a lot of people.
ignoramous 1 days ago [-]
Surprising that a founding team as strong as Tailscale's had to go door-to-door to get seed funding in. Glad that Tailscale did & changed the industry like it ought to, though I'm sure, y'all would have managed to self-fund it either way.
dgb23 3 days ago [-]
You can still do that. Not every business needs to be a hyperscaling startup.
qudat 3 days ago [-]
Agreed! Over at https://pico.sh we are chugging along and having a blast. Profitable but at a scale that is manageable for us. Cheers
simonebrunozzi 2 days ago [-]
> Just shows I'm the Dropbox commentator.
Ha! This made me smile :)
poly2it 3 days ago [-]
Why is an imperative SSH interface a better way of setting cloud resources than something like OpenTofu? In my experience humans and agents work better in declarative environments. If an OpenTofu integration is offered in the future, will exe.dev offer any value over existing cost-effective VPS providers like Hetzner? Technically, Hetzner, for example, also allows you to set up shared disk volumes:
It also has a CLI, hcloud. Am I getting any value with exe.dev I couldn't get with an 80 line hcloud wrapper?
ZihangZ 3 days ago [-]
I don't think SSH vs OpenTofu is the core issue here.
For agents, declarative plans are still valuable because they are reviewable. The interesting question is whether exe.dev changes the primitive: resource pools for many isolated VM-like processes, or just nicer VPS provisioning.
poly2it 3 days ago [-]
It doesn't do either at competitive rates by the looks of it.
dbmikus 2 days ago [-]
I really like exe.dev's pricing model where I pay a fixed monthly fee for compute and then can split it up into as many VMs as I want. I use exe.dev to run little vibe-coded apps and it's nice to just leave them running without a spend meter ticking up.
We're thinking about switching to this pricing model for our own startup[1] (we run sandboxed coding agents for dev teams). We run on Daytona right now for sandboxes. Sometimes I spin up a sandboxed agent to make changes to an app, and then I leave it running so my teammate can poke around and test the running app in the VM, but each second it's running we (and our users) incur costs.
We can either build a bunch of complicated tech to hibernate running sandboxes (there's a lot of tricky edge cases for detecting when a sandbox is active vs. should be hibernated) or we can just provision fixed blocks of compute. I think I prefer the latter.
Europe is crying out for sovereign clouds. If this is to be a viable alt cloud, US jurisdiction is a no.
Not sure we can move away from cpu/memory/io budgeting towards total metal saturation because code isn't what it used to be because no one handles malloc failure any more, we just crash OOM
Quothling 3 days ago [-]
Europe is already moving into the EU cloud. Hetzner, OGH Cloud and so on as well as local data centers where partner companies set up own cloud with various things to rival office 365. So far it's mainly the public sector. My own city cut their IT budget by 70% by switching from Microsoft.
The key point is the partner companies. Almost nobody is actually running their own clouds the way they would with various 365 products, AWS or Azure. They buy the cloud from partners, similar to how they used to (and still do) buy solutions from Microsoft partners. So if you want to "sell cloud" you're probably going to struggle unless you get some of these onboard. Which again would probably be hard because I imagine a lot of what they sell is sort of a package which basically runs on VM's setup as part of the package that they already have.
effisfor 3 days ago [-]
For anybody interested, the meat of 'EU sovereign' means EU companies, not US or UK companies with EU servers. (because of CLOUD Act and the UK-US bilateral arrangement connected to it).
International visitors might tell us more about benefits of non EU, US or UK nexus companies/legal/rights.
overfeed 2 days ago [-]
People who don't understand the software bloat cycle are doomed to repeat it.
Lean software -> missing features users want -> add features over time -> bloated mess -> we need a smaller rewrite -> Lean software -> ...
chr15m 2 days ago [-]
True. The solution is to make a different piece of software customised for each person.
chanux 21 hours ago [-]
We already have the tech for this. It's large! j/k
chr15m 19 hours ago [-]
Ha ha true. It's literally so complicated we don't know how it works.
mohamedkoubaa 2 days ago [-]
It's more of a spiral than a loop, usually the reboot either flops or gets something crucial right and progresses to the point where it challenges the incumbent.
tamimio 3 days ago [-]
> $20/month for your VMs
>One price, no surprises. You get 2 CPUs, 8 GB of RAM, and 25 GB of disk—shared across up to 25 VMs.
This might sounds like a good thing compared to the current state of clouds, but what’s better than that is having your own. The other day I got a used optiplex for $20, it had 2TB hdd, 265gb ssd, 16gb, and corei7. This is a one time payment, not monthly. You can setup proxmox, have dozens of lxc and vm, and even nest inside them whatever more lxc too, your hardware, physically with you, backed up by you, monitored by you, and accessed only by you. If you have stable internet and electricity, there’s really no excuse not to invest on your own hardware. A small business can even invest in that as well, not just as a personal one. Go to rackrat.net and grab a used server if you are a business, or a good station for personal use.
sroussey 3 days ago [-]
> The standard price for a GB of egress from a cloud provider is 10x what you pay racking a server in a normal data center.
Oh, that’s too kind. More like 100x to 1000x. Raw bandwidth is cheap.
ac29 3 days ago [-]
It was a weird point to make in the post given that exe.dev charges $0.07/GB for transfer. That's arguably worse than the major clouds, who charge about the same for egress but give you free ingress.
crawshaw 3 days ago [-]
Author here.
I need to fix our transfer pricing. (In fact I'm going to go look at it now.) I set that number when we launched in December, and we were still considering building on top of AWS, so we put a conservative limit based on what wouldn't break the bank on AWS. Now that we are doing our own thing, we can be far more reasonable.
simonebrunozzi 2 days ago [-]
Love the attitude!! Well done, and good luck with all this. I sent you an email offering help, if any is needed/welcome.
pxc 2 days ago [-]
This is the problem for me with the cloud:
> Finally, clouds have painful APIs. This is where projects like K8S come in, papering over the pain so engineers suffer a bit less from using the cloud. But VMs are hard with Kubernetes because the cloud makes you do it all yourself with lumpy nested virtualization. Disk is hard because back when they were designing K8S Google didn’t really even do usable remote block devices, and even if you can find a common pattern among clouds today to paper over, it will be slow. Networking is hard because if it were easy you would private link in a few systems from a neighboring open DC and drop a zero from your cloud spend. It is tempting to dismiss Kubernetes as a scam, artificial make work designed to avoid doing real product work, but the truth is worse: it is a product attempting to solve an impossible problem: make clouds portable and usable. It cannot be done.
Please learn from Unix's mistakes. Learn from Nix. Support create-before-destroy patterns everywhere. Forego all global namespaces you can. Support rollbacks everywhere.
If any cloud provider can do that, cloud IaC will finally stop feeling so fake/empty compared to a sane system like NixOS.
st-keller 3 days ago [-]
Hahaha! Have fun! I‘m doing the same - together with Claude Code. Since August. With https (mTLS1.3) everywhere, because i can. Just my money, just my servers, just for me. Just for fun. And what a fun it is!
anonzzzies 3 days ago [-]
Me too. I already moved our products to it and it is getting fairly robust. Guess many smaller companies got tired with the big guys asking a lot of money for things that should be cheap.
setnone 3 days ago [-]
Yeah i feel like it's getting cloudy
boesboes 3 days ago [-]
I have mixed feelings about this concept, I agree that the way clouds work now is far from great and stronger abstractions are possible. But this article offers nothing of the sort, it just handwaves 'we solve some problem and that saves you tokens'???
Checking the current offering, it's just prepaid cloud-capacity with rather low flexibility. It's cheap though, so that is nice I guess. But does this solve anything new? Anything fly.io orso doesn't solve?
What is the new idea here? Or is it just the vibes?
ignoramous 2 days ago [-]
> It's cheap though
As another user notes in this thread, exe.dev isn't that cheap. Their bandwidth pricing is £7/100gb. The lowest compute tier is £20/mo (Fly.io machines/sprites can go for less than £2/mo).
> Anything fly.io also doesn't solve?
exe.dev is comparable to sprites.dev Fly.io launched recently; but with a different pricing model.
David, by the way of Tailscale, themselves were among early users of Fly.io. I read some of David's commentary on "Cloud 1.0" as taking a dig at their friends at Fly.io, too. This is going to be interesting...
bradley13 2 days ago [-]
Ok, what am I missing? exe.dev says: "$20/month for your VMs
One price, no surprises. You get 2 CPUs, 8 GB of RAM, and 25 GB of disk".
Fine, their UI is different, but I don't see any real difference from other providers.
joevandyk 2 days ago [-]
You get one machine.
On that machine you can (easily) make an arbitrary number of VMs.
Each VM has their own URL that you can share (or make private).
I relate to the points about obscure platform limits, and leaky abstractions, but when I look at the exe.dev platform, it might be the most obscure PaaS I've seen, and has it's own strange abstractions.
The shell command to start a new vm, has a --prompt flag to get an LLM to configure the VM for you.
VM's have no public ipv4 IP, and the ipv6 IP doesn't seem to allow incoming connections.
The only supported inbound connections are via their HTTP proxy.
There is no private networking.
At first I interpreted the complaint about cloud providers not offering nested-virtualization, as something he intends to address by offering it as a feature, but no, instead he means that exe.dev's VM abstraction eschews the need for it.
neom 2 days ago [-]
Yeah I had a giggle about that also. He argues: “cloud abstractions are the wrong shape”, then what they actually ship is: a different abstraction, with even more hidden constraints.
I'm very curious how they deal with subscription levels/noisy neighbors.
jFriedensreich 3 days ago [-]
I have trouble seeing how this is different to linode, if i invest time in a new VM api, this has to work for cloud or my own machines transparently. Lastly as much as i share the disappointment in k8s promise, this seems a bit too simple, there is a reason homelabs mostly standardised on compose files.
iqihs 2 days ago [-]
linode came to mind as well but OP seems to be much more focused on building cloud as a hardware service rather than VM
bedstefar 3 days ago [-]
This looks like an excellent platform for running a "homelab" in the cloud (no, the irony is not lost on me) for lighter stuff like Readeck, Calibre-web, Immich. Maybe even Home Assistant too if we can find a way (Tailscale?) to get the mDNS/multicast traffic tunnelled.
omnimus 3 days ago [-]
With pricing 100gb/8usd Immich would be wildly uneconomical. Better to wait for upcoming immich hosting to support the project or use ente.io - those are 1tb/10usd.
bedstefar 3 days ago [-]
That's a good tip, thanks. What I meant to say was that there's probably at least a handful of self-hosted services you could run to offset that $20/mo.
Another one could be Bitwarden, although I don't host my own password manager personally. Or netbird. You get the point
ianpurton 3 days ago [-]
I don't get it, what is this, how is it different?
szszrk 3 days ago [-]
You choose a region. Then you pay for some compute size (vcpu and mem), and then you can create a lot of VMs using those limits. If some VM's don't consume all resources, others can consume it in burst.
VMs have a built-in gateway to cloud providers with a fixed url with no auth. You can top that in via the service itself. No need for your own keys.
So likely a good tool for managing AI agents.
And "cloud" is a bit of a stretch, the service is very narrow.
The complete lack of more detailed description of the regions except city name makes it really only suitable for ephemeral/temporary deployments. We don't know what the datacenters are, what redundancy is in place, no backups or anything like that.
saltmate 3 days ago [-]
As I understand, a cloud provider where instead of paying for each VM (with a set of resources), you pay for the resources, and can get as many VMs as you can fit on these resources.
JokerDan 3 days ago [-]
I have had an eye on this for a while (found via pi.dev) but I don't really have a solid use case for it, but the idea/concept of is appealing where the price is not. I can buy a £100-150 mini-pc with better hardware to run 24/7 for my own VMs extending my homelab (granted my ISP doesn't put any restrictions on me, I know many others can't say the same).
I'm not sure if this is the direction the OP is going, but I would love to see a world where local small-time investors can get a bank loan, rent a facility, set up a bunch of computers, and run open-source cloud software on them that provides 95% of the features that most businesses need.
Running a cloud data center could be a business like operating a self-storage facility or a car wash. Small investors love this kind of operation.
DASD 2 days ago [-]
This brings back painful yesteryear memories when consultants would push Microsoft Small Business Server for an on-prem all in one IT solution and charge extra to move to the cloud(colocation). A Linux-based alternative appeared named Clark Connect.
mrhottakes 2 days ago [-]
then private equity moves in, buys up the small and inefficient local data centers, and consolidates them into a massive conglomerate that can charge whatever they want
jacobsenscott 2 days ago [-]
These are called co-los (co-location facilities). Probably any medium to large city has a few.
47872324 3 days ago [-]
exe.dev. 111 IN A 52.35.87.134
52.35.87.134 <- Amazon Technologies Inc. (AT-88-Z)
crawshaw 3 days ago [-]
Hello, author here.
Our exe.dev web UI still runs on AWS. We also have a few users left on our VM hosts there, as when we launched in December we were considering building on AWS. Now almost all customer VMs are on other bare metal providers or machines we are racking ourselves. We built our own GLB with the help of another vendor's anycast network. You can see that if you try any of the exe.xyz names generated for user VMs.
We would move exe.dev too, but we have a few customers who are compliance sensitive going through it, so we need to get the compliance story right with our own hardware before we can. It is a little annoying being tied to AWS just for that, but very little of our traffic goes through them, so in practice it works.
skybrian 3 days ago [-]
Their first location (PDX) is on Amazon I believe and not accepting new customers. They’ve said it’s much more expensive for them than the others. Their other locations are listed here:
Well yes, because they needed high availability and flexibility and tons of features…
Hey wait a minute!
awhitty 3 days ago [-]
"I am white labeling a cloud"
transitorykris 3 days ago [-]
FTA “Hence the Series A: we have some computers to buy.”
pelasaco 3 days ago [-]
Such statement is so off:
"In some tech circles, that is an unusual statement. (“In this house, we curse computers!”) I get it, computers can be really frustrating. But I like computers. I always have. It is really fun getting computers to do things. Painful, sure, but the results are worth it. Small microcontrollers are fun, desktops are fun, phones are fun, and servers are fun, whether racked in your basement or in a data center across the world. I like them all."
The reality: Everyone reading his blog or this HN entry loves computers.
sudo_cowsay 3 days ago [-]
I'm still new to cloud computing. I've only ever used linode. What is this supposed to be? I couldn't figure out a specific design through the article well. Pls help
rbren 2 days ago [-]
Lots of negativity towards k8s in here. It's always funny to me when $WILDLY_POPULAR_TECH gets ripped apart like this, as though no one has ever had a positive experience with it. I've seen similar pile-ons for React, microservices, git, PHP, JavaScript, cloud services, really anything that's been adopted at scale.
layer8 2 days ago [-]
It’s only natural that seeing frequent complaints mostly happens for tech that has high adoption. Stuff that nobody uses doesn’t get many complaints.
adamors 2 days ago [-]
HN has had a hate boner for K8s for as long as I can remember.
In my experience, K8s is a million times better than legacy shit it is usually replacing. The Herokus, the Ansible soup, the Chef/Puppet soup before that etc. The legacy infra that was held together by glue and sweat that everybody was afraid to touch.
stackskipton 2 days ago [-]
As SRE, totally agree. Most companies I've been at where we implement K8S, which is around 30-50 VMs, ends up building their own, shittier Kubernetes. This blog post: https://www.macchaffee.com/blog/2024/you-have-built-a-kubern... is a favorite of mine.
theideaofcoffee 2 days ago [-]
"But it's my legacy shit and I understand it and I haven't taken the time to learn something new because that's scary so I'll just piss on anything attempting to replace it."
Human nature, really.
mrhottakes 1 days ago [-]
people love to make posts like "I used k8s to host my personal website from my homelab and it was too complicated!"
stego-tech 2 days ago [-]
I'm excited to see what they put together, because this raises a number of similar gripes I have with public cloud in its current state:
* Insistence on adding costly abstractions to overcome the limitations of non-fungible resources
* Deliberate creation of over or under-sized resource "pieces" instead of letting folks consume what they need
* Deliberate incompatibility with other vendors to enforce lock-in
I pitched a "Universal Cloud" abstraction layer years ago that never got any traction, and honestly this sounds like a much better solution anyhow. When modern virtualization is baked into OS kernels, it doesn't make a whole lot of sense to enforce arbitrary resource sizes or limits other than to inflate consumption.
Kubernetes without all the stuff that makes it a bugbear to administrate, in other words. Let me buy/rent a pool of stuff and use it how I see fit, be it containers or VMs or what-have-you.
thelastgallon 3 days ago [-]
This is being accurately called "cloud for developers". If it were for enterprise, it should cost 1000x to create thousands of positions, multiple VPs, executives, etc with a bill in 100s of millions of dollars. Execs wants high capex/opex and a massive headcount. BIGG numbers mean bigger titles and compensation.
nojvek 3 days ago [-]
If someone is building a new cloud, worth learning a few lessons from Cloudflare.
Perhaps the VM idea is old. The unit is a worker encapsulated in some deployable container.
In the world of Cloudflare workers - especially durable objects that are guaranteed to have one of them running in the world with a tightly bound database.
The way I think of apps has changed.
My take is devs want a way to say “run this code, persist this info, microsecond latency, never go down, scale within this $ budget”
It’s crazy how good a deal $5/mo cloudflare standard plan is.
Obviously many startups raise millions and they gotta spend millions.
However the new age of scale to zero, wake up in millisecond, process the request and go back to sleep is a new paradigm.
Vs old school of over provision for max capacity you will ever need.
Google has a similar, scale to zero container story but their cold startup time is in seconds. Too slow.
999900000999 3 days ago [-]
I really want an open source version of Firebase with feature parity.
I don’t care about how the backend works.
Superbase requires magical luck to self host.
A lot of cloud providers have very generous free tiers to hook you and then the moment things take off , it’s a small fortune to keep the servers on.
xixixao 3 days ago [-]
Convex's open source version is OK as long as you don't expect huge load.
k9294 3 days ago [-]
That's really cool!
One thing I'm confused with is how to create a shared resources like e.g. a redis server and connect to it from other vms? It looks now quite cumbersome to setup tailscale or connect via ssh between VMS. Also what about egress? My guess is that all traffic billed at 0.07$ per GB. It looks like this cloud is made to run statefull agents and personal isolated projects and distributed systems or horizontal scaling isn't a good fit for it?
Also I'm curious why not railway like billing per resource utilization pricing model? It’s very convenient and I would argue is made for agents era.
I did setup for my friends and family a railway project that spawns a vm with disk (statefull service) via a tg bot and runs an openclaw like agent - it costs me something like 2$ to run 9 vms like this.
I'd like write program / run program / debug program to be as easy as it is in roblox. It isn't that easy though, the set of things you need to do it well is extensive. I wouldn't be averse to a new platform, one in which all io is over highly performant queues, but the moras of existing software tied to unix is large, just look at compilers and all the child processes they launch. It was always shims and it will always be shims.
Ot actually doesn't change their recommendation. There will be less, market share among people really worried about the American government. Others would be happy to pay for a better cloud run locally mostly under local laws.
kelvinjps10 3 days ago [-]
hezner, OVH? in terms of price and just having a vps that works european clouds are better than the american ones, for me is easier to understand a vps that is just *Linux* that whatever AWS or GCP are doing.
pjc50 3 days ago [-]
The "one price" is oddly small for a cloud company. I'm sure it's nice and fast but the $20/mo seems smaller than some companies' free tiers, especially for disk.
The main reason clouds offer network block devices is abstraction.
imafish 3 days ago [-]
Don’t worry - that will certainly change in the future if they have any kind of success :)
pokpokpok 2 days ago [-]
As a happy customer, godspeed to exe.dev.
I've found the quality and simplicity to be an attractive solution for lazy devops when I need to reach for a second computer
nipponese 3 days ago [-]
It seems really cool, but the entry-level tier just seems too expensive. I can get a single pain-in-the-ass OVH VPS for $7. I just need something better than that for the same price.
rldjbpin 2 days ago [-]
all of the grievances resulting in this move is a simple outcome of the cost of convenience. but it should not need going full opposite end to get something good enough.
dedicated servers, as hinted by others here, addresses the vast majority of issues one may face for any non-enterprise needs. if you know about IOPS and care about them, odds are that running a simple open-source project [1] on top of one is all you need to do to move on with your day.
need redundancy, etc.? can complement with another one in another provider/region or put CF in front of your box. this is clearly working well enough for some of the commenters who are able to sell their own service on top of this approach.
This makes me wonder if I could get a few million in funding to rent out some Oxide racks. I'd love to touch some Oxide hardware and this seems like a good way to do it.
PunchyHamster 3 days ago [-]
The author seems to have no clue what is cloud problem, and what is k8s problem, and is blaming everything on k8s. The whole post reeks of ignorance. I have no love to k8s but he is just flat out putting out false information.
> Finally, clouds have painful APIs. This is where projects like K8S come in, papering over the pain so engineers suffer a bit less from using the cloud.
K8s's main function isn't to paint over existing cloud APIs, that is just necessity when you deploy it in cloud. On normal hardware it's just an orchestration layer, and often just a way to pass config from one app to another in structured format.
> But VMs are hard with Kubernetes because the cloud makes you do it all yourself with lumpy nested virtualization.
Man discovered system designed for containers is good with containers, not VMs. More news at 10
> Disk is hard because back when they were designing K8S Google didn’t really even do usable remote block devices, and even if you can find a common pattern among clouds today to paper over, it will be slow.
Ignorance. k8s have abstractions over a bunch of types of storage, for example using Ceph as backend will just use KVM's Ceph backend, no extra overhead. It also supports "oldschool" protocols used for VM storage like NFS or iSCSI. It might be slow in some cases for cloud if cloud doesn't provide enough control, but that's not k8s fault.
> Networking is hard because if it were easy you would private link in a few systems from a neighboring open DC and drop a zero from your cloud spend.
He mistakes cloud problems with k8s problems(again). All k8s needs is visibility between nodes. There are multiple providers to achieve that, some with zero tunelling, just routing. It's still complex, but no more than "run a routing daemon".
I expect his project to slowly reinvent cloud APIs and copying what k8s and other projects did once he starts hitting problems those solutions solved. And do it worse, because instead of researching of why and why not that person seems to want to throw everything out with learning no lessons.
Do not give him money
anitil 2 days ago [-]
In classic soft-launch style, their blog is quite good. I really liked reading how they're doing ssh forwarding similar to http reverse proxies using the 'host' header [0]
I love this and agree with it almost in entirety. One thing that would be good to retain is 'regions' e.g 3 DCs under 10KM apart linked with free/very low cost internal network. It is cheap to build in colo these days with the advent of campus XCs and ever lowering cost of DF, optics and 400G/800G switches.
import 3 days ago [-]
Article doesn’t really tell what fundamental problems will be solved, except fancy VM allocation. Nothing about hardware, networking, reliability, tooling and such. Well, nice, good luck.
qaq 3 days ago [-]
With LLMs there is no real dev velocity penalty of using high perf. langs like say Rust. A pair of 192 Core AMD EPYC boxes will have enough headroom for 99.9% of projects.
kennywinker 3 days ago [-]
That’ll be true for the 0.1% of project that were limited by the speed of their programming language. For the other 99.9% of projects their vibe coded rust can fly and their database, network, or raw computation will still be the bottleneck.
(Percentages cited above are tongue-in-cheek, actual numbers are probably different)
qaq 2 days ago [-]
sqlite when used properly can be pushed pretty far.
Few biz apps are raw compute limited
throwawayapr23 2 days ago [-]
I don’t get it.
Starting a digital ocean droplet is a single curl call. Starting a hetzner server is as well. Their api’s are completely fine and known to llm’s.
Why would agents learn exe’s way of setting up / deploying / binding to ports / auth, rather than just ssh’ing into a vm..?
oxag3n 2 days ago [-]
Comparing laptop SSD to cloud network drive is misleading.
EC2 provides the *d VMs that have SSDs with high IOPS at much lower cost than network SSDs. They are ephemeral, but so is laptop and its SSD - it can loose the data. From AWS docs "If you stop, hibernate, or terminate an instance, data on instance store volumes is lost.".
senko 2 days ago [-]
If I stop, hibernate or power off the laptop, the data is most definitely not lost.
Abby_101 2 days ago [-]
Half the work of pricing a solo SaaS is modeling what a bad month looks like if egress or IOPS spike. You end up pricing defensively to protect against your own infra bill instead of pricing to the value you provide. A fixed bucket of compute with clear limits is way easier to build a business on than a meter that could run anywhere.
arjie 3 days ago [-]
I can't see why I would want this, but I do love Tailscale so I'm excited to see what new stuff he comes up with here.
tee-es-gee 3 days ago [-]
I will follow this one for sure. There are a few more companies with the extremely ambitious goal of "a better AWS", and I am interested in the various strategies they take to approach that goal incrementally.
A service offering VMs for $20 is a long way from AWS, but I see how it makes sense as a first step. AWS also started with EC2, but in a completely different environment with no competition.
pclark 2 days ago [-]
I think I am interested in this? I run a bunch of small web apps, currently as fly.io machines. I love fly, but it adds up when I have a bunch of small things that I want isolated — I wish I could have even smaller Fly instances. Exe.dev seems like a good middleground where I can allocate the compute from tiny to large. (?)
indigodaddy 2 days ago [-]
I use both fly and exe. Exe isn't really "docker image as the app"-focused like fly, but if you want to sort of mimic the fly deploy process you kinda sorta could make it work for you I would think. This might help:
I think clouds pay a huge abstraction penalty to allow tiny VMs. I guess it helps with onboarding and $10 personal VPNs. But I have never needed a fraction of a computer. I want to rent some number of full computers of various sizes, consisting of CPU, memory, and flash disk. Hetzner is closer than AWS, and I think/hope that’s what Crawshaw is aiming for.
ButlerianJihad 3 days ago [-]
The key to renting a fraction of a computer is scaling up. If I can rent 1/8th of a computer, I can also rent 3/8ths and 1/2 and then go to a full computer, if that capacity is necessary.
The key to scaling up is to have big-enough hardware on the backend. If Hetzner is renting out bare metal instances then they can only rent out the sizes that they have. If a cloud provider invests in really big single systems, they can offer fractions of those systems to multiple tenants, some of whom scale up to use the entire system, and some who don't. I think that is a win-win.
A fractional VM is also a fungible VM. If the tenant calls to spin up a certain size VM, then the backend can find suitable hardware for it from a menu of sizes. Smaller VMs can slot in anywhere there is room, not just on a designated bare-metal system.
A cloud provider is always going to want to maximize their rack space, wattage/heat, and resource usage. So they will invest in high-density systems at every chance. On the other hand, cloud tenants will have diverse needs, including some fraction of those big computers.
phrotoma 3 days ago [-]
Allow? I understood tiny VM's to be something (at least AWS) added to try to squeeze more utilization out of idle hardware.
tlb 3 days ago [-]
I understand the appeal from AWS's perspective. Customer A pays for a 32 vCPU VM, which they run on 32-core hardware. Then they can also squeeze in customer B's 1 vCPU instance running a blog, and no one notices. Free money!
But I don't want to be either of those customers. It means the whole system has an extra layer of abstraction, so they can juggle VMs around. It's why you need slow EBS instead of just getting a flash drive in the same case as the CPU, with 0.01x the latency.
jeffrallen 3 days ago [-]
So much good stuff is happening at https://exe.dev, keep it up guys!
Growtika 3 days ago [-]
Congrats. Just checked your homepage. I love the fact you also show this comment
"That must be worst website ever made"
Made me love the site and style even more
vaylian 1 days ago [-]
But it was made!
_nhh 3 days ago [-]
just take a look at hetzner cloud. Its everything 99% of the people need, good pricing. Convert that ux to terminal and you done
esher 3 days ago [-]
Much respect for the ambitous plan, I wish I could do such bold thinking. I am running a small PHP PaaS (fortrabbit) for more than 10 years. For me, it's not only "scratch your own itch", but also "know your audience". So, a limited feature set with a high level of abstraction can also be useful for some users > clear path.
vasco 3 days ago [-]
I know its a personal blog but the writing style is really full of himself. What a martyr, starting a second company.
Animats 3 days ago [-]
It's hard to see the scale of what he's doing. Could be:
- I'm building a server farm in my homelab.
- I'm doing a small startup to see if this idea works.
- We're taking on AWS by being more cost effective. Funding secured.
I don’t understand the point you’re trying to make.
Cloud is bad?
3 days ago [-]
synack 3 days ago [-]
Nevermind, I misread their HTTPS proxy documentation. Cloud is fine.
sealthedeal 2 days ago [-]
I just want to say this has been the greatest experience I have ever had for signing up for a new service etc. I really really loved this entire experience. It has truly inspired me!
> The standard price for a GB of egress from a cloud provider is 10x what you pay racking a server in a normal data center.
From the exe.dev pricing page:
> additional data transfer $0.07/GB/month
So at least on the network price promise they don't seem to deliver, still costs an arm and a leg like your neighbourhood hyperscaler.
Overall service looks interesting, I like simplicity with convenience, something which packet.net deliberately decided not to offer at the time.
ignoramous 2 days ago [-]
If it was anyone else, I'd have totally lost trust after seeing such a cynically diabolical take. I do eventually expect the co-founders at exe.dev, David & Josh, to workout someway to meet the promises laid out in TFA.
schpet 3 days ago [-]
as an exe customer i'm really happy to see this. i don't even use half of their features (such as the https proxy, or the LLM agent) but it's just a reliable computer that i can ssh into from my laptop or phone. i use hetzner too in the same way for a bit of redundancy but exe seems less likely to delete all my machines and data.
every time i've had an issue or question, it's been the same sympathetic people helping me out. over email, in plain text.
moralestapia 3 days ago [-]
Tangential.
Is there a name for this style of writing? I come across it regularly.
I'd describe it as forcefully modest, "I'm just a simple guy" kind of thing. With a dash of "still a child on the inside". I always picture it as if the guy from the King of Queens meme wrote it.
"I guess I'm just really into books, heh" - Bezos (obviously non-real, hypothetical quote, meant to illustrate the concept)
This style is also very prevalent in Twitter bios.
Since it's a "literary" style that is quite common, I'm sure it has been characterized and named.
GPT says it's "aw-shucks", but I think that's a different thing.
piokoch 3 days ago [-]
How this is different from getting dedicated server from any other provider? Typically you need to pay a bit more - $40-$50 but you get more RAM and cores.
And what it has to do with the "cloud"? Cloud means one use cloud-provided services - security, queue, managed database, etc. and that's their selling point. This exe.dev is a bare server where I can install what I want, this is fine, but this is not a cloud and, frankly speaking, nothing new.
Razengan 3 days ago [-]
Isn't it high time to figure out a distributed physical layer / swarm internet or whatever the buzzword is? Would be perfect for distributed AI too..
dzonga 3 days ago [-]
finally a cloud 'vendor' that understands that modern computers are fast.
if we go back to the principle that modern computers are really fast, SSDs are crazy fast
and we remove the extra cruft of abstractions - software will be easier to develop - and we wouldn't have people shilling 'agents' as a way for faster development.
ultimately the bottleneck is our own thinking.
simple primitives, simpler thinking.
kjok 3 days ago [-]
How difficult is it to build a second startup on the side?
z3t4 3 days ago [-]
You can run several VM's or containers with isolation on your phone hardware, why even use the cloud when you just want to show your friends?
skybrian 3 days ago [-]
For me it’s so my coding agent keeps running when I close my laptop lid and it goes to sleep. VM in the cloud because I’m too lazy to set up a computer to be running as a server all the time.
speedgoose 3 days ago [-]
I welcome the initiative but it’s pretty costly compared to the bare metal cloud providers. So the value as to be the platform as service too.
irusensei 2 days ago [-]
I use kubernetes extensively at work. I don't manage the kubernetes cluster anymore since now we have a team that runs centralized services and you can request a namespace with a quota. But back when my team had a dedicated Azure Kubernetes cluster it was not that bad as people says it is and the biggest hassle was the extremely short lived support for each version.
Then I started to realize most people who complain are rolling their own which is also not bad since there are products like k3s that are very simple to use.
It seems things start to fall apart when they try to stuff it with all kinds of crazy idiotic controllers and the favorite of the month CNI and CSI. I always shake my head when I see people creating sand castles by setting up stuff like Ceph from within the cluster.
If you want to play with it keep things simple and have all the persistent data outside of the cluster. Use good old NFS instead of the latest longceph horngluster version. Keep databases and the container registry out. Treat it like a compute pool not a virtual datacenter. Stop recursing chickens inside eggs.
achille 3 days ago [-]
What will happen to my "Grandfathered Plan" I signed up to test it, don't recall if I gave you my credit card
ndr 3 days ago [-]
exe.dev landing page is sublime. The call to action is "ssh exe.dev" and you can bet it works.
joshgel 3 days ago [-]
I love the line on the landing page with a link back to hn:
> That must be worst website ever made.
the level of confidence (this is a second time founder after all) to put that on their website gives me confidence that they can make this work
aayushdutt 3 days ago [-]
Wondering what runtime is the infra under the hood. Firecracker? Traditional VM? Docker Containers?
crawshaw 3 days ago [-]
Author here. Most of our infra is custom, the VMM is based on cloud-hypervisor (a project spiritually similar to Firecracker). We have a lot of work to do, including on the VMM, but right now there is more value for users if we spend our time on the VM management layer and GLB.
hbhhh 3 days ago [-]
HeavyBit is absolutely gross. I've heard lots of horrible things about them from multiple founders.
One of my friends was told to come to a sex party that was all male and he is straight. It soured his relationship with the firm so much he ended up winding down the business.
furyofantares 3 days ago [-]
Does that any anything to do with exe.dev?
pratyahava 2 days ago [-]
the first paragraph in the linked post says "Major investors are Amplify, CRV, and HeavyBit."
but i know nothing about what the comment says, just answering your question.
iqihs 2 days ago [-]
o.O
ianberdin 2 days ago [-]
“Everything is shit. Believe me. We will do something better, just believe me.”
Jokes aside:
- k8s is insane peace of software. A right tool for a big problem. Not for your toys. Yes, it is crazy difficult to setup and manage. Then what?
- cloud has bad and slow disk. BS. They have perfectly fast NVME.
Something else? That’s it.
Why I am so confident? I used to setup and manage kubernetes for 2 years. I have some experience. Do I use it more? Nope. Not a right tool for me. Ansible with some custom Linux tools fits better for Me.
I also build my own cloud. But if I say it less loud: hosting to host websites for https://playcode.io.
Yea, it is hard and with a lot of compromises. Like networking, yes I want to communicate between vms in any region. Or disks and reliability. What about snapshots? And many bare metal renters gives only 1Gbt/s. Which is not fine. Or they ask way more for 10Gbt uplink.
So it is easy to build some limited and unreliable shit or non scalable.
ryanisnan 2 days ago [-]
I wish you great luck! I want this to succeed.
jrflowers 3 days ago [-]
> The standard price for a GB of egress from a cloud provider is 10x what you pay racking a server in a normal data center.
> $160/month
50 VM
25 GB disk+
100 GB data transfer+
100GB/mo is <1mbps sustained
lmao
mrcwinn 2 days ago [-]
I appreciate the confidence that comes with a clear vision - but please make docs useful from day 1. But remember, while you know what's in your mind, the user does not.
These are nice declarative statements but have almost no meaningful substance.
> Setup scripts have a maximum size. Use indirection. [What's the maximum size?]
> Shelley is a coding agent. It is web-based, works on mobile. [Cool model bro. Any details you want to share?]
Let me see if I understand it. The TL;DR is that instead of asking for VMs and fit things there you reserve the CPU and RAM and do with that whatever you want? Number of mVMs, etc.?
rambambram 3 days ago [-]
Now that we're talking about clouds... what happened to the word 'webhosting'?
duriantaco 3 days ago [-]
should log the journey down and os it!
syngrog66 2 days ago [-]
I want to like a Crawshaw startup but seeing anything related to Clawdbot or LLMs is a hard showstopper. Poor judgment or grift.
ludjer 3 days ago [-]
I mean the whole ebs complaint is invalid you are complaining about a san disk vs local disk. If you want high speed local storage use a d instance with nvme storage.
_joel 3 days ago [-]
As someone who has built and managed clouds, good luck to them, you'll need it :)
throwawaypath 2 days ago [-]
What a refreshing read. Not a single woke/DEI religious shibboleth, nor CURRENTYEAR/CURRENTTHING pandering in sight.
3 days ago [-]
0xbadcafebee 3 days ago [-]
Hi David, thanks for trying to fix the cloud. There is a persistent problem with all cloud providers that none of them has fixed yet (and I don't expect any ever will). I imagine users will not care about this issue, so this might not be worth solving. But if you'd like to have the only cloud provider (or technology in general) that can solve this problem, it would make cloud computers less annoying.
If you want to run a website in the cloud, you start with an API, right? A CRUD API with commands like "make me a VPC with subnet 1.2.3.4/24", "make me a VM with 2GB RAM and 1 vCPU", "allow tcp port 80 and 443 to my VM", etc. Over time you create and change more things; things work, everybody's happy. At some point, one of the things changes, and now the website is broken. You could use Terraform or Ansible to try to fix this, by first creating all the configs to hopefully be in the right state, then re-running the IaC to re-apply the right set of parameters. But your website is already down and you don't really want to maintain a complex config and tool.
You can't avoid this problem because the cloud's design is bad. The CRUD method works at first to get things going. But eventually VMs stop, things get deleted, parameters of resources get changed. K8s was (partly) made to address this, with a declarative config and server which constantly "fixes" the resources back to the declared state. But K8s is hell because it uses a million abstractions to do a simple thing: ensure my stuff stays working. I should be able to point and click to set it up, and the cloud should remember it. Then if I try to change something like the security group, it should error saying "my dude, if you remove port 443 from the security group, your website will go down". Of course the cloud can't really know what will break what, unless the user defines their application's architecture. So the cloud should let the user define that architecture, have a server component that keeps ensuring everything's there and works, and stops people from footgunning themselves.
Everything that affects the user is a distributed system with mutable state. When that state changes, it can break something. So the system should continuously manage itself to fix issues that could break it. Part of that requires tracking dependencies, with guardrails to determine if a change might break something. Another part requires versioning the changes, so the user (or system) can easily roll back the whole system state to before it broke. This abstraction is complicated, but it's a solution to a complex problem: keeping the system working.
No cloud deals with this because it's too hard. But your cloud is extremely simple, so it might work. Ideally, every resource in your cloud (exe.dev) should work this way. From your team membership settings, to whether a proxy is public, the state of your VM, your DNS settings, the ssh keys allowed, email settings, http proxy integration / repo integration settings / their attachments, VM tags & disk sizes, etc. Over time your system will add more pieces and get more complex, to the point that implementing these system protections will be too complex and you won't even consider it. But your system is small right now, so you might be able to get it working. The end result should be less pain for the user because the system protects them from pain (fixing broken things, preventing breaking things), and more money for you because people like systems that don't break. But it's also possible nobody cares about this stuff until the system gets really big, so maybe your users won't care. It would be nice to have a cloud that fixes this tho.
achiles 2 days ago [-]
[dead]
adrianojunior 2 days ago [-]
[dead]
ZihangZ 3 days ago [-]
[dead]
hani1808 3 days ago [-]
[dead]
handsometong 3 days ago [-]
[dead]
asiffareed 3 days ago [-]
[dead]
3 days ago [-]
WhereIsTheTruth 3 days ago [-]
> 100 GB data transfer+
> $20 a month
2025 or 2005, what's the difference?
sudo_cowsay 3 days ago [-]
inflation
kitsune1 2 days ago [-]
[dead]
dfsff 3 days ago [-]
[dead]
AashiyaShaikh 3 days ago [-]
Ilike to working with reap name
satnhak 3 days ago [-]
AWS. Months of complex dev work to build using their CDK. Terrible disk speed. Frustrating permissions systems. Tiny deployments that take 30 minutes. Rollbacks that get stuck for hours. What you end up with is about 4 CPUs and 16Gb of RAM for $1000+ per month. No wonder Bezos could send his wife and Katie Perry on a jolly into space. The world's richest man 1 IOP at a time.
For that money I can get 5 big bare metal boxes on OVH with fast SSDs, put k0s on them, fast deploy with kluctl, cloudflare tunnels for egress. Backups to a cheap S3 bucket somewhere. I'll never look at another cloud provider.
BirAdam 3 days ago [-]
If you're using cloudflare tunnels, you don't even need to be on OVH. You could seriously host anywhere, like your own basement.
skullone 2 days ago [-]
I have a basement, and a garage, and 2 sheds... I still dont wanna run servers at home. I use OVH, I just rent a server at their Hillsboro OR, US datacenter, just a few miles from my home, i know the twists and turns the fiber goes between there even.
So well put, my good sir, this describes exactly my feelings with k8s. It always starts off all good with just managing a couple of containers to run your web app. Then before you know it, the devops folks have decided that they need to put a gazillion other services and an entire software-defined networking layer on top of it.
After spending a lot of time "optimizing" or "hardening" the cluster, cloud spend has doubled or tripled. Incidents have also doubled or tripled, as has downtime. Debugging effort has doubled or tripled as well.
I ended up saying goodbye to those devops folks, nuking the cluster, booted up a single VM with debian, enabled the firewall and used Kamal to deploy the app with docker. Despite having only a single VM rather than a cluster, things have never been more stable and reliable from an infrastructure point of view. Costs have plummeted as well, it's so much cheaper to run. It's also so much easier and more fun to debug.
And yes, a single VM really is fine, you can get REALLY big VMs which is fine for most business applications like we run. Most business applications only have hundreds to thousands of users. The cloud provider (Google in our case) manages hardware failures. In case we need to upgrade with downtime, we spin up a second VM next to it, provision it, and update the IP address in Cloudflare. Not even any need for a load balancer.
People use Kubernetes for way too small things, and it sounds like you don't have the scale for actually running Kubernetes.
My app is fairly simple node process with some side car worker processes. k8s enables me to deploy it 30 times for 30 PRs, trivially, in a standard way, with standard cleanup.
Can I do that without k8s? Yes. To the same standard with the same amount of effort? Probably not. Here, I'd argue the k8s APIs and interfaces are better than trying to do this on AWS ( or your preferred cloud provider ).
Where things get complicated is k8s itself is borderline cloud provider software. So teams who were previously good using a managed service are now owning more of the stack, and these random devops heros aren't necessarily making good decisions everywhere.
So you really have three obvious use cases:
a) You're doing something interesting with the k8s APIs, that aren't easy to do on a cloud provider. Essentially, you're a power user. b) You want a cloud abstraction layer because you're multi-cloud or you want a lock-in bargaining chip. c) You want cloud semantics without being on a cloud provider.
However, if you're a single developer with a single machine, or a very small team and you're happy working through contended static environments, you can pretty much just put a process on a box and call it done. k8s is overkill here, though not as much as people claim until the devops heros start their work.
k8s appears to be a corporate welfare jobs program where trillion dollar multinational monopolistic companies are the only ones who can collectively spend 100s of millions sustaining. Since most companies aren't trillion dollar monopolies, adopting such measures seems extremely poor.
All it signals to me is that we have to stop letting SV + VC dictate the direction of tech in our industry, because their solutions are unsustainable and borderline useless for the vast majority of use cases.
I'll never forget the insurance companies I worked at that orchestrated every single repo with a k8s deployment whose cloud spend was easily in the high six figures a month to handle a work load of 100k/MAU where the concurrent peak never went more than 5,000 users, something the company did know with 40 years of records. Literally had a 20 person team whose entire existence was managing the companies k8s setup. Only reason the company could sustain this was that it's an insurance company (insurance companies are highly profitable, don't let them convince you otherwise; so profitable that the government has to regulate how much profit they're legally allowed to make).
Absolute insanity, unsustainable, and a tremendous waste of limited human resources.
Glad you like it for your node app tho, happy for you.
Is it complex? Yes, but so is the problem it's trying to solve. Is its complexity still nicer and easier to use than the previous generation of multimachine deployment systems? Also yes.
It really confuses me how someone can argue for cloud providers over a decent open solution without realising their argument is simply they don't want to be managing the thing.
And that's fine, most teams shouldn't be neck deep in managing a platform. But that doesn't make the solution bad.
You're going to want most of what K8s has anyway: blue-green deployments, some way to specify how many replicas you want, health checks, etc.
The initial setup cost is annoying if you've never done it before, but in terms of maintenance it's very very easy.
That being said I love exe.dev and have been a happy customer since launch. It's a different use case but they do an amazing job at it. Very, very easy personal cloud dev box. But K8s is very very good too, just for production workloads rather than personal ones!
I run it at home and at work, and while I do hate installing it, once that part is done I've never run into these problems that people claim requires a 20 person(!) team to baby sit it. Maybe my scale is too small or whatever, but its hard not to think that maybe they are just "holding it wrong"...
Coolify is full of features, but the UX suffers and they had a nasty breaking bug at one point (related to Traefik if you want to search it.) Dockge is just a simple interface into your running Docker containers and Komodo is a bit harder to understand/come up with a viable deployment model, and has no built-in support for things like databases.
Dokploy is more Heroku-styled: while you can deploy third-party apps (it's just Docker after all), it seems really geared towards and intended for you to be deploying your own apps that you developed, alongside a "managed" database (meaning, the DB is exposed in the UI, includes backup functionality, and can even be temporarily exposed publicly on the internet for debugging.)
Coolify feels a bit like a mix of the two deployment models, while Dockge is "bring your own deployment" and Komodo offers to replace Terraform/Ansible/docker-compose through its own declarative GitOps-style file-based config but lacks features like managed databases, or built-in subdomain provisioning.
Whether it's a worthy mention or not, I'm not sure. I'd like to think its worthy :)
Disclaimer: I am the maintainer.
Point being, it's not the tools the causes the probem.
Or that guy is just a really bad programmer.
But the point was it was in a comparble situations without the microservices / k8s / whatever pet tech you want to hate on.
I think Amazon ECS is within striking distance, at least. It does less than K8S, but if it fits your needs, I find it an easier deployment target than K8S. There's just a lot less going on.
The deployment files / structure were mostly equivalent with the main differences being I can't shell into ECS and I lose kubectl in favour of looking at the AWS GUI ( which for me is a loss, for others maybe not ).
The main difference is k8s has a lot of optionality, and folks get analysis paralysis with all the potential there. You quickly hit this in k8s when you have to actually need the addon to get cloudwatch logs.
This is also where k8s has sharp edges. Since amazon takes care of the rest of the infrastructure for you in ECS, you don't really need to worry about contention and starving node resources resulting in killing your logging daemon, which you could technically do in k8s.
However, you'll note that this is a vendor choice. EKS Auto Mode does away with most of the addons you need to run yourself, simplifying k8s, moving it significantly closer to a vendor supported solution.
Is there a specific reason why you can't shell into ECS? IIRC, I was able to do so by following the guide [0].
[0] https://aws.amazon.com/blogs/containers/new-using-amazon-ecs...
The last 20 years has given us a lot of great primitives for folks to plug in, I think that lots of people don't want to wrangle those primitives, they just want to use them.
This is well put and it's very similar to the arguments made when comparing programming languages. At the end of the day you can accomplish the same tasks no matter which interface you choose.
Personally I've never found kubernetes that difficult to use[1]. It has some weird, unpredictable bits, but so does sysvinit or docker, that just ends up being whatever you're used to.
[1] except for having to install your own network mesh plugin. That part sucked.
So having everyone use the same deployment model (and that’s typically k8s) saves effort. I don’t like it for sure
I like to think if we had a K8s environment a lot of this would be built out within it. Having that functionality abstracted away from the developer would be a huge win in my opinion.
This is certainly the case from all the third person accounts I hear. Online. I never actually met a single one that is like that, if anything, those same people are the ones that are first to tell me about their Hetzner setups.
The trouble is that we are literally expected to do this everywhere we go. I've personally advocated for approaches which use say, a pair of dedicated servers, or VMs as in GPs example. If you want it outside of AWS/GCP/Azure, you're regarded as a crazy person. If you don't adopt "best practices" (as defined by vendors) then management are scared. Management very often trust the sales and marketing departments of big vendors more than their own staff. Many of us have given up fighting this, because what it comes down to is a massive asymmetry of information and trust.
The challenge is convincing people that "golden images" and containers share a history, and that kubernetes didn't invent containers: they just solved load balancing and storage abstraction for stateless message architectures in a nice way.
If you're doing something highly stateful, or that requires a heavy deployment (game servers are typically 10's of GB and have rich dynamic configuration in my experience) then kubernetes starts to become round-peg-square-hole. But people buy into it because the surrounding tooling is just so nice; and like GP says: those cloud sales guys are really good at their jobs, and kubernetes is so difficult to run reliably yourself that it gets you hooked on cloud.
There's a literal army of highly charismatic, charming people who are economically incentivised to push this technology and it can be made to work so- the odds, as they say, are against you.
I think this is the crux of the matter. Also, "everybody is doing it, so they must be right" is also a very common way of thinking amongst this population.
Around the time of the pandemic, a company wanted to make some Javascript code do a kind of transformation over large number of web-pages (a billion or so, fetched as WARC files from the web archive). Their engineers suggested setting up SmartOS VMs and deploying Manta (which would have allowed the use of the Javascript code in a totally unmodified way -- map-reduce from the command-line, that scales with the number storage/processing nodes) which should have taken a few weeks at most.
After a bit of googling and meeting, the higher ups decided to use AWS Lambdas and Google Cloud Functions, because that's what everyone else was doing, and they figured that this was a sensible business move because the job-market must be full of people who know how to modify/maintain Lambda/GCF code.
Needless to say, Lambda/GCF were not built for this kind of workload, and they could not scale. In fact, the workload was so out-of-distribution, that the GCP folks moved the instances (if you can call them that) to a completely different data-center, because the workload was causing performance problems, for _other_ customers in the original data-center.
Once it became clear that this approach cannot scale to a billion or so web-pages, it was decided to -- no, not to deploy Manta or an equivalent -- but to build a custom "pipeline" from scratch, that would do this. This system was in development for 6 months or so, and never really worked correctly/reliably.
This is the kind of thing that happens when non-engineers can override or veto engineering decisions -- and the only reason they can do that, is because the non-engineers sign the paychecks (it does not matter how big the paycheck is, because market will find a way to extract all of it).
One of the fallacies of the tech-industry (I do not mean to paint with too broad a brush, there are obviously companies out there that know what they are doing) is that there are trade-offs to be made between business-decisions and engineering-decisions. I think this is more a kind of psychological distortion or a false-choice (forcing an engineering decision on the basis of what the job market will be like some day in the future -- during a pandemic no less -- is practically delusional). Also, if such trade-offs are true trade-offs, then maybe the company is not really an engineering company (which is fine, but that is kind of like a shoe-store having a few podiatrists on staff -- it is wasteful, but they can now walk around in white lab-coats, and pretend to be a healthcare institution instead of a shoe-store).
Personally, I believe that the tech industry sustains itself via technical debt, much like the real economy sustains itself on real debt. In some sense, everyone is trying to gaslight everyone else into incurring as much technical debt as possible, so that a way to service the debt can be sold. Most of the technical debt is not necessary, and if people were empowered to just not incur it, I suspect it would orient tech companies towards making things that actually push the state of the art forward.
A lot of criticism of k8s is always centered about some imagined perfect PaaS, or related to being in very narrow goldilocks zone where the costs of "serverless" are easier to bear...
This feels like a reminder that everything "Cloud" is still basically the same as IBM's ancient business model. We've always just been renting time on someone else's computers, and those someone else people are always trying to rent more time. The landlords shift, but the game stays the same.
…IBM
…Microsoft
…AWS.”
They're getting kickbacks from cloud vendors. Prove me wrong.
better than nothing, I don't blame em.
Money ain't got no owners, only spenders.
But yeah, let's just spin-up a shadow IT VM with Debian like GP said, it's easy!
That’s literally how they sold AWS in the beginning.
Cloud won not because of costs or flexibility but because it allowed teams to provision their own machines from their budget instead of going through all the red tape with their IT departments creating… a bunch of shadow IT VMs!
Everything old is new again, except it works on an accelerated ten year cycle in the IT industry.
And yes, a dev that's able to do that properly (stress on properly) is indeed a signal of a better overall developer but they are a minority and anyway as orgs scale up there is just too much of "side salad" that it becomes a separated dish.
If you'd know Kubernetes, you know not to use it. I say that as someone who used to do consulting for it.
The reality is that yet again "making money" completely collides with efficient, quality, sane productive work.
For me one of the main reasons to leave that space is that I couldn't really deal with the fact that my work collides with a client's success. That said I have helped to get off that stuff and other things that they thought they needed, that just wasted time and money. It just feels odd going into a company that hired you to consult on a topic only to end up telling them "The best approach for you is not doing that at all". Often never. Like some people thought "Well, if we have hundreds of thousands or even millions of users" and the reality was that even in these scenarios if you went away from that abstract thought and discussed a hypothetical based on their product they realized that they'd still be better off without it. Besides the fact that this hypothetical often was in a future that made it likely that they said they'd likely have completely different setup so preparing for that didn't even make sense.
I think a big thing related to that was/is the microservice craze where people end up moving to a complex architecture for not many good reasons and then they increase complexity way faster than what they actually deliver in terms of the product, because it somehow feels good. I know it does, I've been there. When in reality the outcome often is just a complex mess with what could have been a relatively simple monolith. And these monoliths do work. And in the vast majority of cases they are easy to scale, because your problem switches from "how do we best allocate that huge amount of very different services across our infrastructure" to (for the most part) "how do we spin up our monolith on one more server" which tends to be a way easier to tackle service.
And nothing stops you from still using everything else if you want. Just because it's a monolith doesn't mean you need to skip on any of the cloud offerings, etc. For some reason there seems to be that idea that if you write a monolith you are somehow barred from using modern tooling, infrastructure, services, etc. Not sure where that comes from.
imo this should achieve a nice balance?
SOAs have most utility in scaling teams, not software: creating independent services allows autonomy to independent teams if they apply a few simple patterns for good SOA.
Though imagining the unholy existence of an init system who's only job is to spin up containers, that can contain other inits, OS images, or whatever ..... turtles all the way down.
Tech like Flatpak or Snap is closer to Docker than "machines" are — except that Docker has local, virtualized networking built-in as the IPC layer.
I'm not surprised even in the slightest that DevOps workers will slap k8s on everything, to show "real industry experience" in a job market where the resume matches the tools.
Using new technology in something small and unimportant like a setup script is a perfect way to experiment and learn. It would be irresponsible to build something important as the first thing you do in a new language.
But if you're working with others, you should default to using standard industry tools (absent a compelling reason not to) because your work will be handed off to others and passed on to new team members. It's unreasonable to expect that a new Windows or Linux sysadmin or desktop support tech must learn Rust to maintain a workstation setup workflow.
(Apologies to Cake. And coders.)
I mean, I worked with people who were suprised that you can run more applications inside ec2 vm than just 1 app.
To be fair though, that's true for every profession or skill.
> I mean, I worked with people who were suprised that you can run more applications inside ec2 vm than just 1 app.
I've seen something similar where people were surprised that you can use an object storage (so effectively "make HTTP requests") from every server.
Every company used to have a bespoke collection of build, deployment, monitoring, scaling, etc concerns. Everyone had their own practices, their own wikis to try to make sense of what they had.
I think we critically under-appreciate that k8s is a social technology that is broadly applicable. Not just for hosting containers, but as a cloud-native form of thinking, where it becomes much easier to ask: what do we have here, and is it running well, and to have systems that are helping you keep that all on track (autonomic behavior/control loops).
I see such rebellion & disdain for where we are now, but so few people who seem able to recognize and grapple with what absolute muck we so recently have crawled out of.
This is a problem I've run into enterprise deployments. K8s is often the lowest common denominator semi small platform engineering teams arrive on. At my current employer, a platform managed K8s namespace is the only thing we got in terms of PaaS offering, so it is what we use. Is it overpowered? Yes. Is it overly complex for our usecase? Definitely. Could we basically get by hosting our services on a few cheap mini computers with no performance penalty? Also yes.
Value is not that I got job done at a day's notice. It is black mark that I couldn't package it as per industry best practices.
Not doing would mean out of job/work. Whether it is happening correctly is not something decision makers care as long it is getting done anyhow.
That's basically why k8s is so compelling. It's tech is fine but it's a social technology that is known and can be rallied behind, that has consistent patterns that apply to anything you might dream of making "cloud native". What you did to get this script available for use will closely mirror how anyone else would also get any piece of software available.
Meanwhile conventional sys-op stuff was cobbling together "right sized" solutions that work well for the company, maybe. These threads are overrun with "you might not need k8s" and "use the solution that fits your needs", but man, I pity the companies doing their own frontiers-ing to explore their own bespoke "simple" paths.
I do think you are on to something with there not being food taste making, with not good oversight always.
Everyone else is communicating they are doing Agile while being very far away from it ;)
No Kubernetes whatsoever.
I agree with you.
My personal strategy has always been to start off in docker compose, and break out to a k8s configuration later if I have to start scaling beyond single box.
You don't set up k8s because your current load can't be handled, you do for future growth. Sometimes that growth doesn't pan out and now you're left with a complex infrastructure that is expensive to maintain and not getting any of the benefit.
This is why you get many folks over-thinking the solution and picking the most hyped technologies and using them to solve the wrong problems without thinking about what they are selling.
You don't need K8s + AWS EC2 + S3 just to host a web app. That tells me they like lighting money on fire and bankrupting the company and moving to the next one.
But given how I always see "you don't need k8s because you're not going to scale so fast" I am feel like even professional k8s operators have missed the fundamental design goals of it :/ (maximizing utilization of finite compute)
All of those other tools are complicated and fragile
1. People expect k8s to be an opinionated platform and it's very happy to let you make a mess
2. People think k8s is supposed to be a cross platform portability layer and ... it maybe can be if you're very careful, but it's mostly not that
3. People compare k8s/cloud/etc to some monolithic application with admin permissions to everything and they compare that to the "difficulty" of dealing with RBAC/IAM/networking/secrets management
4. People don't realize how much more complicated vanilla Linux tooling and how much more accidental complexity is involved
and then also package this so that you and other developers can get the infrastructure running locally or on other machines.
Nomad has neither of these problems.
But if its use was confined to this use case, pretty much nobody would be using it (unless as a customer of the organization's infra) and barely would be talking about it (like how there isn't too much talk about Borg).
The reason k8s is a thing in the first place is because it's being used by way too many people for their own goods. (Most people having worked in startups have met too many architecture astronauts in our lives).
If I had to bet, I'd wager that 99% of k8s users are in the “spin a few containers to run your web app” category (for the simple reason that for one billion-dollar tech business using it for legit reasons, there's many thousands early startups who do not).
But to quote someone: "you are not Google".
Teams are free to use EKS internally.
Maybe those devops folks only pay attention to k8s clusters and you're flying under their radar with your single debian VM + Kamal. But the same thinking that results in an overtly complex, impossible to debug, expensive to run k8s cluster can absolutely result in the same using regular VMs unless, again, you are just left to your own devices because their policies don't apply to VMs, yet.
The problem usually is you're one mistake away from someone shoving their nose in it. "What are you doing again? What about HA and redundancy? slow rollout and rollback? You must have at least 3 VMs (ideally 5) and can't expose all VMs to the internet of course. You must define a virtual network with policies that we can control and no wireguard isn't approved. You must split the internet facing load balancer from the backend resources and assign different identities with proper scoping to them. Install these 4 different security scanners, these 2 log processors, this watchdog and this network monitor. Are you doing mtls between the VMs on the private network? what if there is an attacker that gains access to your network? What if your proxy is compromised? do you have visibility into all traffic on the network? everything must flow throw this appliance"
And I’m building and happily using Uncloud (https://github.com/psviderski/uncloud) for this (inspired by Kamal). It makes multi-machine setups as simple as a single VM. Creates a zero-config WireGuard overlay network and uses the standard Docker Compose spec to deploy to multiple VMs. There is no orchestrator or control plane complexity. Start with one VM, then add another when needed, can even mix cloud VMs and on-prem.
I use ansible to set up my hosts, and from there each app just brings it's own caddy & compose file. A bit of scripting and SOPS wraps the lint / build / deploy steps. Compared to my past life which was all k8s infra projects in helm... it is SO NICE to use these lightweight files instead.
If you have an app and you want to run a single app yeah silly to look for K8s.
If you have a beefy server or two you want to utilize fully and put as many apps on it without clashing dependencies you want to use K8s or docker or other containers. Where K8s enables you to go further.
I bet you can do it in some other way but that's built in feature of k8s.
There are no benefits to scaling down in this case. And scaling up won't help handle more load if you've already allocated all resources to running replicas. You need more machines, not more replicas on the existing machine(s).
It all comes down to simple, boring capacity planning and static resource allocation. Fewer moving parts results in fewer failure modes, hence more robust infra and less ops and maintenance work.
You have apps A, B and C (you have N teams and N products) each developed by different teams - that you want to run on that one server, when app A doesn't have much traffic apps B and C can use more of compute. Then having deployment management aligned for all teams/products.
Radboud University recently announced they're rolling it out for managing containers across the faculty which is the most "serious install" I know about, but there could be other: https://cncz.science.ru.nl/en/news/2026-04-15_uncloud/
https://uncloud.run/docs/getting-started/install-cli/#instal...
Spend some time learning it, using it to deploy simple apps, and you won't go back to deploying in a VM again imo.
This only gets better with ai-assisted development, any model is going to produce much better results for k8s given the huge training set vs someone's bespoke build rube-goldberg machine.
how could k8s improve my deployment process?
split deployments -- perhaps you want to see how an update impacts something: if error rates change, if conversion rates change, w/e. K8s makes this pretty easy to do via something like a canary or blue green deployment. Likewise, if you need to rollback, you can do this easily as well from a known good image.
Perhaps you need multiple servers -- not for scale -- but to be closer to your users geographically. 1 server in each of -5-10 AZs makes the updates a bit more complicated, especially if you need to do something like a db schema update.
Perhaps your traffic is lumpy and peaks during specific times of the year. Instead of provisioning a bigger VM during these times, your would prefer to scale horizontally automatically. Likewise, depending on the predictable-ness of the distribution of traffic, running a larger machine all the time might be very expensive for only the occasional burst of traffic.
To be very clear, you can do all of this without k8s. The question is, is it easier to do it with or without? IMO, it is a personal decision, and k8s makes a lot of sense to me. If it doesn't make a ton of sense for your app, don't use it.
Also, Kubernetes uses immutable images and containers so you don't have to worry about dependencies or partial deploys.
when in reality, you can go very bare-bones with k8s, but people pretend like only the most extreme complexity is what's possible because it's not easy to admit that k8s is actually quite practical in a lot of ways, especially for avoiding drift and automation
that's my take on it
This feels like what us Brits would call "damning with faint praise".
Windows 95 was terrible. Really bad. If you really mean to say that Kubernetes is revolutionary and well-engineered, Windows 2000 would be a much better example.
knowing when and when not to use k8s, is also a skill
Scale vertically until you can't because you're unlikely to hit a limit and if you do you'll have enough money to pay someone else to solve it.
Docker is amazing development tooling but it makes for horrible production infrastructure.
Docker Compose is good for running things on a single server as well.
Docker Swarm and Hashicorp Nomad are good for multi-server setups.
Kubernetes is... enterprise and I guess there's a scale where it makes sense. K3s and similar sort of fill the gap, but I guess it's a matter of what you know and prefer at that point.
Throw on Portainer on a server and the DX is pretty casual (when it works and doesn't have weird networking issues).
Of course, there's also other options for OCI containers, like Podman.
IS that a thing still?
> Kubernetes is... enterprise
I would contest that. Its complex, but not enterprise.
Nomad is a great tool for running processes on things. The problem is attaching loadbalancers/reverse proxies to those processes requires engineering. It comes for "free" with k8s with ingress controllers.
Yeah, using it in production. If you don't need the equivalent of CRDs or other complex stuff like network meshes, it's stable and pretty okay! My ingress is just a regular web server image, for example.
> It comes for "free" with k8s with ingress controllers.
Ingress Controllers will keep working but the API is frozen, I think nowadays you're supposed to use Gateway instead: https://gateway-api.sigs.k8s.io/
I'd also contest the k8s is enterprise. Unless by enterprise you just mean over engineered in which case I agree.
Show me a Docker in use where build caching was solved optimally for development builds (like eg. make did for C 40 or 50 years ago)?
Perhaps you consider Docker layers one of the "rough edges", but I believe instant, iterative development builds are a minimum required for "great development tooling".
I did have great fun optimizing Docker build times, but more in the "it's a great engineering challenge to make this shitty thing build fast" sense.
Something like the following works well in practice:
If you don't need or want to build your own images, you can fold steps 1-4 into just using upstream images off of Docker Hub or whatever you prefer, but in practice it works pretty okay across numerous stacks. Of course, it's also possible to easily have very high standards in regards to what you mean as "optimal", so Docker probably won't live up to that.- I changed one source file, I need to rebuild the whole app
- I upped one dependency by a minor version
...
How long is the rebuild time? As I said, compare it to simple, standard make-based setup.
The problem is that layers are really — in the end — sequential, and even a simple change in an earlier one invalidates the latter cached layers.
I do not consider up-to-few-seconds for cases like above "very high standards", but it is a lot of work and care with Docker — especially as you need it not to mess up development or production environments as people skip versions and similar (if you use tricks like reading some config from a file).
Let's say you're a team of 1-3 technical people building something as an MVP, but don't necessarily want to throw everything away and rewrite or re-architect if it gets traction.
What are your day 1 decisions that let you scale later without over-engineering early?
I'm not disagreeing with you btw. I genuinely don't know a "right" answer here.
I use k3s/Rancher with Ansible and use dedicated VMs on various providers. Using Flannel with wireguard connects them all together.
This I think is reasonable solution as the main problem with cloud providers is they are just price gouging.
Even if you just run on 2 nodes with k3s it seems worth it to me for the standardized tooling. Yes, it is not a $5 a month setup but frankly if what you host can be served by a single $5 a month VM I don't particularly care about your insights, they are irrelevant in a work context.
Do you have experience with Kubernetes solving these issues? Would love to hear more if so.
Currently running podman containers at work and trying to figure out better solutions for monitoring, alerting, etc. Not so worried about scale (my simple python scripts don't need it) but abstracting away the monitoring, alerting, secure secret injection, etc. seems like it'd be a huge win.
All of this just adds so much extra complexity. If I'm running Amazon.com then sure, but your average app is just fine on a single VM.
I have been building https://github.com/openrundev/openrun, which provides a declarative solution to deploy internal web apps for teams (with SAML/OAuth and RBAC). OpenRun runs on a single-machine with Docker or it can deploy apps to Kubernetes.
If you have actual need to deploy few dozen services all talking with eachother k8s isn't bad way to do it, it has its problems but it allows your devs to mostly self-service their infrastructure needs vs having to process ticket for each vm and firewall rules they need. That is saying from perspective of migrating from "old way" to 14 node actual hardware k8s cluster.
It does make debugging harder as you pretty much need central logging solution, but at that scale you want central logging solution anyway so it isn't big jump, and developers like it.
Main problem with k8s is frankly nothing technical, just the "ooh shiny" problem developers have where they see tech and want to use tech regardless of anything
* Built the app (into a self contained .jar, it was a JVM shop)
* Put the app into a Ubuntu Docker image. This step was arguably unnecessary, but the same way Maven is used to isolate JVM dependencies ("it works on my machine"), the purpose of the Docker image was to isolate dependencies on the OS environment.
* Put the Docker image onto an AWS .ami that only had Docker on it, and the sole purpose of which was to run the Docker image.
* Combined the AWS .ami with an appropriately sized EC2.
* Spun up the EC2s and flipped the AWS ELBs to point to the new ones, blue green style.
The beauty of this was the stupidly simple process and complete isolation of all the apps. No cluster that ran multiple diverse CPU and memory requirement apps simultaneously. No K8s complexity. Still had all the horizontal scaling benefits etc.
I took it to its maximum: every service is a piece that can break ---> fewer pieces, fewer potential breakages.
When I can (which is 95% of the time, I add certain other services inside the processed themselves inside the own server exes and make them activatable at startup (though I want all my infra not to drift so I use the same set of subservices in each).
But the idea is -- the fewer services, the fewer problems. I just think, even with the trade-offs, it is operationally much more manageable and robust in the end.
There are situations where a single VM, no matter how powerful is, can do the job.
So I guess I'm a fan. I use a monolith for most of my stuff if I have the choice, but if I'm working somewhere or on something where I have to manage a bunch of services I'm most certainly going to reach for k8s.
Similarly, I suspect (based on your "hardening" grievance) that a lot of your tedium is just that cloud APIs generally push you toward least-privileges with IAM, which is tedious but more secure. And if you implement a comparably secure system on your single VM (isolating different processes and ensuring they each have minimal permissions, firewall rules, etc) then you will probably have strictly more incidents and debugging effort. But you could go the other way and make a god role for all of your services to share and you will spend much less time debugging or dealing with incidents.
Even with a single VM, you could throw k3s on it and get many of the benefits of Kubernetes (a single, unified, standardized, extensible control plane that lots of software already supports) rather than having to memorize dozens of different CLI utilities, their configuration file formats, their path preferences, their logging locations, etc. And as a nice bonus, you have a pretty easy path toward high availability if you decide you ever want your software to run when Google decides to upgrade the underlying hardware.
That is not what kube is designed for.
I don't work that closely with k8s, but have toyed with a cluster in my homelab, etc. Way back before it really got going, I observed some OpenStack folks make the jump to k8s.
Knowing what I knew about OpenStack, that gave me an inkling that what you describe would happen and we'd end up in this place where a reasonable thing exists but it has all of this crud layered on top. There are places where k8s makes sense and works well, but the people surrounding any project are the most important factor in the end result.
Today we have an industry around k8s. It keeps a lot of people busy and employed. These same folks will repeat k8s the next time, so the best thing people that who feel they have superior taste is to press forward with their own ideas as the behavior won't change.
I'm not familiar with kubernetes, but doesn't it already do SDN out of the box?
Yes and no. Kubernetes defines specification about network behavior (in form of CNI), but it contains no actual implementation. You have to install the network plugin basically as the first setup step.
The irony is that "DevOps" was supposed to be a culture and a set of practices, not a job title. The tools that came with it (=Kubernetes) turned out to be so complex that most developers didn't want to deal with them and the DevOps became a siloed role that the movement was trying to eliminate.
That's why I have an ick when someone uses devops as a job title. Just say "System Admin" or "Infrastrcutre Engineer". Admit that you failed to eliminate the siloes.
I am primarily a backend developer but I do a lot of ops / infra work because nobody else wants to do it. I stay as far away from k8s as possible.
Absolutely brilliant. Love it.
It's obvious to you, me and the other 2 presumably techie people who've responded within 15 mins that you shouldn't have been using Kubernetes. But you probably work in a company of full of techie people, who ended up using Kubernetes.
We have HN, an environment full of techie people here who immediately recognise not to use k8s in 99% of cases, yet in actually paid professional environments, in 99% of cases, the same techie people will tolerate, support and converge on the idea they should use k8s.
I feel like there's an element of the emperors new clothes here.
I once tried to build a simple setup using VM images and the complexity exploded to the point where I'm not sure why anyone should bother.
When building a container you can just throw everything into it and keep the mess isolated from other containers. If you use a VM, you can't use the OCI format, you need to build custom packages for the OS in question. The easiest way to build a custom package is to use docker. After that you need to build the VM images which requires a convoluted QEMU and libvirt setup and a distro specific script and a way to integrate your custom packages. Then after all of this is done you still need to test it, which means you need to have a VM and you need to make it set itself up upon booting, meaning you need to learn how to use cloud-init.
Just because something is "mature" doesn't mean it is usable.
The overhead of docker is basically insignificant and imperceptible (especially if you use host networking) compared to the day to day annoyances you've invited into your life by using VM images. Starting a a VM for testing purposes is much slower than starting a container.
FROM scratch
COPY my-static-binary /my-static-binary
ENTRYPOINT “/my-static-binary”
Having multiple processes inside one container is a bit of an anti-pattern imo
Do you pair it with some orchestration (to spin up the necessary VM)?
The tools in this space can really help get a few containers in dev/staging/production much more manageable.
Not long after, I found that the pods were CONSTANTLY getting into some weird state where K8s couldn't rebuild, so I had to forcibly delete the pods and rebuild. I blamed myself, not knowing much about K8s, but it also was extremely frustrating because, as I understood/understand it, the entire purpose of Kubernetes is to ensure a reliable deployment of some combination of pods. If it couldn't do that and instead I had to manually rebuild my cluster, then what was the point?
In the end, I ended up nuking the entire project -- K8s, Docker containers, Python, and Dask -- and instead went with a single Rust binary deployed to an Azure Function. The result was faster (by probably an order of magnitude), less memory, cheaper (maybe -80% cost), and much more reliable (I think around four nines).
Most companies aren't "web scale" ™ and don't need an orchestrator built for google level elasticity, they need a vm autoscaling group if anything.
Most apps don't need such granular control over fs access, network policies, root access, etc, they need `ufw allow 80 && ufw enable`
Most apps don't need a 15 stage, docker layer caching optimized, archive promotion build pipeline that takes 30 minutes to get a copy change shipped to prod, they need a `git clone me@github.com:me/mine.git release_01 && ln -s release_01 /var/www/me/mine/current`
This is coming from someone who has had roles both as a backend product engineer and as a devops/platform engineer, who has been around long enough to remember "deploy" to prod was eclipse ftping php files straight to the prod server on file save. I manage clusters for a living for companies that went full k8s and never should have gone full k8s. ECS would have worked for 99% of these apps, if they even needed that.
Just like the js ecosystem went bat shit insane until things started to swing back towards sanity and people started to trim the needless bloat, the same is coming or due for the overcomplexity of devops/backend deployments
"ECS would have worked for 99% of these apps, if they even needed that."
I used to agree with that but is EKS really that much more complicated? Yes you pay for the k8s control plane but you gain tooling that is imho much easier to work with than IaC.
As a devops/cloud engineer coming from a pure sysadmin background (you've got a cluster of n machines running RHEL and that's it) i feel this.
The issues i see however are of different nature:
1. resumeè-driven development (people get higher-paying job if you have the buzzwords in your cv)
2. a general lack of core-linux skills. people don't actually understand how linux and kubernetes work, so they can't build the things they need, so they install off-the-shelf products that do 1000 things including the single one they need.
3. marketing, trendy stuff and FOMO... that tell you that you absolutely can't live without product X or that you must absolutely be doing Y
to give you an example of 3: fluxcd/argocd. they're large and clunky, and we're getting pushed to adopt that for managing the services that we run inside the cluster (not developer workloads, but mostly-static stuff like the LGTM stack and a few more things - core services, basically). they're messy, they add another layer of complexity, other software to run and troubleshoot, more cognitive load.
i'm pushing back on that, and frankly for our needs i'm fairly sure we're better off using terraform to manage kubernetes stuff via the kubernetes and helm provider. i've done some tests and frankly it works beautifully.
it's also the same tool we use to manage infrastructure, so we get to reuse a lot of skills we already have.
also it's fairly easy to inspect... I'm doing some tests using https://pkg.go.dev/github.com/hashicorp/hcl/v2/hclparse and i'm building some internal tooling to do static analysis of our terraform code and automated refactoring.
i still think kubernetes is worth the hassle, though (i mostly run EKS, which by the way has been working very good for me)
This is one of the main fuckups of k8s, the networking is batshit.
The other problems is that secrets management is still an afterthought.
The thing that really winds me up is that it doesn't even scale up that much. 2k nodes and it starts to really fall apart.
> Traditional Cloud 1.0 companies sell you a VM with a default of 3000 IOPS, while your laptop has 500k. Getting the defaults right (and the cost of those defaults right) requires careful thinking through the stack.
I wish them a lot of luck! I admire the vision and am definitely a target customer, I'm just afraid this goes the way things always go: start with great ideals, but as success grows, so must profit.
Cloud vendor pricing often isn't based on cost. Some services they lose money on, others they profit heavily from. These things are often carefully chosen: the type of costs that only go up when customers are heavily committed—bandwidth, NAT gateway, etc.
But I'm fairly certain OP knows this.
There's not enough redundancy. You could raid1 those NVME's when before they get attached to a VM and that helps with hardware failures, but you get less of them to attach. Even if you RAID them, there's not a good way to move that VM to another host if there's a RAM or CPU or other hardware issue on that host.
These VM's with NVME's directly attached have to basically be treated as bare metal servers and you have to do redundancy at the application layer (like database replication).
But again, all of the major cloud services offer these types of machines if you NEED NVME IO speed. There are quirks though. For example, in Azure it seems like you have to be able to expect the VM to be moved whenever Azure feels like it and expect that ephemeral data to be wiped. Whereas in Openstack, we would do local block level migrations if we HAD to move the VM to another host. That block level migration required the VM to be turned off but it did copy the local NVME data to another host. If this happened it was all planned and the particular application had app level redundancy built in so it was not a problem. If the host crashed, that particular VM would just be down till the host was fixed and came back online.
The trick is building a block storage system that treats the local disk as write-back cache with async replication to networked storage. Like the blog post says they'll be doing.
The async replication has some integrity/recovery concerns for sure, but it the trick that enables local speeds. And people have been happy with async replication for their database for a very long time. Just need good observability for the durability delay.
Once you have that, you can do live VM migration if you're careful enough about dirty data. The new node just starts out with an empty cache.
It's not exactly trivial, but it's also probably not the biggest challenge if you're genuinely building a brand new cloud and going to compete against the hyperscalers. (Hell, hire me and I can write it for you. It'll take time and CPU hours to get stable, but the magic required is only mildly arcane.)
For example: https://dl.acm.org/doi/10.1145/3492321.3524271
This is the critical point. All hardware fails eventually. The CPU and RAM are, in a real sense, also ephemeral. The only relevant question is what the risk tolerance of the use-case is. If restoring from async backup is sufficient, then embrace ephemerality and keep backups. If you need round-the-clock availability, pick an architecture that lets you fall over gracefully to another machine, and embrace the ephemerality when you inevitably need to do so.
So build resiliency into your application layer.
When you're an OpenStack cloud provider, your customers choose.
When you're a customer using Open Source software, your vendors choose.
Using a mixture of directly attached NVMe and network-attached volumes with backup is the sweet spot for me.
I don't need to maintain my own network filesystem (Ceph), and I can put applications that mirrors its database natively on NVMe and everything I don't have much control over on network-attached volumes.
I feel like there's something better not yet made.
Using fio
Hetzner (cx23, 2vCPU, 4 GB) ~3900 IOPS (read/write) ~15.3 MB/s avg latency ~2.1 ms 99.9th percentile ≈ ~5 ms max ≈ ~7 ms
DigitalOcean (SFO1 / 2 GB RAM / 30 GB Disk) ~3900 IOPS (same!) ~15.7 MB/s (same!) avg latency ~2.1 ms (same!) 99.9th percentile ≈ ~18 ms max ≈ ~85 ms (!!)
using sequential dd
Hetzner: 1.9 GB/s DO: 850 MB/s
Using low end plan on both but this Hetzner is 4 euro and DO instance is $18.
RS 1000 G12 AMD EPYC™ 9645 8 GB DDR5 RAM (ECC) 4 dedicated cores 256 GB NVMe
Costs 12,79 €
Results with the follwing command:
fio --name=randreadwrite \ --filename=testfile \ --size=5G \ --bs=4k \ --rw=randrw \ --rwmixread=70 \ --iodepth=32 \ --ioengine=libaio \ --direct=1 \ --numjobs=4 \ --runtime=60 \ --time_based \ --group_reporting
IOPS Read: 70.1k IOPS Write: 30.1k IOPS ~100k IOPS total
Throughput Read: 274 MiB/s Write: 117 MiB/s
Latency Read avg: 1.66 ms, P99.9: 2.61 ms, max 5.644 ms Write avg: 0.39 ms, P99.9: 2.97 ms, max 15.307 ms
IOPS: read 325k, write 139k
Throughput: read 1271MB/s, write 545MB/s
Latency: read avg 0.3ms, P99.9 2.7ms, max 20ms; write: 0.14ms, P99.9 0.35ms max 3.3ms
so roughly 100 times iops and throughput of the cloud VMs
Using a Netcup VPS 1000 G12 is more comparable.
read: IOPS=18.7k, BW=73.1MiB/s
write: IOPS=8053, BW=31.5MiB/s
Latency Read avg: 5.39 ms, P99.9: 85.4 ms, max 482.6 ms
Write avg: 3.36 ms, P99.9: 86.5 ms, max 488.7 ms
Here are some "Regular Performance" shared resource stats
Hetzner CPX11 (Ashburn, 2 CPUs, 2GB, 5.49€ or $6.99/month before VAT)
read: IOPS=36.7k, BW=144MiB/s, avg/p99.9/max 2.4/6.1/19.5ms
write: IOPS=15.8k, BW=61.7MiB/s, avg/p99.9/max 2.4/6.1/18.7ms
Hetzner CPX22 (Helsinki, 2 CPUs, 4GB, 7.99€ or $9.49/month before VAT)
read: IOPS=48.2k, BW=188MiB/s, avg/p99.9/max 1.9/5.7/10.8ms
write: IOPS=20.7k, BW=80.8MiB/s, avg/p99.9/max 1.8/5.8/10.9ms
Hetzner CPX32 (Helsinki, 4 CPUs, 8GB, 13.99€ or $16.49/month before VAT)
read: IOPS=48.3k, BW=189MiB/s, avg/p99.9/max 1.9/6.2/36.1ms
write: IOPS=20.7k, BW=81.0MiB/s, avg/p99.9/max 1.8/6.3/36.1ms
Business 101 teaches us that pricing isn't based on cost. Call it top down vs bottom up pricing, but the first principles "it costs me $X to make a widget, so 1.y * $X = sell the product for $Y is not how pricing works in practice.
The price is what the customer will pay, regardless of your costs.
For example I calculated the cost of a solar install to be approximately: Material + Labour + Generous overhead + Very tidy profit = 10,000€
In practice I keep getting offers for ~14,000€, which will be reduced to 10,000€ with a government subsidy and my request for an itemized invoice is always met with radio silence.
Which it won't be, if at every turn you choose the hyperscaler.
It kinda is, but obscured by GP's formula.
More simply; if it costs you $X to produce a product and the market is willing to pay $Y (which has no relation to $X), why would you price it as a function of $X?
If it costs me $10 to make a widget and the market is happy to pay $100, why would I base my pricing on $10 * 1.$MARGIN?
But that is an equilibrium result, and famously does not apply to monopolies, where elasticity of substitution will determine the premium over the rental rate of capital.
Edit: I posted this before reading, and these two are the same he points out.
And yes, IO typically happens in 4kb blocks, so you need a decent amount of IOPS to get the full bandwidth.
That latter part is a big deal, too. If I buy 1PB of block storage, I’m decently likely to be running a fancy journaled or WAL-ed or rollback-logged thing on top, and that thing might be completely unable to read from a read only snapshot. So actually reading from a PIT snapshot is a pain regardless of what I paid for it. Even using EBS or similar snapshots is far from being an amazing experience.
If that's true, I wonder if this is a deliberate decision by cloud providers to push users towards microservice architectures with proprietary cloud storage like S3, so you can't do on-machine dbs even for simple servers.
Instead they make the default "meager IOPS" and then charge more to the people who need more.
I remember my worked laptop's IOPS beating a single VM on the first SSD based SAN I deployed as well. Of course, the SAN scaled well beyond it with 1,000 VMs.
I see the same thing happen with Kubernetes. I've run clusters from various sizes for about half a decade now. I've never once had an incident that wasn't caused by the product itself. I recall one particular incident where we had a complete blackout for about an hour. The people predisposed to hating Kubernetes did everything they could to blame it all on that "shitty k8s system." Turns out the service in question simply DOS'd by opening up tens of thousands of ports in a matter of seconds when a particular scenario occurred.
I'm neither in the k8s is the future nor k8s is total trash. It's a good system for when you genuinely need it. I've never understand the other two sides of the equation.
Usually they go hand in hand.
By the time I left, the developers didn't really know anything about how the underlying infrastructure worked. They wrote their Dockerfiles, a tiny little file to declare their deployment needs, and then they opened a platform webpage to watch the full lifecycle.
If you're a single service shop, then yeah, put Docker Compose on it and run an Ansible playbook via GitHub Actions. Done. But for a larger org moving off cloud to bare-metal, I really couldn't see not having k8s there to help buffer some of the pain.
I agree that Kubernetes can help simplify the deployment model for large organizations with a mature DevOps team. It is also a model that many organizations share, and so you can hire for talent already familiar with it. But it's not the only viable deployment model, and it's very possible to build a deployment system that behaves similarly without bringing in Kubernetes. Yes, including automatic preview deployments. This doesn't mean I'm provided a VM and told to figure it out. There are still paved-path deployment patterns.
As a developer, I do need to understand the environment my code runs in, whether it is bare-metal, Kubernetes, Docker Swarm, or a single-node Docker host. It impacts how config is deployed and how services communicate with each other. The fact that developers wrote Dockerfiles is proof that they needed to understand the environment. This is purely a tradeoff (abstracting one system, but now you need to learn a new one.)
Also, those simpler deployments usually burn more money per utilized compute, or involve reinventing 80% of kids, often badly
It's up to the individual to choose how much knowledge they want to trade away for convenience. All the containers are just forms of that trade.
You surely meant "much less efficient than"
There also seems to be confusion about what I meant by "bare-metal." I wasn't intending to refer to the server ownership model, but rather the deployment model where you deploy software directly onto an operating system.
When all you have is a hammer, every problem starts to look like a nail. And the people with axes are wondering how (or indeed even why) so many people are trying to chop wood with a hammer. Further, some axewielders are wondering why they are losing their jobs to people with hammers when an axe is the right tool for the job. Easy to hate the hammer in this case.
And the end result is often that you have two tribes that have totally incorrect idea of even what tools they are using themselves and how, and it's like you swapped them an intentionally wrong dictionary like in a Monthy Python sketch.
We run k8s with several VMs in a couple different cloud providers. I’d love it if I could forget about the VMs entirely.
Is there a simpler thing than k8s that gets you all that? Probably. But if you don’t use k8s, aren’t you doomed to reimplement half of it?
Like these things:
- Service discovery or ingress/routing (“what port was the auth service deployed on again?”)
- Declarative configuration across the board, including for scale-out
- Each service gets its own service account for interacting with external systems
- Blue/green deployments, readiness checks, health checks
- Strong auditing of what was deployed and mutated, when, and by whom
I ended up buying a cheap auctioned Hetzner server and using my self-hostable Firecracker orchestrator on top of it (https://github.com/sahil-shubham/bhatti, https://bhatti.sh) specifically because I wanted the thing he’s describing — buy some hardware, carve it into as many VMs as I want, and not think about provisioning or their lifecycle. Idle VMs snapshot to disk and free all RAM automatically. The hardware is mine, the VMs are disposable, and idle costs nothing.
The thing that, although obvious, surprised me most is that once you have memory-state snapshots, everything becomes resumable. I make a browser sandbox, get Chromium to a logged-in state, snapshot it, and resume copies of that session on demand. My agents work inside sandboxes, I run docker compose in them for preview environments, and when nothing’s active the server is basically idle. One $100/month box does all of it.
Thank you for sharing!
At home, I've done that with a Zellij session (everything is tied to the session, and quitting Zellij completely means "I'm done with this". Merely disconnecting keeps it running).
My only feedback so far is that a lot of the documentation, though thorough and useful, looks clearly AI-written. That's not bad in and of itself, but it could be more concise. I especially love the "design decisions" section as I learned something new already.
Have you posted it on "Show HN" already? If not, you should.
I am aware of the documentation, it’s what I have been focusing on before I can post on HN. I want to make it a delight to read for other people!
As for the design decisions, I have tried keeping all the plans I made in the repo too. I wouldn’t have been able to make bhatti in a month without LLMs.
Out of interest, what sandboxing solution do you use?
There is already so much software out there, which isn't used by anyone. Just take a look at any appstore. I don't understand why we are so obsessed with cranking out even more, whereas the obvious usecase for LLMs should be to write better software. Let's hope the focus shifts from code generation to something else. There are many ways LLMs can assist in writing better code.
I believe right now we are still in the phase of “how can AI help engineers write better software”, but are slowly shifting to “how can engineers help AI write better software.” This will bring in a new herd of engineers with completely different views on what software is, and how to best go about building computer interactions.
That said, if you look at the apps on your phone, I wager a large proportion don't have these moats. Translation, passwords, budget, reminders, email, to do, project management, messaging, browser, calendar, fitness, games, game tracking, etc.
Jevons paradox would be if despite software becoming cheaper to produce the total spend on producing software would increase because the increase in production outruns the savings
Jevons paradox applies when demand is very elastic, i.e. small changes in price cause large changes in quantity demanded. It's a property of the market.
He's saying that agents make code much cheaper, therefore there will be a large increase in demand for code. This appears to be exactly what you're describing.
I honestly think this is ideal. Video games aside, I think one day we'll look back and realize just how insane it was that we built software for millions or even billions of users to use. People can now finally build the software that does exactly what they've wanted their software to do without competing priorities and misaligned revenue models working against them. One could argue this kind of software, by definition, is higher quality.
I could see maybe more customization of said software, but not totally fresh. I do agree that people will invent more one-off throwaway software, though.
Tinkering? Even today, people don’t need to understand software. They just need to be able to describe their problems and goals to create an app.
> I mean, are you envisioning that everyone would have their own custom messaging app, for example? Or email?
Well first I think there’s a good chance that most apps as we know them today won’t even exist, and most “apps” will be tool use on APIs. But even then, shopping apps, for example, could be so highly personalized that no two people have the same one.
> I mean, I think most people's demands for those things are all extremely homogenous.
They aren’t, as evidenced by the fact there are many dozens of popular messaging apps with millions of users. Despite the network effects for a messaging app to even be viable.
Also, I’m not talking one-off throwaway apps… these are living, breathing pieces of production-grade software users will mold to fit their needs and evolve with them for years.
I’m not sure what “totally fresh” means
My view is actually the opposite. Software now belongs to cattle, not pet. We should use one-offs. We should use micro-scale snippets. Speaking language should be equivalent to programming. (I know, it's a bit of pipe dream)
In that sense, exe.dev (and tailscale) is a bit like pet-driven projects.
Vibe coding or LLM accelerated development is going to turn this on its head. Everyone will be able to afford custom software to fit their specific needs and preferences. Where Salesforce currently has 150,000 customers, imagine 150,000 customers all using their own customised CRM. The scope for software expansion is unbelievably large right now.
In the 70s, it was called "time-sharing". Instead of buying a mainframe, you got a CICS application instance on a mainframe and used that. (tangentially, spare time on these built-out nation-wide dialup-supported networks is what gave birth to CompuServe and GEnie).
In the dot-com era, it was called "application service providers". Salesforce and actually started in this era (1999). So did NetSuite. This was the first attempt to be browser-based but bandwidth and browsers sucked then.
I think PaaS is a more recent software paradigm, albeit a far less successful one.
If you're doing anything complicated, Excel just doesn't make sense anymore. it'll still the be data exchange format (at least, something more advanced than csv), but it's no longer the only frontend.
"No one uses" is no longer the insult it once was. I don't need or want to make software for every last person on the world to use. I have a very very small list of users (aka me) that I serve very well with most of the software that I generate these days outside of work.
It certainly is for lots of businesses, otherwise they go out of business.
There is something called 'revenue' which they need to make from customers which are their 'users', and that revenue pays for the 'operating costs' which includes payroll, office rent, infrastructure etc.
This just means that it is important than ever to know what to build just as how it is built. It is unrealistic for a business to disregard that and to build anything they want and end up with zero users.
No users, No revenue. No revenue, No business.
As for the average quality: it’s unclear.
My intuition is that agents lift up the floor to some degree, but at the same time will lead to more software being produced that’s of mediocre quality, with outliers of higher quality emerging at a higher rate than before.
If you wanted to, you could make an argument about the principal-agent problem - that as hunter-gatherers or subsistence, farmers, our quality versus quantity decisions only affected us, whereas in a market economy, you could argue that one person’s quality versus quantity decision affects someone else.
But dismantling capitalism will not solve this problem. It just moves the decision-making to a different group of people. Those people will face the same trade-offs and the same incentives. After the Revolution, even the most loyal comrade will have to contend with the fact that they can choose to provide the honourable working class with more of a thing if they drop the quality.
I agree there is opportunity in making LLM development flows smooth, paired with the flexibility of root-on-a-Linux-machine.
> Time and again I have said “this is the one” only to be betrayed by some half-assed, half-implemented, or half-thought-through abstraction. No thank you.
The irony is that this is my experience of Tailscale.
Finally, networking made easy. Oh god, why is my battery doing so poorly. Oh god, it's modified my firewall rules in a way that's incompatible with some other tool, and the bug tracker is silent. Now I have to understand their implementation, oh dear.
No thank you.
Tags permanently erase the user identity from a device, and disable things like Taildrop. When I tried to assign a tag for ACLs, I found that I then could not remove it and had to endure a very laborous process to re-register a Tailscale device that I added to Tailscale for the express purpose of remotely accessing
But yes I don’t think you can ACL based o the hostname
Part of the reason that we don't (currently) let you do this is that a hostname is a user-reported field, and can change over time; it's not a durable form of identity that you can write ACLs on. One could imagine, for example:
1. Creating an ACL rule that allows hostname "webserver" to hostname "db".
2. (time passes)
3. Hostname "webserver" is deleted/changed to "web"/etc.
4. Someone can now register a user device with the system hostname set to "webserver"
Should they be allowed to inherit the pre-existing ACL rule?
However, you can accomplish something very close to what you're asking for, I think, by defining a "host" in the policy file (https://tailscale.com/docs/reference/syntax/policy-file#host...) that points to a single Tailscale IP. Since we don't allow non-admins to change their Tailscale IP, this uniquely identifies a single device even if the hostname changes, and thus you can write a policy similar to:
Could you rephrase that / elaborate on that? Isn't Tailscale's selling point precisely that they do identity-based networking?
EDIT: Never mind, now I see the sibling comment to which you also responded – I should have reloaded the page. Let's continue there!
I hope this wasn't interpreted towards exe.dev. That really is a cool service!
I think that's startup-thinking, at least in my experience. Maybe in a small company the DevOps guy does all infra.
In my experience, especially in financial services, who runs the show are platform engineering MDs - these people want maximum control for their software engineers, who they split up into a thousand little groups who all want to manage their own repos, their own deployments, their own everything. It's believed that microservices gives them that power.
I guarantee you devops people hate complexity, they're the ones getting called at night and on the weekend, because it's supposedly always an "infrastructure issue" until proven otherwise.
Also the deployment logs end up in a log aggregation system, and god forbid software developers troubleshoot their own deployments by checking logs. It's an Incident.
Are microservices a past fad yet?
Everything which cloud companies provide just cost so much, my own postgres running with HA setup and backup cost me 1/10th the price of RDS or CloudSQL service running in production over 10 years with no downtime.
i directly autoscales instances off of the Metrics harvested from graphana it works fine for us, we've autoscaler configured via webhooks. Very simple and never failed us.
i don't know why would i even ever use GCP or AWS anymore.
All my services are fully HA and backup works like charm everyday.
Does a regular 20-something software engineer still know how to turn some eBay servers & routers into a platform for hosting a high-traffic web application? Because that is still a thing you can do! (I've done it last year to make a 50PiB+ data store). I'm genuinely curious how popular it is for medium-to-big projects.
And Hetzner gives you almost all of that economic upside while taking away much of the physical hassle! Why are they not kings of the hosting world, rather than turning over a modest €367M (2021).
I find it hard to believe that the knowledge to manage a bunch of dedicated servers is that arcane that people wouldn't choose it for this kind of gigantic saving.
Managing servers is fine. Managing servers well is hard for the average person. Many hand-rolled hosting setups I've encountered includes fun gems such as:
- undocumented config drift.
- one unit of availability (downtime required for offline upgrades, resizing or maintenance)
- very out of date OS/libraries (usually due to the first two issues)
- generally awful security configurations. The easiest configuration being open ports for SSH and/or database connections, which probably have passwords (if they didn't you'd immediately be pwned)
Cloud architecture might be annoying and complex for many use-cases, but if you've ever been the person who had to pick up someone else's "pet" and start making changes or just maintaining it you'll know why the it can be nice to have cloud arch put some of their constraints on how infra is provisioned and be willing to pay for it.
Hetzner is an oldschool German company, it is not surprising to see them act this way. They are very profitable (165M Euro in 2024) and have very little debt. They also seem to be mostly bootstrapped and are not VC funded
https://www.northdata.com/Hetzner%20Online%20GmbH,%20Gunzenh...
Whether or not cloud is viable for a company is very individual. It's very hard to pin point a size or a use case that will always make cloud the "correct" choice.
OP is not saying they push new versions at such a high frequency they need checks every one minute.
The choice of one minute vs 15 minute is implementation detail and when architected like this costs nothing.
I hope that helps. Again this is my own take.
Maybe you meant to say "automatically" instead of "immediately"? Because if you really mean "immediately" then there is still plenty of low-hanging fruit to be had.
But I came across Mythic Beasts (https://www.mythic-beasts.com/) yesterday, similar idea, UK based. Not used them yet but made the account for the next VPS.
It is like 4 lines of config for Postgres, the only line you need to change is on which path Postgres should store the data.
Maybe change the filesystem?
But there is middleground in form of VPS, where hardware is managed by the provider. It's still way way cheaper than some cloud magic service.
You can use block storage if data matters to you.
Many services do not need to care about data reliability or can use multiple nodes, network storage or many other HA setups.
An employee is going to cost anywhere between 8k and 50k per month. Hiring an employee to save 200/month on servers by using a shitty VPS provider is not saving you any money.
I am sure it's luck but we have few hetzner VPSes in both German locations and in last 5 years afaik they've never been down. On our http monitor service they have 100s of days uptime only because we restarted them ourselves.
If you're looking to invest im fine with only $5M :)
Running Shellbox 24/7 is ~25% cheaper than Exe, with 2x storage but 50% of RAM. Exe seems to provide additional features (which I don't need). Not presenting this information upfront and in an easily digestible format makes me suspicious.
I dig the overall aesthetic and may give Shellbox v2 a try.
I don't want to make that public, it's my way of an isolated dev environment and it runs on my private raspberry behind my tv. Costs me nothing.
I hope you have a good success with your service.
`ssh you/repo/branch@box.clawk.work` → jump directly into Claude Code (or Codex) with your repo cloned and credentials injected. Firecracker VMs, 19€/mo.
POC, please be kind.
If you want to try it: code `HNPRELAUNCH` on checkout, first month free, then 19€/mo (cancel anytime from your Stripe receipt). Limited to the first 20 redemptions, expires in a week.
Honest feedback on what breaks would mean a lot.
at 19€/mo are you subsidizing it given the sharp rise of LLM costs lately?
or are you heavily restricting model access. surely there is no Opus?
Just shows I'm the Dropbox commentator. I have what exe provides on my own and am shocked by the value these abstractions provide everyone else!! One off containers on my own hardware spin up spin down run async agents, etc, tailscale auth, team can share or connect easily by name.
The technology itself in its current form is not valuable
Almost every VC rejected us when we went to get seed funding for Tailscale, we knew none of them. Friends of friends of acquaintances got us meetings. Fundraising is very possible for you if you are committed to building a business. Most important thing is don't think of fundraising as the goal, it is just a tool for building a business. (And some businesses don't need VC funding to work. Some do.)
The biggest challenge is personal: do you want to build a business or do you want to work with cool tech? Sometimes those goals are aligned, but usually they are not. Threading the needle and doing both is difficult, and you always have to prioritize the business because you have to make payroll.
Ha! This made me smile :)
https://github.com/hetzneronline/community-content/blob/mast...
It also has a CLI, hcloud. Am I getting any value with exe.dev I couldn't get with an 80 line hcloud wrapper?
For agents, declarative plans are still valuable because they are reviewable. The interesting question is whether exe.dev changes the primitive: resource pools for many isolated VM-like processes, or just nicer VPS provisioning.
We're thinking about switching to this pricing model for our own startup[1] (we run sandboxed coding agents for dev teams). We run on Daytona right now for sandboxes. Sometimes I spin up a sandboxed agent to make changes to an app, and then I leave it running so my teammate can poke around and test the running app in the VM, but each second it's running we (and our users) incur costs.
We can either build a bunch of complicated tech to hibernate running sandboxes (there's a lot of tricky edge cases for detecting when a sandbox is active vs. should be hibernated) or we can just provision fixed blocks of compute. I think I prefer the latter.
[1] https://github.com/gofixpoint/amika
Not sure we can move away from cpu/memory/io budgeting towards total metal saturation because code isn't what it used to be because no one handles malloc failure any more, we just crash OOM
The key point is the partner companies. Almost nobody is actually running their own clouds the way they would with various 365 products, AWS or Azure. They buy the cloud from partners, similar to how they used to (and still do) buy solutions from Microsoft partners. So if you want to "sell cloud" you're probably going to struggle unless you get some of these onboard. Which again would probably be hard because I imagine a lot of what they sell is sort of a package which basically runs on VM's setup as part of the package that they already have.
International visitors might tell us more about benefits of non EU, US or UK nexus companies/legal/rights.
Lean software -> missing features users want -> add features over time -> bloated mess -> we need a smaller rewrite -> Lean software -> ...
>One price, no surprises. You get 2 CPUs, 8 GB of RAM, and 25 GB of disk—shared across up to 25 VMs.
This might sounds like a good thing compared to the current state of clouds, but what’s better than that is having your own. The other day I got a used optiplex for $20, it had 2TB hdd, 265gb ssd, 16gb, and corei7. This is a one time payment, not monthly. You can setup proxmox, have dozens of lxc and vm, and even nest inside them whatever more lxc too, your hardware, physically with you, backed up by you, monitored by you, and accessed only by you. If you have stable internet and electricity, there’s really no excuse not to invest on your own hardware. A small business can even invest in that as well, not just as a personal one. Go to rackrat.net and grab a used server if you are a business, or a good station for personal use.
Oh, that’s too kind. More like 100x to 1000x. Raw bandwidth is cheap.
I need to fix our transfer pricing. (In fact I'm going to go look at it now.) I set that number when we launched in December, and we were still considering building on top of AWS, so we put a conservative limit based on what wouldn't break the bank on AWS. Now that we are doing our own thing, we can be far more reasonable.
> Finally, clouds have painful APIs. This is where projects like K8S come in, papering over the pain so engineers suffer a bit less from using the cloud. But VMs are hard with Kubernetes because the cloud makes you do it all yourself with lumpy nested virtualization. Disk is hard because back when they were designing K8S Google didn’t really even do usable remote block devices, and even if you can find a common pattern among clouds today to paper over, it will be slow. Networking is hard because if it were easy you would private link in a few systems from a neighboring open DC and drop a zero from your cloud spend. It is tempting to dismiss Kubernetes as a scam, artificial make work designed to avoid doing real product work, but the truth is worse: it is a product attempting to solve an impossible problem: make clouds portable and usable. It cannot be done.
Please learn from Unix's mistakes. Learn from Nix. Support create-before-destroy patterns everywhere. Forego all global namespaces you can. Support rollbacks everywhere.
If any cloud provider can do that, cloud IaC will finally stop feeling so fake/empty compared to a sane system like NixOS.
Checking the current offering, it's just prepaid cloud-capacity with rather low flexibility. It's cheap though, so that is nice I guess. But does this solve anything new? Anything fly.io orso doesn't solve?
What is the new idea here? Or is it just the vibes?
As another user notes in this thread, exe.dev isn't that cheap. Their bandwidth pricing is £7/100gb. The lowest compute tier is £20/mo (Fly.io machines/sprites can go for less than £2/mo).
> Anything fly.io also doesn't solve?
exe.dev is comparable to sprites.dev Fly.io launched recently; but with a different pricing model.
David, by the way of Tailscale, themselves were among early users of Fly.io. I read some of David's commentary on "Cloud 1.0" as taking a dig at their friends at Fly.io, too. This is going to be interesting...
Fine, their UI is different, but I don't see any real difference from other providers.
On that machine you can (easily) make an arbitrary number of VMs.
Each VM has their own URL that you can share (or make private).
See features: https://exe.dev/docs/customization
The shell command to start a new vm, has a --prompt flag to get an LLM to configure the VM for you.
VM's have no public ipv4 IP, and the ipv6 IP doesn't seem to allow incoming connections.
The only supported inbound connections are via their HTTP proxy.
There is no private networking.
At first I interpreted the complaint about cloud providers not offering nested-virtualization, as something he intends to address by offering it as a feature, but no, instead he means that exe.dev's VM abstraction eschews the need for it.
I'm very curious how they deal with subscription levels/noisy neighbors.
Another one could be Bitwarden, although I don't host my own password manager personally. Or netbird. You get the point
VMs have a built-in gateway to cloud providers with a fixed url with no auth. You can top that in via the service itself. No need for your own keys.
So likely a good tool for managing AI agents. And "cloud" is a bit of a stretch, the service is very narrow.
The complete lack of more detailed description of the regions except city name makes it really only suitable for ephemeral/temporary deployments. We don't know what the datacenters are, what redundancy is in place, no backups or anything like that.
You can see their base docker image here - https://github.com/boldsoftware/exeuntu
Running a cloud data center could be a business like operating a self-storage facility or a car wash. Small investors love this kind of operation.
52.35.87.134 <- Amazon Technologies Inc. (AT-88-Z)
Our exe.dev web UI still runs on AWS. We also have a few users left on our VM hosts there, as when we launched in December we were considering building on AWS. Now almost all customer VMs are on other bare metal providers or machines we are racking ourselves. We built our own GLB with the help of another vendor's anycast network. You can see that if you try any of the exe.xyz names generated for user VMs.
We would move exe.dev too, but we have a few customers who are compliance sensitive going through it, so we need to get the compliance story right with our own hardware before we can. It is a little annoying being tied to AWS just for that, but very little of our traffic goes through them, so in practice it works.
https://exe.dev/docs/regions
Hey wait a minute!
"In some tech circles, that is an unusual statement. (“In this house, we curse computers!”) I get it, computers can be really frustrating. But I like computers. I always have. It is really fun getting computers to do things. Painful, sure, but the results are worth it. Small microcontrollers are fun, desktops are fun, phones are fun, and servers are fun, whether racked in your basement or in a data center across the world. I like them all."
The reality: Everyone reading his blog or this HN entry loves computers.
In my experience, K8s is a million times better than legacy shit it is usually replacing. The Herokus, the Ansible soup, the Chef/Puppet soup before that etc. The legacy infra that was held together by glue and sweat that everybody was afraid to touch.
Human nature, really.
* Insistence on adding costly abstractions to overcome the limitations of non-fungible resources
* Deliberate creation of over or under-sized resource "pieces" instead of letting folks consume what they need
* Deliberate incompatibility with other vendors to enforce lock-in
I pitched a "Universal Cloud" abstraction layer years ago that never got any traction, and honestly this sounds like a much better solution anyhow. When modern virtualization is baked into OS kernels, it doesn't make a whole lot of sense to enforce arbitrary resource sizes or limits other than to inflate consumption.
Kubernetes without all the stuff that makes it a bugbear to administrate, in other words. Let me buy/rent a pool of stuff and use it how I see fit, be it containers or VMs or what-have-you.
Perhaps the VM idea is old. The unit is a worker encapsulated in some deployable container.
In the world of Cloudflare workers - especially durable objects that are guaranteed to have one of them running in the world with a tightly bound database.
The way I think of apps has changed.
My take is devs want a way to say “run this code, persist this info, microsecond latency, never go down, scale within this $ budget”
It’s crazy how good a deal $5/mo cloudflare standard plan is.
Obviously many startups raise millions and they gotta spend millions.
However the new age of scale to zero, wake up in millisecond, process the request and go back to sleep is a new paradigm.
Vs old school of over provision for max capacity you will ever need.
Google has a similar, scale to zero container story but their cold startup time is in seconds. Too slow.
I don’t care about how the backend works. Superbase requires magical luck to self host.
A lot of cloud providers have very generous free tiers to hook you and then the moment things take off , it’s a small fortune to keep the servers on.
One thing I'm confused with is how to create a shared resources like e.g. a redis server and connect to it from other vms? It looks now quite cumbersome to setup tailscale or connect via ssh between VMS. Also what about egress? My guess is that all traffic billed at 0.07$ per GB. It looks like this cloud is made to run statefull agents and personal isolated projects and distributed systems or horizontal scaling isn't a good fit for it?
Also I'm curious why not railway like billing per resource utilization pricing model? It’s very convenient and I would argue is made for agents era.
I did setup for my friends and family a railway project that spawns a vm with disk (statefull service) via a tg bot and runs an openclaw like agent - it costs me something like 2$ to run 9 vms like this.
https://exe-muttha-fukken-dev.exe.xyz/
The main reason clouds offer network block devices is abstraction.
I've found the quality and simplicity to be an attractive solution for lazy devops when I need to reach for a second computer
dedicated servers, as hinted by others here, addresses the vast majority of issues one may face for any non-enterprise needs. if you know about IOPS and care about them, odds are that running a simple open-source project [1] on top of one is all you need to do to move on with your day.
need redundancy, etc.? can complement with another one in another provider/region or put CF in front of your box. this is clearly working well enough for some of the commenters who are able to sell their own service on top of this approach.
[1] https://disco.cloud/
> Finally, clouds have painful APIs. This is where projects like K8S come in, papering over the pain so engineers suffer a bit less from using the cloud.
K8s's main function isn't to paint over existing cloud APIs, that is just necessity when you deploy it in cloud. On normal hardware it's just an orchestration layer, and often just a way to pass config from one app to another in structured format.
> But VMs are hard with Kubernetes because the cloud makes you do it all yourself with lumpy nested virtualization.
Man discovered system designed for containers is good with containers, not VMs. More news at 10
> Disk is hard because back when they were designing K8S Google didn’t really even do usable remote block devices, and even if you can find a common pattern among clouds today to paper over, it will be slow.
Ignorance. k8s have abstractions over a bunch of types of storage, for example using Ceph as backend will just use KVM's Ceph backend, no extra overhead. It also supports "oldschool" protocols used for VM storage like NFS or iSCSI. It might be slow in some cases for cloud if cloud doesn't provide enough control, but that's not k8s fault.
> Networking is hard because if it were easy you would private link in a few systems from a neighboring open DC and drop a zero from your cloud spend.
He mistakes cloud problems with k8s problems(again). All k8s needs is visibility between nodes. There are multiple providers to achieve that, some with zero tunelling, just routing. It's still complex, but no more than "run a routing daemon".
I expect his project to slowly reinvent cloud APIs and copying what k8s and other projects did once he starts hitting problems those solutions solved. And do it worse, because instead of researching of why and why not that person seems to want to throw everything out with learning no lessons.
Do not give him money
[0] https://blog.exe.dev/ssh-host-header
(Percentages cited above are tongue-in-cheek, actual numbers are probably different)
Starting a digital ocean droplet is a single curl call. Starting a hetzner server is as well. Their api’s are completely fine and known to llm’s.
Why would agents learn exe’s way of setting up / deploying / binding to ports / auth, rather than just ssh’ing into a vm..?
EC2 provides the *d VMs that have SSDs with high IOPS at much lower cost than network SSDs. They are ephemeral, but so is laptop and its SSD - it can loose the data. From AWS docs "If you stop, hibernate, or terminate an instance, data on instance store volumes is lost.".
A service offering VMs for $20 is a long way from AWS, but I see how it makes sense as a first step. AWS also started with EC2, but in a completely different environment with no competition.
https://exe.dev/docs/customization
The key to scaling up is to have big-enough hardware on the backend. If Hetzner is renting out bare metal instances then they can only rent out the sizes that they have. If a cloud provider invests in really big single systems, they can offer fractions of those systems to multiple tenants, some of whom scale up to use the entire system, and some who don't. I think that is a win-win.
A fractional VM is also a fungible VM. If the tenant calls to spin up a certain size VM, then the backend can find suitable hardware for it from a menu of sizes. Smaller VMs can slot in anywhere there is room, not just on a designated bare-metal system.
A cloud provider is always going to want to maximize their rack space, wattage/heat, and resource usage. So they will invest in high-density systems at every chance. On the other hand, cloud tenants will have diverse needs, including some fraction of those big computers.
But I don't want to be either of those customers. It means the whole system has an extra layer of abstraction, so they can juggle VMs around. It's why you need slow EBS instead of just getting a flash drive in the same case as the CPU, with 0.01x the latency.
"That must be worst website ever made"
Made me love the site and style even more
- I'm building a server farm in my homelab.
- I'm doing a small startup to see if this idea works.
- We're taking on AWS by being more cost effective. Funding secured.
https://news.ycombinator.com/item?id=6641378
Cloud is bad?
> The standard price for a GB of egress from a cloud provider is 10x what you pay racking a server in a normal data center.
From the exe.dev pricing page:
> additional data transfer $0.07/GB/month
So at least on the network price promise they don't seem to deliver, still costs an arm and a leg like your neighbourhood hyperscaler.
Overall service looks interesting, I like simplicity with convenience, something which packet.net deliberately decided not to offer at the time.
every time i've had an issue or question, it's been the same sympathetic people helping me out. over email, in plain text.
Is there a name for this style of writing? I come across it regularly.
I'd describe it as forcefully modest, "I'm just a simple guy" kind of thing. With a dash of "still a child on the inside". I always picture it as if the guy from the King of Queens meme wrote it.
"I guess I'm just really into books, heh" - Bezos (obviously non-real, hypothetical quote, meant to illustrate the concept)
This style is also very prevalent in Twitter bios.
Since it's a "literary" style that is quite common, I'm sure it has been characterized and named.
GPT says it's "aw-shucks", but I think that's a different thing.
And what it has to do with the "cloud"? Cloud means one use cloud-provided services - security, queue, managed database, etc. and that's their selling point. This exe.dev is a bare server where I can install what I want, this is fine, but this is not a cloud and, frankly speaking, nothing new.
if we go back to the principle that modern computers are really fast, SSDs are crazy fast
and we remove the extra cruft of abstractions - software will be easier to develop - and we wouldn't have people shilling 'agents' as a way for faster development.
ultimately the bottleneck is our own thinking.
simple primitives, simpler thinking.
Then I started to realize most people who complain are rolling their own which is also not bad since there are products like k3s that are very simple to use.
It seems things start to fall apart when they try to stuff it with all kinds of crazy idiotic controllers and the favorite of the month CNI and CSI. I always shake my head when I see people creating sand castles by setting up stuff like Ceph from within the cluster.
If you want to play with it keep things simple and have all the persistent data outside of the cluster. Use good old NFS instead of the latest longceph horngluster version. Keep databases and the container registry out. Treat it like a compute pool not a virtual datacenter. Stop recursing chickens inside eggs.
> That must be worst website ever made.
the level of confidence (this is a second time founder after all) to put that on their website gives me confidence that they can make this work
One of my friends was told to come to a sex party that was all male and he is straight. It soured his relationship with the firm so much he ended up winding down the business.
but i know nothing about what the comment says, just answering your question.
Jokes aside: - k8s is insane peace of software. A right tool for a big problem. Not for your toys. Yes, it is crazy difficult to setup and manage. Then what?
- cloud has bad and slow disk. BS. They have perfectly fast NVME.
Something else? That’s it.
Why I am so confident? I used to setup and manage kubernetes for 2 years. I have some experience. Do I use it more? Nope. Not a right tool for me. Ansible with some custom Linux tools fits better for Me.
I also build my own cloud. But if I say it less loud: hosting to host websites for https://playcode.io. Yea, it is hard and with a lot of compromises. Like networking, yes I want to communicate between vms in any region. Or disks and reliability. What about snapshots? And many bare metal renters gives only 1Gbt/s. Which is not fine. Or they ask way more for 10Gbt uplink. So it is easy to build some limited and unreliable shit or non scalable.
> $160/month
100GB/mo is <1mbps sustained lmaoThese are nice declarative statements but have almost no meaningful substance.
> Setup scripts have a maximum size. Use indirection. [What's the maximum size?] > Shelley is a coding agent. It is web-based, works on mobile. [Cool model bro. Any details you want to share?]
I like the way you can tell it what you want and it makes it. Very cool.
https://exe.dev/docs/proxy
If you want to run a website in the cloud, you start with an API, right? A CRUD API with commands like "make me a VPC with subnet 1.2.3.4/24", "make me a VM with 2GB RAM and 1 vCPU", "allow tcp port 80 and 443 to my VM", etc. Over time you create and change more things; things work, everybody's happy. At some point, one of the things changes, and now the website is broken. You could use Terraform or Ansible to try to fix this, by first creating all the configs to hopefully be in the right state, then re-running the IaC to re-apply the right set of parameters. But your website is already down and you don't really want to maintain a complex config and tool.
You can't avoid this problem because the cloud's design is bad. The CRUD method works at first to get things going. But eventually VMs stop, things get deleted, parameters of resources get changed. K8s was (partly) made to address this, with a declarative config and server which constantly "fixes" the resources back to the declared state. But K8s is hell because it uses a million abstractions to do a simple thing: ensure my stuff stays working. I should be able to point and click to set it up, and the cloud should remember it. Then if I try to change something like the security group, it should error saying "my dude, if you remove port 443 from the security group, your website will go down". Of course the cloud can't really know what will break what, unless the user defines their application's architecture. So the cloud should let the user define that architecture, have a server component that keeps ensuring everything's there and works, and stops people from footgunning themselves.
Everything that affects the user is a distributed system with mutable state. When that state changes, it can break something. So the system should continuously manage itself to fix issues that could break it. Part of that requires tracking dependencies, with guardrails to determine if a change might break something. Another part requires versioning the changes, so the user (or system) can easily roll back the whole system state to before it broke. This abstraction is complicated, but it's a solution to a complex problem: keeping the system working.
No cloud deals with this because it's too hard. But your cloud is extremely simple, so it might work. Ideally, every resource in your cloud (exe.dev) should work this way. From your team membership settings, to whether a proxy is public, the state of your VM, your DNS settings, the ssh keys allowed, email settings, http proxy integration / repo integration settings / their attachments, VM tags & disk sizes, etc. Over time your system will add more pieces and get more complex, to the point that implementing these system protections will be too complex and you won't even consider it. But your system is small right now, so you might be able to get it working. The end result should be less pain for the user because the system protects them from pain (fixing broken things, preventing breaking things), and more money for you because people like systems that don't break. But it's also possible nobody cares about this stuff until the system gets really big, so maybe your users won't care. It would be nice to have a cloud that fixes this tho.
> $20 a month
2025 or 2005, what's the difference?
For that money I can get 5 big bare metal boxes on OVH with fast SSDs, put k0s on them, fast deploy with kluctl, cloudflare tunnels for egress. Backups to a cheap S3 bucket somewhere. I'll never look at another cloud provider.