It already has it [1]. Native litestream that RPC's to the primary sounds interesting though!

[1] https://github.com/phoenixframework/phoenix/pull/4268

We're not shying away from "boring" stuff at all. We just have a small team with bigger priorities that's spread too thin. There's a million things like resizable volumes we need to ship and we're aggressively hiring to get them done.

> there’s no obvious way to resize disks

Yes, this sucks right now. Resizable disks is on our list, we just need somebody to spend a few days on it. Luckily we're hiring platform engineers [1] to work on fun problems like that.

> I actually am bothered that I basically don’t get billed.

We actually had a bug that skipped charging a bunch of accounts. :) Regardless, we're not overly concerned about making $1/mo from small accounts. Large customers more than make up for it. Turns out building something devs _choose_ to use on their free time often leads to using it at work too.

[1] https://fly.io/jobs/platform-product-engineer/

Node, Ruby, and Python come to mind. Oddly enough the alpine versions of haproxy and nginx on Docker hub had issues too.

Failing in one AZ probably means they changed something with DNS servers for that region. We had a similar issue recently when we rolled out a new DNS server that returned longer SOA records which broke python's mysql driver in only some regions during the rollout. Debugging nightmare fuel.

I really want to like Alpine, but we (Fly.io) have seen so many DNS issues with customer images that we’re now recommending Ubuntu or Debian slim. The extra ~50mb is a worthwhile trade off to avoid hard to debug musl-libc issues.

eg https://www.linkedin.com/pulse/musl-libc-alpines-greatest-we...

You control which region your app and it’s volumes are in. Metrics, logs, and volume snapshots end up on servers in the USA. We haven’t addressed data residency for those platform services yet, but we might someday if there’s enough interest.

We'll write about it when the time comes. To be fair, Nomad and Consul have served us well. Most of our troubles stem from abusing them in ways they weren't designed to handle.

As for autoscaling, our hands are tied as long as we're running on Nomad. Right now our autoscaler is nothing more than some ruby that loops over data from prometheus and changes counts in Nomad. It's slow and buggy, but worse we don't have control over where Nomad places VMs or which ones it stops when scaling down.

We're working on a replacement for Nomad (called flyd) that gives us full control over VMs. Once apps are running on that we can do a lot of cool things. Better autoscaling is one, but I'm really excited about suspending idle VMs that our proxy wakes up on demand. That'll cover most use cases without forcing customers to worry about counts or blowing through a budget.

It's unlikely we'll open source the Elixir app, most of it isn't that interesting. But extracting the machine code into an example app is a good idea! Until then, here's a prototype go proxy that launches suspended machines when requests come in: https://github.com/superfly/machine-proxy

We'll be baking that into our proxy soon so we can handle lambda-type workloads.

Image size impacts the cold startup time since it gets fetched and converted into a rootfs before launching. Caching speeds up subsequent launches. Once a machines is created, it can be stopped and resumed, which is _much_ faster. I don't remember the numbers, but it's in the several hundred ms range.

We'll have more to share soon.

The build executes untrusted code and temporarily authenticates with your fly.io and Heroku accounts, so we need a sandbox for isolation. Our Elixir app launches ephemeral VMs using our Machines API so it can offload that can of worms.

We use wireguard-go in our cli for connecting to vms over a private network. It is slower than wireguard in the kernel, but the convenience is hard to beat.

Here's an overview https://fly.io/blog/ssh-and-user-mode-ip-wireguard/

Our remote builders are just fly apps based on this image with an auth proxy in front. Works great https://github.com/superfly/rchab

Markdown + middleman. We use it for docs and landing pages in addition to the blog.

Fwiw we run as many services as we can on our own platform. Mission critical systems like our registry, api, redis servers, and much more are all running as fly apps in firecracker.

That's great to hear!

We're using Nomad to manage a large fleet of firecracker vms at fly.io. It's not as robust as k8s, but I think that's a feature. It's well documented, extensible, and predictable. Not a big community, but hashicorp folks are responsive on GitHub.

We're going to expand there as soon as we can

Not at the moment but it's certainly doable. Our CLI is written in go (github.com/superfly/flyctl) and could be ran as a go-plugin for terraform without rewriting the whole thing. I'd like that :)

Thanks for the kind words! Cockroach is awesome and it's something we'd like to offer someday. Until then checkout FaunaDB.

Your push example is interesting. We don't have a way to connect to redis from outside fly, but you could certainly boot up a tiny app on fly that acts as a proxy from external apps into the fly redis.