Self-hosting fixes this

If you're a regular Mutiny user you might've noticed some downtime over the last few weeks. This sucks for any product, but it particularly sucks for a product that's been designed from day one to be self-custodial and sovereign.

What happened

Basically, our original VSS service that provides end-to-end encrypted cloud storage for user wallets was down. The wallet can operate without VSS, but we've encountered bugs and inconsistencies when relying on browser storage alone, especially on iOS. By making sure all state was backed up to the cloud we can be confident all data is saved. Cloud backups also have the advantage of allowing for easy wallet restores on any device with just your 12-word seed phrase.

Consistent storage is integral to operating a lightning node. Each lightning channel is a constantly updating consensus between two nodes. If one node has incorrect, out of date, or missing knowledge about the channel state, the best recourse is usually a force close of that channel. This costs fees and wastes everyone's time. In the worst case, incorrect channel information can result in loss of funds for the faulty party.

VSS fixes this.

And it was all working great, but then the database that backs VSS started to go down randomly. Which brought the lightning functionality of our wallets to a halt.

This is where we really went wrong: we built the VSS storage on top of a beta database product from Cloudflare called D1. We loved the developer experience, performance, and scaling we got with D1. But when our D1 instance started going offline randomly, for indeterminate amounts of time, that naturally ruined our fondness for the technology.

The problems started when we were storing roughly 50 megabytes in D1, but by the time we hit 100 megabytes, the outages stretched to multi-day. We're not sure exactly what went wrong with our storage instance, but one potential problem is the high quantity of writes the current version of LDK does for every update.

Thankfully no user state was lost or corrupted during these outages. But something had to change.

The solution

We (Ben) rewrote the VSS service so it can be deployed to any normal host and backed by a standard Postgres database. We learned our lesson.

This new storage has now been deployed and is working great. The database is highly available, and the service is across multiple regions for redundancy and latency reasons. We put a lot of work in to make sure this is solid. Like we said, this is an essential service for our wallet.

Performance for storage will also likely improve with the release of LDK 0.0.117, which reduces the number of writes required to save lightning state and should be released days from now.

Self-hosting

One great upside of how we rewrote storage is it's now much easier to self-host.

Our long-term plan with Mutiny has always been to make the wallet self-hostable. You should be able to deploy your own copy of Mutiny to yourownwebsite.com and remove a whole lot of trust from the equation of using Mutiny as your wallet. You control what code is deployed when, and where it's run.

Of course, we have to balance that goal with the immediate challenge of giving non-self-hosters a great experience.

Now that VSS is just a normal service with a normal database, it's much easier to recommend self-hosting as an option.

In fact, we've used this opportunity to put together deploy instructions for the primary components of Mutiny wallet. So far it's fairly easy to deploy:

1. The frontend (it's just a static website)
2. VSS (our new vss service)
3. Websocket Proxy (how Mutiny communicates to the wider Lightning network from a web context)

Some services are still a bit more challenging, or more expensive to deploy:

1. Rapid Gossip Sync
2. Esplora (also self-hostable, but you need a beefy server with a lot of storage)
3. LSP (Mutiny works without an LSP, just remove the LSP in the settings, but you have to manage liquidity manually which it's not optimized for. We're working on adding additional LSP options.)

For any of those services you don't want to self-host, you can just rely on the existing services we provide for Mutiny Wallet.

If self-hosting Mutiny sounds like a good time to you, check out mutiny-deploy for instructions. We'll work on getting one-click deploy options for services like Fly.io and Clovyr. We're also working to make Mutiny available as an Umbrel app soon. For now, the best way to get your own copy of Mutiny running is probably a VPS like Digital Ocean.

Self-hosting will likely always be more work than using our official instance at app.mutinywallet.com. But the upside is self-determination. Our code is MIT licensed, once you grab your own copy there are no take backs.

Just like deploying our new version of VSS gave us more control over the uptime and scalability of a key part of our infrastructure, deploying your own copy of Mutiny gives you more control over your own destiny. Our goal is to make Mutiny the best way to use bitcoin. Software so good it deserves to be self-hosted.