I have a few applications running on fly.io, and some of them need to keep data in the file system persistently (more precisely, an SQLite database file and user-submitted data) so that it is not lost after a redeploy or when the Fly Machine running my application is restarted.
To achieve that, I use Fly Volumes which are local persistent storage for Fly Machines, mounted in my server just like a regular directory. This setup works fine, but I began considering how to back up the data stored there.
Volume snapshots are created automatically on a daily basis and
retained for 5 days by default. However, there doesn’t seem to be an easy (or well-documented) way to implement
a custom backup policy. I wanted the ability to copy the entire directory’s content using tools like rsync or upload
it to an S3 bucket on my own schedule.
I explored solutions involving cron jobs running inside my Fly Machine, but they became overly complicated.
These approaches required modifying my Dockerfile to install additional applications, and I wasn’t sure
how to manage the schedule effectively, especially since I configured my machines to auto-stop to save resources.
Direct SSH connections requires me to use flyctl CLI and it wasn’t clear to me how to handle authentication
in this case. After some research, I found that I can use access tokens
to connect to the machines using SSH allowing me to send commands there in an automated way.
Generating your access token
First step is to create an access token that allows me to send commands to my machine without requiring any manual form of authentication. This can be done using the following command:
fly tokens create ssh -n my-token-name
Check the command documentation for more options. The output of
this command will be as the following, where <TOKEN_CONTENT_STRING> will be a very long string that
you need to store and don’t share it publicly.
FlyV1 <TOKEN_CONTENT_STRING>
Add the token to an environment var in the machine you will run the backup script:
export FLY_SSH_TOKEN=<TOKEN_CONTENT_STRING>
Data location
The volume is mounted in /data directory, defined in our application fly.toml file:
[[mounts]]
source = 'app_data'
destination = '/data'
Creating a backup script
With the token, we can now create two scripts: one to run locally on the machine that will receive the backup data, and another to be executed remotely on your Fly Machine. The example scripts are very simple, but you can improve them by adding more capabilities, error handling, uploading the data to S3 buckets, and so on.
# local_backup.sh
# This call is needed to "wake-up" your machine if stopped
curl -s -o /dev/null https://your-app.fly.dev/
# Execute the script that will generate a tarball of /data/ directory
fly ssh console -C 'sh remote_backup.sh' -t $FLY_SSH_TOKEN
# Copy the generated tarball to our local machine
fly ssh sftp get "/root/data_content_$(date +%F).tar.gz" -t $FLY_SSH_TOKEN
# remote_backup.sh
tar czf "/root/data_content_$(date +%F).tar.gz" data/
You need to send remote_backup.sh to your Fly Machine. Adding the following line to your Dockerfile should be enough:
COPY remote_backup.sh /remote_backup.sh
Deploy your application again, and you can run local_backup.sh.
Run it periodically
Now you can add local_backup.sh to your crontab schedule, or even adapt the procedure described here
to be executed in other environments, like defining a GitHub Action or another way to schedule jobs.
I know this is not the most complete way to implement a backup policy, but it is working for my current projects. In the future, as I improve my scripts, I will possibly update this post to make it more complete.