Self-Host on Renne Rocha

Using rclone on Backblaze without spending all your money

blog@rocha.dev.br (Renne Rocha) — Thu, 14 Aug 2025 00:00:00 +0000

Backblaze is an object storage company that offers a S3 compatible cloud object storage with very competitive prices. With prices starting at USD6 / TB / month (or less, as you pay proportionally to the amount of data stored), it was a perfect choice to store live data and backups of my projects and self-hosted services.

One of the services that I use is Immich, a self-hosted photo and video management solution that allows me to have fully control on my photos (even in mobile). It doesn’t allows me to store my photos directly in a object storage as Backblaze, but I decided that it would be a good idea to keep a backup of the files of the server (photos, videos, and some other data that I need to keep safe).

One tool that simplifies the process is rclone, a tool that make it easier to sync data in cloud providers (their maintainers call it as “rsync for cloud storage”). I followed of the instructions they provide to configure Backblaze and then I was able to run the following command:

rclone sync /home/immich/library/ b2:my-immich-backup-bucket

This command syncs automatically only new and changed files (e.g. updated photos metadata), reducing the time needed to backup my photos and keeping them updated, uploading to my Backblaze bucket.

There are many other options that I can use with rclone, and scenarios where this backup scheme doesn’t cover. However, as a starting point to give me a bit more of confidence that I will not lose my photos if my VPS provider suddenly decides to stop providing me services, it is good enough.

I configured crontab to run it hourly, and then I start to get my files synchronized.

So I got the first charge after enabling this scheme! Usually I was paying less than USD2/month, but the first month I got a USD10 bill, then USD60, then USD70. I wasn´t storing terabytes of data. Looking at my billing report I noticed that now I have a not negligible amount of Class C (Charged) transactions, that are responsible for me to pay that much.

We interact with Backblaze using API calls to manage our storage, so we can upload and download files, get information about them and the buckets, and many other actions. Each call is grouped in a different class of transaction (A, B and C), and they are charged differently. Class A transactions are always free, but Class B and C transactions have a daily limit of 2,500 calls each and they charge for extra calls.

When rclone executes, it gets information from the remote location about the files already there to decide if the local files need to be uploaded or not. This is done by the Class C API call b2_list_file_names that returns a list of filenames in chunks of 1.000 files each call.

But when we have our files in nested directories, a call is made for each of them. Immich organizes the uploaded files in hundreds of nested directories, and because of that, every time my cron was executed, thousands of b2_list_file_names calls were made. Given that I was running it hourly, I reached 17M calls in a month easily (around 500k a day). As I have only 2.5K free calls, I found the reason of the amount I was charged.

The solution is to add --fast-list parameter to the rclone command, that requires fewer transactions for highly recursive operations as we have in this scenario. This parameter is mentioned in rclone documentation that explains how to setup it with Backblaze, but I didn’t consider that this is basically a mandatory parameter based on how things are charged there!

So I updated my rclone command to:

rclone sync /home/immich/library/ b2:my-immich-backup-bucket --fast-list

I reactivated cron again and after almost a day, instead of 500K calls, less than 1K calls were made (which is within the free amount I am entitled to). Probably when I have the double of the number of files I have right now, the daily quota I have will not be enough, but then I will know how to deal with it (possibly with a better backup strategy).

Creating backups for fly.io Volumes

blog@rocha.dev.br (Renne Rocha) — Wed, 30 Apr 2025 00:00:00 +0000

I have a few applications running on fly.io, and some of them need to keep data in the file system persistently (more precisely, an SQLite database file and user-submitted data) so that it is not lost after a redeploy or when the Fly Machine running my application is restarted.

To achieve that, I use Fly Volumes which are local persistent storage for Fly Machines, mounted in my server just like a regular directory. This setup works fine, but I began considering how to back up the data stored there.

Volume snapshots are created automatically on a daily basis and retained for 5 days by default. However, there doesn’t seem to be an easy (or well-documented) way to implement a custom backup policy. I wanted the ability to copy the entire directory’s content using tools like rsync or upload it to an S3 bucket on my own schedule.

I explored solutions involving cron jobs running inside my Fly Machine, but they became overly complicated. These approaches required modifying my Dockerfile to install additional applications, and I wasn’t sure how to manage the schedule effectively, especially since I configured my machines to auto-stop to save resources.

Direct SSH connections requires me to use flyctl CLI and it wasn’t clear to me how to handle authentication in this case. After some research, I found that I can use access tokens to connect to the machines using SSH allowing me to send commands there in an automated way.

Generating your access token

First step is to create an access token that allows me to send commands to my machine without requiring any manual form of authentication. This can be done using the following command:

`1`	`fly tokens create ssh -n my-token-name`

Check the command documentation for more options. The output of this command will be as the following, where <TOKEN_CONTENT_STRING> will be a very long string that you need to store and don’t share it publicly.

`1`	`FlyV1 <TOKEN_CONTENT_STRING>`

Add the token to an environment var in the machine you will run the backup script:

`1`	`export FLY_SSH_TOKEN=<TOKEN_CONTENT_STRING>`

Data location

The volume is mounted in /data directory, defined in our application fly.toml file:

[[mounts]]
  source = 'app_data'
  destination = '/data'

Creating a backup script

Given that we have a token, we are now able to execute SSH commands remotely on our Fly Machine. In our scenario, I am compacting the whole content of /data/ directory (where all the data that I want to backup is located) generating a tarball, then I download it locally.

I could create a custom script and copy it to the remote machine if I want to perform more complex tasks, or you can extend/modify this script to perform other tasks (e.g. download the tarball and upload to a S3 bucket).

#!/bin/bash
# backup.sh

# Need to start container in fly.io if it was stopped by inactivity
curl -s -o /dev/null https://your-app.fly.dev/

filename="data_backup_$(date +%F).tar.gz"
fly ssh console -C 'tar cvz /data' -t $FLY_SSH_TOKEN > $filename

Run it periodically

Now you can add backup.sh to your crontab schedule, or even adapt the procedure described here to be executed in other environments, like defining a GitHub Action or another way to schedule jobs.

I know this is not the most complete way to implement a backup policy, but it is working for my current projects. In the future, as I improve my scripts, I will possibly update this post to make it more complete.

Configuring a self-hosted calendar for your small community

blog@rocha.dev.br (Renne Rocha) — Thu, 08 Feb 2024 00:00:00 +0000

I am one of the co-founders of Laboratório Hacker de Campinas (LHC), one of the first hackerspaces in Brazil and during more than a decade we always struggled how to publish and publicize our events.

We started publishing them in our Wiki and crafting some custom scripts to convert certain special pages (following a very specific pattern) into entries of a ICS calendar (people usually didn’t follow the pattern, so many events were missing).

For other years, we use meetup.com. It worked well and the net effect was good to make more people to know about the hackerspace. However they started to charge a not reasonable amount of money that we couldn’t afford to keep our community there.

I was always looking for some kind of open source and (possibly) self-hosted solution. Then I found Gancio, described as “A shared agenda for local communities.”. Created in Italy by people from political hacking movements, it looked exactly the solution we are looking for. It is simple to use, (relatively) easy to maintain, privacy focused and federated.

To deploy it, I decided to use fly.io, a Platform-as-a-Service (PaaS) company that with one Dockerfile and a few configuration files I was able to quickly deploy Gancio there. They require a credit card to create an account, but if your application doesn’t use much CPU|memory|network|storage (which is our case with our calendar), it is very likely that you will never reach the minimum threshold to start paying. So it will be basically free to host it.

UPDATE (2025-03-04): fly.io did some changes in how they charge their users since this post was written (and to be honest, I still didn’t completely understand how it works), so perhaps following the steps that I shared here will not be totally free as before. But due to the ease of configuration and maintenance, I still believe it is a low-cost alternative to be considered.

Disclaimer I am not being sponsored by fly.io. It is just a service that worked well for my needs.

My hackerspace calendar running on fly.io

Pre-requisites

To follow the next steps, you should have a basic knowledge with Linux and being comfortable to use command-line. Also I am assuming that you have a custom domain and know how to configure a DNS hosting to point it to a server.

Preparing the environment

We need three files that contains everything to deploy Gancio:

Dockerfile to build the application)
config.json to configure the application
fly.toml to deploy the application

Start create a directory where you will place all these files. The first file you need is a Dockerfile. I copied it from the project documentation directory and did a few changes to give more control to the configuration file.

FROM node:18-slim AS nodejs-base
RUN apt-get -q update && \
    env DEBIAN_FRONTEND=noninteractive apt-get -y install git && \
    apt-get clean && rm -fr /var/lib/apt/lists/*

FROM nodejs-base AS build
RUN yarnpkg global add --network-timeout 1000000000 --latest --production --silent https://gancio.org/latest.tgz && \
    apt-get clean && rm -fr /var/lib/apt/lists/* && \
    yarnpkg cache clean

FROM nodejs-base
COPY --from=build /usr/local/share/.config/yarn/ /usr/local/share/.config/yarn/
COPY config.json /config.json
RUN ln -s ../share/.config/yarn/global/node_modules/.bin/gancio /usr/local/bin/gancio

ENTRYPOINT ["/usr/local/bin/gancio"]

Next, create a config.json file. The following is similar to the one that I am using, but you can check other configuration options information in Gancio’s docs.

Given that I am not expecting that it will be a server with a high number of connections per second, I choose to keep every data stored in a SQLite file. This also make it easier for me to perform backups.

{
  "baseurl": "https://your.domain.com",
  "hostname": "your.domain.com",
  "server": {
    "host": "0.0.0.0",
    "port": 13120
  },
  "log_level": "debug",
  "log_path": "/data/logs",
  "db": {
    "dialect": "sqlite",
    "storage": "/data/gancio.sqlite",
    "logging": false,
    "dialectOptions": {
      "autoJsonMap": true
    },
    "retry": {
      "match": [
        null,
        null,
        null,
        {},
        {}
      ],
      "max": 15
    }
  },
  "user_locale": "/data/user_locale",
  "upload_path": "/data/uploads",
  "proxy": {
    "protocol": "",
    "hostname": "",
    "host": "",
    "port": "",
    "auth": {
      "username": "",
      "password": ""
    },
    "headers": {}
  }
}

Finally we need to configure the application to deploy it to fly.io using a fly.toml file.

app = 'gancio-my-events'
primary_region = 'gru'

[build]

[[mounts]]
  source = 'gancio_data'
  destination = '/data'

[http_service]
  internal_port = 13120
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0
  processes = ['app']

[[vm]]
  cpu_kind = 'shared'
  cpus = 1
  memory_mb = 1024

This is a minimal configuration that will allow you to deploy the application. But you can check fly.io docs to understand all available options.

Important to know is that we are using Fly Volumes for persistent storage. In this volume we will have the configuration files, database and uploaded data (images) stored.

Launching the application

We are ready to launch our application. I am considering that you already have a fly.io account created and installed flyctl command-line utility that will be used in the next steps.

After sign-in to your account, you can launch your application using:

`1`	`fly launch`

You will be asked a few questions. Answer them and wait. This may take a few minutes to finish. If no errors happen, you will see a message with an internal URL provided by fly.io:

Visit your newly deployed app at https://xxxxxxxxx.fly.dev/

Now you already have Gancio up and running! It is time to configure your domain and create your first admin user.

Domain configuration

To get the IPs your application server is using, use the following command:

`1`	`fly ips list`

You will get a result like the following:

configure DNS to point to these IPs with your domain

v6      xxxx:xxxx:x::xx:xxxx  public (dedicated)  global  51s ago
v4      xx.xxx.xxx.xxx        public (shared)             Jan 1 0001 00:00

Configure a A record (v4) and a AAAA record (v6) in your DNS hosting with your domain. After you change that, it may take minutes (or hours) to be able to access it.

Now it is time to configure the SSL certificates (don’t forget to replace the command with your real domain):

`1`	`flyctl certs create your.domain.com`

And we are done! Your domain is configured and you can securely access your application.

Admin user

Now we need an admin user to manage our instance. Gancio provides a CLI that allows us to manage our calendar, approve events, etc.

Open our application container console:

`1`	`fly ssh console -C bash`

s The CLI binary is located at /usr/local/bin/gancio. Inside the container, create a new user using:

`1`	`/usr/local/bin/gancio users create your@email.com yourpassword --admin`

Now you can login using your@email.com as username and yourpassword as password (change the values with the ones you want to use).

Now you are ready to use Gancio!

Maintenance

After deploying the application, follow their mastodon account so you can be notified about new versions of Gancio.

To deploy a new version, you just need to use the following command:

`1`	`fly deploy`

This command will rebuild your container and publish the latest version of Gancio.

Before doing that, it is highly recommended to do a backup of the current state of the application. As we are keeping all the relevant files in a persistent volume from fly.io you can simply copy the files from there to your local computer.

Unfortunately, there is not a simple command to copy an entire volume from a fly.io volume. The solution I am using now, is to connect into the container, create a tar file of the /data/ directory, and use SFTP to copy it locally.

If I find a better solution to perform backups, I will update this post. If you find anything that doesn’t work anymore, please let me know and I will fix the post.