One RaspBerry to replace my k8s cluster

I have a raspberry Pi 4 that helps me to host several websites and utilities tools, using docker and ansible to provision everything.

I’ll describe here my joys and struggles, in the hope that this might encourage you to get one too :)

Hosting a website

This is the main reason why I got this RaspBerry. I have a website for beekeepers (well just one, my father). It has been running on a k8s cluster hosted on digital ocean for some time, and as you can guess, it was quite expensive for a website of that size (50$/month - 2 nodes + 1 load balancer). I was planning to use the RaspBerry to save me some money.

The website itself is not really big. But it does run using several python/rust micro-services I created. I use this website as a way to ramp up my skills as well (hence why I chose to run it on a k8s cluster… to learn more about the beast, to my wallet expense).

How to start that migration ? Well, first off I tried to run some of those services inside my RaspBerry !

Building docker images for ARM

I use docker to run all of my stack, and I thought this was going to be a piece of cake. How innocent of me !

Building my services’ images was my first big challenge. Up till then, I was building my images from and for an x86 processor architecture. But RaspBerry runs on ARM… so I needed to build my image for that specific platform.

And since I wanted to build my images using a CI, it had to check how to build an image both for both platforms in an automated way.

That’s how I learnt about Qemu, a tool that emulates a bunch of different architectures. It’s able to take a set of instructions and convert it to any other architecture it supports, and allows docker to build multi-platform images.

You will need to use docker buildx for building multiplatform images. You will also not make it out alive without reading the official documentation from docker about multi-platform images.

Overall the experience was nice with python images. But more, much more challenging with rust ones. The binary built by cargo should also be compiled for a specific architecture as well, and it does need a bit more twicking to get it working correctly.

After struggling a lot, learning about the linking phase and so on, I was finally able to run all of my services in my RaspBerry. Now I just needed start them and test that my website is running well… That leads me to the next section, about networking.

Network configuration

Getting reachable from the Internet

I had to configure my internet box in order for it to be reachable from the public Internet, and to redirect all traffic to my RaspBerry. That part depends on your internet provider and the box you are using. It should not be difficult to put in place as long as you have admin access to your network.

DNS and SSL

Being able to be reached through an IP address is good, but you probably want to own a domain that redirect to that IP. And that always means having a SSL certificate to allow https traffic. I wouldn’t do it otherwise anyway.

So I bought my domain on google domains. I had to transfer it to digital ocean because google doesn’t support wildcard certificate with more than 1 level of subdomain (things like *.yyy.mydomain.com).

I used certbot to do handle the certificate creation and renewals. It takes a bit of configuration and then does everything automatically for me. I configured it once and never looked back at it. Works like a charm !

Small note: Certbot is the only service I installed on my raspberry that is not running in a docker container. It’s possible to run it in a docker image but takes more effort to properly move around the certificates.

A load balancer

Once you receive traffic, unless you are only hosting one service, you will need a load balancer acting as a reverse proxy. It takes the traffic from Internet and redirect it to one of your services, based on the configuration from your load balancer.

There’s plenty of choices over there, I selected HAProxy because I never used it before and it’s fully open source.

And it’s really not complex. I set it up quickly and was able to redirect the traffic to my services based on subdomains. It handles traffics coming from Internet (using my public DNS) but also from my home network (with another internal DNS configured on my ISP box). It’s kind of neat because it allows me to have some services only reachable from my home network.

Securing your network

My raspberry is now available to the whole Internet. This is a good news, but it is also really scary because anyone can come knock at the door. And you should really be careful about what you’re making publicly available and how secured it is. There’s a lot of bad actors out there who are scanning the whole internet looking for exploits.

Those bots usually target range of IPs and try known vulnerabilities. To protect myself against them, I reduced the number of ports opened on the raspberry to limit the attack surface. The only open ports are related to http traffic (port 80 and 443) and the ssh server (port 22). The rest is closed.

As of my services, they are all running in docker, in a private docker network and are not exposed to the outside.

The load balancer is, naturally, the only service exposed outside of my docker network. It has been configured to take in only SSL trafic. Once in my docker network, everything is working through http.

Password manager

I am hosting my own bitwarden server in my RaspBerry. I actually use Vaultwarden because it’s less greedy on resources. And I never had any issue with it.

It took me one evening to set it up and I hadn’t look back since then. The thing is rock solid. Passwords are saved locally, on the RasoBerry, so this might trigger a warning in your head about backup… Good thing it’s the next section !

Backup of data

Everything is saved in my RaspBerry. Database, password managers, grafana dashboard… Would be a shame to loose this data.

Initially I was scared of doing that, and I had my postgres database hosted in a vm from Digital Ocean. But after some time, I realized that I could hugely improve performances by having postgres running directly in my RaspBerry (rather than some server in another country).

To reassure me a bit about the data, I started to backup things in a S3 bucket. I came up with a litle script that would basically save my postgres data on a S3 bucket, and it’s running on a daily basis… Given the activity on the website and its number of users, a daily backup is definitively acceptable.

I use the aws cli to do that, with the æws sync command.

That script is running daily through a cronjob and can backup any kind of data to S3, I use it to store all kind of data. Of course I pay a bit (5€ per month maybe ?). Most of the cost is coming from my NAS backup (lots of photos/videos) and the cost keeps increasing a bit moth after month. I’ll probably move to something like AWS glacier to save some bucks.

Monitoring

I wanted to know what’s happening in the RaspBerry. For some time I felt blind in term of everything (cpu/mem usages, incoming/outgoing traffic, services health…).

So I installed a prometheus and grafana on it. Promotheus is allowing me to collect various metrics, whether it’s from docker containers or the RaspBerry itself. The grafana instance allows me to have a nice view on it.

To collect metrics on the RaspBerry itself, I am using the prometheus node_exporter tool. It’s very convenient, and even allows me to send some custom metrics. For example I have some metrics sent everytime the backup script from above is running, with a success flag.

If on top of that I manage to add some alerting, that would be perfect.

Ansible and RaspBerry, that’s a match

Well, it starts to add up quite a bit no ? All of this configuration, it would be a shame to have to redo it over again. So I wanted a way to automatically provision my RaspBerry, and here comes Ansible !

As I was looking for a tool to manage my RaspBerry infrastructure, I started to play with Ansible and it looked exactly like what I wanted. It uses ssh to connect to my RaspBerry and to apply a bunch of commands that I have configured myself.

Literrally everything I have installed/configured/executed in my RaspBerry is done through Ansible. Except for the user used by Ansible. I created it manually and generated its ssh keys myself.

Ansible allows you to execute everything. It has a whole sea of plugins available to manage stuff (docker, file system, git, cron… you name it), but it can also run your own bash commands.

Another thing I really like about Ansible is that there is a native way to manage secrets. You can encrypt files with secrets and reuse them in different parts of the infrastructure. Not that it matters a lot, but I try to open source everything I do as a way for me to follow best practices.

Conclusion

I think I am already making a decent usage of my RaspBerry. It already helps me to save money, and it helps me to learn a lot of stuff I wouldn’t have known otherwise. Being able to monitor the RaspBerry showed me I was barely using any CPU/memory, and this still surprises to this day given all the things I already have running inside.

I consider having a RaspBerry is a great opportunity for developers to keep a hand on servers hosted on premise. In a world where things are most commonly in the cloud, it’s a nice way to diverge from that trend and discover new ways of doing things :) But keep in mind that it’s a direct entrypoint in your home and you should really be careful about how you configure it.