One RaspBerry to replace my k8s cluster

My raspberry Pi 4 helps me to host several websites and utilities tools, using docker and ansible to provision everything.

I’ll describe here my joys and struggles, in the hope that this will encourage you to get one :)

Hosting a website

That’s the main reason why I got this RaspBerry. I have a website for beekeepers (well just one, my father). It was running in a k8s cluster hosted on digital ocean, and as you can guess it was quite expensive for a website of that size (50$/month - 2 nodes + 1 load balancer). I was planning to use the RaspBerry to save me some money.

The website itself is not really big. But it does run using several python/rust micro-services I created. I use this website as a way to ramp up my skills as well (hence why I chose to run it on a k8s cluster… to learn more about the beast, to my wallet expense).

How to start that migration ? Well, first off I tried to run some of those micro-services inside my RaspBerry !

Building docker images for ARM

I use docker to run all of my stack, and I thought this was going to be a piece of cake. Innocent me !

Building my services’ images was my first big challenge. Up till then, I was building my images from and for an x86 processor architecture. But RaspBerry runs on ARM… so I needed to build my image for that specific platform.

And since I wanted to build my images using a CI, it had to check how to build an image both for both platforms in an automated way.

That’s how I learnt about Qemu, a tool that emulates a bunch of different processor architectures. It’s able to take a set of instructions and convert it to any other architecture it supports, and allows docker to build multi-platform images.

You will need to use docker buildx for building multiplatform images. You will also not make it out alive without reading the official documentation from docker about multi-platform images.

Overall the experience was nice with python images. But more, much more challenging with rust ones. The binary built by rust should also be compiled for a target architecture as well, and it does need a bit more twicking to get it working correctly.

After struggling a lot, learnt about the linking phase and so on, before finally being able to run all of my services in my RaspBerry. Now I just needed start them and test that my website is running well… That leads me to the next ection, about networking.

Network configuration

Getting reachable from the Internet

I had to configure my internet box in order for it to be reachable from the public Internet, and to redirect all traffic to my RaspBerry. That part depends on your internet provider and the box you are using. It should not be difficult to put in place as long as you have admin access to your network.

DNS and SSL

Being able to be reached through an IP address is good, but you probably want to own a domain that redirect to that IP. And that always means having a SSL certificate to allow https traffic. I wouldn’t do it otherwise anyway.

So I bought my domain on google domains. I had to transfer it to digital ocean because google doesn’t support wildcard certificate with more than 1 level of subdomain (things like *.yyy.mydomain.com), and this is what I wanted.

I used certbot to do that. It handles all the stuff automatically for me, including renewals. I configured it once and never looked back at it. Works like a charm !

Small note: Certbot is the only service I installed on my raspberry that is not running in a docker container.

A load balancer

Once you receive traffic, unless you are only hosting one service, you will need a load balancer acting as a reverse proxy. It takes the traffic from Internet and redirect it to one of your services, based on the routing configured in the load balancer.

There’s plenty of choices over there, I selected HAProxy because I never used it before and it’s fully open source.

And it’s really not complex. I set it up quickly and was able to redirect the traffic to my services based on subdomains. It handles traffics coming from Internet (using my public DNS) but also from my home network (with another internal DNS configured on my ISP box). It’s kind of neat because it allows me to have configuration about public/private endpoints in the same place.

Securing your network

My raspberry is now available to the whole Internet. This is a good news, but it is also really scary because anyone can come knock at the door. You should really be careful about what you’re making publicly available and how secured it is. There’s a lot of bad actors out there who are scanning the whole internet looking for exploits.

Those bots usually target range of IPs and try known vulnerabilities. To protect myself against them, I reduced the number of port opened on the raspberry to limit the attack surface. The only open ports are related to http traffic (port 80 and 443) and the ssh server (port 22). The rest is closed.

As of my services, they are all running in docker, in a private docker network and are not exposed to the outside.

The reverse proxy is the only service exposed outside of my docker network. It has been configured to take in SSL trafic and decrypt it. Once in my docker network, everything is working through http.

Backup of data

Initially I had my postgres database hosted in a vm from Digital Ocean. Initially, it was the only service I didn’t want to host in my RaspBerry because I didn’t want to manage the backup of my data.

It’s only after some time with my services running in the RaspBerry that I realized I could hugely improve performances by having postgres running directly in my RaspBerry (rather than some server in another country). But I was still scared about storing data inside the RaspBerry. So I came up with a litle script that would basically save my postgres data on a S3 bucket on a daily basis… Given the activity on the website and its number of users, a daily backup is definitively acceptable.

I created that through a python script. I did try to use the aws cli instead but I didn’t manage to install it on the RaspBerry (ARM again…) and the python alternative using botocore seemed easier to me.

That script is running daily through a cronjob and can backup any kind of data to S3, I use it for other use cases than postgres as well :)

Password manager

I am hosting my own bitwarden server in my RaspBerry. I actually use Vaultwarden because it’s less greedy on resources. And I never had any issue with it.

It took me one evening to set it up and I hadn’t look back since then. The thing is rock solid. I am also backing up my encrypted passwords in a s3 bucket as well !

Monitoring

I wanted to know what’s happening in the RaspBerry. For some time I felt blind in term of everything (cpu/mem usages, incoming/outgoing traffic, services health…).

So I installed a prometheus and grafana on it. Promotheus is allowing me to collect various metrics, whether it’s from docker containers or the RaspBerry itself. The grafana instance allows me to have a nice view on it.

I didn’t set up everything yet but I plan to monitor upgrades and cronjob executions (the backup) thanks ot it. If on top of that I manage to add some alerting, that would be perfect.

Ansible and RaspBerry, that’s a match

Well, it starts to add up quite a bit no ? All of this configuration, it would be a shame to have to redo it over again. So I thought about a way to automatically provision my RaspBerry, and here comes Ansible !

As I was looking for a tool to manage my RaspBerry infrastructure, I started to play with Ansible and it looked exactly like what I wanted. It uses ssh to connect to my RaspBerry and to apply a bunch of commands that I have configured myself.

Literrally everything I have installed/configured/executed in my RaspBerry is done through Ansible. Except for the user used by Ansible. I created it manually and generated its ssh keys myself.

Ansible allows you to execute everything. It has a whole sea of plugins available to manage stuff (docker, file system, git, cron… you name it), but it can also run your own bash commands.

Another thing I really like about Ansible is that there is a native way to manage secrets. You can encrypt files with secrets and reuse them in different parts of the infrastructure. Not that it matters a lot, but I try to open source everything I do as a way for me to follow best practices.

Conclusion

I think I am already making a decent usage of my RaspBerry. It already helps me to save lots of money, and it helps me to learn a lot of stuff I wouldn’t have known otherwise. Being able to monitor the RaspBerry showed me I was barely using any CPU/memory, and this still surprises to this day given all the things I already had running inside.

I consider having a RaspBerry is a great opportunity for developers to keep a hand on servers hosted on premise. In a world where things are most commonly in the cloud, it’s a nice way to diverge from that trend and discover new ways of doing things :)