Personal Microservices Infrastructure Project

Join me as I build a personal datacenter, a “homelab” suitable for playing with microservices infrastructure. This is the first chapter of a story where people reclaim their power from their digital lords, and grow into equal peers in the digital realm.

I am starting my portion of this journey by arranging computer and software components to build a datacenter to serve as the foundation of a "Personal Cloud." This personal cloud will create my own digital sovereign territory from which I can steward my information and define my interaction with my peers. This personal cloud is intended as a proof of concept of what a better internet could look like.

Introduction

There was a time when only a few wealthy people had books. There was a time when only a few people had telephones. There was a time when only a few people had computers. We are now in an era where many people have easy access to these modern information technologies. In this era, there are only a few people who have their own datacenter. Everything is in “the cloud”, a new form of digital feudalism reigns supreme. Most people life as serfs, graciously accepting the few crumbs of value that trickle down in the form of apps and services while those people controlling them accumulate wealth and power.

The Nitty Gritty

This project aims to create a comprehensive, real-world microservices architecture, focusing on automation, scalability, and best practices in modern DevOps.

I'll write developer journal entries as blog posts, as well as organize them into a more structured "how-to" documentation suitable for recreating the project or following along. I’ll also write the occasional essay about the meaning and ‘philosophy’ motivating my actions. 

As of now, I've done some preliminary research and have come up with this plan of action. As I go ahead and my hands dirty, I imagine some of this plan will change, either slightly or dramatically as I learn from my mistakes and refine my understanding.

Tentative Plan of Action

1. Physical Server Setup

• Personal Infrastructure Part 1: Introduction and Basic Ansible Setup
• Personal Infrastructure Part 2: Setting up Secret Storage for Ansible
• blog/03-simple-automation
• Personal Infrastructure Part 4: Creating and Storing Initial Secrets

2. CI Runner Setup:

• Install and configure a CI tool (likely Drone CI)
• Migrate legacy services to Ansible, deployed by CI
• Automatically run Ansible on infrastructure code changes.

3. Core Services

In this phase, I will deploy a small cluster of virtual machines that will host “core services” that will make deploying production services easier and more repeatable.

• Use Packer for creating standardized VM images
• Implement Terraform for infrastructure provisioning
• Setup Headscale for secure networking
• Set up Nomad for initial workload orchestration
• Deploy Vault for secrets management
• Set up monitoring tools (e.g., Prometheus, Grafana)
• Implement logging solution (e.g., ELK Stack)

4. Advanced Orchestration:

• Create a separate Nomad cluster for application workloads
• Potentially set up K3s (lightweight Kubernetes) within Nomad

5. Network and Security:

• Implement ingress controllers and load balancers
• Set up network policies and firewalls

6. Automation and Scalability:

• Develop scripts and workflows for rapid cluster creation
• Implement auto-scaling and self-healing capabilities

7. Begin deploying Services

In this phase, I will begin with the working foundation to deploy modern applications that I’ve build in subsequent steps. From here I will begin to make this cloud more personal, and develop applications and services on top of it that are valuable to me, and demonstrate the value of a personal cloud.

Learning Objectives

Why am I doing this? I'd like to learn these tools, and build something useful for myself and others. I have a large project in mind, and I will discuss it after I build the initial proof of concept "personal cloud".

Put into dry bullet points, I'd like to:

• Gain hands-on experience with modern DevOps tools and practices
• Understand whether microservices architecture is a good fit for my project
• Develop skills in automation, security, and scalability in distributed systems
• Build something useful for myself and others

Sharing Knowledge

Throughout this project, I will:

• Document each step in this developer journal or blog
• Create some guides and tutorials
• Share challenges faced and solutions implemented. I'd like to specifically share my perspective, from someone new to modern DevOps, though no stranger to software engineering.
• Possibly create video content