Building autoscaling CI infrastructure with Azure Kubernetes

Ever wanted to create a build agent factory where you do not have to care about how many build agents you need at a given point? With this post I want to share my experience setting up a dedicated CI runner infrastructure with the Azure + Pipelines ecosystem. The main features of the solution are automated scaling, ephemeral build agents, docker based environments, minimal operation responsible and strong pay-per-use billing concepts. Basic knowledge of Docker and Kubernetes should exists - you should know what they are.

Hosting Gitea and Drone with Docker

This post will walk you through setting up a self hosted git based continuous integration environment on a two machine setup - assuming you already have two virtual machines at your disposal. Using Gitea* for git hosting and contribution management and Drone for docker-based build jobs, this will guide you through creating docker-compose* files as well as configuring the individual services and getting SSL certificates via traefik. Docker and docker-compose knowledge is required for this tutorial. It mostly focuses on the correct configuration of all the services at play here and not explaining their basic functionality.

Configure Azure Active Directory as Authentication Provider for Gitea

Gitea is an open source git hosting platform, written in Go forked from the Gogs project and seems to be highly inspired by GitHub. If you want to host your own git platform and want to use Azure Active Directory as an authentication provider - this post might help you.

Automated dev workflow for using Data Science VM on Azure

tl;dr; I put together a bunch of scripts on Github that let you deploy a VM from your command line as well as sync code from your local directory to the VM easily to be able to use local IDE and git but execute on the powerful remote machine. Perfect for Data Science applications based around jupyter notebook.

In my previous blog post I explained how to do Terraform deployment of an Azure Data Science Virtual Machine.

Deploy Datascience infrastructure on Azure using Terraform

In this article I will talk about my experience building my first infrastructure deployment using Terraform that does (a little) more than combining off-the-shelf resources.

The stack we will deploy 📦

Lately I’ve been looking at a lot of Microsoft Azure services in the big data area. I am looking for something to replace a Hadoop based 🐘 data analytics environment consisting mainly of HDFS, Spark & Jupyter.