Infrastructure, tools, and systems from the trenches

Note : This is part four of a series of posts describing how to write “Production Grade Webservices in Go”. Here’s Part - 1, The Service , Part - 2, The Store , Part - 3, Transports. In part-3, we built a working HTTP service that passed the tests. But we’re still lacking the option to actually “run” it. While, this may appear like a trivial task, it’s actually quite important to get this right. Something I feel rather strongly about is the need to avoid package level variables and global state in Go programs. Like plenty of other things, Peter Bourgon says it perfectly in this post: ...

Note : This is part three of a series of posts describing how to write “Production Grade Webservices in Go”. Here’s Part - 1, The Service and Part - 2 The Store if you haven’t read those. We’ve reached a point where we have properly laid out the business logic and the storage implementation for our service. Now, we’re going to move on and talk about transports. As the name suggests, a transport is essentially transporting data over the network in a pre-defined format. JSON over HTTP is one such transport. gRPC is another. We’re going to start with JSON over HTTP simply because of how popular it is. I may write a separate post that adds a gRPC transport to this very service later, but for now, HTTP-JSON. ...

Note : This is part two of a series of posts describing how to write “Production Grade Webservice in Go”. Here’s Part - 1, The Service if you haven’t read it. The previous post ended with a defined structured for our service and some basic testing. It was, however, lacking what is a very important component for most webservices, a datastore. If you’ve used frameworks to write services in the past, you’re probably familiar with abstractions like Hibernate/DjangoORM etc. While Go has the option of working with similar alternatives (GORM, Pop), I generally find building a simple custom abstraction over the database to be better. Unless you have a lot CRUD like APIs with plenty of models, direct is, in my opinion, better. See this post for more information. ...

Production Grade is a term thrown around a lot these days. Every organization seems to have it’s own definition for what qualifies as “Production Grade” or “Production Ready”. In these series of posts, I’m going to present my take on the topic and the bare minimum of what I think qualifies under this definition. These posts assume a working knowledge of Go and are not meant for total beginners to the language. ...

So, I’ve recently had to work with the Azure-SDK for a few small tasks. I’ve worked with AWS and GCP SDKs before, and while they have their problems it wasn’t too hard to figure them out. Azure wasn’t quite the same. Now, I am new to Azure, but I did not expect to spend 30 minutes to get a simple VM listing to work. The Authentication docs seemed fairly straight forward. I just need my credentials now. Okay, google, tell me how to do that. I read the official docs and they did not seem very easy to navigate. Eventually, I figured it out. Here’s the gist of it for anyone still wondering: ...

I’ve always been a big fan of Caddy, and it just got a 2.0 release! In the meantime, I have developed another throwaway service called Archy, and naturally, it deserves the best Ops can offer. So, here’s the scenario. I need a simple HTTPS service exposed to the world. I have a VPS with a public IP. Of course, I’ll celebrate Caddy’s 2nd birth and write a simple Caddyfile. archy.tux-sudo.com reverse_proxy /* 127.0.0.1:15999 Done. Cool. See you later. ...

Code Coverage, Static Code Analysis, Tests, Releases? What else? Probably a few more things, but you get the point. These are all things that are expected to be part of a modern day development workflow. While this is becoming the norm in companies, a lot of pet open-source projects still skimp on these things. And frankly, I did too. I wasn’t going to bother hosting my own Jenkins behemoth just so my soon-to-be abandoned project could run a few builds. But with practically everything available as a SaaS (and in most cases, free for open source projects), I had no excuse anymore. Travis/Circle/GitlabCI/AzurePipelines etc. were already quite common and with GitHub’s entry into the fray, there really is no reason not to have atleast a simple CI workflow on your project. ...

One thing has always bothered me while writing tests is the lack of a real datasource to run tests against. Most projects I’ve worked with in the past have either mocked responses from a datastore or used a “common” datastore to perform tests against. While mocks are good for quick unit tests, I still prefer using a “real” datasource, especially for integration tests. While taking Bill Kennedy’s Ultimate Go training a while ago, I saw Bill recommend a testing approach which involved “spinning up” a database container right from within your test code, run your tests, and clean-up. See this for reference. I love this approach. It provides a wonderful way to do everything from within go test without meddling with external scripts, thereby providing much greater interoperability. My only gripe with this is the fact that it uses os.exec to command communicate with the docker daemon. The approach is sound, however, I find forking out to use docker-cli to be a bit clunky. Docker’s client for go is pretty good and offers imho, a better way to achieve the same. It adds a dependency to the code, however, I believe that is a good thing. The real dependency here is the daemon on the underlying system. Taking a dependency in Go code, makes the management of the API easier and offers more control than relying on the cli api silently. Moreover, if you only use the dependency in your _test.go files, it won’t impact the final build binary (or that’s my understanding atleast, someone correct me if I’m wrong). In most cases the exec approach is perhaps good enough and the right thing to do, but I have an alternative. ...

So then, here is the deal. I write tiny microservices that may or may not serve any purpose. Despite how meaningless they might be, I give them the perfect operations treatment. ...

This one is a gamechanger. It really is. On a recent Golang obsessed Github browsing spree, I read the following project description: “Fast, cross-platform HTTP/2 web server with automatic HTTPS”. I’ve read that before, and more often than not ended up disappointing myself and resorting to hacks like [this] (https://blog.tux-sudo.com/posts/letsencrypt-nginx-docker/) to create an automatic TLS system for my websites. There are other ways, but almost all of them were semi-automatic in the long run (yay cron!). I decided to try out [Caddy] (https://caddyserver.com/). ...