[{"content":"I\u0026rsquo;m Tanmay Chaudhry. I write about platform engineering, infrastructure, and developer tools.\nGitHub · LinkedIn · X · RSS\n","permalink":"https://blog.tux-sudo.com/about/","summary":"\u003cp\u003eI\u0026rsquo;m Tanmay Chaudhry. I write about platform engineering, infrastructure, and developer tools.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/tchaudhry91\"\u003eGitHub\u003c/a\u003e · \u003ca href=\"https://linkedin.com/in/tanmay-chaudhry-72b0b827\"\u003eLinkedIn\u003c/a\u003e · \u003ca href=\"https://twitter.com/Tuxybuzz13\"\u003eX\u003c/a\u003e · \u003ca href=\"/index.xml\"\u003eRSS\u003c/a\u003e\u003c/p\u003e","title":"About"},{"content":"Note : This is part four of a series of posts describing how to write \u0026ldquo;Production Grade Webservices in Go\u0026rdquo;. Here\u0026rsquo;s Part - 1, The Service , Part - 2, The Store , Part - 3, Transports.\nIn part-3, we built a working HTTP service that passed the tests. But we\u0026rsquo;re still lacking the option to actually \u0026ldquo;run\u0026rdquo; it. While, this may appear like a trivial task, it\u0026rsquo;s actually quite important to get this right. Something I feel rather strongly about is the need to avoid package level variables and global state in Go programs. Like plenty of other things, Peter Bourgon says it perfectly in this post:\ntl;dr: magic is bad; global state is magic → no package level vars; no func init\nIn my experience, while writing services, this principle gets violated most often in the main package. Especially with initializations and configuration. So, I\u0026rsquo;m going to present a simple method, where everything is initialized in func main itself and flows down from there. And by everything, I do mean everything. There is no magic. Unless there is an explicit parameter on something, it\u0026rsquo;s not receiving that resource. In other words, all the dependencies are explicitly visible and injected from func main.\nStarting off, we\u0026rsquo;re attempt to initialize our HTTP Server (since that\u0026rsquo;s the final goal). Through this, you\u0026rsquo;ll realize everything you need to get the server up. Check out the constructor for the http server:\nfunc NewRainbowHTTP(listenAddr string, svc RainbowService) *RainbowHTTP Hm, so we need a listenAddr, which should probably be an input parameter. So, let\u0026rsquo;s begin with that. There are plenty of flag/environment parsing libraries out there and any one of them should fit the bill as long as they don\u0026rsquo;t violate the \u0026ldquo;no package scoped variables\u0026rdquo; principle. The one I prefer is ff. It\u0026rsquo;s extremely simple, yet effective. So let\u0026rsquo;s write a simple flag/env parser to fetch our listenAddr.\nfunc main() { fs := flag.NewFlagSet(\u0026#34;rainbow\u0026#34;, flag.ExitOnError) var ( listenAddr = fs.String(\u0026#34;listen-addr\u0026#34;, \u0026#34;localhost:8080\u0026#34;, \u0026#34;listen address\u0026#34;) ) ff.Parse(fs, os.Args[1:], ff.WithEnvVarPrefix(\u0026#34;RAINBOW\u0026#34;), ) } Here, we populate the variable listenAddr based on the flag value (or the environment variable RAINBOW_LISTEN_ADDR if the flag isn\u0026rsquo;t specified). This can be expanded to all the configuration values you need to get the application up and running. If you have too many options, it\u0026rsquo;s probably best to encapsulate this parsing in another function and call that from main instead. But again, no shared variables please, pass arguments, receive returns.\nLet\u0026rsquo;s move on to the second bit now. The transport requires an implementation of the RainbowService in addition to this listen address. This is our RainbowSHA256Service struct that we fleshed out earlier. Let\u0026rsquo;s try to initialize that now:\nfunc NewSHA256RainbowService(store Store) *SHA256RainbowService { return \u0026amp;SHA256RainbowService{store} } Hm, this in turn requires an implementation of a Store. Alright let\u0026rsquo;s get a store first. We could use either our InMemStore or the RedisStore, but it\u0026rsquo;s probably better to go with Redis if we want persistance. Let\u0026rsquo;s see what a store needs:\nfunc NewRedisStore(addr string, password string, db int) *RedisStore Only configuration parameters. Okay, this is stuff we can add to our flag set.\nvar ( listenAddr = fs.String(\u0026#34;listen-addr\u0026#34;, \u0026#34;localhost:8080\u0026#34;, \u0026#34;listen address\u0026#34;) dbAddr = fs.String(\u0026#34;db-addr\u0026#34;, \u0026#34;localhost:6379\u0026#34;, \u0026#34;Database address\u0026#34;) dbPassword = fs.String(\u0026#34;db-password\u0026#34;, \u0026#34;\u0026#34;, \u0026#34;Database password\u0026#34;) dbNumber = fs.String(\u0026#34;db-number\u0026#34;, 1, \u0026#34;Database number\u0026#34;) ) With this, we get a nice cascading effect. We have enough to initialize a store. Which means we have enough to initialize our RainbowService, which in turn will allow us to initialize the HTTP Transport. No magic. Everything flows directly and transparently. Let\u0026rsquo;s tie it together:\nstore := service.NewRedisStore(*dbAddr, *dbPassword, *dbNumber) svc := service.NewSHA256RainbowService(store) server := service.NewRainbowHTTP(*listenAddr, svc) server.Start() And there you go. That\u0026rsquo;s a working func main for you. I also like to put this in a package of it\u0026rsquo;s own. See the repository structure and the main file here at this commit which shows the progress so far.\nIf you go ahead and build this with go build -o rainbow-server ./cmd/server, you\u0026rsquo;ll have a working configurable service. Sure, it\u0026rsquo;s awful quiet in the output and logging department (we\u0026rsquo;ll cover that in Part 5 soon), but it works. Try it out.\nThere are a few more things to make this service proper. We need to make sure our service is interruptible, shuts down gracefully and displays some basic start-up information. To do this, we must listen for Operating System signals that ask the application to terminate.\nshutdown := make(chan error, 1 interrupt := make(chan os.Signal, 1) signal.Notify(interrupt, os.Interrupt, syscall.SIGTERM) The snippet above is going to create a new interrupt Channel and signal.Notify is going to post to that channel whenever an interrupt signal is received. We also create a shutdown channel to track fatal errors that should lead to the application shutting down.\nSo, we need to listen to this channel for signals AND run our server as before. This means concurrency, and that means go routines.\ngo func() { err = server.Start() shutdown \u0026lt;- err }() Okay, that\u0026rsquo;s one part handled. Our service is up and running. Now, let\u0026rsquo;s listen to the channels:\nselect { case signalKill := \u0026lt;-interrupt: // log to taste case err := \u0026lt;-shutdown: // log to taste } This select statement is going to block until one of two things happens. Either we receive an interrupt signal or we have an error on our shutdown channel. In both cases, we need to go ahead and shutdown our service gracefully, which is going to look something like this:\nerr = server.Shutdown(context.TODO()) if err != nil { // log to taste } The actual graceful shutdown is left the to the shutdown method on our transport. This exercise is to ensure that it gets called.\nAlso, there are logging statements that need to be sprinkled in. The library choice is up to you again, however, I like using kit/log. Go ahead an initialize a logger, and fill out this the log statements. A sample implementation can be found in this commit with a completed func main\nAll done! We\u0026rsquo;re going to use the logger we initialized for request/response logging with Middlewares in the next post. See you!\n","permalink":"https://blog.tux-sudo.com/posts/production-grade-svc-4/","summary":"\u003cp\u003eNote : This is part four of a series of posts describing how to write \u0026ldquo;Production Grade Webservices in Go\u0026rdquo;. Here\u0026rsquo;s \u003ca href=\"/posts/production-grade-svc-1/\"\u003ePart - 1, The Service \u003c/a\u003e, \u003ca href=\"/posts/production-grade-svc-2/\"\u003ePart - 2, The Store \u003c/a\u003e, \u003ca href=\"/posts/production-grade-svc-3\"\u003ePart - 3, Transports\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eIn part-3, we built a working HTTP service that passed the tests. But we\u0026rsquo;re still lacking the option to actually \u0026ldquo;run\u0026rdquo; it. While, this may appear like a trivial task, it\u0026rsquo;s actually quite important to get this right.\nSomething I feel rather strongly about is the need to avoid package level variables and global state in Go programs. Like plenty of other things, Peter Bourgon says it perfectly in \u003ca href=\"https://peter.bourgon.org/blog/2017/06/09/theory-of-modern-go.html\"\u003ethis post\u003c/a\u003e:\u003c/p\u003e","title":"Production Grade Web Services with Go - Tying it Together"},{"content":"Note : This is part three of a series of posts describing how to write \u0026ldquo;Production Grade Webservices in Go\u0026rdquo;. Here\u0026rsquo;s Part - 1, The Service and Part - 2 The Store if you haven\u0026rsquo;t read those.\nWe\u0026rsquo;ve reached a point where we have properly laid out the business logic and the storage implementation for our service. Now, we\u0026rsquo;re going to move on and talk about transports. As the name suggests, a transport is essentially transporting data over the network in a pre-defined format. JSON over HTTP is one such transport. gRPC is another. We\u0026rsquo;re going to start with JSON over HTTP simply because of how popular it is. I may write a separate post that adds a gRPC transport to this very service later, but for now, HTTP-JSON.\nGo comes with a very nice and usable HTTP server right out of the box. We\u0026rsquo;ll use the server that net/http provides us with a third-party router (gorilla/mux) from some added features.\nLet\u0026rsquo;s define a struct that\u0026rsquo;s going to hold all of this together. The bare-minimum looks something like this:\n// RainbowHTTP is an HTTP server for the underlying RainbowService type RainbowHTTP struct { svc RainbowService server *http.Server router *mux.Router } The business-logic is contained in svc. The server is the actual HTTP Server from the net/http package. And finally, we\u0026rsquo;ve got our router from gorilla/mux. I\u0026rsquo;m sure you\u0026rsquo;ve noticed that this is where our service goes from a Go interface to a webservice. The transport is going to glue all the things together and do the proper plumbing.\nAlright, let\u0026rsquo;s define some methods to deal with HTTP requests. In Go, a method that follows the signature func(w http.ResponseWriter, r *http.Request) is potentially capable of serving web requests. Read more about it here. So let\u0026rsquo;s write a method with a signature like this:\nfunc (s *RainbowHTTP) GetHash() http.HandlerFunc There\u0026rsquo;s a reason why we actually return a HandlerFunc instead of making this method a handler func itself. Doing it this way, allows us to do some setup before the actual handler begins. Mat Ryer explains this concept in his famous post.\nThe handler that we return needs to do a bunch of stuff:\nExtract the business logic request from HTTP params/body to a single struct. Call the underlying business logic service with the appropriate parameters. Form and return a suitable response with proper error handling. All three of the above can be seen the code below. Check out the full function (along with a helper utility) definition and see if you can decipher what it\u0026rsquo;s doing.\n// respond is a internal utility to set proper HTTP responses func (s *RainbowHTTP) respond(w http.ResponseWriter, req *http.Request, data interface{}, statusCode int, err error) { w.WriteHeader(statusCode) // Log the supplied error later if it\u0026#39;s not nil if data != nil { err := json.NewEncoder(w).Encode(data) if err != nil { // Log this later } } } // GetHash returns the HandlerFunc for the Hash route. func (s *RainbowHTTP) GetHash() http.HandlerFunc { // Do some potential setup here. // Not required in this case. return func(w http.ResponseWriter, req *http.Request) { // These structs are limited to the handler scope. // They may seem overly verbose in this case, but they\u0026#39;re worth it when you\u0026#39;re dealing with more complex requests. type Request struct { Str string `json:\u0026#34;str,omitempty\u0026#34;` } type Response struct { Hash string `json:\u0026#34;hash,omitempty\u0026#34;` Err string `json:\u0026#34;err,omitempty\u0026#34;` } r := Request{} qparams := req.URL.Query() if _, ok := qparams[\u0026#34;str\u0026#34;]; !ok { s.respond(w, req, nil, http.StatusBadRequest, errors.New(\u0026#34;No string supplied\u0026#34;)) return } // Grab the first \u0026#34;str\u0026#34; query parameter r.Str = qparams[\u0026#34;str\u0026#34;][0] resp := Response{} hash, err := s.svc.Hash(r.Str) if err != nil { resp.Err = err.Error() s.respond(w, req, resp, http.StatusInternalServerError, err) return } resp.Hash = hash s.respond(w, req, resp, http.StatusOK, nil) } } A similar method, GetReverseHash should complete the service. It\u0026rsquo;s going to be identical, so take a try at it yourself. You\u0026rsquo;ll find it in the final code anyway.\nGreat, so we have two methods that can tied to HTTP routes. To do that, we\u0026rsquo;ll use the router from the struct above and define the routes there. I like to do this in a separate file (usually called routes.go) so that I can see an overview of all the routes in my service in one place. It also serves as a good starting place for anyone new to this code. In this service we have two simple GET methods but the same logic applies to any method.\n// routes registers the handlers to the specific routes func (s *RainbowHTTP) routes() { s.router.HandleFunc(\u0026#34;/hash\u0026#34;, s.GetHash()).Methods(\u0026#34;GET\u0026#34;) s.router.HandleFunc(\u0026#34;/reverse\u0026#34;, s.GetReverseHash()).Methods(\u0026#34;GET\u0026#34;) } Almost there now. Like any good server, we\u0026rsquo;ll need a Start method and Shutdown method. These aren\u0026rsquo;t too complex and something as basic as the following works well for me:\n// Start begins listening for requests on the bindAddr. Blocks. func (s *RainbowHTTP) Start() error { return s.server.ListenAndServe() } // Shutdown gracefully terminates the server func (s *RainbowHTTP) Shutdown(ctx context.Context) error { return s.server.Shutdown(ctx) } If you\u0026rsquo;re confused about the context parameter, you may want to read a little bit about what it does here. It\u0026rsquo;ll play an important role later. The rest is fairly straight forward. We use the ListenAndServe method that the Go std lib provides us (no need for Tomcat etc).\nLet\u0026rsquo;s finish off with a constructor for our server and we\u0026rsquo;ll be done.\n// NewRainbowHTTP returns a new HTTP server for the Rainbow Service. func NewRainbowHTTP(listenAddr string, svc RainbowService) *RainbowHTTP { router := mux.NewRouter() // Build the server server := \u0026amp;RainbowHTTP{ svc: svc, router: router, server: \u0026amp;http.Server{ Addr: listenAddr, Handler: router, }, } // Initialize the routes server.routes() return server } We have a working webservice! (far from production ready, but still). But is it really working? Only one way to tell. Tests! Testing webservices is something that I\u0026rsquo;ve usually done externally (via something like Postman). That was until I started writing Go services. Go has a pretty nice testing mechanism built straight into the standard library and that\u0026rsquo;s what we\u0026rsquo;ll use. Here\u0026rsquo;s a snippet to show how the httptest package works:\nfunc TestHashHandler(t *testing.T) { // This is a method to demonstrate how to test the handlers. // Should ideally be a table driven test with multiple requests later. server := getServer() rr := httptest.NewRecorder() // Create a request req, err := http.NewRequest(\u0026#34;GET\u0026#34;, \u0026#34;/hash\u0026#34;, nil) if err != nil { t.Errorf(\u0026#34;Failed to create an HTTP Request: %v\u0026#34;, err) t.FailNow() } q := req.URL.Query() q.Add(\u0026#34;str\u0026#34;, \u0026#34;thisisastring\u0026#34;) req.URL.RawQuery = q.Encode() server.Handler().ServeHTTP(rr, req) if status := rr.Code; status != http.StatusOK { t.Errorf(\u0026#34;Handler returned non OK status: got %d, want %d\u0026#34;, rr.Code, http.StatusOK) t.FailNow() } // Check the data expectedHash := \u0026#34;572642d5581b8b466da59e87bf267ceb7b2afd880b59ed7573edff4d980eb1d5\u0026#34; resp := struct { Hash string Err string }{} err = json.NewDecoder(rr.Body).Decode(\u0026amp;resp) if err != nil { t.Errorf(\u0026#34;Error Decoding Response JSON:%v\u0026#34;, err) t.FailNow() } if resp.Hash != expectedHash { t.Errorf(\u0026#34;Wrong hash returned in response. Got - %s Want - %s\u0026#34;, resp.Hash, expectedHash) } } While this is not a great test, it has been kept simple intentionally to demonstrate the point. It creates a request and then tests against the response against the expected by actually using the same handler that our HTTP server would. Let\u0026rsquo;s give it a spin:\n17:35:27 tchaudhry@QuasExort93 rainbow master ? go test -v ./... 1 ↵ === RUN TestServiceHash --- PASS: TestServiceHash (0.00s) === RUN TestServiceHashReverse --- PASS: TestServiceHashReverse (0.00s) === RUN TestInMemStore --- PASS: TestInMemStore (0.00s) === RUN TestHashHandler --- PASS: TestHashHandler (0.00s) PASS ok github.com/tchaudhry91/rainbow/service Mmmm..Nice. Play around with the tests for a while and see if you can write them in a more table driven way.\nNow, that we have our transport done. We only need to tie all of this together in our main function. Let\u0026rsquo;s break here and do that next time.\nHere\u0026rsquo;s the git tree for the code that we\u0026rsquo;ve already written.\n","permalink":"https://blog.tux-sudo.com/posts/production-grade-svc-3/","summary":"\u003cp\u003eNote : This is part three of a series of posts describing how to write \u0026ldquo;Production Grade Webservices in Go\u0026rdquo;. Here\u0026rsquo;s \u003ca href=\"/posts/production-grade-svc-1/\"\u003ePart - 1, The Service \u003c/a\u003e and \u003ca href=\"/posts/production-grade-svc-2\"\u003ePart - 2 The Store \u003c/a\u003e if you haven\u0026rsquo;t read those.\u003c/p\u003e\n\u003cp\u003eWe\u0026rsquo;ve reached a point where we have properly laid out the business logic and the storage implementation for our service. Now, we\u0026rsquo;re going to move on and talk about \u003cstrong\u003etransports\u003c/strong\u003e. As the name suggests, a transport is essentially transporting data over the network in a pre-defined format. JSON over HTTP is one such transport. gRPC is another. We\u0026rsquo;re going to start with JSON over HTTP simply because of how popular it is. I may write a separate post that adds a gRPC transport to this very service later, but for now, HTTP-JSON.\u003c/p\u003e","title":"Production Grade Web Services with Go - Transports"},{"content":"Note : This is part two of a series of posts describing how to write \u0026ldquo;Production Grade Webservice in Go\u0026rdquo;. Here\u0026rsquo;s Part - 1, The Service if you haven\u0026rsquo;t read it.\nThe previous post ended with a defined structured for our service and some basic testing. It was, however, lacking what is a very important component for most webservices, a datastore. If you\u0026rsquo;ve used frameworks to write services in the past, you\u0026rsquo;re probably familiar with abstractions like Hibernate/DjangoORM etc. While Go has the option of working with similar alternatives (GORM, Pop), I generally find building a simple custom abstraction over the database to be better. Unless you have a lot CRUD like APIs with plenty of models, direct is, in my opinion, better. See this post for more information.\nThis point is specifically emphasized with our RPC like service where resources are not at the forefront. There is effectively no resource involved and modelling this with an ORM sounds rather clunky. Just think about it in terms of behaviour, what does our datastore really need to provide? Hopefully, you\u0026rsquo;ll find an answer similar to the interface below:\n// Store is an interface defining the operations required from a Rainbow Store type Store interface { Put(blob string, hash string) error Get(hash string) (blob string, err error) } This resembles a key-value store, but we\u0026rsquo;re not obligated to use one. We can use any concrete implementation of the store. It could be any of Redis, Postgres, Mongo, Cosmos etc. As long as you build a wrapper that provides the given behaviour, the underlying database does not matter. I do however recommend, that proper research be done while choosing the database. While, you can technically switch out implementations, in more complicated services it\u0026rsquo;s usually a mess with migrations and what not.\nNevertheless, let\u0026rsquo;s start with the simplest possible datastore, an in-memory map.\n// InMemStore implements an in-memory map that can function as a Rainbow Store type InMemStore struct { db map[string]string } // NewInMemStore instantiates a fresh InMemory Rainbow store func NewInMemStore() *InMemStore { return \u0026amp;InMemStore{db: make(map[string]string)} } // Put stores the value of the blob and associates it with the hash as the key func (store *InMemStore) Put(blob string, hash string) error { store.db[hash] = blob return nil } // Get retrieves the value from the database. Errors if the value is not found. func (store *InMemStore) Get(hash string) (blob string, err error) { if blob, ok := store.db[hash]; ok { return blob, nil } return \u0026#34;\u0026#34;, fmt.Errorf(\u0026#34;Not Found\u0026#34;) } Let\u0026rsquo;s also write some basic tests for it:\nfunc TestInMemStore(t *testing.T) { store := service.NewInMemStore() type TestCase struct { Blob string Hash string } cases := []TestCase{ {\u0026#34;thisisastring\u0026#34;, \u0026#34;572642d5581b8b466da59e87bf267ceb7b2afd880b59ed7573edff4d980eb1d5\u0026#34;}, {\u0026#34;password\u0026#34;, \u0026#34;5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8\u0026#34;}, {\u0026#34;\u0026#34;, \u0026#34;e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\u0026#34;}, // Hash of empty-string https://www.di-mgt.com.au/sha_testvectors.html } // Add a few values for _, c := range cases { err := store.Put(c.Blob, c.Hash) if err != nil { t.Errorf(\u0026#34;Error returned while added entry: %s\u0026#34;, c.Blob) } } // Retrieve the values for _, c := range cases { blob, err := store.Get(c.Hash) if err != nil { t.Errorf(\u0026#34;Error while fetching expected entry: %s\u0026#34;, c.Hash) } if blob != c.Blob { t.Errorf(\u0026#34;Mis-matched Blob for %s, Want=%s, Have=%s\u0026#34;, c.Hash, c.Blob, blob) } } // Check for non existent value _, err := store.Get(\u0026#34;123123\u0026#34;) if err == nil { t.Errorf(\u0026#34;Failed to error for a non-existent value\u0026#34;) } } The tests pass! Great, now let\u0026rsquo;s fix our old test that\u0026rsquo;s still failing. For that we need to use a Store in our service. Back in rainbow.go let\u0026rsquo;s add the dependency in our Service.\n// SHA256RainbowService is a SHA256 implementation for the RainbowService type SHA256RainbowService struct { store Store } // NewSHA256RainbowService instantiates a SHA256RainbowService func NewSHA256RainbowService(store Store) *SHA256RainbowService { return \u0026amp;SHA256RainbowService{store} } We added the store as a dependency to the concrete struct type. Also notice that the dependency is of type Store (the interface) not any particular implementation of the store.\nWith the store available in our service, let\u0026rsquo;s re-think the methods in our service. The hash method was already operational, however, it was not storing the hash as promised. The hashReverse method was unimplemented. Both of these can now be fleshed out now that we have a store!\n// Hash returns the SHA256 sum for the given string func (svc *SHA256RainbowService) Hash(blob string) (hashed string, err error) { sumA := sha256.Sum256([]byte(blob)) hashed = hex.EncodeToString(sumA[:]) err = svc.store.Put(blob, hashed) return } // HashReverse looks up the original string for the given hash func (svc *SHA256RainbowService) HashReverse(hashed string) (blob string, err error) { return svc.store.Get(hashed) } You\u0026rsquo;ll notice that the signature (return value) for the Hash method has changed! While the hash operation cannot error, the storage to the database can! And a failure in the dependency is considered a failure in the method. This means we need to change our interface to reflect the changes as well. The hashReverse method already has the correct signature and can be fleshed out as such.\ntype Hasher interface { Hash(blob string) (hashed string, err error) } We only needed to edit the signature for the method in the Hasher interface. Our Service interface will reflect this change as well.\nNow, that we have all the methods fleshed, let\u0026rsquo;s try our test again after adding the store to the test service and fixing the Hash signature.\ngo test -v ./... === RUN TestServiceHash --- PASS: TestServiceHash (0.00s) === RUN TestServiceHashReverse --- PASS: TestServiceHashReverse (0.00s) === RUN TestInMemStore --- PASS: TestInMemStore (0.00s) PASS ok github.com/tchaudhry91/rainbow/service 0.002s Lovely! Browse the repository at this stage to get a better idea.\nAs an exercise please add the concrete implementation for the database of your choice! Another sample implementation for Redis has been added in the following commit.\nWe\u0026rsquo;re two posts in and we still have not touched on anything that would make this a \u0026ldquo;Web\u0026rdquo; service. I promise you, we\u0026rsquo;ll get there. But how cool is it to work on your business logic unencumbered by any other terminology!? More soon.\nEdit: Part-3 is up!\n","permalink":"https://blog.tux-sudo.com/posts/production-grade-svc-2/","summary":"\u003cp\u003eNote : This is part two of a series of posts describing how to write \u0026ldquo;Production Grade Webservice in Go\u0026rdquo;. Here\u0026rsquo;s \u003ca href=\"/posts/production-grade-svc-1/\"\u003ePart - 1, The Service \u003c/a\u003e if you haven\u0026rsquo;t read it.\u003c/p\u003e\n\u003cp\u003eThe previous post ended with a defined structured for our service and some basic testing. It was, however, lacking what is a very important component for most webservices, a \u003cstrong\u003edatastore\u003c/strong\u003e. If you\u0026rsquo;ve used frameworks to write services in the past, you\u0026rsquo;re probably familiar with abstractions like Hibernate/DjangoORM etc. While Go has the option of working with similar alternatives (\u003ca href=\"https://gorm.io/\"\u003eGORM\u003c/a\u003e, \u003ca href=\"https://github.com/gobuffalo/pop\"\u003ePop\u003c/a\u003e), I generally find building a simple custom abstraction over the database to be better. Unless you have a lot CRUD like APIs with plenty of models, direct is, in my opinion, better. See this \u003ca href=\"https://eli.thegreenplace.net/2019/to-orm-or-not-to-orm/\"\u003epost\u003c/a\u003e for more information.\u003c/p\u003e","title":"Production Grade Web Services with Go - The Store"},{"content":"Production Grade is a term thrown around a lot these days. Every organization seems to have it\u0026rsquo;s own definition for what qualifies as \u0026ldquo;Production Grade\u0026rdquo; or \u0026ldquo;Production Ready\u0026rdquo;. In these series of posts, I\u0026rsquo;m going to present my take on the topic and the bare minimum of what I think qualifies under this definition. These posts assume a working knowledge of Go and are not meant for total beginners to the language.\nAt it\u0026rsquo;s core, a webservice provides some sort of RPC (remote-procedure calls) or REST (Representational State Transfer). REST is often a misused term, that\u0026rsquo;s a story for another time, but in short Actions define RPCs, while REST is all about the resources. For example, /getAllUsers screams RPC, whereas, GET /users is more of a representational state transfer. Both of these may use JSON to serialize their payload and HTTP for underlying transfer, but that does not mean they are the same thing. This isn\u0026rsquo;t going to matter a whole lot with what we\u0026rsquo;re planning to do, but it\u0026rsquo;s something to keep in mind.\nThroughout this series, I\u0026rsquo;m going to be working on a web service that provides two simple RPCs. I have placed strategic commits and linked them in this post to allow for \u0026ldquo;point in time\u0026rdquo; snapshot of the code at that duration. These will be indicated in the text below.\nhash(blob string) (hashed string) // Computes the hash and stores the value in a store. hashReverse(hashed string) (blob string, err error) // Retrieves the \u0026#34;key\u0026#34; from the store by doing a reverse lookup. These are two simple methods, one that returns the hash of a given string and another that attempts to return the reverse of a hash. If you\u0026rsquo;re not very familiar with what Hashes are, the hash method is a simple Put operation and the hashReverse method is a simple Get operation.\nHashing, by definition is a one-way process and a hashReverse should be meaningless. True, but nothing is stopping us pre-computing a lot of hashes and looking them up from a store. This is something called a rainbow-table and it is the reason why all hashes you store should be salted. Read more about it here.\nThat\u0026rsquo;s it. That\u0026rsquo;s the entire business logic right there, and this is what we\u0026rsquo;re going to start with. In fact, in this first post, we\u0026rsquo;re not going to be using ANY of web components. We\u0026rsquo;re focussing entirely on the business logic because that\u0026rsquo;s what is most important. I like to build services from the inside-out and that\u0026rsquo;s what I\u0026rsquo;m going to demonstrate. This a concept promoted by the wonderful Go-Kit toolkit that I find highly effective. We\u0026rsquo;re not going to use Go-Kit for our service (perhaps another series in the future), but the principles apply.\nLet\u0026rsquo;s bootstrap the Go repository to use modules by running:\ngo mod init \u0026#34;github.com/tchaudhry91/rainbow\u0026#34; Next, lay down some structure and create a package service and inside it, a file called rainbow.go.\nAs promised, we start with the business logic definition of our service and place the following interface in our rainbow.go:\ntype RainbowService interface { Hash(blob string) (hashed string) HashReverse(hashed string) (blob string, err error) } That\u0026rsquo;s a nice start. But, we can do better with interface composition. Check out the following and see how it helps adhere to the best practice of keeping interfaces small and composable.\n// Hasher is any type that can return a hash of string type Hasher interface { Hash(blob string) (hashed string) } // HashReverser is any type that can reverse a hash and return the original blob type HashReverser interface { HashReverse(hashed string) (blob string, err error) } // RainbowService is a service to compute hashes and lookup reverse hashes type RainbowService interface { Hasher HashReverser } We now have meaningful interfaces and a composed service. Next up, we\u0026rsquo;re going to create a concrete implementation of this interface for the SHA256 hashing algorithm.\n// SHA256RainbowService is a SHA256 implementation for the RainbowService type SHA256RainbowService struct{} func NewSHA256RainbowService() *SHA256RainbowService { return \u0026amp;SHA256RainbowService{} } // Hash returns the SHA256 sum for the given string func (svc *SHA256RainbowService) Hash(blob string) (hashed string) { sumA := sha256.Sum256([]byte(blob)) return hex.EncodeToString(sumA[:]) } // HashReverse looks up the original string for the given hash func (svc *SHA256RainbowService) HashReverse(hashed string) (blob string, err error) { panic(\u0026#34;unimplemented\u0026#34;) } This is an implementation of Hasher, HashReverser and consequently RainbowService. This service is however, still incomplete, we don\u0026rsquo;t have a working HashReverse method yet due to the absence of a datastore.\nTo demonstrate a bit of TDD (Test Driven Development), I\u0026rsquo;m going to leave the datastore bit off for a little while and instead, let\u0026rsquo;s write some tests! Create a rainbow_test.go and declare it to use a different package service_test. This way we can test our service like a black-box without having access to the internals.\nHere are the two sample tests that test the two methods:\nfunc TestServiceHash(t *testing.T) { svc, err := getService() if err != nil { t.Errorf(\u0026#34;Failed to get sample test service\u0026#34;) t.FailNow() } type TestCase struct { Blob string WantedHash string } cases := []TestCase{ {\u0026#34;thisisastring\u0026#34;, \u0026#34;572642d5581b8b466da59e87bf267ceb7b2afd880b59ed7573edff4d980eb1d5\u0026#34;}, {\u0026#34;password\u0026#34;, \u0026#34;5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8\u0026#34;}, {\u0026#34;\u0026#34;, \u0026#34;e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\u0026#34;}, // Hash of empty-string https://www.di-mgt.com.au/sha_testvectors.html } for _, testCase := range cases { ReceivedHash := svc.Hash(testCase.Blob) if testCase.WantedHash != ReceivedHash { t.Errorf(\u0026#34;Failed for blob - %s, Wanted: %s, Received: %s\u0026#34;, testCase.Blob, testCase.WantedHash, ReceivedHash) } } } func TestServiceHashReverse(t *testing.T) { svc, err := getService() if err != nil { t.Errorf(\u0026#34;Failed to get sample test service\u0026#34;) t.FailNow() } type TestCase struct { Hash string WantedBlob string } cases := []TestCase{ {\u0026#34;572642d5581b8b466da59e87bf267ceb7b2afd880b59ed7573edff4d980eb1d5\u0026#34;, \u0026#34;thisisastring\u0026#34;}, {\u0026#34;5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8\u0026#34;, \u0026#34;password\u0026#34;}, {\u0026#34;e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\u0026#34;, \u0026#34;\u0026#34;}, // Hash of empty-string https://www.di-mgt.com.au/sha_testvectors.html } for _, testCase := range cases { ReceivedBlob, err := svc.HashReverse(testCase.Hash) if err != nil { t.Errorf(\u0026#34;Failed to calculate reverse hash for %s\u0026#34;, testCase.Hash) } if testCase.WantedBlob != ReceivedBlob { t.Errorf(\u0026#34;Failed for Hash - %s, Wanted: %s, Received: %s\u0026#34;, testCase.Hash, testCase.WantedBlob, ReceivedBlob) } } } You can see a couple of methods with some Table driven tests. The first method TestServiceHash passes but the second one TestServiceReverseHash fails as expected. our goal will now be to fix this method so that these tests pass. Hence, TDD.\nThis is a good stopping point and we\u0026rsquo;ll pick-up in the following post for more! Find the entire code tree at this point here: GITHUB - TREE\nEdit: Part 2 is up here!\n","permalink":"https://blog.tux-sudo.com/posts/production-grade-svc-1/","summary":"\u003cp\u003eProduction Grade is a term thrown around a lot these days. Every organization seems to have it\u0026rsquo;s own definition for what qualifies as \u0026ldquo;Production Grade\u0026rdquo; or \u0026ldquo;Production Ready\u0026rdquo;. In these series of posts, I\u0026rsquo;m going to present my take on the topic and the bare minimum of what I think qualifies under this definition. These posts assume a working knowledge of Go and are not meant for total beginners to the language.\u003c/p\u003e","title":"Production Grade Web Services with Go - The Service"},{"content":"So, I\u0026rsquo;ve recently had to work with the Azure-SDK for a few small tasks. I\u0026rsquo;ve worked with AWS and GCP SDKs before, and while they have their problems it wasn\u0026rsquo;t too hard to figure them out. Azure wasn\u0026rsquo;t quite the same.\nNow, I am new to Azure, but I did not expect to spend 30 minutes to get a simple VM listing to work. The Authentication docs seemed fairly straight forward. I just need my credentials now. Okay, google, tell me how to do that. I read the official docs and they did not seem very easy to navigate. Eventually, I figured it out. Here\u0026rsquo;s the gist of it for anyone still wondering:\nNavigate to App Registrations in the Portal and create a new one. Find the newly created \u0026ldquo;App\u0026rdquo; and look for Certificates and Secrets in the left sidebar. You can create a + New Client Secret from here and that should give you what you need to get started. Hang on though, what about RBAC? Yes, so the roles need to be managed from a totally different place. The \u0026ldquo;App\u0026rdquo; is essentially considered a User. So here\u0026rsquo;s what you need to assign a proper role:\nNavigate to the Subscriptions panel on the portal and select the subscription you want to work with. Again, on the left sidebar you should see Access Control (IAM). Here you can create a new Role or use a Built-In and assign it to the \u0026ldquo;App\u0026rdquo; in the Role Assignments tab. Fairly more complex than anything I\u0026rsquo;ve encountered earlier. Perhaps I\u0026rsquo;ll appreciate the benefit for the workflow as I work more with it.\nThe rest was fairly straightforward, except the GoDoc. Despite the GoDoc being more polluted than my eyes could handle, I stuck with it and eventually figured out the basics. Just ctrl + f your way through it. Here\u0026rsquo;s a small example I used to list the VMs in our account:\npackage main import ( \u0026#34;context\u0026#34; \u0026#34;fmt\u0026#34; \u0026#34;time\u0026#34; \u0026#34;github.com/Azure/azure-sdk-for-go/profiles/2019-03-01/compute/mgmt/compute\u0026#34; \u0026#34;github.com/Azure/go-autorest/autorest/azure/auth\u0026#34; ) const subscriptionID = \u0026#34;find-your-own\u0026#34; func main() { authorizer, err := auth.NewAuthorizerFromEnvironment() if err != nil { panic(err) } vmClient := compute.NewVirtualMachinesClient(subscriptionID) vmClient.Authorizer = authorizer ctx, cancel := context.WithTimeout(context.Background(), 100*time.Second) defer cancel() results, err := vmClient.ListAllComplete(ctx) if err != nil { panic(err) } for results.NotDone() { vm := results.Value() fmt.Println(*vm.ID) if err := results.Next(); err != nil { panic(err) } } } Now, while this works, it wasn\u0026rsquo;t before I ran into another problem with version mismatches. I hadn\u0026rsquo;t defined version in my go.mod and that seemed to trip things. So, go ahead declare the versions manually. It\u0026rsquo;s something I\u0026rsquo;d normally do anyway, but not while testing things out. After some rather unhelpful compilation failures and dives into GitHub issues, it was fixed by declaring the dependency explicitly.\n","permalink":"https://blog.tux-sudo.com/posts/azure-sdk-go/","summary":"\u003cp\u003eSo, I\u0026rsquo;ve recently had to work with the \u003ca href=\"https://github.com/Azure/azure-sdk-for-go\"\u003eAzure-SDK\u003c/a\u003e for a few small tasks. I\u0026rsquo;ve worked with AWS and GCP SDKs before, and while they have their problems it wasn\u0026rsquo;t too hard to figure them out. Azure wasn\u0026rsquo;t quite the same.\u003c/p\u003e\n\u003cp\u003eNow, I am new to Azure, but I did not expect to spend 30 minutes to get a simple VM listing to work. The \u003ca href=\"https://github.com/Azure/azure-sdk-for-go#authentication\"\u003eAuthentication\u003c/a\u003e docs seemed fairly straight forward. I just need my credentials now. Okay, google, tell me how to do that. I read the official docs and they did not seem very easy to navigate. Eventually, I figured it out. Here\u0026rsquo;s the gist of it for anyone still wondering:\u003c/p\u003e","title":"Azure SDK Go"},{"content":"I\u0026rsquo;ve always been a big fan of Caddy, and it just got a 2.0 release! In the meantime, I have developed another throwaway service called Archy, and naturally, it deserves the best Ops can offer.\nSo, here\u0026rsquo;s the scenario. I need a simple HTTPS service exposed to the world. I have a VPS with a public IP. Of course, I\u0026rsquo;ll celebrate Caddy\u0026rsquo;s 2nd birth and write a simple Caddyfile.\narchy.tux-sudo.com reverse_proxy /* 127.0.0.1:15999 Done. Cool. See you later.\nI can easily use https://archy.tux-sudo.com for my API using my command line. Everyone (only me) is happy. A coconut fell on my head and I decided to venture into the marvellous world of Javascript. I\u0026rsquo;ll build myself a UI. Why not? Really though, why? Nevertheless, the age old developer horror story of having to resolve CORS struck. Now, it\u0026rsquo;s not my first time, I know what CORS is, how it works etc etc. But my service was released, I didn\u0026rsquo;t want to go back and make changes in there (see, already thrown away). Now, I need to solve the following problems:\nMy service needs to add Access-Control Headers My service needs to respond 200 to OPTIONS on a particular path. Ugh. No, my service doesn\u0026rsquo;t do this and I\u0026rsquo;m happy to keep it that way. BUT Caddy can help. Have a look at my updated Caddyfile:\narchy.tux-sudo.com @entries_options { method OPTIONS path /entries } header { Access-Control-Allow-Headers \u0026#34;*\u0026#34; Access-Control-Allow-Origin \u0026#34;*\u0026#34; Access-Control-Allow-Methods \u0026#34;GET, POST, OPTIONS\u0026#34; } respond @entries_options 200 reverse_proxy /* 127.0.0.1:15999 { } Use the appropriate headers of course and not \u0026ldquo;*\u0026rdquo;, but you get the point. This achieves both. I\u0026rsquo;ve added headers to be put on my final response and it\u0026rsquo;s also handling the OPTIONS call on /entries using my @entries_options matcher properly. My service has no idea. Now, I\u0026rsquo;m sure Nginx or similar might be able to do this as well. But the level of power and clarity in this config is unparalleled imho. Wishing Caddy many many happy years ahead.\n","permalink":"https://blog.tux-sudo.com/posts/caddy-reverse-proxy-ssl-cors/","summary":"\u003cp\u003eI\u0026rsquo;ve always been a big fan of \u003ca href=\"https://caddyserver.com/\"\u003eCaddy\u003c/a\u003e, and it just got a 2.0 release! In the meantime, I have developed another throwaway service called \u003ca href=\"https://github.com/tchaudhry91/archy\"\u003eArchy\u003c/a\u003e, and naturally, it deserves the best Ops can offer.\u003c/p\u003e\n\u003cp\u003eSo, here\u0026rsquo;s the scenario. I need a simple HTTPS service exposed to the world. I have a VPS with a public IP. Of course, I\u0026rsquo;ll celebrate Caddy\u0026rsquo;s 2nd birth and write a simple Caddyfile.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003earchy.tux-sudo.com\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ereverse_proxy /* 127.0.0.1:15999\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eDone. Cool. See you later.\u003c/p\u003e","title":"Easy SSL Termination - CORS Edition"},{"content":"Code Coverage, Static Code Analysis, Tests, Releases? What else? Probably a few more things, but you get the point. These are all things that are expected to be part of a modern day development workflow. While this is becoming the norm in companies, a lot of pet open-source projects still skimp on these things. And frankly, I did too. I wasn\u0026rsquo;t going to bother hosting my own Jenkins behemoth just so my soon-to-be abandoned project could run a few builds. But with practically everything available as a SaaS (and in most cases, free for open source projects), I had no excuse anymore. Travis/Circle/GitlabCI/AzurePipelines etc. were already quite common and with GitHub\u0026rsquo;s entry into the fray, there really is no reason not to have atleast a simple CI workflow on your project.\nWhat I want to talk about in this post however, is about going beyond the regular build and test workflow. Most of the steps below can be extrapolated to pretty much any language, but I\u0026rsquo;ll primarily be operating with Go. All the tools highlighted below cost nothing to operate on open source projects and require no self-hosting.\nBuild Automation Tool Pretty straight forward. If you like writing Makefiles, go ahead and treat yourself. I personally don\u0026rsquo;t. I have a new favourite in Task. It feels very clean and simple. Ships as a single binary and can be easily put inside any public CI system. Here\u0026rsquo;s a sample Taskfile:\n# https://taskfile.dev version: \u0026#34;2\u0026#34; tasks: build-client: cmds: - mkdir release || true - go build -o release/bottle-cli ./cmd/client build-server: cmds: - mkdir release || true - go build -o release/crate ./cmd/server build: cmds: - task: build-server - task: build-client test: cmds: - go fmt ./... - golangci-lint run - go test -v ./... sonar: cmds: - sonar-scanner -Dsonar.login=$SONAR_TOKEN Pretty easy to tell what is going on here. The tasks can then simple be run like task build. Again, if you\u0026rsquo;d like to stick to Make or any other language specific tool, do that instead.\nStatic Analysis / Code Quality Lots of candidates here. I\u0026rsquo;m going to highlight SonarCloud. Again, totally free for open source projects and has been a common feature of companies I\u0026rsquo;ve worked with. You need to install the sonar-scanner to be able to scan your projects. This is easily doable on most CI systems (see example later for GitHub Actions) and a simple configuration file which resembles:\n# Organization and project keys are displayed in the right sidebar of the project homepage sonar.organization=tchaudhry91-github sonar.projectKey=tchaudhry91_bottle sonar.host.url=https://sonarcloud.io sonar.exclusions=crate/pb/crate.pb.go You can setup the organization, projectKey and a grab a sonar token on https://sonarcloud.io.\nI\u0026rsquo;m not going to go deep into all the things you can do with Sonar, but it\u0026rsquo;s really easy to get started and get some nice quality gates and badges set-up.\nOther quality/linting checks tools can also be integrated as required. GolangCI-Lint is another quite nice tool that I use in most of my repos. See example here.\nReleases This, I\u0026rsquo;m going to deliberately keep language specific because of how widely it varies. In short, I\u0026rsquo;ve never had to look beyond GoReleaser for Go. This directly creates a github release based on git tags and also compiles binaries for specified platforms. Lots of knobs available to tweak, but something basic as below should suffice in most cases:\n# This is an example goreleaser.yaml file with some sane defaults. # Make sure to check the documentation at http://goreleaser.com before: hooks: # you may remove this if you don\u0026#39;t use vgo - go mod tidy # you may remove this if you don\u0026#39;t need go generate - go generate ./... builds: - id: \u0026#34;crate\u0026#34; main: ./cmd/server/main.go binary: crate env: - CGO_ENABLED=0 - id: \u0026#34;bottle\u0026#34; main: ./cmd/client/main.go binary: bottle env: - CGO_ENABLED=0 archives: - replacements: darwin: Darwin linux: Linux windows: Windows 386: i386 amd64: x86_64 checksum: name_template: \u0026#34;checksums.txt\u0026#34; snapshot: name_template: \u0026#34;{{ .Tag }}-next\u0026#34; changelog: sort: asc filters: exclude: - \u0026#34;^docs:\u0026#34; - \u0026#34;^test:\u0026#34; Now, we\u0026rsquo;ll trigger the release whenever a Git tag resembling \u0026ldquo;v*\u0026rdquo; gets pushed.\nBind it together I\u0026rsquo;m going to use the example of this project that I started a while ago. We\u0026rsquo;ll see here how to tie up all the tools discussed above and have GitHub actions do it\u0026rsquo;s job.\nI\u0026rsquo;m going to be writing two distinct workflows. The CI workflow that runs on every push/pull request and does all the quality checks and tests. Additionally, there will be a release workflow that will be run only when a specific tag gets pushed.\nCI Workflow\nname: CI on: [push, pull_request] jobs: ci: name: CI runs-on: ubuntu-latest steps: - uses: actions/checkout@v1 - uses: actions/setup-go@v1 with: go-version: \u0026#34;1.13\u0026#34; - run: curl -sL https://taskfile.dev/install.sh | sh - run: curl -sfL https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s v1.21.0 - name: Test run: ./bin/task test env: PATH: $PATH:./bin CGO_ENABLED: 0 - name: Build run: ./bin/task build env: CGO_ENABLED: 0 - name: Sonar Scan uses: sonarsource/sonarcloud-github-action@master env: SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} and a similar Release workflow but with a different on condition:\nname: Release on: push: tags: - \u0026#34;v*\u0026#34; jobs: release: name: Release runs-on: ubuntu-latest steps: - uses: actions/checkout@v1 - run: curl -sL https://taskfile.dev/install.sh | sh - run: curl -sfL https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s v1.21.0 - run: curl -sfL https://install.goreleaser.com/github.com/goreleaser/goreleaser.sh | sh - name: Test run: ./bin/task test env: PATH: $PATH:./bin CGO_ENABLED: 0 - name: Build run: ./bin/task build env: CGO_ENABLED: 0 - name: Sonar Scan uses: sonarsource/sonarcloud-github-action@master env: SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - name: Release run: ./bin/goreleaser env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} Supply the SONARY_TOKEN secret under your github project settings, add both these files into your .github/workflows directory and see the magic happen!\nThere\u0026rsquo;s a lot more you can do with your workflows. Security scans, deployments, perhaps anicillary build and test services (see this post for more details on how to spin databases for testing in CIs). Lots of possibilities and not a $ spent. Have fun!\n","permalink":"https://blog.tux-sudo.com/posts/golang-github-actions/","summary":"\u003cp\u003eCode Coverage, Static Code Analysis, Tests, Releases? What else? Probably a few more things, but you get the point. These are all things that are expected to be part of a modern day development workflow. While this is becoming the norm in companies, a lot of pet open-source projects still skimp on these things. And frankly, I did too. I wasn\u0026rsquo;t going to bother hosting my own Jenkins behemoth just so my soon-to-be abandoned project could run a few builds. But with practically everything available as a SaaS (and in most cases, free for open source projects), I had no excuse anymore.\nTravis/Circle/GitlabCI/AzurePipelines etc. were already quite common and with GitHub\u0026rsquo;s entry into the fray, there really is no reason not to have atleast a simple CI workflow on your project.\u003c/p\u003e","title":"Go: Useful Development Workflows"},{"content":"One thing has always bothered me while writing tests is the lack of a real datasource to run tests against. Most projects I\u0026rsquo;ve worked with in the past have either mocked responses from a datastore or used a \u0026ldquo;common\u0026rdquo; datastore to perform tests against. While mocks are good for quick unit tests, I still prefer using a \u0026ldquo;real\u0026rdquo; datasource, especially for integration tests.\nWhile taking Bill Kennedy\u0026rsquo;s Ultimate Go training a while ago, I saw Bill recommend a testing approach which involved \u0026ldquo;spinning up\u0026rdquo; a database container right from within your test code, run your tests, and clean-up. See this for reference. I love this approach. It provides a wonderful way to do everything from within go test without meddling with external scripts, thereby providing much greater interoperability. My only gripe with this is the fact that it uses os.exec to command communicate with the docker daemon. The approach is sound, however, I find forking out to use docker-cli to be a bit clunky. Docker\u0026rsquo;s client for go is pretty good and offers imho, a better way to achieve the same. It adds a dependency to the code, however, I believe that is a good thing. The real dependency here is the daemon on the underlying system. Taking a dependency in Go code, makes the management of the API easier and offers more control than relying on the cli api silently. Moreover, if you only use the dependency in your _test.go files, it won\u0026rsquo;t impact the final build binary (or that\u0026rsquo;s my understanding atleast, someone correct me if I\u0026rsquo;m wrong). In most cases the exec approach is perhaps good enough and the right thing to do, but I have an alternative.\nRecently, I wrote a small project with a bunch of helpers and an approach to achieve the same using the docker client directly. Spin Me! is a simple tool that can be used to \u0026ldquo;spin up\u0026rdquo; docker database containers followed by auto-configuration, right from within your go code. See the example below for Postgresql:\nout, err := spin.Postgres(context.Background(), nil) if err != nil { fmt.Println(err) return } defer spin.SlashID(context.Background(), out.ID) // Give postgres a few seconds to boot-up, sadly there is no \u0026#34;ready\u0026#34; check yet time.Sleep(5 * time.Second) connStr, err := spin.PostgresConnString(out) if err != nil { fmt.Println(err) return } db, err := sql.Open(\u0026#34;postgres\u0026#34;, connStr) if err != nil { fmt.Println(err) return } defer db.Close() err = db.Ping() if err != nil { fmt.Println(err) return } fmt.Println(\u0026#34;Connected!\u0026#34;) The above example goes through the entire cycle of going through the entire spin-up, connect, clean-up cycle. See the entire API and more examples here Go Doc.\nMost CI systems ship with an environment that allows applications to create docker containers and this allows go test to spawn it\u0026rsquo;s own database on each integration run in a commpletely clean environment.\nAs a bonus, the package comes with a small command-line utility to manage containers.\nspinme -h SpinMe is a wrapper around docker to run common applications. Use this to easily create dependent services such as databases. Usage: spinme [command] Available Commands: down Bring down the given container help Help about any command status Status shows the list of all running services spun via spinme up Start a particular service Flags: --db string Database for local storage (default \u0026#34;/home/tchaudhry/.spinme\u0026#34;) -h, --help help for spinme Use \u0026#34;spinme [command] --help\u0026#34; for more information about a command. See the project repo for more details.\nCheers and happy testing.\n","permalink":"https://blog.tux-sudo.com/posts/ci-docker-dbs/","summary":"\u003cp\u003eOne thing has always bothered me while writing tests is the lack of a real datasource to run tests against. Most projects I\u0026rsquo;ve worked with in the past have either mocked responses from a datastore or used a \u0026ldquo;common\u0026rdquo; datastore to perform tests against. While mocks are good for quick unit tests, I still prefer using a \u0026ldquo;real\u0026rdquo; datasource, especially for integration tests.\u003c/p\u003e\n\u003cp\u003eWhile taking Bill Kennedy\u0026rsquo;s Ultimate Go training a while ago, I saw Bill recommend a testing approach which involved \u0026ldquo;spinning up\u0026rdquo; a database container right from within your test code, run your tests, and clean-up. See \u003ca href=\"https://github.com/ardanlabs/service/blob/master/internal/platform/database/databasetest/docker.go\"\u003ethis\u003c/a\u003e for reference. I love this approach. It provides a wonderful way to do everything from within \u003ccode\u003ego test\u003c/code\u003e without meddling with external scripts, thereby providing much greater interoperability. My only gripe with this is the fact that it uses \u003ccode\u003eos.exec\u003c/code\u003e to command communicate with the docker daemon. The approach is sound, however, I find forking out to use docker-cli to be a bit clunky. Docker\u0026rsquo;s client for go is pretty good and offers imho, a better way to achieve the same. It adds a dependency to the code, however, I believe that is a good thing. The real dependency here is the daemon on the underlying system. Taking a dependency in Go code, makes the management of the API easier and offers more control than relying on the cli api silently. Moreover, if you only use the dependency in your \u003ccode\u003e_test.go\u003c/code\u003e files, it won\u0026rsquo;t impact the final build binary (or that\u0026rsquo;s my understanding atleast, someone correct me if I\u0026rsquo;m wrong). In most cases the exec approach is perhaps good enough and the right thing to do, but I have an alternative.\u003c/p\u003e","title":"Go: Spin-Up Databases for CI Testing"},{"content":"So then, here is the deal. I write tiny microservices that may or may not serve any purpose. Despite how meaningless they might be, I give them the perfect operations treatment.\nHere is an example of such a service: [hash-svc] (https://github.com/tchaudhry91/hash-svc). At it\u0026rsquo;s core, all it does is take a string and returns it hash.\ntchaudhr:cmd/ (master✗) $ ./hash-svc -serverAddr :12000 {\u0026#34;addr\u0026#34;:\u0026#34;:12000\u0026#34;,\u0026#34;msg\u0026#34;:\u0026#34;Started HTTP Server\u0026#34;} {\u0026#34;err\u0026#34;:null,\u0026#34;input\u0026#34;:\u0026#34;123\u0026#34;,\u0026#34;method\u0026#34;:\u0026#34;hashsha256\u0026#34;,\u0026#34;output\u0026#34;:\u0026#34;a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3\u0026#34;,\u0026#34;took\u0026#34;:\u0026#34;41.472µs\u0026#34;} tchaudhr:~/ $ curl http://localhost:12000/hash\\?s\\=123 {\u0026#34;v\u0026#34;:\u0026#34;a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3\u0026#34;} Now, to please my heart, I put this in a container, added a Makefile, hooked up my CI and started pushing out containers with every commit. Check the Makefile, Dockerfile and the travis.yml in this [commit] (https://github.com/tchaudhry91/hash-svc/tree/a6bccb829f18e6d1f9e6f9592e0700d14ad03f4b) to see how I achieved that.\nThere\u0026rsquo;s a problem. I couldn\u0026rsquo;t run this image on my Pi or new bare metal 2.99$/month ARM server on [scaleway] (https://scaleway.com).\nDocker officially supports ARMv7, Golang cross-compiles quite well. Shouldn\u0026rsquo;t be that hard.\nFirst up, modify the Dockerfile to cross-build for ARM instead.\nENV CGO_ENABLED=0 ENV GOOS=linux ENV GOARCH=arm This should now produce binaries that would run on ARMv7-linux (i.e RaspberryPi). CGO is disabled by default when cross-compiling, but in a multi-stage docker build it can cause problems, so we disable it for the non-ARM version as well.\nNext up, modify the final image to be ARM compliant. Replace FROM alpine with an ARM-based image like arm32v7/ubuntu or the more lightweight FROM hypriot/rpi-alpine from the guys over at [Hypriot] (https://blog.hypriot.com/post/setup-simple-ci-pipeline-for-arm-images/)\nAdd the targets to the [Makefile] (https://github.com/tchaudhry91/hash-svc/blob/master/Makefile) and now you should have a build mechanism for ARM containers. There is, however, one problem. While the service itself is cross-compiled and runs on ARM, the final image builder that we use also uses other commands to prepare the image. An example would be RUN apk update \u0026amp;\u0026amp; apk add --no-cache ca-certificates. This command fails to run on my x86 machine because apk itself is an ARM binary. This means we need a processor emulator to be able to build them on TravisCI. Fortunately, the cool guys at [Hypriot] (https://blog.hypriot.com/post/setup-simple-ci-pipeline-for-arm-images/) have documented an easy method to get that done. Read the blog in full if you need more details, but here the command you need to run in your travis.yml to get this to work.\ndocker run --rm --privileged multiarch/qemu-user-static:register --reset\nThis will register the QEMU processor emulator for the ARM binaries (via binfmt_misc). And that\u0026rsquo;s all, custom service containers on your Pi! You can check out the full code over here in this [repo] (https://github.com/tchaudhry91/hash-svc)\n","permalink":"https://blog.tux-sudo.com/posts/armyourcontainers/","summary":"\u003cp\u003eSo then, here is the deal. I write tiny microservices that may or may not serve any purpose. Despite how meaningless they might be, I give them the perfect operations treatment.\u003c/p\u003e","title":"ARM Your Golang Containers"},{"content":"This one is a gamechanger. It really is.\nOn a recent Golang obsessed Github browsing spree, I read the following project description: \u0026ldquo;Fast, cross-platform HTTP/2 web server with automatic HTTPS\u0026rdquo;. I\u0026rsquo;ve read that before, and more often than not ended up disappointing myself and resorting to hacks like [this] (https://blog.tux-sudo.com/posts/letsencrypt-nginx-docker/) to create an automatic TLS system for my websites. There are other ways, but almost all of them were semi-automatic in the long run (yay cron!). I decided to try out [Caddy] (https://caddyserver.com/).\nAfter about an hour, I had the \u0026ldquo;Luke, I\u0026rsquo;m your father\u0026rdquo; moment. Now, I\u0026rsquo;m happily munching secure cookies on the dark side. Check out the line below:\ntls tiredsysadmin@deathstar.com\nThat is all you have to configure to get a truly automatic Let\u0026rsquo;s Encrypt HTTPS-enabled website served by caddy.\nWhat\u0026rsquo;s more? Caddy really feels like a breath of fresh air in terms of configuration. It is written with usability in mind. I thought we already had it pretty good with nginx (but I come from Apache, so make of it what you will), but this is even better.\nCheck out a sample configuration for an entrypoint that I use:\ntux-sudo.com { tls tanmay.chaudhry@gmail.com redir / blog.tux-sudo.com 301 } Yeah, okay it\u0026rsquo;s not much (but isn\u0026rsquo;t that the beauty?). When was the last time you saw an auto-TLS configured in 2 lines? Also, no more returns/redirect to write for HTTP traffic. This does auto-redirects. Stuff like this is why it feels like a true out-of-the-box HTTPS solution. You can still go deep and configure stuff to work the way you want, but more often than not I just want it to work.\nBut I had a slightly more complex use-case. I wanted to proxy to my prometheus instance. This is running on a raspberry-pi on my internal network behind a dynamic IP. For ISP reasons, I was not able to bind to 443 for the incoming internet. That creates a problem for the let\u0026rsquo;s encrypt challenge. Aw, back to DNS records and manual cert installation? Well, no. Check this out:\nprometheus.tux-sudo.com { log / /var/log/caddy/caddy-prometheus.log basicauth / hah nicetry tls { dns route53 } proxy / localhost:9090 { transparent } } Yes, that is all. Set your DNS provider creds in environment variables and keep munching those cookies.\nThe real clincher is that everything here works exactly the same inside containers.\nversion: \u0026#39;3\u0026#39; services: web: image: abiosoft/caddy container_name: web ports: - \u0026#34;80:80\u0026#34; - \u0026#34;443:443\u0026#34; environment: - CADDYPATH=/etc/caddycerts volumes: - $HOME/.caddy:/etc/caddycerts - ./config/caddyfile:/etc/Caddyfile If you\u0026rsquo;re an enthusiast who writes abandonable personal projects every weekend, I urge you to try this out. It is supposed to be production ready as well if you\u0026rsquo;re interested, but that\u0026rsquo;s not something I have tested.\nCaddy : Caddy-Github ","permalink":"https://blog.tux-sudo.com/posts/caddy-autohttps/","summary":"\u003cp\u003eThis one is a gamechanger. It really is.\u003c/p\u003e\n\u003cp\u003eOn a recent Golang obsessed Github browsing spree, I read the following project description: \u0026ldquo;Fast, cross-platform HTTP/2 web server with automatic HTTPS\u0026rdquo;. I\u0026rsquo;ve read that before, and more often than not ended up disappointing myself and resorting to hacks like [this] (\u003ca href=\"https://blog.tux-sudo.com/posts/letsencrypt-nginx-docker/\"\u003ehttps://blog.tux-sudo.com/posts/letsencrypt-nginx-docker/\u003c/a\u003e) to create an automatic TLS system for my websites. There are other ways, but almost all of them were semi-automatic in the long run (yay cron!). I decided to try out [Caddy] (\u003ca href=\"https://caddyserver.com/)\"\u003ehttps://caddyserver.com/)\u003c/a\u003e.\u003c/p\u003e","title":"Caddy! Team 443 FTW"},{"content":"I like nginx. I like docker. I like SSL. I also like not paying for SSL certificates.\nIf you, like me are operating a small time website, you probably don\u0026rsquo;t want to dish out a few hundred dollars for an SSL certificate.Thankfully, LetsEncrypt provides free ssl certificates and is now trusted by most major browsers.\nHowever, integrating certbot to automatically configure my nginx inside a docker container has been a pain. In the past, I generated the standalone cert with \u0026ndash;cert-only and then linked my docker container to use it but this was a far from usable solution given the horrid automation code I had to write. Recently, I tried to come up with a more automation friendly solution and ended up with the following:\nCustom nginx image with certbot pre-installed inside the container. Dockerfile:\nFROM nginx RUN apt-get update \u0026amp;\u0026amp; apt-get install -y \\ certbot \\ python-certbot-nginx ADD certify.sh / RUN chmod +x certify.sh certify.sh:\n#!/bin/bash certbot --nginx -d $DOMAIN -m $EMAIL --agree-tos -n --redirect Github-Source: https://github.com/tchaudhry91/nginx-certbot-docker\nDockerHub Image: https://hub.docker.com/r/tchaudhry/nginx-certbot-docker/\nThen I would simply run the container with the following docker-compose block:\nversion: \u0026#39;3\u0026#39; services: web: image: tchaudhry/nginx-certbot-docker container_name: web ports: - \u0026#34;80:80\u0026#34; - \u0026#34;443:443\u0026#34; environment: - DOMAIN=example.com - EMAIL=xyz@example.com volumes: - letsencrypt:/etc/letsencrypt - ./config/nginx_config.conf:/etc/nginx/conf.d/default.conf volumes: letsencrypt: An equivalent docker run command would also suffice. But do make note of the enviroment variables and the letsencrypt volume.\nThe enviroment variables are used to provide information about the domain for which the certificate is to be requested. The letsencrypt volume is also essential as this will prevent recertification till the old certificate expires. This can also be a host-mounted directory, however I prefer docker volumes. This image does not do ssl yet. Infact, it can be used as a drop in replacement for the regular nginx image. I can \u0026lsquo;certify\u0026rsquo; the container inside it on demand with:\ndocker exec container_name /certify.sh This command can easily be cron-ed on the base system to re-certify and prevent expired certificates.\nPitfalls\nWhile this works for now, a true service is most likely going to be load balanced. This approach would require scaling down a single container to ensure the letsencrypt verification passes before certification. The service can then be scaled back up, and given the shared volume, it will work without requiring re-certification. ","permalink":"https://blog.tux-sudo.com/posts/letsencrypt-nginx-docker/","summary":"\u003cp\u003eI like nginx. I like docker. I like SSL. I also like not paying for SSL certificates.\u003cbr\u003e\nIf you, like me are operating a small time website, you probably don\u0026rsquo;t want to dish out a few hundred dollars for an SSL certificate.Thankfully, \u003ca href=\"https://letsencrypt.org\"\u003eLetsEncrypt\u003c/a\u003e provides \u003cstrong\u003efree\u003c/strong\u003e ssl certificates and is now trusted by most major browsers.\u003cbr\u003e\nHowever, integrating \u003ca href=\"https://certbot.eff.org/\"\u003ecertbot\u003c/a\u003e to automatically configure my nginx inside a docker container has been a pain. In the past, I generated the standalone cert with \u0026ndash;cert-only and then linked my docker container to use it but this was a far from usable solution given the horrid automation code I had to write. \u003cbr\u003e\nRecently, I tried to come up with a more automation friendly solution and ended up with the following:\u003cbr\u003e\u003c/p\u003e","title":"LetsEncrypt Certificates for Dockerized Nginx"}]