A non trivial concurrency example in Go language

Overview

In this blog I describe a non trivial concurrency example in the Go programming language. The code is part of my Dynago project https://github.com/mdkhanga/dynago.

This is not a tutorial on Go or on writing concurrent programs. But if you know a little bit of Go and/or little bit of concurrent programing, then I am hoping this can be a useful example. 

In Dynago, I have so far built a leaderless cluster of peer servers. The command for starting a server needs to only point to one other server. The only exception is the first server, which has nothing to point to.  The servers exchange ping and gossip like messages to share the details on the members of the cluster. After a few messages every server has the cluster membership list. When servers join or leave the cluster, the membership list is updated.

Code

If you would like to skip reading the text and jump to the code, the relevant files are :

https://github.com/mdkhanga/dynago/blob/main/cluster/peer.go
https://github.com/mdkhanga/dynago/blob/main/cluster/ClusterService.go

The relevant tests are 
https://github.com/mdkhanga/dynago/blob/main/e2e-tests/test_membership.py
https://github.com/mdkhanga/dynago/blob/main/e2e-tests/test_membership_chain.py

I show some screenshots of code snippets in this blog for the casual reader. But if you are interested in the code, I recommend directly looking at the code in github.

Go concurrency recap

In Go, you write code that executes concurrently by writing Goroutines.

A goroutine is like a lightweight thread. It is written as a regular function.

You run a goroutine using the go keyword.

Go provides channels for communicating between goroutines. Channels are a safer way to send and receive data. This is safer than using shared memory as is done in Java, C++. You do not have to use mutexes to avoid memory visibility and race conditions.

The select statement lets the goroutine wait on receiving data on a channels.

Though not recommended, the shared memory approach of  data exchange is also supported. You will need to use primitives from the https://pkg.go.dev/sync package to synchronize access to data.

The Examples

There were two features in Dynago where concurrency is relevant.

1. Each server receives a message on a GRPC stream. It has to process the message and sometimes send a response back. This is the classic producer - consumer problem. Some goroutines produce. Other goroutines consume and do work. I use channels for sharing the messages between producers and consumers.

2. A map stores the list of cluster members. 

• The map is to be updated as servers join or leave the cluster.
• Periodically we need to iterate over the map and send out our copy of the membership list to others.
• The receiving server will merge the received list with it own list.
• If the receiving servers list is more up to date, then it sends a response to and the original sender has to merge.
• timestamps are used to determine who is more recent

This the case of multiple goroutines reading and writing a data structure concurrently. 

Example 1: Channels 

The peer struct shown in the code below defines a channel InMessagesChan for incoming messages and a channel OurMessgesChan for outgoing messages.

The receiveMessages goroutine method has for loop with following code. It reads a message from a Grpc stream and sends it to the InMessagesChannel.

The processMessageLoop goroutine method in peer.go has a loop that takes messages from the InMessagesChannel and processes them.  When a response is needed, it processes the message and writes the response that needs to go out to the OutMessagesChannel. The image below is shortened version of the real code.

Lastly the sendLoop goroutine method has a for loop that takes messages from the outMessageChannel and writes them to the outbound Grpc stream.

As you can see, very simple. 3 goroutines working concurrently by exchange data over channels. No locking, no synchronization an fewer problems.

In my opinion, channels is one of the best features of Go.

Example 2 : Shared memory

I have this struct cluster which has the list of peers in the cluster

We need to add /update / remove entries from the map in a thread safe manner. The code below uses the mutex to synchronize access to the map. These methods are either called from both peer.go and cluster.go.

The code below shows a loop from the ClusterInfoGossip method that uses the same mutex to synchronize the map while iterating over it. Typically the calls from add, remove and gossip happen from different goroutines. If you do not synchronize, you will have memory visibility problems. Note that since peer is updated, we need to synchronize access to the peer as well.

You might be wondering, why can I not just do this using channels when it is safer and cleaner ? What you would do is to write a function with like an event loop.  The function can accept commands on a channel and read , update or iterate over the map based on the command. The code would look something like below

Which approach you take is sometimes a personal choice. For the message processing, I preferred channels. But for CRUD on a map, I preferred shared memory.

Conclusion

In conclusion, what I have shown you here is a non trivial concurrency example in Go. Go makes it quite easy to write concurrent programs. The recommended way to share data between goroutines is by channels. By using channels, you can avoid race conditions and memory visibility issues. But the traditional approach of shared memory with synchronization is also supported. 

As I build out Dynago, I plan to blog about the interesting pieces.  If you like my blogs, please follow me on LinkedIn and/or X/twitter.