RabbitMQ vs ZeroMQ

As for OpenSettlersII I want to present an interface to be able to implement for any possible GUI. And as I don’t want to tie anyone with Elixir/Erlang I decided to go with messages. I looked throu the net and my own experience, and there are two picks that I though I could use

RabbitMQ

RabbitMQ is a Your Typical Message Queue. There’s a message broker that handles messaging and whatnot. It’s written in Erlang, which is huge advantage, as I could use it directly from Elixir (you can call any Erlang function from Elixir with no obstacles). Not only that -there’s an official support for Elixir itself. I also used it in my previous work, as a client. It was trivial, as should be using it in OSII.

Or is it? My aim is to be able, after getting interface for GUI, to get started as quick as possible. With RabbitMQ, when you get to the server part, not only you have to install additional software, it can get ugly quickly. As for commercial products it’s not a problem, as there’s company with support behind it, for open source, pet project it is a problem.

ZeroMQ

I really need a citation here, but apparently creator of protocol for RabbitMQ decided that it’s not really that good and did ZMTP (ZeroMQ Message Transport Protocol). It is brokerless message queue and some may argue that it’s just simple socket. The biggest problem is message persistency – if you don’t have anything to hold/save messages while other party is offline – you’ll lost them. This doesn’t really sound like a problem to me, as I think that there are some mechanisms in OTP that can help me here.

ZeroMQ is really easy to configure, I worked a bit more with it and have fond memories with it. Basic communication worked out of the box, no need for any additional installations or configurations.

 

I decided to go with ZeroMQ for reasons stated above, but there’s one more reason. I may not google that much, but I haven’t seend any ZMTP 3.0/3.1 implementations made in Elixir. There are some bindings, sure. But nothing made out of scratch. I won’t lie – if I can learn even more doing this project and possibly make first 3.0/3.1 ZMTP for Elixir – I’m in!

Integration series: Messaging

Last time we spoke about some integration methods we can use.

As we see, there are methods that are not so tight coupled, being able to generate lots of little data packages (like file transfer), easily synchronizable (like shared database), details of storage’s structure hidden from applications (unlike shared database) and being able to send data to invoke behavior in other app (like RPI) but with being resistant to failure (unlike RPI).

And here messaging comes to play. The rules are simple: you create message, send it to message channel and someone waiting for this kind of message will get it. While it has some problems on it’s own, it is reliable, fequent, fast and asynchronous and.

  1. Being asynchronous means you won’t block process while waiting for the result/answer. Calling app can continue with it’s work.
  2. Decoupling. Messages will be sent to message channel without knowing almost anything about receiver. The common interface are the types of messages sent, not the bidings between apps. It also allows separation integration developement from application developement.
  3. Frequent, small messages allow applications to behave almost immediatly by sending more messages.

And many more we’ll explore in the series. Why I will write a series on it? The main disadvantage of messaging is the learning curve. While other methods are fairly easy to use, messaging and async thinking is not something we’re used to. But once learned this concepts will help you not only when integrating lots of enormous applications. You can also apply it to “integrate” classes/functions/actors in your code.

OpenSettlersII #1

Hello and welcome for a short update!

Here’s a commit to look at!

There’s not much and it’s not very nice code. But it’s something, right?

I decided to postpone Settlers II itself as I need transport layer. I decided ZMTP is the best one for me (why is the topic on the other blogpost incoming!) so I started doing something basic with it. According to ZMTP 3.1 specs first octet (8 bits) follows this convention:

  • bits 7-3 are reserved for future use and are 0
  • bit 2 indicates if frame is message (0) or command (1)
  • bit 1 indicates if frame is long(1) or short(0)
  • bit 0 indicates if there are more frames (1) after this one or not (0). For command it’s always 0.

Now for some clarification. “Bit 0” means “right-most bit”. Short message has body of 0 to 255 octets. Long message has body of 0 to 2^63 – 1 octets.

Next we have size field – it’s one or eight octets, depending on LONG flag. What’s important that size doesn’t include flags and itself – so empty frame has a size of 0 even if it has flags and size set.

Lastly we get SIZE octets of body.

In the commit we have simple encode/decode for frame with SIZE = 1, LONG = false. There’s a difference between Command and Message.

I am sure that “reserved” is redundant and code itself is not something clean and nice. But it’s a little start that I will continue working on and either refactor or throw it away.

Bytes are represented in Elixir as <<values>>. It’s quite useful to get working, but we can get better than I got here.

As for functional language specific stuff, please take a look at multiple definitions of private method decode_flags. It’s the pattern matching on functions (yup, can do it better here) – elixir will look for matching function and will call it. It’s simple and quite powerful feature.

In the next commit(s) and post I will make those encode/decode functions more general to be able to work on all sized frames. And hopefully more 😉

Introduction to integration

I started to get more into integration and integration patterns. There are few reasons:

  • Open Settlers II will be created with integration with possible UI integration in mind
  • It will be helpful in my daily job
  • I feel that it’s an important topic in software engineering

Having this set up, let’s briefly talk about some integration methods.

File transfer

We want two (or more) applications to exchange data. We can use simplest solution – write it to file for others to read. (Almost?) every language non-esoteric lanuage has some file read/write function built in. It is also easy to do no matter what environment you’re working with. Coupling is not so tight as application devs can (should?) agree on common file format(s) to work with. Changes in code won’t change the communication as long as output file is the same. With json it’s easier than ever. Even with third party apps it’s still trivial to consume messages from software we don’t have influence on.

There are also some downsides, too. There is a lot of work with deciding on file structure, file processing. Not too mention storage place, naming conventions, delivering file (if one app doesn’t have rights to output location of other), times of reading/writing (and what will happen if one reads while the other writes). But it all is nothing compared to one big problem. Changes propagate slowly (one system can produce file overnight, after “collection” of other). Desynchronization is common and it’s easy for corrupted data be spread before any validation (if it’s even possible).

Shared database

Shared database is a remedy for the synchronization problem. All data is in one central database, so information propagates instantly. Databases also have transaction mechanism to prevent some reading/writing-while-writing errors. You also don’t need to worry about different file formats.

But it also comes with a price. It’s difficult to design a shared database. Usually tables are designed to fit different applications and are a pain to work with. Worse if we’re talking enterprise level solutions and some critial app – its needs will be put higher, making work for others harder. After creating database design there’s a tendency to leave it as it is – changes can be hard to follow. Another problem is third-party software. It will usually work with its own design and it may change with newer verions. Database itself can become a perfomance bottleneck

Remote Procedure Invocation

Sometimes sharing data is not enough, because data changes may require actions in different applications. Think changing address at goverment service – there are a lot of adustments and documents to be generated. Apps maintain integrity of data it owns. It also can modify it without affecting other appliactions. Multiple interfaces to CRUD data can be created (e.g. few methods to update data, depending on caller), which can prevent semantic dissonance and enforces encapsulation.

It may loosen the coupling, but it’s still quite tight. In particular doing things in particular order can lead to muddy mess. While developers know how to write procedures (it’s what we do all the time, right?) and it may seem like a good thing it’s actually not so good. It’s easy to forget that we’re not calling local procedure and that it will take more time or can fail due to multiple reasons. Due to this thinking also quite tight coupling arises (as stated before).

As always, there’s always a tradeoff. But do we have the best approach here? Or can we do even better? I’ll address these questions in the next post in series.

Chicago vs London TDD

Somewhere near my very beggining of my software engineering journey, as a fresh Junior, I happened to talk with a collegue of mine. I remember him saying:

So, when someone says [during the interview]
-I know TDD!
I ask:
-So tell me the difference between Chicago and London style.

This happened to be The Great Filter, as many people didn’t know that. Luckily this wasn’t my interview as I didn’t know either. So, naturally, I did some googling.

It turns out, that it’s not rocket science at all.
Let’s say we’re testing the metods talking with DB (using some injectable context, of course)
Chicago style focuses on results. So here you check, if result that you get back is the same as expected.
London style focuses on behavior. So here you mock the context and then validate if methods you need to call were called defined number of times.

Chicago style focuses on results. London style focuses on behavior.

So, it’s easy, right? Also – you can also mock in Chicago style, and by getting the results you want test behavior, right? And why is it that important?

I’ll start with second question. If you start mocking around and check for results, your setup/arrange parts tend to grow and be more and more complicated. Also, if you want to test results, you have to provide some sensible data. This makes testing more cumbersome than needed and results in greater reluctancy in writing them. In my opinion those kind of tests work best with integration/end-to-end tests and also with unit testing, were you have no “complex” (or maybe any?) side effects. Pure functions are great example – for same input, always same output, without any side effects. It’s very easy to write those tests and arrange part will be small, if almost not existent.

When you verify behavior, you don’t care about carefully setting up mock, populating data, thinking about complex relations. You just want to know that system behaves in a way yout want it to behave. Databases, messaging systems, IO operations etc are fine places. You have other kinds of tests to check if your system is working correctly with those alive elements. Here you want to check if you handle them correctly.

It’s easy, yeah. Don’t seem that important. But it’s really easy to forget London style and check for result everywhere. Writing tests starts to be painful, they take more time and, out of nowhere, you’re dug under pile of complexity.
“But I did unit testing, why is this happening?!”
Because maybe there’s more to writing tests than just Assert.AreEqual(expected, actual) 😉

 

PS: Both ways are equally important and have their own purpose. Don’t just focus only on one and you should be fine.

OpenSettlersII #0

This is the first post in series about OpenSettlersII aka GetNoticed! project.

I will always link commit(s) and describe what’s happening.

First I’ll talk a bit about Elixir itself and why I picked it up from all available functional languages.

It all started over 30 years ago with the creation of Erlang. It’s a functional language which main purpose was telecoms. In this world high availability, scalability and all of this good stuff was needed. While I was talking with a friend, who worked with switches back in the day, he told me that “nine nines*” was the thing everybody talked about. And in optimistic scenario, Erlang delivered that. How? “Let it crash”. You fire up many processes (elrang’s, not OS’) and in a case of failure – you let process fail, die and restart. All without system even noticing it. There are processes which main job is to supervise others (hence the name “supervisor”). Should any process fail, supervisors will restart it, possibly providing some data. Worth noting that you can do all sorts of app in Erlang, not only telecom related ones.

After Erlang came OTP. Long story short it’s a framework that helps building applications with all the goodness described.

And then comes Elixir. Known Rubyist, Jose Valim, found that Ruby doesn’t satisfy his needs anymore. Around same time he came across Erlang and OTP. That resulted in language with nicer syntax than Erlang, really friendly and open community with some additional features/sugars. What’s really worth mentioning is the fact that Elixir itself is mostly written in Elixir, as it has elastic and pleasant metaprogramming.

Now back to OpenSettlersII

First Commit.

This is just a simple project initialization, but it shows the tools Elixir is shipped with. Mix handles creating projects, handling deps, running tests, compiling etc. I just created new, empty, configurable project (please, take a while to look at already existing place to write tests! <3).

Next commits will be more interesting, I promise.

*we’re talking about 99.9999999% availability/uptime here

Hello, world!

I decided to participate in GetNoticed. Short story even shorter, it’s about writing OpenSource code and blog about it (and IT).

I chose to make a port of old game – The Settlers II in Elixir. Being more specific I will focus only (or mainly) on backend. Why?

  • I’ve never made any “serious” app in Elixir/Erlang/F# any other functional language – this seems like a good opportunity
  • I feel like this is the good way of learning Elixir itself
  • Creating backend only will force me to solve some challenging design problems
  • UI will be easily changeable, so anyone can make their own version of game. Imagine The Settlers 2 in space, or Stone Age!
  • I feel like there’s not enough “nice” code in the GameDev world. One project won’t make a difference, but won’t hurt as well
  • It’s a scary challenge
  • I love this game

I will deep dive into details with next posts. Also, expect at least one commit per week! Here’s repository for the project on GitHub for you to follow.