We wanted Palaamon to be a pure Reactive application, following as much as possible the ideas of the Reactive Manifesto.
Palaamon is written in Scala as an Akka cluster. So for now, every feature in Palaamon is either a microservice or a set of microservices within an Akka cluster.
One important decision was the way how to couple workers with the main cluster.
Workers are the pieces of code that are doing the real scientific calculations. They are all packaged as sepratate containers with all their dependencies.
Many workers are using experimental pieces of code written by researchers in Academia or industry. Some of these workers can become unstable as they were often written as scientific experiments. Others are very sensitive to varying input data or file formats. They can crash because a comma has been replaced by a semi-colon. But at the same time, they can provide implementations of very advanced scientific ideas that are very useful for researchers.
Akka has a lot of features that can be very helpful in those tricky situations.
For instance, the Palaamon wokers are very much isolated from the main cluster that manages all other activities and objects.
Workers nodes register to the worker cluster and tells it what kind of service they can offer. The information about each worker is propagated to the controller and to the rest API.
The controller keeps a real time registry with all the workers available and what each can do.
The workers are therefore available to the users but they cannot influence the functioning of the whole cluster which remains resilient.