Organizing Microservices - Modern Integration

Microservices is probably one of the most popular buzz words among my fellow developer friends, and I do like the concept of being flexible, agile and having simply having more choices. But as a person that worked in the software integration space for years, I started to see some resemblance of the old ESB days.

Looking at the problem from ten thousand feet up. A decade ago, we had to come up with a better way of organizing the spaghetti connection in between systems, stop duplicating effort on the same piece of business logic. That is when service-oriented architecture (SOA) became popular. By modularizing services, sharing them among others systems, and organize ways of communication, routing of data. And ESB is one implementation of that, maybe not necessarily how it should be done.

I was very fortunate to participate in many integration projects of this kind and lead some myself. We worked with various middleware vendors; at the time, the solutions were all about ESB. By removing the complex interconnecting logic between systems into ESB, negotiating data model among teams (with application/services leaders), setup WSDL contracts for web services and adding BPEL processes if needed between calls. And I had seemed a fair share amount of these ESBs myself; outside of ESB, everything seems very organized. Lift up the ESB “box” you will find many of them end up with even worst tangled spaghetti code (or diagrams depends on your tooling). And that giant bundle of chaos, then become the bottleneck in delivering changes in the enterprise.

When the idea of independently deployable microservices came along. This giant monolithic monster with the idea of a smart pipe where it tries to do too much, like data model definition, process flow, interconnect protocols and data format transform...etc and dumb endpoint where services just wait to be activated by events was soon cast aside by developers. The relation between services is still messy and the monolithic nature makes the cycle of development long and cumbersome. The microservice allows a developer to be more agile, who needs that piece of “Integration”. So this is where the resemblance of a problem where I once tried to solve :)

And let alone we still need to interact with other services (systems/applications) outside microservices. So learning from the past here are the things we know what NOT to do...

• No more, large monolithic bundle applications to speed up the dev cycle.
• Stop doing complex business process inside system integration code.
• The data model shouldn’t be defined in the integration layer; it only leads to extra negotiation between teams.
• State should not be kept in the integration layer, to allow maximum scalability.
• Create fix complex contracts between services.

Still, there are things we liked about the old "integration" way.

• Organized visualized interconnect view of systems.
• Patterns that are already defined for integrating applications such as aggregations, separation, and normalization of data.
• Event-based approach. A more reactive way and less coupled way to integrate systems.

That is why I came up with the layered architecture for microservice. The main goal that I wanted to achieve in the new integration architecture in the modern integration/application development is flexibility. Not only in a scalable way but also allow developers to make any architectural change with ease.

So first thing first, in the new modern agile integration, the integration code itself should be sorted into different small services that make sense in the business world. And are decoupled and made them be independently deployable microservices units. Then we can start to apply true CI/CD to the integration code. This will avoid becoming a large monolithic ESB.

In order to bring some logical sense to my microservice spaghetti, and avoid repeating the mistake of taking on too much in one single integration bundle. I have defined four layers for my microservice, so each will have its own responsibility making it easier to adopt changes.

• Gateway layer: Provides simple gateway routing capability such as versioning, dealing with a different platform of devices.

Originally I wanted to divide this layer into two but decided not to because the gateway pattern has become so popular now, people have the perception of gateway should achieve certain ability. (That is a whole other story I want to talk some other time). That is why I have two separate forms of the gateway in my diagram; one represents as green other blue.

It’s all about the separation of concerns. This layer is all about generating and putting together what is needed for external API consumption. This is when and where separate application domains try to integrate with each other and route to the correct versions of services. Clearly, the two major concerns were:

• Accessing policy rules, versioning of endpoints. (APIs)
  • Sets traffic throttle applying limits and policies.
  • Security of  APIs.
  • Routing to correct versions.
• Combine; transform return result to the clients.
  • Citizen developers are the main creators here, they use the capability provided by each microservice team to put together meaningful service to external clients.
  • Collect, split, or normalized a canonical data format to external users.
  • Translate data output depending on the clients.
  • Routing between application domains.

• Composite layer: Important middle tier that handles composition of multiple microservices. They do more complex routing from processing the content data and aggregates/split data and populate the split/aggregated result to other microservices by triggering events or simply passing it. This layer hides the complexity of microservices away from clients.

This layer is probably the layer that does most of the old time integration logic where it is responsible.

• Composing microservices
  • By calling the API available from microservices, and transforming data as needed between them and routing data to the corresponding microservices based on its content.

• Triggers events within domain
  • I believe strongly that event base does best at decoupling stickiness between each service. And allowing performance at its best because of its asynchronous nature. The event bus here doesn’t necessary has to be a message broker, but any forms of Bus.  
• Enrich contents
  • The content enrichment here should be specific to the format and avoid any business side enrichment when possible.
• Applying Integration patterns
  • Old integration problem will not just go away in the microservices world. By applying proven patterns, we can avoid making mistakes.
• Caching
  • The REST architecture specifically talks about a caching layer. It makes sense for the composite layer to try to apply caching mechanism because it’s non-business related, but do make a significant impact to the performance and fault tolerance ability of the system.  

The composite services should also be included in the application domain, which I will talk about in the basic layer.

• Base layer: Like the name, which most likely represents the basic components of the system. Handles data retrieval or business logic processing. Each should be independent and self-contained.

Borrowing from the idea of Bounded Context, in this case logically organized group of microservices into what is called application domain. Where each domain represents separate entities of business functions that share the same data model, this eliminates the need for doing too much transformation in the code.  

• Anti-corruption layer - This layer handles interface to a legacy application or anything that works against microservice quick and flexible principle. This layer is built in protection wall to your system, by doing the transforming job and translates between two very distinct implementations of the system.  

Till next time! :)

To build your Java EE Microservice visit WildFly Swarm and download the cheat sheet.