High Availability Architecture (3/4) – Performances

In the previous posts of this serie we’ve addressed the availability and scalability aspects of high availability systems. In this (rather lengthy) one we’ll focus on the performance side of things.

Again, performance is something to be contextually defined quite early in the project. For instance a requirement such as “3s response time” is not precise enough. “3s response time with 200 simultaneous users” is a valid requirement.

History

Performance has been a common issue for the last decade or so with the emergence of multi-tier IT systems. It has not been such a problem in the past or rather it has been addressed as the core issue and fixed once for all during the mainframe years (65-89). It also has been skipped during the client-servers years (90’s) as no real software architecture was in place. Back then, no-one would react if SQL code was found in the presentation layer.

Nowadays, the standard 5 layer software architecture (Client, Application, Business, Integration and Resources) has naturally emerged as the de facto solution. The basic principle is to ensure for isolation between layers to provide the software architecture with a greater modularity and robustness. With the considerable litterature available, architecture software development constraints are much more pregnant – and that’s an excellent thing.

However, such complex software architecture requires to be extremely cautious from the very early stage of architecture and design as a laidback appoach can prove to be quite expensive in terms of performance.

Usual Suspects

From experience, when it comes to performance, the usual suspects in this 5 layers architecture are :

1. Integration tier
2. Database tier : Execution plans, indexes, table schemes, DBA early in the process
3. Business and Application tier : Algorithm, transactions, resource connections and APIs

Integration tier

Java Enterprise frameworks (Hibernate, Entity Beans) have been developed to abstract data persistence from the application developer. On one hand that’s a good thing : application developers can then focus on their own business problems and forget about SQL.

As a result these frameworks usage may not optimised and it could happen that application developers lose control on what they actually do, the number of objects in memory etc … The solution is to keep a fine grained control on these frameworks and the SQL that is actually generated and executed behind the scene. A close collaboration with a experience DBA (Database Administrator) is strongly recommended here, from the early steps of design.

For information, it’s worth mentionning that big internet giant such as Amazon for instance just don’t use such frameworks in order not to lose control on piece of processing that actually is critical in terms of performance. Read this article on that very subject.

Databases tier

Another drawback is that SQL not being at the center of development concerns, we may end up with applications that are not really optimised from the database perspective : database schema and queries execution plans are inappropriate and inefficient.

Then again it is recommended to have a DBA involved from the very early stage of the project (architecture and design) to validate the database schema and suggest execution plan to optimise the database usage.

Stored Procedures

At first, stored procedures may be considered as antiquities inheritated from the client/server years, souding like heretic software components from the 5 layer perspective.

However, from a genuine performance perspective they still are top drawer solutions. Data handling on the server allow to save a lot of time for applications and on the network.

The aim here is obviously not to develop the whole business layer as sored procedures but, rather, to optimise a few heavy process. Besides it proves to be an excellent antidote to the “DB request machin-gun” anti-pattern.

Application and business tier

There are a few standards Java dos and don’t to improve algorithm path length and CPU/memory usage.

The most important ones though regard sessions, transactions, APIs and resource connection

1. Avoid stateful component whenever possible (EJB Stateful sessions are performance killers). Whenever handling sessions (e.g HTTP Session) make sure they don’t contains too many objects.
2. Transactions must be kept as short as possible.
3. Resource connections are very precious asset. There use should be optimised to the actual need and kept as short as possible.
4. Whenever developing services API, always develop both unitary treatment service (i.e perform one action for one given item) and multiple treatments interface (i.e execute multiple request with multiple inputs and return multiple replies). This basic principle allows to save a lot of time by avoiding machin-gun request to the service.

Lastly, whenever building a new enterprise system using platform such as JEE or .Net, we need to keep in mind that these were made for real time process, not for long process. So if the application you are developing takes times (i.e above 5 sec), this component must not be deployed on the application server : it will uses precious resources for long process that are not used in the meatime to serve real-time services. Such a component should be deployed as a stand-alone server instead.

The Asynchronous trick

This is a great tool to load balance. Whenever addressing long process issue, it always is worth considering deploying it as an asynchonous service. This is a very natural and elegant load balancer mechanism that does not impact real time service QOS.

The biggest problem here is to convinced the business to accept to implement some asynchronous solution for rather peripheral services (e.g Reporting).

JMS is the natural solution whenever developing asynchronous solution in the Java world. One has still to be careful when choosing a JMS implementation.

A basic rule of thumb is that whenever you’re dealing with transactional messages with high availability and robustness concerns, you should rule out JMS implementation without persistence mechanism. JBoss JMS for instance is not as mature as JBoss Servlet or EJB implementation. Today the best solution on the market for JMS is SonicMQ.