Chain Testing is Risky Business

In large companies a lot of systems work together. What happens in one system affects other systems? All these systems are there to support processes… and there are a lot of those… which affect each other as well. Now if you want to test a system that has been modified or added, not only do you test that system but also the system that it is connected to. In fact, you have three systems: system A, system B and the connection (the interface). Now this connection often is an orphan without an owner. Nobody wants to be responsible for the connection.

But the issue is not limited to these three systems, these systems are connected to a lot of other systems for which you may want to check whether the changed functionality in system A affects P, Q and R for instance. Things can get out of hand easily when you want to do a chain test or end-to-end test. You need the support of a lot of people from the business who know the systems you want to have checked. Also, these business departments want to be involved since the change could affect their system. Things can become uncontrollable. Also setting up a test environment like this can become quite expensive.

First of all you have to define the responsibilities in the chain. The one who causes the risk by changing or adding functionality should be responsible for the connection to adjacent systems. If you want your new system to work on Windows, then you are responsible for it working correctly on Windows. It is as simple as that. For the rest of the chain you would need to define the responsibilities. Either a chain test manager is responsible or each system is responsible for the outgoing flow. Whatever you define… as long as it is clear what everyone has to do. A chain test manager should define this in the test plan.

The way you organize the chain test is determined by quality, time, and money. If you need to spend a lot of time and money to get a proper test environment (including all the interfaces) up and running, in which all the affected systems are integrated, it may not be worth it. For this we need a proper risk analysis in which all the risks in the whole chain are identified and examined. If there is a high risk, e.g. in case of a vital financial application, then tests should be done. In case of low risks, then it is probably not worth it and if issues occur they can easily be fixed afterwards. Of course it is always wise to keep the developers available for one or two months after going into production. After the risks are identified the risks can be weighed against the implications and a proper chain test strategy can be made. For more on risk analysis and risk management have a look at my article on this.

A lot of testing can be done up front by analyzing the data flow through the whole system and the structure of the systems, databases and interfaces that are affected. For example, if there is a certain field X that is copied from A to B, make sure you follow X and examine what happens to it between A and B, in B, and after B in the rest of the systems. Also make sure that you know what B requires and what other systems require and adjust A and the interface between A and B if necessary.

So reviews and analysis can save a lot of money. The role of reviews and analysis will increase in testing anyway, since more and more companies begin to realize that professor Boehm was right when he said that most system faults originate from faulty requirements and specification. That is something to consider in this increasingly agile world. Nice topic for a separate article.

This article is based on my book Q-Course Quality & Organization.