A Champion Challenger in a decisioning context is a technique that allows testing of operational decision in production from different angles.

In the context of marketing advertisements, you probably have heard about AB testing. For example, when you want to make sure an email you are about to send will have the maximum open rate, you can define two subject lines and the system sends the email using randomly selected subject lines based on your two subject lines. Once a recipient opens the email, the system records the performance (i.e., email open action) against its subject line, then you can measure the success rate of each subject line.

Champion Challenger is like A/B testing. It enables implementing and experimenting on a decision with different implementations of decision logic in order to monitor the results and understand which one works better (i.e., successfully meets KPIs).

This technique enables both monitoring and measuring the performance of decisions ‘live’, by distributing the process requests using different variations of the same decision. The goal is to identify which strategy (i.e., variation of the decision logic) performs better. The “better performance” may have different aspects. For example, from technology metrics (e.g., speed of execution, to business KPIs such as new revenue impact, numbers of accepted offers by the customer, etc.).

Champion Challenger that shows 3 challengers against a champion processing live transactions

Champion Challenger that shows 3 challengers against a champion processing live transactions

Why is it important?

When there are a couple of competing models then there is a need to systematically understand which of the models is performing better than the others. Although companies may still run some historical and test data on new models to ensure they behave as expected, the simulation and tests just show the quality of the implementation. These do not show the impact. This is where champion challenger comes into the picture.

The fact is that companies mostly make decisions (i.e., change offers, add options in bundling, providing a discount, etc.). With historical and test data alone you cannot understand whether an offer is accepted or not.

When competing decision models and implementations of existing strategy should not be like flipping a coin, then the Champion Challenger becomes a necessity.

How does it work?

Champion Challenger takes the guesswork and gambles out of choosing the right operational decision. Choosing the right one is the game of numbers. In reality, two or more competing decision logics may be run to provide a decision service. These multiple implementations (i.e. decision logic) of a decision are responsible for processing transactions. The Champion Challenger in the decision service layer will distribute requests to each of these implementations randomly.

This is where it gets interesting. This randomness should not be biased. Randomness is a very complex topic and it is extensively studied in the fields of mathematics and statistics. These algorithms are used in many gambling games, such as Jackpot.

It is more complex in a Champion Challenger scenario, as the operations staff should be able to control the distribution of transactions between variant decision logic. For example, you may want to keep processing the transactions the same way 80% of the time and distribute the remaining 20% to the other decision logic that is challenging the champion (currently running strategy). That is why it is called Champion Challenger – as there is a champion operating, and there is some decision logic that challenges the champion, a challenger can perform better and eventually becomes the champion.

Conclusion

Although companies use historical data for simulation and testing to ensure the quality of new decisions. These techniques just assert the expected results of decision logic and ensure the correctness of the decision results based on different situations. But in contrast, champion challenger enables testing the impact of the decision based on a KPI by running experiments in production using live transactions.

Last updated December 2nd, 2021 at 08:40 pm, Published January 23rd, 2020 at 08:40 pm