Skip to content
Recruitment BlogSoftware Reliability at Optiver: Test Driven Development
recruitment blog

Software Reliability at Optiver: Test Driven Development

June 15, 2020

Up to this point, much of the discussion around Software Reliability has focused on the macro level of our systems – software correctness, simple architecture and design, people, process and culture. One of the ways we achieve all of this at the micro level of our systems is through the practice of test-driven development (TDD). 

In a nutshell, TDD is the process of writing tests before actually writing production code. This deceptively simple concept addresses both classes of risk mentioned in an earlier post, Erosion of Trust and Automated Trading Risk. It forces a developer to figure out what successful code will do before even writing it. This encourages developers to work with the stakeholder and decide on clear requirements before a single line of production code is written, thus building trust between developer and stakeholder. How can you write successful code without deeply understanding what it should do in the first place? It also bolsters protections put in place for automated trading risk by continually verifying their behavior, helping protect against future code changes and software decay.

Test-driven development has been around for decades, yet it is very rare to see it employed correctly. Many excellent articles have been written about the details of TDD. Details on unit testing frameworks, toolchains and technical details can be found elsewhere. This article will focus mainly on lessons learned from over 15 years of using TDD for real mission-critical systems.

Use Your Judgment – Before digging into any details or bold assertions, it is important to remember that the core of any professional software developer should be to always use your own judgment. There is an exception to every rule; blind application of rules and principles is the path to a horrible system. When reading this article, keep this in mind. I recommend trying every rule and practice, breaking every rule and practice and then reflecting. This article may make statements that sound absolute. This is for emphasis – please realize in practice all of these rules can be judiciously violated. Proper software development is more of a mathematical art than a science or engineering practice.

Invert, Always Invert – The key to TDD done well is inverting your thinking. TDD is a balance between writing the minimal set of unit tests necessary to clearly specify the design and writing enough unit tests to capture the corner cases. By inverting your thinking and writing a test first, you are more likely to think of what could go wrong. For example, if your program needs to divide two numbers, writing the code first is pretty easy. However, if you instead try to come up with a minimal set of unit tests that demonstrate successful division, you are more likely to think of corner cases like division-by-zero, NaNs, etc. In many ways, trying to minimally and formally specify constraints up front that communicate the intent of a program forces one to think of what could go wrong.

Red, Green, Refactor – The heart of TDD is to write a failing test (Red), write code to make the test pass (Green) and finally take a step back and refactor (Refactor). I highly recommend following this pattern with discipline. Adding unit tests after code has been written seldom improves quality and robustness as developers are prone to confirmation bias. To make matters worse, once working production code is written, it is very difficult to find the motivation to go back and add unit tests. Most developers who practice TDD focus on the Green stage and skip the Red and Refactor stage altogether.

The true power of TDD is the Red and Refactor stages. Writing a failing unit test (Red) forces you to add a unit test the current program does not conform to (inversion of thinking). Observing the unit test actually fail proves that the new unit test is, in fact, additive, and not just covering preexisting functionality. It also is a double-check against errors. It makes the achievement of 100% code coverage possible. Writing the minimal amount of production code (Green) necessary to make the test pass is also very important and a delicate art. Developers, in general, tend to future-proof; resist the temptation to write more production code than necessary. Finally, refactoring both production code and unit tests (Refactor) is critical to discovering ways to improve your design. If your unit tests start becoming combinatoric, it is probably time to break the class under test into multiple classes. Use mocking frameworks judiciously, i.e. no fancy features, and remember that finding a problem at compile-time is much easier than at run-time.

TDD is a mathematical art – Imagine a simple thought experiment: you can either keep your unit tests or your code. Which would rather keep? What if someone were to delete all of your production code lacking code coverage? The system should still work. Strive for 100% code coverage. Each line of code should only exist if a test was written first. Think mathematically: imagine the set of all possible programs that could ever be written. Without a single unit test, all programs in this set are valid solutions to your problem. Now write one unit test. With one single unit test, you have already reduced the set of acceptable programs. Keep writing unit tests, until the set of acceptable programs matches your requirements. Use your judgment and ignore pathological programs, i.e. mirrors of your unit tests. This art of carefully selecting simple unit tests one-by-one to collapse the set of acceptable programs to your requirements is the essence of TDD done well.

Listen to your pain – If it is difficult to write unit tests for something you are building, take a step back and consider that your design may be overly complicated. Break the code into more manageable components. Use your pain to guide you to a simple design. Most accepted software development best practices can be self-discovered through the crucible of TDD. The cycle of TDD is an excellent way to harness continuous improvement for yourself as a developer.

When we have followed these rules stringently, the results have spoken for themselves: multiple systems running in production at a world-class trading firm with few issues and a track record of performance, robustness and stability in the midst of extreme market conditions. These systems, composed of small simple components built using the principles of TDD have seldom been revisited and tend to “just work.” Some of these components have even been reused in unexpected ways. For example, some of the core components of a market making system were built using TDD. Due to their simplicity, performance and resilience they have found their way into most of our core infrastructure! Another practical benefit we have observed is these systems seldom need much support and maintenance; when they break, they break in clear ways that we are able to quickly remedy.

I want to leave you with a warning. Adopting the practice of TDD is akin to learning a new skill. At first, it will be very painful and seem pointless. You may even have to force yourself to do it and your productivity will slow down. At some point, you will have the eureka moment where it will all fall into place – your productivity will increase, your bugs and production incidents will decrease, your designs and design skills will improve, you will grow as a developer, and your overall development velocity may actually increase. It may even change your overall thinking about systems, designs, and architecture as these concepts can also be applied to the macro level of systems. In the end, you will help keep complexity at bay and know that you have written solid systems.

Joe Fourness, Development Team Lead

Joe has been programming computers since age seven and his passion for technology and finance led him to pursue a degree in Electrical and Computer Engineering and Mathematics from the University of Wisconsin – Madison and an MBA in finance from Northwestern’s Kellogg School of Management.  Additionally, Joe has earned the Chartered Financial Analyst designation and is a member of the ECE Advisory Board for UW-Madison.  Prior to his career at Optiver, Joe’s career has spanned industries and technologies ranging from automotive, logistics, technology consulting and finance.  At Optiver, he has held a variety of roles ranging from software developer, team lead and architect.  In his spare time, Joe enjoys reading, rowing and investing.