Month: August 2019

Debugging can be a frustrating couple of hours, and it can be the most rewarding work of the month. You might end up in places where you did not even dream of. I have been scrolling through the source code of libraries to understand how a parameter manipulate some function. I have been diving through dependencies of dependencies to find out why a particular piece of code is not working. When debugging, you always look for some way to unearth the missing puzzle piece.

One of the things I have recognized is that different methods of debugging provide different latencies. Going into a shell and pasting in the exact piece of code will give you a result immediately. Committing a piece of code to a repo, getting it reviewed by a coworker, deploying it to test, deploying it to production, and then getting the result takes a bit longer. Depending on the task it might be necessary to do the cumbersome last option, but as a rule of thumb, we always want to keep the latency of our debugging as low as possible.

In this post I will be exploring the various levels of debugging that can be used to debug in, and how they differ regarding the latency of the feedback loop.

Levels of debugging

The following are the levels I have identified as the major methods of debugging, from the lowest latency to the highest.

Shell / script

The ultimate fast feedback delivery method. You write some code and get the feedback. If you want to try the same method with several inputs, you do it right away. This method might get cumbersome if you are debugging something that requires a lot of setup, because you need to find out how to set it up for your particular case.

Unittests

This is very similar to using a script, as mentioned above, but has the advantage that you probably (hopefully!) already have a test suite. You can use existing tests to create a new case that captures the problem, and then use your standard toolchain for unit tests. I use the debugger in Pycharm which helps me step through and inspect variables in the tests. This helps my debugging immensely.

Local instance

Booting up the application and clicking through it to provoke the bug is often an easy way to detect what context the bug happens in, and what methods should be focused on. The difficulty with this method is that it is not always easy to translate a click on a button in the application to the exact path of execution that contains the problem. If the bug is dependent on a particular data configuration or some other edge conditions, it might not be obvious how to replicate it. On the other hand, the actual application might be the best way of finding out how the bug arises.

Test environment

Particularly difficult problems might need to be debugged in a test environment. This is often because the bug has some complex data dependency that is not easy to replicate in your local environment. If your test environment has data similar to reals customers, it might also be easier to find a case that is similar to the bug that was originally reported.

Because the test environment is a remote system, the latency gets really high at this point. The right code needs to be on the system, logs need to be traversed to find the appropriate piece of information, and other users using the same environment might interfere with the debugging.

Because this system is a test system, you can usually do whatever you want, just like on a local instance. The difference is that you need to jump through some hoops that might be more difficult than you initially think. This creates overhead that adds to the inherent latency when using the test environment.

This is a dangerous level because it often feels like you are making progress towards finding a solution, but oftentimes you will end up facing some restriction that complicates the debugging process. Bypassing those restrictions feels necessary, but it would actually be much faster and easier if done on a lower level. When debugging on a test system, always consider if it is really necessary to be on this level.

Production environment

Using the production environment should always be the last resort. If the production environment is not yelling the answer to you, it will not be worth trying to make it do something it was not supposed to. Trying to do that needs special care and you might end up breaking something. Don’t do it!

Keep low on the ladder

Proponents of test-first development will say that you should always create a test case for the bug, and then fix it. I do agree, but sometimes you do not know all details of the case is, and then you need to investigate. If you can capture the case in a unit test, you have captured the problem. If you cannot you need to move to another level and find out what you are missing. When you find that missing piece of information, move down again, and keep as low as possible on the ladder .

It might seem like an obvious piece of advice to always stay on the level with the least latency, but it can be tempting to stay on a high level for too long, because it is more convenient. But the hidden costs often ends up being far greater than the cost of moving to a method with lower latency. The trick is to identify when it is necessary to stay and when it is a waste.