Testing Faith

At 40+ I consider myself an “old” programmer although it feels better to just think of myself as “old-school”. Its an interesting ride because I go back and forth between feeling like I have seen all the things come around 100 times and thinking there is so much new stuff I will never be able to keep up. For example, I have seen people make the mistake of thinking portability has something to do with programming languages instead of realizing its about platforms probably 100 times, but spending too much working on videogames made me miss the whole foundation of web programming and I still feel behind the curve.

One thing that seems to be new in recent years is the religious fervor about testing. Before I say more, I should point out that I think testing is good. More testing is better etc. etc. Testing isn’t new but this view of unit test suites as a panacea for all ills does seem to be new and in my view potentially dangerous.

I first starting thinking about this when I started reading all the discussion about how Unit Testing is better than strong typing. You know how it goes. Strong typing is for small minded cretins who can’t even trust themselves, and besides what matters is having good unit tests. Type checking is nothing more than a pathetically weak test suite. Oh well um whatever. The correct answer about typing is to make it optional. Python is moving this direction. Its a shame that C++ choose to instead inflict templates on us all but none of this has anything to do with unit testing. We need to test programs because people make mistakes even if they are using a language that has closures, lambdas, and currying.

The problem with obsessing about these unit test scripts isn’t that they are bad, its that they are not good enough. No matter how good your suite is there is are things it won’t do for you and you better make sure you realize that or you are never going to write code that is as good as it should be. Testing is not a substitute for some other technique. Its not an alternative to Typing. Its not a new way to do design. It’s not a definition of overall quality. Most of all its not a substitute for THINKING. Its just a way to make sure that none of the things you might have expected to go wrong did. You should make your tests as good as you can, but if you start believing that its good enough, then you are doomed.

Its worth pointing out of course that the value of testing is greater in some projects than others. If you are working on a library with a pretty clean API then a good suite of unit tests can give you a pretty high level of confidence whether or not your code will produce correct results. As things scale up to larger projects with complex, stateful interfaces, this level of confidence will get more elusive since the test harness can become as complex as the program its testing. I’m not sure how to write a test that verifies that my algorithm to find the shortest path always finds this path. I don’t know how many cases is enough and just having 100% coverage isn’t going to help me.

Another good application for testing is when a software project must under go some major changes. Having a good suite of regression tests is a nice way to make sure you haven’t gone too far off the map but its not a real test of equivalence.

Thinking
Let’s say you just finished making some non-trivial changes in your local copy of the software you are working on. There’s alot of things you could do before committing these changes. You can:

• Sit and think hard about what you did, trying to imagine in your mind what the impact of your changes will be on the rest of the system
• Do a little local testing by hand to see if things are going like you thought
• Look over every line of diffs to make sure the code you think you are going to commit is what you are going to commit
• Try to think objectively about whether or not this is the best way to do what you are doing
• Run a unit test suite.

The correct answer is all of the above. If you think that your test suite lets you off the hook from any of the other steps you are wrong. Btw the steps which are just thinking should be done both BEFORE and AFTER you write your code. Even though this process of thinking is imperfect (if we were perfect then we wouldn’t need tests) its still important because it helps you become smarter about the code and helps insure that the code itself doesn’t become too crappy.

TDD
I have tried hard to understand this concept of “write the tests first” as a way to do design. I have followed a few examples on the web of people demonstrating this. All of them suffer from what I call the “factorial effect”. Often when someone is showing off a new programming language they will show the cool way you can write some simple program like computing a factorial in this language. The problem of course is that its dead simple to write factorial in any language so all we have proved is that the new language makes easy things easy. Woot! For a complex system writing the tests seems to me about as hard as writing the system so how about we just do the latter?

I know that most people write these tests as they go and build up the library of tests little by little. My guess is that 90% of the time they are sort of cheating on the idea a bit, maybe even alot as their deadlines approach. But what really worries me is that I’m dead sure there is some large software project being developed right now where a team of programmers is spending months writing tests for a system they haven’t started developing. While such a project would likely end up as a huge mass of Fail, it would however succeed in taking an idea which started as “agile” and turning it into a pathetic kind of Waterfall model. The Test becomes the Spec but any Spec is almost always doomed to become less valuable than the thing its trying to specify. What the program does is reality. What the spec says or what you think the tests validate is a dream.

My other problem with this idea is that coding to the tests seems about as useful as teaching to the test does in the school system. Right now someone is probably putting some ugly if statement in their code “so the damn tests will pass” and that just makes me cry.

Quality
As I said above I don’t think tests are bad its just that they aren’t every going to be a perfect measure of correctness and so we better not make the mistake of thinking they are. However even a Perfect Test Suite (PTS) could not guarantee anything which would be worthy of the word Quality. Even if it would find and report every possible false output it still doesn’t say that much about how good the code is. I don’t mean whether the resulting application is good in the sense that iPhoto is “better” than Picasa or whatever, thats a whole other question. I’m just speaking about software quality. Passing the tests is only step 1 on the way to having good software quality. I’m actually scared there are kids out there picking up some idea that all programs capable of passing PTS are equally good. Um, no.

This would only be true if you work in the sales or marketing dept. since to these people the software is a black box, they only care that it ships on time and it works. That sounds good except that the program is not a black box. If you are programmer its more like an organism. It lives and changes and interacts with a changing environment. If you are the person in charge of maintaining it, its an organism you have to live with, so it better be good and easy to get along with on the inside as well as looking good on the outside.

Overall I just worry that focusing on the test suite is actually setting the bar far too low. Thinking your code is good because it passes the tests is like thinking your life is good because you have HDTV.