One technique that was discussed in school that I thought was really interesting was mutating your software in order to test your test suite. What I'd like is a "basic" mutator that just comments out your line and tells you if that had any effect on your test suite. For everything that I've found, they are mostly papers published by PhD's with more discussion and less implementation, seemly abandoned projects, or projects that I can't actually find code for. They all do one level above of what I'd want for a basic mutator: they change the code rather than commenting it out.
I think that showing lines that "don't matter" to the test suite in conjunction with lines that are hit / not hit with your test suite would be really, really useful.
Interesting idea, but you'd need one hell of a computer to run that suite in reasonable time :)
ReplyDeleteThe suite on the project I'm working on takes over 5min right now (approx 1700 tests, lots that interact with the DB). If each line of code that is hit was commented out, then the suite was run to find out if there was any adverse effect... that would be 5min * (x lines of executed code). Ouch!!!
I think that's the difference between "unit" tests and integration tests. Yes, when you're doing a round trip to a db etc then it'd be far, far too slow.
ReplyDeleteFor a library I'm working on that has 122 tests, it takes under 1 second (including startup time in eclipse).
Having said that, I agree with you - that's probably the reason why I've only found talk about this in university white papers. ;-)
Yes, I think it's a scaling issue for sure. Small suites (especially ones that don't interact with a container or DB) would benefit from this for sure.
ReplyDeleteAnd I agree, I'd much rather know what code I've written is useless, rather than what line of code can be mangled to produce a strange result.
- Andrew