Agileee 2010: J. B. Rainsberger: Integrated Tests are A Scam
Disclaimer: text below is compilation of notes I made on Agileee 2010 conference, listening to different speakers. I do it to keep knowledge I got on conference, share it with my colleagues and anyone else who interested. It is only about how I heard, interpret, write down the original speech. It also includes my subjective opinion on some topics. So it could not 100% reflects author opinion and original ideas.
It a very dramatic beginning - “Integrated Tests are a scam”. a self-replicating virus that takes over your project and burdens you with long-running, fragile, hard-to-understand test suites - a sentence JBrains started his speech!
What is integrated tests?
Integrated tests checks whole system, not only collaboration between some components. It means any tests where the failure of the test with unjustifiable failures. It is bad, because it is a distraction and give no value.
Integrated tests are slow
Integrated tests involves DB, network that makes tests are slow.
Integrated tests are brittle
If you change one small thing you have to fix a lot of tests. JBrains gives a very nice and common example. “Foreign Keys hell” - you have a foreign keys in DB, so to insert/remove record to one table, you have to insert/remove record to another table.. and so on. Preparation tests data is very expensive operation, complexity goes as exponential form. Eventually it came up that change that took 1.5 hour start to take 1 days.
Goal of unit testing
Goal of unit testing is to have quick and correct result. Test failure should point to exact problem in object to be tested. In case of integrated tests, it is possible to have multiple failures, with very few actual understanding of what is wrong.
What to do?
JBrains introduces term of - Collaboration tests (Interaction tests) - using mock objects or tests doubles. Collaboration is observed as Client-Server architecure. The object that is being tested is Client, the object it depends on is Server. We create interfaces, to make it possible to substitute implementation (using test doubles), we simulate behavior for object we testing. Client object is a completely tested, test double simulate behavior of Server.
This is greatly works, tests pass, but we found a problems. The issue is that object doesn’t always work as it is expected by tests and having Collaboration tests only is not enough. We need another type of test - Contract tests.
Contract tests focuses on Server side, this are ones that actually do SQL to DB, requests by network etc. The goal here is to focus on Contract (or interface) of object and test interface, not exact implementation. Since contract tests are defined, now it is possible to reuse them for any particular implementation. As an example, we could review IList class, that could have different implementations (as array list, linked list etc.), but it have one interface with simple operations like - add, remove, at. We create a number of tests, for all contract operations. Such tests could be put to common class, with abstract operation of object creation (template method pattern), so tests of particular implementation (array list for instance) just inherit common class, override creation function to return array list method and reused all already defined contract tests.
What are benefits?
Combining these 2 types of tests give us fast, isolated suites, failure is suite should point to exact issue avoiding creation of complex integration tests.
Later on JBrains gives some mathematical proves of differences between number collaboration/contacts tests and integrated tests, that I just do not want to write here, cause afraid to incorrectly interpret the information. The keynote of the speech is rather simple: forget about complex and heavyweight integration tests, do as much as possible with test doubles running everything in memory, do contract testing for platform depended objects.
You could find more, here: http://blog.jbrains.ca/integrated_tests_are_a_scam
Disclaimer: text below is compilation of notes I made on Agileee 2010 conference, listening to different speakers. I do it to keep knowledge I got on conference, share it with my colleagues and anyone else who interested. It is only about how I heard, interpret, write down the original speech. It also includes my subjective opinion on some topics. So it could not 100% reflects author opinion and original ideas.
It a very dramatic beginning - “Integrated Tests are a scam”. a self-replicating virus that takes over your project and burdens you with long-running, fragile, hard-to-understand test suites - a sentence JBrains started his speech!
What is integrated tests?
Integrated tests checks whole system, not only collaboration between some components. It means any tests where the failure of the test with unjustifiable failures. It is bad, because it is a distraction and give no value.
Integrated tests are slow
Integrated tests involves DB, network that makes tests are slow.
Integrated tests are brittle
If you change one small thing you have to fix a lot of tests. JBrains gives a very nice and common example. “Foreign Keys hell” - you have a foreign keys in DB, so to insert/remove record to one table, you have to insert/remove record to another table.. and so on. Preparation tests data is very expensive operation, complexity goes as exponential form. Eventually it came up that change that took 1.5 hour start to take 1 days.
Goal of unit testing
Goal of unit testing is to have quick and correct result. Test failure should point to exact problem in object to be tested. In case of integrated tests, it is possible to have multiple failures, with very few actual understanding of what is wrong.
What to do?
JBrains introduces term of - Collaboration tests (Interaction tests) - using mock objects or tests doubles. Collaboration is observed as Client-Server architecure. The object that is being tested is Client, the object it depends on is Server. We create interfaces, to make it possible to substitute implementation (using test doubles), we simulate behavior for object we testing. Client object is a completely tested, test double simulate behavior of Server.
This is greatly works, tests pass, but we found a problems. The issue is that object doesn’t always work as it is expected by tests and having Collaboration tests only is not enough. We need another type of test - Contract tests.
Contract tests focuses on Server side, this are ones that actually do SQL to DB, requests by network etc. The goal here is to focus on Contract (or interface) of object and test interface, not exact implementation. Since contract tests are defined, now it is possible to reuse them for any particular implementation. As an example, we could review IList class, that could have different implementations (as array list, linked list etc.), but it have one interface with simple operations like - add, remove, at. We create a number of tests, for all contract operations. Such tests could be put to common class, with abstract operation of object creation (template method pattern), so tests of particular implementation (array list for instance) just inherit common class, override creation function to return array list method and reused all already defined contract tests.
What are benefits?
Combining these 2 types of tests give us fast, isolated suites, failure is suite should point to exact issue avoiding creation of complex integration tests.
Later on JBrains gives some mathematical proves of differences between number collaboration/contacts tests and integrated tests, that I just do not want to write here, cause afraid to incorrectly interpret the information. The keynote of the speech is rather simple: forget about complex and heavyweight integration tests, do as much as possible with test doubles running everything in memory, do contract testing for platform depended objects.
You could find more, here: http://blog.jbrains.ca/integrated_tests_are_a_scam