Lately I've spent some time thinking and discussing pros and cons of unit and integration tests. Basically I think you need both but they have totally different characteristics and should therefore be combined carefully. Before we dive in, lets define the terms for the sake of this discussion since I know there are at least as many definitions of "integration test" as there are projects out there.
Unit test: Test focused on one or a closely related set of classes. Defined and run outside any run-time environment (container). All dependencies are mocked/stubbed.
Integration test: Test focused on one or several (sub)systems. Defined in code and run in the run-time environment. Some, but not all, dependencies might be stubbed.
With definitions done, let's go for some characteristics I've found to be true.
Unit test - pros:
1: Small units (single classes or a small set) are tested independently without interferences from the surrounding environment.
2: Can be run instantly with a few keystrokes - no deployment necessary, the whole (sub-) system doesn't even have to compile. Makes it easy to test incrementally.
3: Supports Test Driven Development.
4: With small units it is easier to cover all interesting execution paths.
Unit test - cons:
1: Mocking frameworks could be a challenge to master, but they are oh-so useful when you get to know them.
2: Different tests must be in sync to ensure that individual units work together. Can be tricky, especially when dealing with semantic changes in the interface of a class/unit. Requires disciplined top-down TDD.
A colleague of mine pointed out that my con #1 actually could be a pro if using TDD (as in writing test and production code simultanouosly) since having a hard time mocking dependencies would be an incentive towards writing code with fewer dependencies, which in general is a good design practice. However, my experience is that a good mocking framework is necessary to effectively write unit tests, and as with every tool there is learning curve. Now for integration tests.
Integration test - pros:
1: Tests the code in its true context with few or no stubbed dependencies.
2: Supports testing of non-functional requirements, such as performance, security and transaction boundaries.
Integration test - cons:
1: Can only be run after the (sub-) system has been fully developed, built and deployed. Tests are written as an afterthought instead of driving the development.
2: The code-build-deploy-test-cycle takes a long time which make debugging cumbersome.
3: Might require other parts of the system than those under test to be fully functional in order to bring the system into the state for the test to start. This creates strong dependencies on parts of the system that are not targeted by the test.
Most of my cons for integration tests only become severe when integration tests are used to test business logic, which I've seen from programmers claiming unit tests with mocked dependencies being too hard to write. When used to test real integration issues (such as inter-system communication or database connectivity), the cons sort of goes away since those issues cannot be detected anyway prior to having at least a "walking skeleton" implementation of the architecture.
In conclusion, I think both unit and integration tests (as well as automated systems/acceptance tests) are really needed in a software project. However, each need to be used propely. For ensuring implementation of business logic I think unit tests are the proper choice. It might be a bit harder to write because you would have to mock dependencies instead of relying on the state currently set up in the database, but at the time of adding functionality or refactoring your domain code unit tests are so much easier and thereby cheaper to run since you do not need to bring your complete system into a known state, you do not even have to compile, let alone install, your complete system. Striving for a good coverage (some say a 100%, I don't) of your domain logic, preferably using a test-first approach, you also tend to achieve a better design since well designed objects are easier to test.
Side note: Today we got lucky like a crazy when we found a bug in the business logic because we by chance happened to run the integration test designed to test that feature. It could be, and was, claimed to be an argument for the benefit of using integration tests for testing business logic. However, I think it is not. For starters, the bug had been in the code for about five months. Not even the guy writing the feature knew were in the system the code was located. We had to spend several hours to find the implementation. After that, the bug was easy to spot. It gives a perspective on cycle time and why quick feedback is important. Secondly, the bug could not be found by unit test since there was no coverage on this part of the system. I'm pretty sure the reason for that is that it is really hard to write unit tests for code directly manipulating a JPA entity manager. So, the problem was really that business logic was burried in the database access code and hence not unit tested. If a unit test had been attempted I'm pretty sure the logic had been moved into a more test friendly design. Which of course would have been better from other perspectives as well.
Unit test: Test focused on one or a closely related set of classes. Defined and run outside any run-time environment (container). All dependencies are mocked/stubbed.
Integration test: Test focused on one or several (sub)systems. Defined in code and run in the run-time environment. Some, but not all, dependencies might be stubbed.
With definitions done, let's go for some characteristics I've found to be true.
Unit test - pros:
1: Small units (single classes or a small set) are tested independently without interferences from the surrounding environment.
2: Can be run instantly with a few keystrokes - no deployment necessary, the whole (sub-) system doesn't even have to compile. Makes it easy to test incrementally.
3: Supports Test Driven Development.
4: With small units it is easier to cover all interesting execution paths.
Unit test - cons:
1: Mocking frameworks could be a challenge to master, but they are oh-so useful when you get to know them.
2: Different tests must be in sync to ensure that individual units work together. Can be tricky, especially when dealing with semantic changes in the interface of a class/unit. Requires disciplined top-down TDD.
A colleague of mine pointed out that my con #1 actually could be a pro if using TDD (as in writing test and production code simultanouosly) since having a hard time mocking dependencies would be an incentive towards writing code with fewer dependencies, which in general is a good design practice. However, my experience is that a good mocking framework is necessary to effectively write unit tests, and as with every tool there is learning curve. Now for integration tests.
Integration test - pros:
1: Tests the code in its true context with few or no stubbed dependencies.
2: Supports testing of non-functional requirements, such as performance, security and transaction boundaries.
Integration test - cons:
1: Can only be run after the (sub-) system has been fully developed, built and deployed. Tests are written as an afterthought instead of driving the development.
2: The code-build-deploy-test-cycle takes a long time which make debugging cumbersome.
3: Might require other parts of the system than those under test to be fully functional in order to bring the system into the state for the test to start. This creates strong dependencies on parts of the system that are not targeted by the test.
Most of my cons for integration tests only become severe when integration tests are used to test business logic, which I've seen from programmers claiming unit tests with mocked dependencies being too hard to write. When used to test real integration issues (such as inter-system communication or database connectivity), the cons sort of goes away since those issues cannot be detected anyway prior to having at least a "walking skeleton" implementation of the architecture.
In conclusion, I think both unit and integration tests (as well as automated systems/acceptance tests) are really needed in a software project. However, each need to be used propely. For ensuring implementation of business logic I think unit tests are the proper choice. It might be a bit harder to write because you would have to mock dependencies instead of relying on the state currently set up in the database, but at the time of adding functionality or refactoring your domain code unit tests are so much easier and thereby cheaper to run since you do not need to bring your complete system into a known state, you do not even have to compile, let alone install, your complete system. Striving for a good coverage (some say a 100%, I don't) of your domain logic, preferably using a test-first approach, you also tend to achieve a better design since well designed objects are easier to test.
Side note: Today we got lucky like a crazy when we found a bug in the business logic because we by chance happened to run the integration test designed to test that feature. It could be, and was, claimed to be an argument for the benefit of using integration tests for testing business logic. However, I think it is not. For starters, the bug had been in the code for about five months. Not even the guy writing the feature knew were in the system the code was located. We had to spend several hours to find the implementation. After that, the bug was easy to spot. It gives a perspective on cycle time and why quick feedback is important. Secondly, the bug could not be found by unit test since there was no coverage on this part of the system. I'm pretty sure the reason for that is that it is really hard to write unit tests for code directly manipulating a JPA entity manager. So, the problem was really that business logic was burried in the database access code and hence not unit tested. If a unit test had been attempted I'm pretty sure the logic had been moved into a more test friendly design. Which of course would have been better from other perspectives as well.
