Separate Unit, Integration, and Functional Tests for Continuous Delivery
GUI Tests in the top of this pyramid is the Functional test in my meaning. It is sometime called End-to-End Tests or System Tests. In addition, let’s treat API, Integration, and Component Tests in the pyramid as Integration Tests. Typically, most apps require Unit and Functional Tests. For complex app, it may also require Integration Test as well. We can call all of them as Test Suite.
In practical, you might feel that these 3 main layers are not totally clear-cut and kind of overlapped with each other. One big unit (which call another smaller units) can be a component or an API. Testing via UI could be either functional or API test.
However, when you want to automate these tests (e.g. for continuous delivery). It is very important to separate them especially the Unit Tests because of their different behaviors and purposes you expect from their test results.
First, let’s learn the differences of them:
- Validate individual component of the app
- Assertions test the component’s input and output parameters
- Testing is done from developer’s perspective
- Real-time developer feedback
- Simple, fast, and has characteristics of F.I.R.S.T. Principle
- No uncontrollable dependencies or interfaces to other component e.g. database access, file/log read/write, remote/API calls, etc.
- Always give the same consistent result when executing anywhere (DEV/QA/Prod servers), anytime (next day, next month, next year, next 5 years).
- Generate good bug report which could answer these 4 questions:
- Which component is under test?
- What is the expected behavior?
- What was the actual result?
- What is the expected result?
- Validate components collaboration
- Assertions may test component’s input and output parameters (API), UI, or side-effects (e.g. database, file, log, etc.)
- Testing may be done from developer’s (e.g. API) or end-user’s perspective (e.g. UI)
- They are simply Unit Tests (no matter how big it is) with uncontrollable dependencies or interfaces e.g. database access, file/log read/write, remote/API calls, etc.
- Give unconsistent result when executing due to their unpredictable dependencies
Unit vs. Integration Test
To understand more the difference between these twos, let’s take a look into this example:
Suppose you have several functions (in a modular program) with calling relationships as shown in below diagram. Function E needs to read a table from database to fulfill its operation so it has dependency as well as B and A which calling E.
In this case, you can only write Unit Tests for C, D, F, and G. If you write tests for E, B, or A, then they are not Unit Tests but Integration Tests which are not the purpose of xUnit framework which is the tool for writing Unit Tests. The most dangerous thing was you mix Unit and Integration Tests together and how do you trust the bug report?
So if you want E, B, and A to be “testable” by Unit Tests, there are ways to workaround. One way is using a mock/fake database if available in your platform to ensure that E will always get the same result.
You may create another function H which only does selecting data from the database which is required by E. You let the caller calls H first to get the required data and then pass (inject) this data (dependency) into E (through B and A). Now, you can write Unit Tests for E, B, and A.
Integration Tests should always be kept separate from Unit Tests.
- Ensure the whole app or system works as expected
- Also known as End-to-End Tests (from Front-end UI to the back-end database) or System Tests (set of Functional Tests)
- Assertions primarily test the actual user interface output
- Testing is done from end-user’s perspective
- Typically have thorough tests for “happy paths” — ensuring the critical app capabilities, such as user logins, signups, purchase work flows, and all the critical user workflows all behave as expected.
There is another kind of test which is also a Functional Tests that I would like to mentioned i.e. Smoke Tests
- A subset of production-safe functional tests
- Run to ensure that none of the critical functionality was broken during the deploy process.
- The purpose is “ You don’t want your users to find the bugs before you do”
- You should also maintain this kind of functional test separated from others in your automation test suite so you can run them in production after you deploy.
When you have automated test suite, you might want to run each type of tests differently in each environment, for example:
- In Development: Unit Tests
- In Staging/QA: Full Test Suite
- In Production: Smoke Tests