The heart of the argument presented is that using mocks in unit tests is problematic because if an interface is changed, that will possibly break every test that involves mocking that interface and that's friction to making changes. This is silly. If you use real implementations and change the interface, you have the exact same problem except that your configuration/setup is also likely to have to change in subtle ways.
Let's just assume the absurd notion that a choice of creating tech debt or changing a common interface is a real choice. If there was a serious change that could not be accounted for with easy test changes, I'm not the only one to see the tests commented out with a "TODO: fix these". Developers tend to be pragmatic.
If you want to change code you will always measure the effect it will have on the code and tests are incidental, not the primary concern. Make it work. Make it right. Make it fast. I want to be able to trigger all code paths and exceptions (make it good). Using a real dependency, I would be left in the unfortunate situation of depending on knowledge of the internals of that dependency. It may not allow me to execute specific paths via pure configuration at all.
I don't think using real dependencies is a good idea at all for unit tests. Integration tests are a different and I do fear that they are being confused.Reply
Agreed. Push all external dependencies (DBs, APIs, etc) to the edge. That's it ! Why do you need a functional core though ? You can use decoupled classes and an ioc container that does the wiring for you.Reply
inb4 someone saying something about mocking or dependency injection for testingReply
I’ve spent the last few years working on test infrastructure for blockchain projects so this question is regularly on my mind, and I’ve found the best solution is to just include the dependencies. Compute and storage is cheap enough that you can spin up these external tools and the extra level of assurance makes me sleep easier, and while flakey tests are theoretically an issue I rarely find them (and they tend to be programmatically fixable).
If a unit test requires an external dependency then just use an integration test (or check it’s covered by system tests) and leave it at that.Reply
As a potential counter-argument, the use of mocks can enable testing of functionality that the current concrete implementation doesn't exercise. It's easier than one would think to accidentally rely on implementation details rather than coding just to the interface (and optionally any documented restrictions to that interface).
They explicitly call out clocks as a source of non-determinism that probably should be mocked, but I'll re-use them as an example anyway because everyone is familiar with them: it's extraordinarily useful for the tests to execute nearly immediately rather than actually waiting on a clock, and rare behavior like a clock running backward, two consecutive timings being identical within the clock's resolution, or whatever other weird artifacts that your code should handle are definitely better explicitly tested rather than not mocking the clock. Other domain-specific interfaces are often similarly able to exhibit a weird edge case that ought to be explicitly tested (rather than accidentally relying on a "nice" implementation) if you really want to unit test the callers and not integration test the coupled system.Reply
Finally! Somebody that has the guts to say the truth.Reply
I see another soul has rediscovered the universal truth that unit testing is only pleasant in functional programsReply
So TLDR - write integration tests as much as possible and only mock external dependencies like network calls?Reply
If some method has several dependencies that are tangentially related with what I'm testing, I'm going to mock the shit out of them, and let people know we should do something about it.
Testing "the real thing" sounds fun until the budget for new features skyrockets, because several man-weeks are needed to get decent coverage. Your client will hate it, and your client's clients will hate it even more.Reply
So... in order to do unit testing, you should shift your entire system to be written in functional style, and then write integration tests instead of unit tests? Because that's what you get when you use the real components.Reply
This article espouses what is known as "classicist testing" school of thought and rejects "mockist/London-style testing". I wholeheartedly agree with it. I have been championing it in my team of 20+ devs since many years now, to great effect. You can read more about it in the excellent book by Vladimir Khorikov from January 2020 titled "Unit Testing Principles, Practices, and Patterns".Reply
There are also Function Tests and Integration Tests. Seems the author took Unit Tests for all three.Reply
I 100% agree, but rarely have found much to put in a functional core. Most everything these days is distributed microservices talking to each other (different argument) and having more than a couple lines in a row that don't call some other service, even in a well-factored app, is almost a cause for celebration.Reply
This is almost right except for two points:
For fakes, spin up the real thing. If you’re not able to model your database transactions deterministically, then your transactions could themselves be flawed and tests are great way to catch that.
Deterministic tests are not a goal in and of themselves. Controlled non determinism is valuable. This is popularized in various frameworks under the names of property checking and fuzzing which will let you know the seed to use for the failure for example so that while the runs don’t have the exact same input/output for every invocation, you get better coverage of your test space and can revisit the problematic points at any time. If you’re doing numeric simulation, make sure you are using a PRNG that’s seedable and that you log the seed at the start if you’re using a seed (and make sure time is an input parameter). Why is this technique valuable? You transitively get increasing code coverage for free through CI/coworkers running the tests AND you have a way to investigate issues sanely.Reply
In other words, shouldn't we prefer integration testing, or perhaps partial integration testing. This requires an initial effort of setting up your test framework/environment, but in my experience integration tests provide good value for the time you put into testing.
Rather than a granular test on a single class, test the orchestration of many classes. You end up hitting big % of code. As always, depends on the project. If we're building a rocket ship, you need both granular testing and coarse testing.Reply
I prefer mocks over real services because I onboard many people to a repo. Too many services attached to unit tests turns into a fatigue until eventually no one runs the tests any more I found.Reply
I agree that fakes are very good for testing, with one caveat: the fakes must be owned by the same folks that own the real implementation.
When fakes are owned by anyone else, e.g. the folks writing the system under test, there's a high risk that the semantics of the fake and the real implementation will diverge, rendering the tests that use the fakes much less useful.Reply
I have two services that communicate over the network. Neither does anything useful without data from the other. So I write my tests like this article suggests and inject at the boundaries of the services, the network boundary in this case. I've built tests that now only test my idea of what the services will return. IMO these aren't particularly useful tests: what happens when the remote service starts returning unexpected data or errors due load problems? I guess my point is just this article is good advice: prefer testing your actual code and dependencies, but unit testing isn't a replacement for integration testing.Reply
I first learned this concept from Gary Bernhardt’s video Boundaries.Reply
When you build and sell a network security appliance, the appliance has to work properly all the time. You have to give guarantees of how it will perform, guarantee that its functions do what you claim they do, and eliminate all possible bugs. So you have to test it in every configuration possible.
So you make models and build automation test frameworks and labs full of prototype equipment so you can run 10,000 tests an hour, 24/7. You automate setting up networks and nodes and passing live traffic to test detectors and rule sets. You use custom hardware to emulate ISP-levels of real traffic.
You can't really mock anything. You have to be sure each function will perform as you describe. So most of this testing is end-to-end testing. Using a mock wouldn't tell you if the millions of code paths are working correctly; they'd mostly just tell you if the syntax/arguments/etc of a call were correct. Unit tests are basically one step above evaluating that your code compiles correctly, but it's not testing the code works as expected. End-to-end tests are what you can rely on, because it's real traffic to real devices.
That's gonna look different for a lot of your apps, but for web apps that means getting real familiar with synthetic tests and how to test with different browsers. For SDKs it means end-to-end tests for the whole stack of functions, which is a lot more testing than you may have expected. For APIs it means getting the consumers of your APIs involved in end-to-end testing. It also means "testing in production", in the sense of spinning up new production deployments and using those as your end-to-end testing targets (rather than having a dedicated "dev/test/cert" environment which is always deviating from production). This can take a significant effort to adapt in legacy systems, but for Greenfield IaC projects is not very hard to set up.Reply
Related: Why I don't mock (2014). https://blog.metaobject.com/2014/05/why-i-don-mock.htmlReply
In database land you can even freely bring up a number of _proprietary_ databases without needing an account or license. I spin up MySQL, Postgres, Oracle and SQL Server databases in containers in Github Actions for tests. Unfortunately IBM DB2 seems to require a license key to spin up their container. :(Reply
Amen to the idea.
- Prefer real objects, or fakes over mocks. It will make your tests usually more robust.
- Use mocks when you must: to avoid networking, or other flaky things such as storage.
- Use mocks for “output only objects”, for example listeners, or when verifying the output for some logging. (But, prefer a good fake)
- Use mocks when you “need to get shit done”, it’s the easiest way to add tests in an area that has almost none, and the code is not designed to be easily testable. But remember this is tech debt, and try to migrate towards real objects over time.
That’s my short advice I told many times. So might as well comment with it here.Reply
If you are testing the externalities, they aren't unit tests, but integration tests.
The better advice is: continue to isolate unit tests away from real dependencies, and ALSO have integration tests that test the way the package connects to dependencies (and the other packages in the software dependent on it).Reply
Great article, not the first time I read this argument against the usage of mocks; I never understood though how to solve the problem of the explosion of the path that must be tested (usually exponential).
As an example, consider a single API that uses ~3 services, and these 3 services have underneath from 1 to 5 other internal or external dependencies (such as time, an external API service, a DB repository, and so on). How can I test this API, the 3 services, and their underneath dependencies without exponential paths to test - I want to be able to cover all the paths of my code, and ensure that my test only tests a single thing (either the API, the service, or the dependency interface); otherwise, it is not an unit test.
I always felt like that these type of tests without mocks works super-nice in nice situations without any external, or even complex but internal, dependency; otherwise, it becomes very very hard to test ONLY what I want, and not all the dependencies underneath.
Mocks allow me to stub the behaviour of a service/dependency that I can test in a separate fashion, covering all the paths, and ensuring that each unit test covers a single unit of my code, and not the integration of all my components.Reply
I have used this approach and now I perceive mocks as a bad code smell. It's worth noting that long and complicated flows need to be wrapped in sagas (which are just imperative shells) and things can get not so easy. I would love to hear what are other clean alternatives in such cases. Still, "functional core, imperative shell" is the way to go, code really fits in the head and tests actually make sense.Reply
I worked somewhere with a hard rule that you are not allowed to test internal classes.
So if I write my own sort algorithm I am not allowed to unit test that class unless I make it public.
But if the sort algorithm is to solve a specific sub problem in a library there is no need to make it public - it may not make much sense.
So I had to test it “through” its consumer class(es) that is public. For illustration sake lets say a MVC controller mocked up to the eyeballs with ORM mocks, logging mocks etc.
This bugged me because having direct access to a functional core I can quickly amplify the number of test cases against it and find hidden potential bugs much quicker.Reply
the thing i see that makes this test brittle is the use of relative dates and durations that break across day light savings time changesReply
I don’t get it. When I unit test, I want to only test what I’m testing. A dependency or the result of a dependency is not what I’m testing. So I mock the result of that dependency to unlink the test from the dependency. If an interface changes I generally want to know anyway, as the test may be different or even obsolete depending on the change.Reply