Yacfg tests use mocking too much

[jiridanek]

Description

One lesson we learned the hard way is the danger of overusing mocking frameworks, which allow you to easily create test doubles (we will discuss mocking frameworks in more detail later in this chapter). When mocking frameworks first came into use at Google, they seemed like a hammer fit for every nail—they made it very easy to write highly focused tests against isolated pieces of code without having to worry about how to construct the dependencies of that code. It wasn’t until several years and countless tests later that we began to realize the cost of such tests: though these tests were easy to write, we suffered greatly given that they required constant effort to maintain while rarely finding bugs. The pendulum at Google has now begun swinging in the other direction, with many engineers avoiding mocking frameworks in favor of writing more realistic tests.

Even though the practices discussed in this chapter are generally agreed upon at Google, the actual application of them varies widely from team to team. This variance stems from engineers having inconsistent knowledge of these practices, inertia in an existing codebase that doesn’t conform to these practices, or teams doing what is easiest for the short term without thinking about the long-term implications.

https://abseil.io/resources/swe-book/html/ch13.html

Steps to Reproduce the Problem

yacfg/tests/test_yacfg/files/test_select_profile_file.py

Lines 37 to 46 in e58bad3

	@mock.patch("yacfg.files.get_profiles_path", side_effect=fake_profiles_path)
	@mock.patch("os.path.isfile", side_effect=(True, True, True))
	@mock.patch("os.path.abspath", side_effect=fake_os_abspath)
	def test_user_true(*_):
	"""User Specified profile selection"""
	expected_name = "user_profile.yaml"
	expected_path = os.path.dirname(fake_os_abspath(""))
	result_name, result_path = yacfg.files.select_profile_file(expected_name)
	assert result_name == expected_name
	assert result_path == expected_path

Let me explain

@mock.patch("os.path.isfile", side_effect=(True, True, True))

This thing drives some random helper function through a desired codepath when the tested function does three isfile() calls.
(There are some other tests with different combinations of True/False for isfile()).
Such test is extremely tightly coupled to the particular implementation and is extremely fragile.

Actual Behavior

The primary issue with interaction testing is that it can’t tell you that the system under test is working properly; it can only validate that certain functions are called as expected. It requires you to make an assumption about the behavior of the code; for example, “If database.save(item) is called, we assume the item will be saved to the database.” State testing is preferred because it actually validates this assumption (such as by saving an item to a database and then querying the database to validate that the item exists).

Another downside of interaction testing is that it utilizes implementation details of the system under test—to validate that a function was called, you are exposing to the test that the system under test calls this function. Similar to stubbing, this extra code makes tests brittle because it leaks implementation details of your production code into tests. Some people at Google jokingly refer to tests that overuse interaction testing as change-detector tests because they fail in response to any change to the production code, even if the behavior of the system under test remains unchanged.

Specifications

• yacfg Version: e58bad3
• python Version:
• Platform:

Notes