Tải bản đầy đủ
Chapter 4. Philosophy of Test Automation
Philosophy of Test Automation
• “Test after” versus “test ﬁrst”
• Test-by-test versus test all-at-once
• “Outside-in” versus “inside-out” (applies independently to design and
• Behavior veriﬁcation versus state veriﬁcation
• “Fixture designed test-by-test” versus “big ﬁxture design upfront”
Some Philosophical Differences
Test First or Last?
Traditional software development prepares and executes tests after all software
is designed and coded. This order of steps holds true for both customer tests and
unit tests. In contrast, the agile community has made writing the tests ﬁrst the
standard way of doing things. Why is the order in which testing and development take place important? Anyone who has tried to retroﬁt Fully Automated
Tests (page 22) onto a legacy system will tell you how much more difﬁcult it is
to write automated tests after the fact. Just having the discipline to write automated unit tests after the software is “already ﬁnished” is challenging, whether
or not the tests themselves are easy to construct. Even if we design for testability,
the likelihood that we can write the tests easily and naturally without modifying
the production code is low. When tests are written ﬁrst, however, the design of
the system is inherently testable.
Writing the tests ﬁrst has some other advantages. When tests are written ﬁrst
and we write only enough code to make the tests pass, the production code tends
to be more minimalist. Functionality that is optional tends not to be written;
no extra effort goes into fancy error-handling code that doesn’t work. The tests
tend to be more robust because only the necessary methods are provided on each
object based on the tests’ needs.
Access to the state of the object for the purposes of ﬁxture setup and result
veriﬁcation comes much more naturally if the software is written “test ﬁrst.”
For example, we may avoid the test smell Sensitive Equality (see Fragile Test on
page 239) entirely because the correct attributes of objects are used in assertions
rather than comparing the string representations of those objects. We may even
ﬁnd that we don’t need to implement a String representation at all because we
Some Philosophical Differences
have no real need for it. The ability to substitute dependencies with Test Doubles
(page 522) for the purpose of verifying the outcome is also greatly enhanced because substitutable dependency is designed into the software from the start.
Tests or Examples?
Whenever I mention the concept of writing automated tests for software before the
software has been written, some listeners get strange looks on their faces. They ask,
“How can you possibly write tests for software that doesn’t exist?” In these cases,
I follow Brian Marrick’s lead by reframing the discussion to talk about “examples”
and example-driven development (EDD). It seems that examples are much easier
for some people to envision writing before code than are “tests.” The fact that the
examples are executable and reveal whether the requirements have been satisﬁed
can be left for a later discussion or a discussion with people who have a bit more
By the time this book is in your hands, a family of EDD frameworks is likely
to have emerged. The Ruby-based RSpec kicked off the reframing of TDD to
EDD, and the Java-based JBehave followed shortly thereafter. The basic design
of these “unit test frameworks” is the same as xUnit but the terminology has
changed to reﬂect the Executable Speciﬁcation (see Goals of Test Automation on
page 21) mindset.
Another popular alternative for specifying components that contain business
logic is to use Fit tests. These will invariably be more readable by nontechnical
people than something written in a programming language regardless of how
“business friendly” we make the programming language syntax!
Test-by-Test or Test All-at-Once?
The test-driven development process encourages us to “write a test” and then
“write some code” to pass that test. This process isn’t a case of all tests being
written before any code, but rather the writing of tests and code being interleaved in a very ﬁne-grained way. “Test a bit, code a bit, test a bit more”—this
is incremental development at its ﬁnest. Is this approach the only way to do
things? Not at all! Some developers prefer to identify all tests needed by the
current feature before starting any coding. This strategy enables them to “think
like a client” or “think like a tester” and lets developers avoid being sucked into
“solution mode” too early.
Test-driven purists argue that we can design more incrementally if we build
the software one test at a time. “It’s easier to stay focused if only a single test is
failing,” they say. Many test drivers report not using the debugger very much
Philosophy of Test Automation
because the ﬁne-grained testing and incremental development leave little doubt
about why tests are failing; the tests provide Defect Localization (see Goals of
Test Automation on page 22) while the last change we made (which caused the
problem) is still fresh in our minds.
This consideration is especially relevant when we are talking about unit tests
because we can choose when to enumerate the detailed requirements (tests) of
each object or method. A reasonable compromise is to identify all unit tests at
the beginning of a task—possibly roughing in empty Test Method (page 348)
skeletons, but coding only a single Test Method body at a time. We could also
code all Test Method bodies and then disable all but one of the tests so that we
can focus on building the production code one test at a time.
With customer tests, we probably don’t want to feed the tests to the developer one by one within a user story. Therefore, it makes sense to prepare all the
tests for a single story before we begin development of that story. Some teams
prefer to have the customer tests for the story identiﬁed—although not necessarily ﬂeshed out—before they are asked to estimate the effort needed to build
the story, because the tests help frame the story.
Outside-In or Inside-Out?
Designing the software from the outside inward implies that we think ﬁrst about
black-box customer tests (also known as storytests) for the entire system and
then think about unit tests for each piece of software we design. Along the way,
we may also implement component tests for the large-grained components we
decide to build.
Each of these sets of tests inspires us to “think like the client” well before we
start thinking like a software developer. We focus ﬁrst on the interface provided
to the user of the software, whether that user is a person or another piece of
software. The tests capture these usage patterns and help us enumerate the various scenarios we need to support. Only when we have identiﬁed all the tests are
we “ﬁnished” with the speciﬁcation. Some people prefer to design outside-in
but then code inside-out to avoid dealing with the “dependency problem.” This
tactic requires anticipating the needs of the outer software when writing the
tests for the inner software. It also means that we don’t actually test the outer
software in isolation from the inner software. Figure 4.1 illustrates this concept.
The top-to-bottom progression in the diagram implies the order in which we
write the software. Tests for the middle and lower classes can take advantage of
the already-built classes above them—a strategy that avoids the need for Test
Stubs (page 529) or Mock Objects in many of the tests. We may still need to use
Test Stubs in those tests where the inner components could potentially return
Some Philosophical Differences
speciﬁc values or throw exceptions, but cannot be made to do so on cue. In
such a case, a Saboteur (see Test Stub) comes in very handy.
Figure 4.1 “Inside-out” development of functionality. Development starts with
the innermost components and proceeds toward the user interface, building on
the previously constructed components.
Other test drivers prefer to design and code from the outside-in. Writing the
code outside-in forces us to deal with the “dependency problem.” We can use
Test Stubs to stand in for the software we haven’t yet written, so that the outer
layer of software can be executed and tested. We can also use Test Stubs to inject
“impossible” indirect inputs (return values, out parameters, or exceptions) into
the SUT to verify that it handles these cases correctly.
In Figure 4.2, we have reversed the order in which we build our classes. Because the subordinate classes don’t exist yet, we used Test Doubles to stand in
Figure 4.2 “Outside-in” development of functionality supported by Test
Doubles. Development starts at the outside using Test Doubles in place of the
depended-on components (DOCs) and proceeds inward as requirements for
each DOC are identiﬁed.
Philosophy of Test Automation
Once the subordinate classes have been built, we could remove the Test Doubles
from many of the tests. Keeping them provides better Defect Localization at the
cost of potentially higher test maintenance cost.
State or Behavior Veriﬁcation?
From writing code outside-in, it is but a small step to verifying behavior rather
than just state. The “statist” view suggests that it is sufﬁcient to put the SUT
into a speciﬁc state, exercise it, and verify that the SUT is in the expected state
at the end of the test. The “behaviorist” view says that we should specify not
only the start and end states of the SUT, but also the calls the SUT makes to its
dependencies. That is, we should specify the details of the calls to the “outgoing
interfaces” of the SUT. These indirect outputs of the SUT are outputs just like
the values returned by functions, except that we must use special measures to
trap them because they do not come directly back to the client or test.
The behaviorist school of thought is sometimes called behavior-driven
development. It is evidenced by the copious use of Mock Objects or Test
Spies (page 538) throughout the tests. Behavior verification does a better
job of testing each unit of software in isolation, albeit at a possible cost of
more difficult refactoring. Martin Fowler provides a detailed discussion of
the statist and behaviorist approaches in [MAS].
Fixture Design Upfront or Test-by-Test?
In the traditional test community, a popular approach is to deﬁne a “test bed”
consisting of the application and a database already populated with a variety of
test data. The content of the database is carefully designed to allow many different test scenarios to be exercised.
When the ﬁxture for xUnit tests is approached in a similar manner, the test
automater may deﬁne a Standard Fixture (page 305) that is then used for all
the Test Methods of one or more Testcase Classes (page 373). This ﬁxture may
be set up as a Fresh Fixture (page 311) in each Test Method using Delegated
Setup (page 411) or in the setUp method using Implicit Setup (page 424). Alternatively, it can be set up as a Shared Fixture (page 317) that is reused by many
tests. Either way, the test reader may ﬁnd it difﬁcult to determine which parts of
the ﬁxture are truly pre-conditions for a particular Test Method.
The more agile approach is to custom design a Minimal Fixture (page 302)
for each Test Method. With this perspective, there is no “big ﬁxture design upfront” activity. This approach is most consistent with using a Fresh Fixture.
When Philosophies Differ
We cannot always persuade the people we work with to adopt our philosophy,
of course. Even so, understanding that others subscribe to a different philosophy
helps us appreciate why they do things differently. It’s not that these individuals
don’t share the same goals as ours;1 it’s just that they make the decisions about
how to achieve those goals using a different philosophy. Understanding that different philosophies exist and recognizing which ones we subscribe to are good
ﬁrst steps toward ﬁnding some common ground between us.
In case you were wondering what my personal philosophy is, here it is:
• Write the tests ﬁrst!
• Tests are examples!
• I usually write tests one at a time, but sometimes I list all the tests I can
think of as skeletons upfront.
• Outside-in development helps clarify which tests are needed for the
next layer inward.
• I use primarily State Veriﬁcation (page 462) but will resort to Behavior
Veriﬁcation (page 468) when needed to get good code coverage.
• I perform ﬁxture design on a test-by-test basis.
There! Now you know where I’m coming from.
This chapter introduced the philosophies that anchor software design, construction, testing, and test automation. Chapter 5, Principles of Test Automation,
describes key principles that will help us achieve the goals described in Chapter
3, Goals of Test Automation. We will then be ready to start looking at the overall test automation strategy and the individual patterns.
For example, high-quality software, ﬁt for purpose, on time, under budget.
This page intentionally left blank
Principles of Test
About This Chapter
Chapter 3, Goals of Test Automation, described the goals we should strive to
achieve to help us be successful at automating our unit tests and customer tests.
Chapter 4, Philosophy of Test Automation, discussed some of the differences in
the way people approach software design, construction, and testing. This provides the background for the principles that experienced test automaters follow
while automating their tests. I call them “principles” for two reasons: They are
too high level to be patterns and they represent a value system that not everyone
will share. A different value system may cause you to choose different patterns
than the ones presented in this book. Making this value system explicit will, I
hope, accelerate the process of understanding where we disagree and why.
When Shaun Smith and I came up with the list in the original Test Automation
Manifesto [TAM], we considered what was driving us to write tests the way we
did. The Manifesto is a list of the qualities we would like to see in a test—not a
set of patterns that can be directly applied. However, those principles have led us
to identify a number of somewhat more concrete principles, some of which are
described in this chapter. What makes these principles different from the goals is
that there is more debate about them.
Principles are more “prescriptive” than patterns and higher level in nature. Unlike patterns, they don’t have alternatives, but rather are presented in a “do this
because” fashion. To distinguish them from patterns, I have given them imperative
names rather than the noun-phrase names I use for goals, patterns, and smells.
Principles of Test Automation
For the most part, these principles apply equally well to unit tests and storytests. A possible exception is the principle Verify One Condition per Test, which
may not be practical for customer tests that exercise more involved chunks of
functionality. It is, however, still worth striving to follow these principles and to
deviate from them only when you are fully cognizant of the consequences.
Also known as:
Principle: Write the Tests First
Test-driven development is very much an acquired habit. Once one has “gotten
the hang of it,” writing code in any other way can seem just as strange as TDD
seems to those who have never done it. There are two major arguments in favor
of doing TDD:
1. The unit tests save us a lot of debugging effort—effort that often fully
offsets the cost of automating the tests.
2. Writing the tests before we write the code forces the code to be designed
for testability. We don’t need to think about testability as a separate
design condition; it just happens because we have written tests.
Principle: Design for Testability
Given the last principle, this principle may seem redundant. For developers
who choose to ignore Write the Tests First, Design for Testability becomes an
even more important principle because they won’t be able to write automated
tests after the fact if the testability wasn’t designed in. Anyone who has tried
to retroﬁt automated unit tests onto legacy software can testify to the difﬁculty
this raises. Mike Feathers talks about special techniques for introducing tests in
this case in [WEwLC].
Also known as:
Front Door First
Principle: Use the Front Door First
Objects have several kinds of interfaces. There is the “public” interface that clients
are expected to use. There may also be a “private” interface that only close friends
should use. Many objects also have an “outgoing interface” consisting of the used
part of the interfaces of any objects on which they depend.
The types of interfaces we use inﬂuence the robustness of our tests. The use of
Back Door Manipulation (page 327) to set up the ﬁxture or verify the expected
outcome or a test can result in Overcoupled Software (see Fragile Test on page
239) that needs more frequent test maintenance. Overuse of Behavior Veriﬁcation (page 468) and Mock Objects (page 544) can result in Overspeciﬁed Software
(see Fragile Test) and tests that are more brittle and may discourage developers
from doing desirable refactorings.