Acceptance Testing
Andy Lindeman
Levels of Testing
Roughly speaking...
- Unit Tests
- Integration Tests
- Acceptance Tests
Unit Tests
- Isolation
- Dependencies often mocked or stubbed
- No filesystem, network, database (
Rails "unit" tests) - Programmer Tests
Acceptance Tests
- End-to-end (as much as is feasible)
- No implementation details (black box)
- Often based on a user story
- Can be run locally or remotely
- Customer Tests
A few quick examples...
Unit Test
describe BankAccount do
specify "new accounts have a zero balance" do
account = BankAccount.new
expect(account.balance).to eq(BigDecimal("0.00"))
end
end
Acceptance Test
(Cucumber/Gherkin)
As a depositor
In order to safeguard my money
I want to hold it in a bank account
Given I am a depositor
When I open a new bank account
Then my balance is $0.00
Acceptance Test
Given /I am a depositor/ do
@user = Factory(:depositor)
sign_in(@user)
end
When /I open a new bank account/ do
click_button "New Account"
end
Then /my balance is $([0-9.]+)/ do |bal|
expect(page).to have_text("Balance: $#{bal}")
end
Workflow
(roughly)
- Translate a user story into an acceptance test (or few)
- Watch the acceptance test fail
- Write a failing unit test
- Watch the unit test fail
- Make the unit test pass
- Refactor
- Watch the acceptance test get a bit farther
- Repeat until all tests are passing
- Confidently deliver the feature
Good Practices
In my very humble opinion ...
Seriously, I'm still learning this stuff too
But here's some things I've observed
Hide Implementation Details
Given I am a newsletter publisher
When I send out an electronic newsletter
Then when Resque jobs are run
And after 5 minutes passes
Then the newsletter is received by all recipients
Hide Implementation Details
Given I am a newsletter publisher
When I send out an electronic newsletter
Then all recipients receive my newsletter within 5 minutes
How would your customer talk to others about this feature? How would a salesperson?
Avoid "Programming"
Given I am a user named "Alice"
And there is a user "Bob" who is my friend
And there is a user "Charlie" who is a friend of "Bob"
When "Alice" invites "Bob" to a party
But "Alice" does not invite "Charlie" to the same party
Then "Alice" receives a message
"Do you also want to invite Charlie?"
No customer or user would ever describe the feature like this ...
Avoid "Programming"
As a party planner
In order to have the best party
I want to invite as many fun people as possible
Given that I am planning a party
When I invite my friend Alice
Then I am asked if I might want to invite her friend Bob
Focus
Given I do not yet have an account with bank.example.com
When I register for an account
And I correctly decipher the CAPTCHA image proving
I am a human
And I correctly input the code on my RSA SecurID dongle
And I deposit $20.00
Then my account balance is $20.00
What is this feature really about?
Focus
Given I am a depositor
When I deposit $20.00
Then my account balance is increased by $20.00
Some other features should describe that complicated sign up
process if it is important.
... but not every feature that implicitly requires signing
up.
Use Consistent Terminology
Given I am a user who has forgotten his password
When I request that my password be reset
Then I receive an email with instructions
to replace my passcode
This example is obvious ("password" vs. "passcode") and ("reset"
vs. "replace").
... but others are less so. Multiple terms for the same thing are
confusing for everyone.
... or might be telling you that the feature is not fully
fleshed out yet.
Don't Exhaustively Test Edge Cases
Given I am a depositor
And my account balance is $50.00
When I attempt to withdraw $49.99
Then the transaction is allowed
And my balance is $0.01
Given I am a depositor
And my account balance is $50.00
When I attempt to withdraw $50.01
Then the transaction is not allowed
And my balance remains $50.00
Test many edge cases if they are important, but not as acceptance
tests.
But Why?
This is tedious and nobody but me will ever read these.
Your customers can write them
Your customers can write them
- They can do this. You may just need to clean up their prose a bit.
- Now impress them: "the stories you write are being used to verify I implement features correctly"
- "... and that these features do not break in the future"
My customer doesn't care
Are you sure?
Have you asked them?
Have they seen good acceptance tests in action?
Seriously, they don't care
Admittedly, that's sad
But what if writing them still had benefits?
Other Benefits
- You as a developer are forced to think as a user
- ... or a business person
- "I need this database table and this model and this controller" vs. "what value is this feature providing?" is hard
- Acceptance tests help formalize that transition
Other Benefits
- One-stop shop for "what does this application do?" in language everyone can read
- New developers
- New project managers
- Salespeople (oh, we have to know what this thing does to make money?
Other Benefits
- Who has tried writing real isolated unit tests?
- It was awesome, right?
- But did you ever run into issues integrating those isolated objects, esp. after refactoring?
- Did you have good higher-level tests?
- ... ones that you trusted and were not brittle?
Tests Should Make You Go Fast
- Good acceptance tests: have your cake and eat it too.
- Fast unit tests that drive good design, facilitate refactoring and encourage cohesion.
- Slower, end-to-end tests that verify everything fits together properly and provides value to customers.
By the way ...
You can write acceptance tests without Cucumber
RSpec and Capybara
feature "Accounts" do
include AccountSteps
scenario "new accounts start with a zero balance" do
login_as_depositor
open_new_account
balance_should_be("0.00")
end
end
RSpec and Capybara
module AccountSteps
def login_as_depositor
# click_on ...
# fill_in ...
end
def open_new_account
# ...
end
def balance_should_be(balance)
expect(page).to have_text("Balance: $#{balance}")
end
end
Be wary of
feature "Accounts" do
scenario "transferring money between accounts" do
visit root_path
click_on "Login"
fill_in "username", with: "andy@example.com"
page.execute_script "$('omg_js').hack()"
# ...
end
end
Be wary of
- Acceptance tests themselves should not change unless the feature itself changed appreciably
- The code behind may change slightly as the view layer changes