PyCharm and pytest-bdd

Paul Everitt

Last week we published a blog post on the new pytest fixture support in PyCharm 2018.2. This feature was a big part of the 2018.2 work and proved to be a hit. But it wasn’t the only notable pytest work: 2018.2 also delivered support for behavior testing with pytest-bdd, a project that provides behavior testing with pytest as the test framework. Our What’s New in 2018.2 video gave a quick glimpse of this support.

Let’s rewrite the tutorial from last week, showing the pytest-bdd support instead of simply working with fixtures. Behavior-driven-development (BDD) support is a part of PyCharm Professional Edition, previously covered in a blog post about using Behave and an older blog post about BDD. This tutorial puts pytest-bdd to use.

Want the finished code? It’s in available in a GitHub repo.

What and Why for BDD

If you’re new to BDD, it can appear a bit strange. You write a programming-free specification, known as a “feature”, that describes how the software is supposed to behave. You then implement that specification with test code.

This two-step process shines in several workflows:

• Large teams with “test engineers”, business analysts or/and domain experts. That is, those who understand business and can read English but not Python code.
• Complex software in which you want to think about what’s happening in a simpler fashion than test code

BDD can seem like overhead. But if you find yourself lost in your test code, the feature spec can help you step back from the details and follow the high-level intent. Even better, when you come back to your code months later, it is much easier to get up-to-date. Finally, if you feel committed to the “Given/When/Then” test style, pytest-bdd moves that commitment from “dating” to “marriage”.

Setup

We’ll use the same example application as the previous blog post.

To follow along at home, make sure you have Python 3.7 (our example uses dataclasses) and Pipenv installed. Clone the repo at https://github.com/pauleveritt/laxleague and make sure Pipenv has created an interpreter for you. (You can do both of those steps from within PyCharm.) Make sure Pipenv has installed pytest and pytest-bdd into your interpreter.

Then, open the directory in PyCharm and make sure you have set the Python Integrated Tools -> Default Test Runner to pytest, as done in the previous blog post.

This time, though, we also need to set Languages & Frameworks -> BDD -> Preferred BD framework to pytest-bdd. This will enable many of the features discussed in this tutorial. More information is available in the PyCharm Help on pytest-bdd.

Now you’re ready to follow the material below. Right-click on the tests directory and choose Run ‘pytest in tests’. If all the tests pass correctly, you’re setup.

Let’s Make a Feature

What parts of our project deliver business value? We’re going to take these requirements, as “features”, and write them in feature files. These features are specified using a subset of the Gherkin language.

Right click on the tests folder and select New, then choose Gherkin feature file.
Name the file games. Note: Some like to group BDD tests under tests/features.

Here’s the default file contents generated by PyCharm:

# Created by pauleveritt at 8/6/18
Feature: #Enter feature name here
 # Enter feature description here

 Scenario: # Enter scenario name here
   # Enter steps here

Let’s change the feature file’s specification to say the following:

Feature: Games
 laxleague games are between two teams and have a score resulting
 in a winner or a tie.

 Scenario: Determine The Winner
   Given a home team of Blue
   And a visiting team of Red
   When the score is 5 for Blue to 4 for Red
   Then Blue is the winner

This Scenario shows the basics of BDD:

• Each Feature can have multiple Scenarios
• Each Scenario has multiple steps
• The steps revolve around the “given/when/then” approach to test specification
• Step types can be continued with the And keyword
• Given specifies inputs
• When specifies the logic being tested
• Then specifies the result

We might have a number of other scenarios relating to games. Specifically, the “winner” when the score is tied. For now, this provides enough to implement.

As we type this feature in, we see some of the features PyCharm Professional provides in its Gherkin support:

• Syntax highlighting of Gherkin keywords
• Reformat Code fixes indentation
• Autocomplete on keywords
• Warnings on unimplemented steps

Implement The Steps

If you run the tests, you’ll see….no difference. The games.feature file isn’t a test: it’s a specification. We need to implement, in test code, each of the scenarios and scenario steps.

PyCharm can help on this. As mentioned above, PyCharm warns you about an “Undefined step reference”:

If you Alt-Enter on the warnings, PyCharm will offer to either Create step definition or Create all steps definition. Since 2018.2 does the latter in an unexpected way (note: it’s being worked on), let’s choose the former, and provide a File name: of test_games_feature:

Here’s what the generated test file test_games_feature.py looks like:

from pytest_bdd import scenario, given, when, then

@given("a home team of Blue")
def step_impl():
   raise NotImplementedError(u'STEP: Given a home team of Blue')

These are just stubs, of course, which we’ll have to come back and name/implement. We could implement the other steps by hand. Let’s instead let PyCharm continue generating the stubs, into the same file.

2018.2 doesn’t generate the scenario, which is what actually triggers the running of the test. Let’s provide the scenario, as well as implement each step, resulting in the following for the test_games_feature.py implementation:

from pytest_bdd import scenario, given, when, then

from laxleague.games import Game
from laxleague.teams import Team

@scenario('games.feature', 'Determine The Winner')
def test_games_feature():
   pass

@given('a home team of Blue')
def blue():
   return Team('Blue')

@given('a visiting team of Red')
def red():
   return Team('Red')

@given('a game between them')
def game_red_blue(blue, red):
   return Game(home_team=blue, visitor_team=red)

@when('the score is 10 for Blue to 5 for Red')
def record_score(game_red_blue):
   game_red_blue.record_score(10, 5)

@then('Blue is the winner')
def winner(game_red_blue):
   assert 'Blue' == game_red_blue.winner.name

To see this in action, let’s run the test then take a look at what pytest-bdd is doing.

Run the Tests

You’ve already run your regular pytest tests, with fixtures and the like. What extra does it take to also run your pytest-bdd tests? Nothing! Just run your tests:

Your pytest-bdd tests show up just like any other test.

Let’s take a look at some things pytest-bdd is doing:

1. The @scenario decorated function test_games_feature is the only function in the file with the test_ prefix. That means, this file only has one test. And guess what? The function itself doesn’t do anything. It’s just a marker.
2. We need two teams, so we implement the two Given steps by making a Blue and Red team.
3. We also need a game between these two teams. This is the third step in our games.feature scenario. Not that this function takes two arguments. As it turns out, pytest-bdd steps are pytest fixtures which can be injected into the functions for each step. (In fact, any pytest fixture can be injected.)
4. Now that we have a game setup, we use @when to run the logic being tested by recording a score.
5. Did our logic work? We use @then to do our assertion. This is where our test passes or fails.

It’s an interesting approach. It’s verbose, but it clearly delineates the “Given/When/Then” triad of good test cases. Plus, you can read the almost-human-language Gherkin file to quickly understand what’s the behavior being tested.

Test Parameters

You might have noticed that the feature file specified a score for the game but it was ignored in the implemented tests. pytest-bdd has a feature where you can extract parameters from strings. It has several parsing schemes. We’ll use the simplest, starting by adding parsers to the import from pytest_bdd:

from pytest_bdd import scenario, given, when, then, parsers

Note: You could also do this the productive way by using the symbol and letting PyCharm generate the import for you.

The feature file says this:

When the score is 10 for Blue to 5 for Red

Let’s change our @when decorator to parse out the score:

@when(parsers.parse('the score is {home:d} for Blue to {visitor:d} for Red'))

When we do so, PyCharm warns us that Not all arguments.... were used. You can type them in, but PyCharm knows how to do it. Hit Alt-Enter and accept the first item:

After changing record_score to use these values, our function looks like this:

@when(parsers.parse('the score is {home:d} for Blue to {visitor:d} for Red'))
def record_score(game_red_blue, home, visitor):
   game_red_blue.record_score(home, visitor)

As we experiment with lots of combinations, moving this to the feature file is very helpful, particularly for test engineers.

Want more? Gherkin and pytest-bdd support “Scenario Outlines” where you can batch parameterize your inputs and outputs, just like we saw previously with pytest.mark.parametrize.

Use pytest Fixtures and Features

As noted above, pytest-bdd treats scenario steps as pytest fixtures. But any pytest fixture can be used, which makes pytest-bdd a powerful approach for BDD.

For example, imagine we have an indirection where there are different types of Game implementations. We’d like to put the choice of which class is used behind a fixture, in conftest.py:

@pytest.fixture
def game_type():
   return Game

We can then use this fixture in the code>@given step when we instantiate a Game:

@given('a game between them')
def game_red_blue(game_type, blue, red):
   return game_type(home_team=blue, visitor_team=red)

Prior to 2018.2, PyCharm provided BDD for Behave, but nothing for pytest-bdd. Admittedly, there are still loose ends being worked on. Still, what’s available now is quite productive, with IDE features brought to bear when getting in the pytest-bdd flow.

Warnings

Forgot to implement a step? Typo in your @scenario text identifier? PyCharm flags these with the (configurable) warning and error infrastructure.

Autocomplete

Obviously we autocomplete keywords from Gherkin and pytest-bdd. And as expected, we autocomplete function names as fixture parameters. But we also autocomplete strings from the feature file…for example, to fix the kind of error just mentioned for the scenario identifier. Also, parameters parsed out of strings can autocomplete.

Quick Info

Wonder what is a particular symbol and where it came from? You can use Quick Info or on-hover type information with Cmd-hover to see more information:

Conclusion

Doing behavior-driven development is a very different approach to testing. With pytest-bdd, you can keep much of what you know and use from the very-active pytest ecosystem when doing BDD. PyCharm Professional 2018.2 provides a “visual testing” frontend to keep you in a productive BDD flow.

Subscribe to PyCharm Blog updates

Subscribe form

By submitting this form, I agree to the JetBrains Privacy Policy Notification icon

By submitting this form, I agree that JetBrains s.r.o. ("JetBrains") may use my name, email address, and location data to send me newsletters, including commercial communications, and to process my personal data for this purpose. I agree that JetBrains may process said data using third-party services for this purpose in accordance with the JetBrains Privacy Policy. I understand that I can revoke this consent at any time in my profile. In addition, an unsubscribe link is included in each email.

Submit

Thanks, we've got you!