Python testing framework pytest high-level usage in full detail

preamble

Python testing framework has been used before is unittest + HTMLTestRunner, heard someone say pytest is very good to use, so this period of time to look at pytest documentation, here to make a record.

Official Documentation Description:

Pytest is a framework that makes building simple and scalable tests easy. Tests are expressive and readable—no boilerplate code required. Get started in minutes with a small unit test or complex functional test for your application or library.

pytest is a very mature full-featured Python testing framework with the following main features:

Simple, flexible and easy to use
Support for parameterization
Able to support simple unit testing and complex functional testing, can also be used to do selenium/appnium and other automated testing, interface self
Dynamized tests (pytest+requests)
pytest has a lot of third-party plug-ins, and can be customized to extend the more useful such as pytest-selenium (integration)
selenium), pytest-html (perfect html test report generation), pytest-rerunfailures (repeated execution of failed cases), pytest-html (perfect html test report generation), pytest-rerunfailures (repeated execution of failed cases).
lines), pytest-xdist (multi-CPU distribution), etc.
Skip and xfail handling of test cases
Integrates well with jenkins
report framework ----allure also supports pytest

mounting

1.1 Installation

pip install -U pytest

1.2 Verify installation

pytest --version # The currently installed version is displayed

1.3 pytest documentation

Official Documentation:/en/latest/

In the pytest framework, there are the following constraints:

All single test filenames need to fulfill the test_*.py format or *_test.py format.
In a single test file, the test class starts with Test and cannot have an init
method (note: when defining a class, it needs to start with a T, otherwise pytest won't bother to run the class)
In a single test class, one or more functions starting with test_ can be included.
When the pytest command is executed, it automatically looks for test functions from the current directory and subdirectories that meet the above constraints to execute.

1.4 How Pytest works

 # file_name: test_abc.py
 import pytest # Introducing the pytest package
 def test_a(): # test function starting with test
     print("------->test_a")
     assert 1 # Asserting success
 def test_b():
     print("------->test_b")
     assert 0 # Assertion failure
 if __name__ == '__main__':
        ("-s  test_abc.py") # call (programming)pytest(used form a nominal expression)mainFunction Execution Tests

1. Test class main function mode

("-s test_abc.py")

2. Command line mode

  pytest file path／Test File Name
  for example：pytest ./test_abc.py

1.5 List of meanings of Pytest Exit Code

Exit code 0 All use cases have been executed and passed.

Exit code 1 All use cases are executed and there is a Failed test case.

Exit code 2 The user interrupted the execution of the test.

Exit code 3 An internal error occurred during test execution.

Exit code 4 pytest command line usage error

Exit code 5 No available test case files captured

1.6 How to get help information

View pytest version

pytest --version

Show available built-in function parameters

pytest --fixtures

Viewing Help Information and Configuration File Options from the Command Line

pytest --help

1.7 Controlling Test Case Execution

1. End test execution after the Nth use case fails

pytest -x                    # Stop the test on the 01st failure
pytest --maxfail=2     # appeared2Terminate the test if one fails.

2. Specify the test module

pytest test_mod.py

3. Specify the test directory

pytest testing/

4. Filtering execution by keyword expressions

pytest -k "MyClass and not method"

This command will match the file name, class name, and method name to the use case of the matching expression, where this command will run TestMyClass.test_something, but not TestMyClass.test_method_simple.

5. Specify the test case by node id.

The nodeid consists of the module filename, separator, class name, method name, and parameters, as exemplified below:
Run the specified use cases in the module

pytest test_mod.py::test_func

Run the specified method in the module

ytest test_mod.py::TestClass::test_method

6. Execution through markup expressions

pytest -m slow

This command executes all the test cases decorated by the decorator @

7. Execute the test through the package

pytest --pyargs

This command automatically imports the package , and uses the directory where the package is located to execute the following use case.

1.8 Multi-process running cases

When there are a lot of cases, the running time becomes very long. If you want to shorten the running time of the script, you can use multi-process to run it.

Install pytest-xdist:

pip install -U pytest-xdist

Running mode:

pytest test_se.py -n NUM

Where NUM fills in the number of concurrent processes.

1.9 Retry run cases

When doing interface testing, something will encounter 503 or short network fluctuations, resulting in cases run failed, and this is not the result we expect, then you can can retry to run cases to solve the way.

Install pytest-rerunfailures:

pip install -U pytest-rerunfailures

Running mode:

pytest test_se.py --reruns NUM

NUM Fill in the number of retries.

1.10 Displaying the contents of the print

When running a test script, to debug or print something, we will put some print content in the code, but when running pytest, this content will not be shown. If you put -s, it will be displayed.

Running mode:

pytest test_se.py -s

Also, pytest's multiple run modes are stackable for execution, for example, you want to run 4 processes at the same time and you want to print out the contents of print. You can use:

pytest test_se.py -s -n 4

The setup and teardown functions of the

And teardown is mainly divided into: module level, class level, function level, function level.

2. Exists inside the test class

Code Example:

Function level setup()/teardown()

Runs at the beginning and end of the test method, i.e.: running a test function will run setup and teardown once.

import pytest
class Test_ABC:
  # Starting at the function level
  def setup(self):
      print("------->setup_method")
  # End of function level
  def teardown(self):
      print("------->teardown_method")
  def test_a(self):
      print("------->test_a")
      assert 1
  def test_b(self):
      print("------->test_b")
if __name__ == '__main__':
              ("-s  test_abc.py")

Implementation results：
  test_abc.py 
  ------->setup_method # First time setup()
  ------->test_a
  .
  ------->teardown_method # First teardown()
  ------->setup_method # second setup()
  ------->test_b
  .
          ------->teardown_method # number two teardown()

class level

Runs at the beginning and end of the test class, i.e.: runs setup_class and teardown_class only once within a test, doesn't care how many test functions are in the test class.

Code Example:

import pytest
class Test_ABC:
   # Test class level start
   def setup_class(self):
       print("------->setup_class")
   # End of test class level
   def teardown_class(self):
       print("------->teardown_class")
   def test_a(self):
       print("------->test_a")
       assert 1
   def test_b(self):
       print("------->test_b")
          if __name__ == '__main__':
              ("-s  test_abc.py")

Implementation results：
  test_abc.py 
  ------->setup_class # First setup_class()
  ------->test_a
  .
  ------->test_b
  F 
          ------->teardown_class # number one teardown_class()

configuration file

The configuration file for pytest is usually placed in the test directory under the name and is used by the command line when running.

# Configure pytest command line run parameters
   [pytest]
    addopts = -s ... # space-separated, can add multiple command line arguments - all arguments are paths to be searched by the plugin package's parameter configuration test
    testpaths = ./scripts  # The scripts folder in the current directory - customizable
# Configure the name of the file to be searched by the test
    python_files = test*.py 
# All files in the scripts folder in the current directory that start with test and end with .py - customizable
Configuring test class names for test searches
    python_classes = Test_*  
   # Classes starting with Test in all files starting with test and ending with .py in the scripts folder in the current directory - customizable
Configuring test function names for test searches
    python_functions = test_*
# Methods starting with test_ in all files starting with test and ending with .py in the scripts folder in the current directory, within classes starting with test - customizable

4 Pytest commonly used plug-ins

Plugin List URL:

Includes many plug-in packages, you can choose to use according to the needs of the work.

4.1 Preconditions:

1. File path:

- Test_App
- - test_abc.py
- -

Configuration file contents:

  [pytest]
# Command line arguments
 addopts = -s
# Search for filenames
 python_files = test_*.py
 # Class name to search for
 python_classes = Test_*
 # The name of the function to search for
    python_functions = test_*

4.2 Pytest Test Report

pytest-HTML is a plugin pytest used to generate HTML reports of test results. Compatible with Python 2.7,3.6

Installation: pip install pytest-html

pip install pytest-html

Generate test reports in xml/html format via command line and store them in user specified paths. Plugin name: pytest-html

Usage: Command line format: pytest --html=User Path/

Example:

import pytest
class Test_ABC:
    def setup_class(self):
        print("------->setup_class")
    def teardown_class(self):
        print("------->teardown_class")
    def test_a(self):
        print("------->test_a")
        assert 1
    def test_b(self):
            print("------->test_b")
            assert 0 # Assertion failed ``
running mode：
1.modificationsTest_App/file，Adding Reporting Parameters，assume (office)：addopts = -s --html=./ 
    # -s: output program runtime information
    # --html=. / generate file in current directory
    ️ To generate thexmlfile，connects--html=./ adapt (a story to another medium) --html=./
2.Command line entryTest_Appcatalogs
3.execute a command： pytest
Implementation results：
    1.在当前catalogs会生成assetsfile夹和file

Higher-order usage of (i)

Preconditions:

1. File path:

Test_App
    - - test_abc.py
    - -

Configuration file contents:

 [pytest]
  command-line parameter
 addopts = -s
 Search for filenames
 python_files = test*.py
  Class name to search for
 python_classes = Test*
Function name to search
 python_functions = test_*

5.1 Pytest's fixture

The fixture modifier marks a fixed factory function that is activated and prioritized when it is called by another function, module, class, or the entire project, and is often used to accomplish preprocessing and repetitive operations.

Methods:

fixture(scope="function", params=None, autouse=False, ids=None, name=None)

Common Parameters.

scope: the scope of the tagged method

function(default): works on each test method, runs once for each test

class: works on the whole class, all tests of each class are run only once.

module: works on the whole module, all tests of each module will be run only once.

session: works on the entire session (use with caution), each session only runs once

params: (list type) provides parameter data to be used by the function calling the tagged method

autouse: if or not auto-run, default is False, set to True to auto-run.

5.2 First example of fixture (referenced by parameters)

Example:

class Test_ABC:
    @()
    def before(self):
        print("------->before")
    def test_a(self,before): # ️ The test_a method passes in the function identified by fixture, in the form of the variable
        print("------->test_a")
        assert 1
if __name__ == '__main__':
    ("-s  test_abc.py")
Implementation results：
    test_abc.py 
        ------->before # Discover that before takes precedence over test function running
        ------->test_a
         .

5. Second example (via function references)

Example:

import pytest
@() # fixture-tagged functions can be applied outside the test class
def before():
    print("------->before")
@("before")
class Test_ABC:
    def setup(self):
        print("------->setup")
    def test_a(self):
        print("------->test_a")
        assert 1
if __name__ == '__main__':
          ("-s  test_abc.py")
  Implementation results：
      test_abc.py 
      ------->before # Discover that before takes precedence over the test class
      ------->setup
      ------->test_a
      .

5. Third example (default setting is run)

Example:

 import pytest
 @(autouse=True) # Set to run by default
 def before():
     print("------->before")
 class Test_ABC:
     def setup(self):
         print("------->setup")
     def test_a(self):
         print("------->test_a")
         assert 1
 if __name__ == '__main__':
     ("-s  test_abc.py")
Implementation results：
    test_abc.py 
    ------->before # Discover that before automatically takes precedence over test classes
    ------->setup
    ------->test_a
        .

5. Fourth example (set the scope to function)

Example:

    import pytest
    @(scope='function',autouse=True) # Scope set to function, run automatically
    def before():
        print("------->before")
    class Test_ABC:
        def setup(self):
            print("------->setup")
        def test_a(self):
            print("------->test_a")
            assert 1
        def test_b(self):
            print("------->test_b")
            assert 1
    if __name__ == '__main__':
        ("-s  test_abc.py")
Implementation results：
    test_abc.py
        ------->before # Run the first time
        ------->setup
        ------->test_a
        .------->before # Run a second time
        ------->setup
        ------->test_b
        .

5. Fifth example (set the scope to class)

Example:

    import pytest
    @(scope='class',autouse=True) # Scope set to class, run automatically
    def before():
        print("------->before")
    class Test_ABC:
        def setup(self):
            print("------->setup")
        def test_a(self):
            print("------->test_a")
            assert 1
        def test_b(self):
            print("------->test_b")
            assert 1
    if __name__ == '__main__':
        ("-s  test_abc.py")
Implementation results：
    test_abc.py
    ------->before # Found only one run
    ------->setup
        ------->test_a
        .
        ------->setup
        ------->test_b
        .

5. Sixth example (return value)

Example one.

     import pytest
    @()
    def need_data():
        return 2 # Return the number 2
    class Test_ABC:
        def test_a(self,need_data):
            print("------->test_a")
            assert need_data != 3 # Take the return value and make an assertion
    if __name__ == '__main__':
        ("-s  test_abc.py")
Implementation results：
    test_abc.py 
    ------->test_a
    .
``

Example two.

import pytest
@(params=[1, 2, 3])
def need_data(request): # Incoming parameters request System wrapped parameters
    return  # Take a single value from the list, the default way of taking values
class Test_ABC:
    def test_a(self,need_data):
        print("------->test_a")
        assert need_data != 3 # Assert that need_data is not equal to 3
if __name__ == '__main__':
    ("-s  test_abc.py")
 Implementation results：
      # You can see the results run three times
      test_abc.py 
      1
      ------->test_a
      .
      2
      ------->test_a
      .
      3
      ------->test_a
      F

Advanced Usage (II)

Preconditions:

1. File path:

- Test_App
- - test_abc.py
- -

Configuration file contents:

[pytest]
command-line parameter
addopts = -s
Search for filenames
python_files = test_*.py
 Class name to search for
python_classes = Test_*
 Function name to search
python_functions = test_*

6.1. Skip Test Functions

Test functions that do not execute identifiers under certain conditions.

Methods:

skipif(condition, reason=None)

Parameters:

condition: Skip condition, must pass parameter

reason: labeling reasons, must pass the parameter

Usage:

@(condition, reason="xxx")

Example:

import pytest
class Test_ABC:
    def setup_class(self):
        print("------->setup_class")
    def teardown_class(self):
        print("------->teardown_class")
    def test_a(self):
        print("------->test_a")
        assert 1
    @(condition=2>1,reason = "Skip this function") # Skip test function test_b
    def test_b(self):
        print("------->test_b")
            assert 0
Implementation results：
   test_abc.py 
   ------->setup_class
   ------->test_a # Only the function test_a is executed
   .
   ------->teardown_class
       s # Skip functions ``

6.2 Marking as an Expected Failure Function

Marking a Test Function as a Failed Function

Methods:

xfail(condition=None, reason=None, raises=None, run=True, strict=False)

Commonly used parameters:

condition: expected failure condition, must pass parameter

reason: reason for failure, must pass parameter

Usage:

@(condition, reason="xx")

Example:

import pytest
class Test_ABC:
    def setup_class(self):
        print("------->setup_class")
    def teardown_class(self):
        print("------->teardown_class")
    def test_a(self):
        print("------->test_a")
        assert 1
    @(2 > 1, reason="Marked as Expected Failure") # Mark as expected failure function test_b
       def test_b(self):
           print("------->test_b")
          assert 0
   Implementation results：
       test_abc.py 
       ------->setup_class
       ------->test_a
       .
       ------->test_b
       ------->teardown_class
       x  # fail mark

6.3 Function Data Parameterization

Facilitates test function access to test belongings.

Methods:

parametrize(argnames, argvalues, indirect=False, ids=None, scope=None)

Commonly used parameters:

argnames: Parameter name

argvalues: the corresponding value of the parameter, which must be of type list

Format when one parameter: [value]

When the number of parameters is greater than one, the format is.

[(param_value1,param_value2.....),(param_value1,param_value2.....)]

Usage.

@(argnames,argvalues)

With N parameter values, the test method will run N times

Single parameter example:

import pytest
class Test_ABC:
    def setup_class(self):
        print("------->setup_class")
    def teardown_class(self):
        print("------->teardown_class")

@("a",[3,6]) # a parameter is given two values, the function is run twice
def test_a(self,a): # Parameters must be the same as in parametrize.
    print("test data:a=%d"%a)
    assert a%3 == 0
    Implementation results:
    test_abc.py 
    ------->setup_class
    test data:a=3 # Run first take a=3
    .
    test data:a=6 # Run a second time for value a=6
    . 
    ------->teardown_class

Multiple parameter examples:

import pytest
class Test_ABC:
    def setup_class(self):
        print("------->setup_class")
    def teardown_class(self):
        print("------->teardown_class")

@("a,b",[(1,2),(0,3)]) # Parameters a,b are given two values and the function is run two times
def test_a(self,a,b): # Parameters must be the same as in parametrize.
    print("test data:a=%d,b=%d"%(a,b))
    assert a+b == 3
    Implementation results：
    test_abc.py 
    ------->setup_class
    test data:a=1,b=2 # Run first take a=1,b=2
    .
    test data:a=0,b=3 # Run second time for values a=0,b=3
    .
    ------->teardown_class

Examples of function return value types:

import pytest
def return_test_data():
    return [(1,2),(0,3)]
class Test_ABC:
    def setup_class(self):
        print("------->setup_class")
    def teardown_class(self):
            print("------->teardown_class")

@("a,b",return_test_data()) # Pass in parameter values in the form of function return values
def test_a(self,a,b):
    print("test data:a=%d,b=%d"%(a,b))
    assert a+b == 3
    Implementation results：
    test_abc.py 
    ------->setup_class
    test data:a=1,b=2 # Run first take a=1,b=2
    .
    test data:a=0,b=3 # Run second time for values a=0,b=3
    .
        ------->teardown_class

6.4 Modifying Python traceback output

pytest --showlocals     # show local variables in tracebacks
pytest -l               # show local variables (shortcut)
pytest --tb=auto        # (default) 'long' tracebacks for the first and last
                        # entry, but 'short' style for the other entries
pytest --tb=long        # exhaustive, informative traceback formatting
pytest --tb=short       # shorter traceback format
pytest --tb=line        # only one line per failure
pytest --tb=native      # Python standard library formatting
pytest --tb=no          # no traceback at all

The --full-trace parameter prints more error output messages than the --tb=long parameter, even if the error was triggered by Ctrl+C

6.5 Jump to PDB when execution fails

When executing the use case, follow the argument --pdb, so that when it fails, every time it encounters a failure, it will automatically jump to the PDB

pytest --pdb              # Jump to the PDB every time you encounter a failure #
pytest -x --pdb           # Jump to the PDB the first time it encounters a failure, ending the test execution
pytest --pdb --maxfail=3  # Only the first three failures jump to PDB

6.6 Setting breakpoints

In the use case script add the following python code, pytest will automatically close the execution of the output of the crawl, here, the other test script will not be affected, with breakpoints on the test of the previous test normal output

import pdb; pdb.set_trace()

6.7 Obtaining Use Case Execution Performance Data

Get the execution elapsed time for the 10 slowest use cases

pytest --durations=10

6.8 Generating a JUnitXML Format Result File

Result files in this format can be parsed by Jenkins or other CI tools.

pytest --junitxml=path

6.9 Disabling plug-ins

For example, disabling the doctest plugin

pytest -p no:doctest

6.10 Calling pytest from Python code

()                      # Basic usage
(['-x', 'mytestdir'])   # Pass in configuration parameters
// Specify custom or additional plug-ins
# content of 
import pytest
class MyPlugin(object):
    def pytest_sessionfinish(self):
        print("*** test run reporting finishing")
(["-qq"], plugins=[MyPlugin()])

6.11 Rapid deployment of virtualenv with pytest after test script migration

For example, if you clone a test script written by a fellow project team member, Dagger, from the Gitlab repository to your own computer, and you want to modify something and debug it, what should you do? You can quickly create a VirtualEnv by doing the following

cd <repository>
pip install -e .

This will set up a symlink to your code in site-packages, allowing you to edit your code while
your tests run against it as if it were installed.
Setting up your project in development mode lets you avoid having to reinstall every time you want to run your tests,
and is less brittle than mucking about with to point your tests at local code.
Also consider using tox

Problems encountered

pytest can output an html report of coverage

Use the command as follows:

pytest -vv --cov=./ --cov-report=html
open htmlcov/

It is possible to encounter an error:

(venv) zhangxiaofans-MacBook-Pro:mgap-mendel joe$ pytest --cov-report=html
usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --cov-report=html
  inifile: None
  rootdir: /Users/joe/workspace/platform/mgap-mendel/mgap-mendel

Reason:

Missing packages for pytest cov

cure

pip install pytest-cov

The above is Python testing framework pytest high-level usage comprehensive details, more information about Python testing framework pytest please pay attention to my other related articles!