Technical Test Creation Guidelines
Helpful Definitions
Test category - A test category is a grouping of related tests (subtests) that all test against the same vulnerability. For example, the category may be SQL Injection, Cross Site Scripting (XSS), Authentication, etc.
Test / Subtest - This is a specific method to attack or exploit some vulnerability on the system. Each test has its own CVSS score.
Steps - Given, When, Then, And, But
Gherkin is a syntax using “Given,” “When,” and “Then” steps to write plain English.
Behavior-Driven Development (BDD) is a system of describing the behavior of a system in plain English.
Feature files (.feature) use the Gherkin syntax to write security tests in a BDD style.
PyTest files (.py with prefix of “test_”) use decorators in Python to connect the plain English of the feature file to an actual implementation of code.
Playwright is a PyTest plugin used to tests websites.
Creating a Test
Pick a specific test and test category to add to the OpenMRS system.
Make sure to review the dashboard or the tests/ subdirectories to ensure you are not recreating an existing test.
Decide whether the test you are writing already has a created category.
If it does (in the form of a feature file and directory) then skip to step 9, where a new scenario is added.
If it doesn’t, then continue following these steps.
Copy the example_category/ folder and paste it in the tests/ subdirectory
Rename the copy to the test category
Rename the example.feature file to <test_category>.feature
We will rename the test_example.py test file later.
At this point, each file has boilerplate and comments to help guide you through writing each file. Even so, the following will go through how to write each portion.
__init__.py
The __init__.py file does not need to be edited. Its function is to allow python to treat the entire project as a package, which allows importing files in different directories. Do not edit this.
example.feature
Here are the sections of the feature file that will need to be filled out.
Feature: The Feature describes a vulnerability or test category that a user is testing. For example, XSS, SQL Injection, etc. In this way, it groups related scenarios, or related tests.
Background: Background holds any amount of Given steps to run before each scenario. These should be used to consolidate shared behavior.
Given: Given steps are used to describe the initial context of the system. This will run before the test and sets up the system to be in a known state. Such as, "Given I am logged in" When written in the background, the Given will run before each scenario.
Scenario: A scenario is a specific method to test against the vulnerability. Each scenario will be converted into its own test file in Python.
Make sure to give the Scenario a description underneath it. This will be what the dashboard reads in as the description.
Given: See above description.
When: The attacker tries some attack method,
Then: The system should do this in response as a proper defense.
The “Then” step should be specific and mention how something is done, but not delve too much into implementation.
A good example is “Then the user should be on the login page” and not “Then the authentication test was properly defended.” The latter can be assumed, as that is the point of the “Then” step and it gives no explanation for how the system is defended.
And: The system may optionally do this. Any amount of “And’s” may be used.
Here is how the feature file is able to integrate with the dashboard
The “Given a CVSS score is calculated and printed” step is automatically in the feature file and the test file. This should always be kept in. It allows the dashboard to access a specific test’s CVSS score and severity.
The feature text will become the category in the dashboard.
The scenario text will become the test name in the dashboard.
The text directly under the scenario will become the test description in the dashboard.
test_example.py
For each scenario written, a corresponding test file will need to be made. Simply copy and paste this test_example.py for more than one test.
Rename this file to test_<scenario description>.py
It is important to keep the prefix test_, or pytest will not run the test.
In this file, it explains how to choose the CVSS scores, as well as how to connect each step in the feature file to the py file.
Each decorator, starting with @pytest_bdd.<step>(‘’) needs the corresponding text of that step inside of the quotes exactly. Copy and pasting from the feature file is recommended.
Under each decorator, the functions should be renamed to reflect what is happening in the step above.
Note that the scenario’s function name must start with test_.
As well, It is important that each scenario function name must be unique or else pytest will confuse them and the results may be invalid.
To pick the CVSS score, follow along with the descriptions in the Given function. For supplementary guidance, see also https://www.first.org/cvss/v4.0/specification-document
If any steps are repeated from another scenario, like if two scenarios share the same given statement, then it can be removed from the specific py files and added to the directory’s conftest.py file. See below.
conftest.py
For any shared steps that have the same implementation, put them in this conftest file. When pytest runs, it will be able to figure out what to run based on the pytest_bdd decorators.
If a step is shared outside of the feature file, meaning two different feature files share a step, then that step can be added to the conftest.py file in the tests/ directory.
Do not put the CVSS step in here because its implementation should be different for each scenario. Even if it is the same for a subset of scenarios, keep in mind that if a new scenario is added in the future, it may have a different CVSS score.
As well, fixtures can be added here or in the specific test file to hold information between steps. Please seek further documentation for information about this feature.
For more utilization of pytest’s and bdd’s features, see also pytest hooks and parametrized steps. Please seek further documentation for information about these features.