What are End-to-End Tests?
End-to-end (E2E) testing is a software testing methodology used to verify the complete flow of an application from start to finish. It involves testing the entire system from the user's perspective and simulating real-world scenarios to ensure that the software works as intended.
E2E tests are important in software development because they ensure that the application meets the user's requirements and specifications. They help to identify any bugs, defects or inconsistencies in the application, and prevent problems that could arise in production.
Why do we need E2E tests?
Automated end-to-end (E2E) testing is an essential practice in software development for several reasons. Firstly, it increases efficiency and productivity by simulating the user's behavior and testing the entire system's functionality in one go, eliminating the need for manual testing of individual components. Additionally, automated E2E testing provides improved accuracy and reliability compared to manual testing. It can simulate complex scenarios and edge cases that are difficult to replicate manually, resulting in more accurate and reliable results. This practice is also cost-effective, as it can identify issues early in the development cycle, thereby preventing costly errors from making it to production. Automated E2E testing provides a faster feedback loop on the system's performance and functionality, allowing the team to fix issues promptly and avoid delays in the development process. Lastly, it leads to better quality software by identifying defects and bugs early in the development cycle, resulting in better customer satisfaction and retention.
Getting Started
Before starting any testing, ensure that you have a complete understanding of the feature and its functionality. To get started, follow these steps:
- Reviewing the feature's specifications and requirements to get an overview of what it is expected to do.
- Discussing with the O3 squad to get a more detailed understanding of the feature, its intended purpose, and how it is supposed to work.
- Creating a test plan that outlines the features that need to be tested and the scenarios that will be used to test them.
Identifying Test Cases
The following steps will help you identify test cases for the relevant feature:
- Identify all the possible scenarios that the user could encounter while using the feature.
- Prioritise the critical paths and ensure that they are covered in the testing process.
- Create test cases that encompass different scenarios, including best-case scenarios, worst-case scenarios, and edge cases.
Writing tests with playwright
Getting Started
Please ensure that you have followed the basic installation guide in the README based on the esm module you are working on. Once everything is set up, make sure the dev server is running by using:
yarn start --sources 'packages/esm-*-app/'Then, in a separate terminal, run:
yarn test-e2e --headed
By default, the test suite will run against the http://localhost:8080. You can override this by exporting E2E_BASE_URL environment variables beforehand:
# Ex: Set the server URL to dev3: export E2E_BASE_URL=https://dev3.openmrs.org/openmrs # Run all e2e tests: yarn test-e2e --headed
To run a specific test by title:
yarn test-e2e --headed -g "title of the test"
Check this documentation for more running options.
It is also highly recommended to install the companion VS Code extension: https://playwright.dev/docs/getting-started-vscode
Writing New Tests
In general, it is recommended to read through the official Playwright docs before writing new test cases. The project uses the official Playwright test runner and, generally, follows a very simple project structure:
e2e
|__ commands
| ^ Contains "commands" (simple reusable functions) that can be used in test cases/specs,
| e.g. generate a random patient.
|__ core
| ^ Contains code related to the test runner itself, e.g. setting up the custom fixtures.
| You probably need to touch this infrequently.
|__ fixtures
| ^ Contains fixtures (https://playwright.dev/docs/test-fixtures) which are used
| to run reusable setup/teardown tasks
|__ pages
| ^ Contains page object model classes for interacting with the frontend.
| See https://playwright.dev/docs/test-pom for details.
|__ specs
| ^ Contains the actual test cases/specs. New tests should be placed in this folder.
|__ support
^ Contains support files that requires to run e2e tests, e.g. docker compose files.
When you want to write a new test case, start by creating a new spec in ./specs. Depending on what you want to achieve, you might want to create new fixtures and/or page object models. To see examples, have a look at the existing code to see how these different concepts play together.
Demo data usage
To ensure that the test contains the necessary metadata, you may follow the procedures outlined below:
Utilize the user interface - Suppose the test scenario involves editing patient information. In this case, you can use the UI to create a new patient record for testing purposes.
Leverage demo data - If the test case only requires data viewing, the demo data available in the RefApp can suffice. Check the demo data module to know more.
Generate the required data - In case the above solutions are inadequate, you can use the API to create the necessary data ahead of the test.
Open reports from GitHub Actions / Bamboo
To download the report from the GitHub action/Bamboo plan, follow these steps:
- Go to the artifact section of the action/plan and locate the report file.
- Download the report file and unzip it using a tool of your choice.
- Open the index.html file in a web browser to view the report.
The report will show you a full summary of your tests, including information on which tests passed, failed, were skipped, or were flaky. You can filter the report by browser and explore the details of individual tests, including any errors or failures, video recordings, and the steps involved in each test. Simply click on a test to view its details.
Debugging Tests
Refer to this documentation on how to debug a test.
Troubleshooting tips
On MacOS, you might run into the following error: browserType.launch: Executable doesn't exist at /Users/<user>/Library/Caches/ms-playwright/chromium-1015/chrome-mac/Chromium.app/Contents/MacOS/Chromium In order to fix this, you can attempt to force the browser reinstallation by running: PLAYWRIGHT_BROWSERS_PATH=/Users/$USER/Library/Caches/ms-playwright npx playwright install
Do's and Don'ts
- Do ensure that all test cases are written clearly and concisely, with step-by-step instructions that can be easily understood.
- Do use a variety of test cases to cover all possible scenarios, including best-case scenarios, worst-case scenarios, and edge cases.
- Do ensure that all tests are executed in a timely and efficient manner to save time and resources.
- Don't assume that a feature is working just because it seems to be functioning correctly. Test it thoroughly to ensure that all its features and functionalities are working as expected.
- Don't ignore any errors or issues that arise during testing, even if they seem minor. Report them to the development team so that they can be addressed promptly.
- Don't skip any critical paths or scenarios. Ensure that all scenarios are tested thoroughly to identify any potential issues or defects.
Best Practices
- Start testing early in the development process to identify and address issues before they become more challenging and expensive to fix.
- Utilize automated testing whenever possible to save time and increase efficiency.
- Use real-world data and scenarios to create accurate and relevant test cases.
- Ensure that all test cases are repeatable and easily reproducible to ensure that results can be verified and tested again if necessary.
- Continuously review and update the testing plan to ensure that it covers all relevant features and scenarios.
- Work collaboratively with the O3 team to ensure that any issues or defects are identified and resolved quickly.
Resources
- Playwright docs: https://playwright.dev/docs/intro
- O3 Refapp workflow cases