| Info |
|---|
| This is an attempt to reorganize and update Testing and it's child pages. It's still under construction. |
...
Our integration tests are run by Travis-CI using Saucelabs. The setup is described in detail at https://github.com/openmrs/openmrs-distro-referenceapplication#running-ui-tests-on-travis-ci-with-saucelabs2
The key points are:
- We are using docker to startup an OpenMRS server on Travis-CI before running tests (fresh instance including database for the whole test suite)
- We run all tests against two servers in parallel. One using MySQL and the other MariaDB.
- Tests are executed by Travis-CI.
- Saucelabs is used as a client with a browser driven by tests. Saucelabs connects to the server instance running on Travis-CI through a tunnel (no access to the test server from the outside world). Saucelabs records screencasts and takes screenshots when running tests, which can be used for debugging.
- We test on Firefox 42 and Chrome 48.
- We run tests in parallel (currently 5 at a time).
- A failing test is executed again twice to verify, if it is a reproducible issue. If the test passes in consecutive runs, it is not failing a build.
When writing UI tests you have to follow guidelines, which are aimed to improve tests stability, readability and maintainability.
...
In case of questions, please write on talk.openmrs.org.
Debugging
If you are notified about a test failure, the following should help you figure out why:
- Builds on Travis CI are triggered by https://ci.openmrs.org/browse/REFAPP-OMODDISTRO. The Bamboo build waits for the results from the Travis CI build before proceeding.
- Visit https://saucelabs.com/u/openmrs (open the Automated Tests tab) to watch a recording or step by step screenshots to see why a particular test failed.
- Open the failing build at https://travis-ci.org/openmrs/openmrs-distro-referenceapplication and see the build logs. We include server logs at the end of each build log, which is also helpful at times. At times the build log is too long to be displayed in Travis CI, so look for the Raw Log button at the top.
- Note that a failing test is executed two more times to confirm it is a reproducible issue. If the test passes in consecutive runs, it is not failing a build. Previous runs, which failed will still appear in Saucelabs as failing.
- Finally you can try running UI tests locally against a test server of your choice e.g. run locally. Note that UI tests that are run against local servers may not be failing even though they fail on remote servers. It is usually caused by the network latency and indicates a test need to wait before taking an action. The UI test framework, if used as outlined above, prevents such situations in most cases.
Performance Tests
To be addressed...
...