Source code
Revision control
Copy as Markdown
Other Tools
# IndexedDB Performance Testing
## How to run tests on CI:
- Windows: `mach try perf --full -q "test-windows10-64-shippable-qr/opt-browsertime-indexeddb"`
- Linux: `mach try perf --full -q "test-linux1804-64-shippable-qr/opt-browsertime-indexeddb"`
- Mac: `mach try perf --full -q "test-macosx1015-64-shippable-qr/opt-browsertime-indexeddb"`
- All but 32-bit jobs: `mach try perf --chrome --safari --full -q 'shippable-browsertime-indexeddb !32'`
- In general:
- Open test selection interface with `mach try perf --full`
- Filter out the preferred tests by typing letters which are expected to be part of the test job name string (as in the -q argument above)
- Note down the string used as a filter for rerunning the job (or rerun it with `mach try again --list-tasks` and `mach try again --index`)
## How to run tests locally with the profiler?
- Build the browser with release or release with debug symbols flags (not in debug mode)
- Use `mach raptor --browsertime -t $(test_name) --gecko-profile --post-startup-delay=1000` where test name, such as `addMarN` is one of the items listed in `testing/raptor/raptor/tests/custom/browsertime-indexeddb.ini`
- After the test is complete, the generated profile is opened with the default browser.
- The generated profile file path is listed also in the command line output.
- For best symbolication results, it may help to
- run the same browser build that was used for the tests with `./mach run`
- navigate to "profiler.firefox.com"
- use the "Load a profile from file" button
## How to compare performance to a different browser?
- The test outputs a `time_duration` value for all supported browsers
- Using Chrome as an example,
- `mach raptor --browsertime -t $(test_name) --post-startup-delay=1000 --app=chrome -b "/c/Program Files/Google/Chrome/Application/chrome.exe"`
- where test name, such as `addMarN` is one of the items listed in `testing/raptor/raptor/tests/custom/browsertime-indexeddb.ini`
- browser executable path after the `-b` argument varies locally
- in some cases, a test driver argument such as `--browsertime-chromedriver` may be required
## How to add more tests?
- For the test boilerplate, copy and rename an old test script such as `testing/raptor/browsertime/indexeddb_write.js` under the `testing/raptor/browsertime/` directory
- Modify the test case script argument of `commands.js.run` / Selenium's `executeAsyncScript`
- Test parameters can be passed to such script with syntax `${variable_name}` where `variable_name` represents the parameter in the context of `executeAsyncScript` or `commands.js.run`.
- Use quotes to capture a string value, for example `"${variable_name}"`
- TIP: Debugging the test case could be simpler by serving it locally without the boilerplate
- Add `[test_name]` section to file `testing/raptor/raptor/tests/custom/browsertime-indexeddb.ini` where `test_name` **must be 10 characters or less** in order to be a valid `Treeherder` test name
- Under the `[test_name]` section, specify the test script name as a value of `test_script=`
- Under the `[test_name]` section, specify the test parameters as a sequence of `--browsertime.key=value` arguments as a value of `browsertime_args =`
- Under the `[test_name]` section, override any other values as needed
- Add test as a subtest to run for Desktop `taskcluster/kinds/browsertime/desktop.yml` (maybe also for mobile)
- Add test documentation to `testing/raptor/raptor/perfdocs/config.yml`
- Generated files:
- Run `./mach lint --warnings --outgoing --fix` to regenerate the documentation and task files, and warn about omissions
- Running `./mach perfdocs --generate` may also be needed
- Testing:
- Test the new test by running it with the profiler
- Test the new test by running it with a different browser
- Test the new test by triggering it on CI