Automation Frameworks

Below you will find a list of the various automated testing frameworks in use at Mozilla, along with a brief description and a link to read more. The letters in parentheses are the abbreviations used by TBPL.

Generally, you should pick the lowest-level framework that you can. If you are testing JavaScript but don't need a window, use XPCShell. If you're testing page layout, try to use Reftest. The advantage is that you don't drag in a lot of other components that might have their own problems, so you can hone in quickly on any bugs in what you are specifically testing.

If you have questions, join #automation on or post to the public mailing list.

AutoPhone a platform for running automated tests on physical mobile devices.
Compiled-Code (B) written in C++, compiled-code tests can test pretty much anything but are difficult to write properly.
Crashtest (C) If you've found a page that crashes Firefox, add a crashtest to make sure future versions don't experience this crash again.
Eideticker an automated test harness that captures and analyzes browser output via HDMI.
Firebug Test Runner automates the process of running the Firebug tests.
JS Reftest used to run the JavaScript tests in the browser using the Reftest framework.
JS Shell (J) test every piece of the JavaScript engine.
Marionette used to drive the UI and JS/XPCOM layer of remote or local instances.
Mochitest: A11y (Moth) tests accessibility interfaces.
Mochitest: Browser Chrome (Moth) runs in the scope of the browser window, this is a rough UI automation tool testing how the browser interacts with itself and with content.
Mochitest: Chrome (Moth) run with high privileges that can test a lot of the browser's functionality. Tests that verify JavaScript interactions with chrome-level objects should go here.
Mochitest: IPC (Moth) tests plugin APIs, particularly out-of-process plugins.
Mochitest: Plain (M) used to test DOM APIs and other pieces of functionality exposed to web content (i.e. requiring no special permissions).
Mochitest: Robocop (Mrc) designed for testing the native UI of Android devices by sending events to the front end.
Mozmill an extensive framework for browser automation.
Peptest measures responsiveness, how "snappy" Firefox/Thunderbird feels, by issuing alerts when the event loop is stuck for more than 50 ms.
Reftest (R) compares the display of two web pages.
Reftest: IPC (Ripc) identical to Reftest, but running in a separate process.
SpeedTests a framework for executing arbitrary tests, which report results via JavaScript calls, in several browsers.
Talos (T) a framework for performance testing.
TPS a multi-profile test automation framework for Firefox Sync.
XPCShell (X) console JavaScript tests. There is no chrome, no content, no window. XPCShell is useful for testing low-level things, such as XPCOM components.

See also

Automation Tools