Awesome
flutter_convenient_test
Write and debug tests easily, with full action history, time travel, screenshots, rapid re-execution, video records, retryability, interactivity, isolation and more.
🚀 Advantages
- Enjoy 5x speedup: Able to execute your tests not only on simulator/device, but also on host computer (much faster), without changing any code.
- Full action history: Gain insight on what has happened to the tests - View all actions/assertions taken in the tests, with friendly descriptions.
- Time travel, with screenshots: What did the UI look like when that button was tapped 50 steps ago? Now you know everything
- Rapid re-execution: Edit code, save, run - done within seconds, not minutes.
- Videos recorded: Watch what has happened, in full detail.
- Retryability: No manual
pump
, wait or retry anymore. Just write down what you want, and the framework will retry and wait. - Being interactive: Play with the app interactively, again within seconds.
- Isolated: One test will no longer destroy environment of other tests - run each with full isolation (optional feature).
- Abundant information for errors: When something is not working, it tries to print out as much information as possible to help you quickly debug.
- Enhanced golden: GUI to check golden changes, allow a few pixels to be different, screenshot widgets above your widget, etc (Standalone feature, does not require full convenient_test)
- Utilities: Enter text without replacing, pump with run async, etc. (Standalone feature, does not require full convenient_test)
And also...
integration_test
still there: Freely use everything fromintegration_test
,mockito
,test
, etc.- Flaky test awareness: No false alarms, while no blindly ignoring.
- Suitable for CI: Run headlessly, with logs to be visualized when needed.
- Miscellaneous: Visually see target regions, run single test/group within seconds, raw logs...
📷 Quick demo
NOTE: The video only contains features up to 2022.05. New features not included in the video: The 5x speedup, the enhanced golden, etc.
<!--README_VIDEO_REPLACEMENT_PLACEHOLDER_ONE--> <!--README_VIDEO_REPLACEMENT_PLACEHOLDER_TWO-->https://user-images.githubusercontent.com/5236035/167066810-d0aa36ba-0113-4140-92f9-cec0a9e77ed1.mov
Stability
It has been used extensively in my own 200kloc Flutter app in production environment, and I integrate it with my CI infra so it is run everyday.
P.S. Surely, this package - especially the doc - is not yet perfect. If having problems, just create an issue and I usually reply quite quickly.
Unreleased features
I have been using these components internally for a long time, but have not found time to make cleanup (e.g. remove internal dependencies) in order to open-source it. If you like it, feel free to tell me and I will prioritize it!
- Monkey tester: Randomly interact with tappable/draggable/... regions of your app, trying to find crashes. I even find a bug of Flutter itself using this simple monkey!
- Benchmark helper: Using one command to automatically benchmark on all devices, upload to storage, and generate reports. (I run it in pseudo-CI and read the reports to see performance regressions.)
📚 Features
5x speedup & remove mobile devices
In addition to the standard way of integration tests, i.e. run them on a real mobile device / a simulator, this package supports running the exact same testing code on your host computer (Windows/MacOS/Linux) with similar experience.
This leads to a big speedup and reduction of needed resources. As we all know, running code on simulators / real devices are slow, and hard to parallelize as well. Running on host computer, on the other hand, does not have this problem.
<sup><sub>Remark: In my own app in production, it is >10x faster, but without creating public and reproducible tests, I do not want to claim such a big improvement - so test by youself and see how fast it provides :)</sub></sup>
How to use it: Just write code normally, and execute your code as if it is a widget test. If you see weird fonts, you may want loadAppFonts.
<br>Full action history
See actions/taps/assertions taken in the tests, with friendly descriptions (in the left panel)
Time travel with screenshots
Tap an action to see its screenshots
P.S. Use mouse wheels and dragging to zoom and pan the screenshots.
P.S. The gif is outdated - new version has (1) an overview of all screenshots (2) a ruler for each screenshot.
<br>Rapid re-execution
Edit code, save, run - within seconds
Videos recorded
Watch what has happened in full detail (in right panel)
P.S. Highlighted action is in sync with the playing video. Tap action to seek video.
<br>Being interactive
Temporarily play with the app, interactively. (Flutter originally cannot interact with app in test mode)
Retryability
No manual pump
, wait or retry anymore. Just write down what you want, and the framework will retry and wait.
Example:
await t.get(find.byTooltip('Fetch Data From Backend')).tap();
// OK even if "fetch data" needs undeterministic time interval. Will automatically pump, wait and retry.
await t.get(find.text('New Data')).should(findsOneWidget);
More in quickstart and tutorials below.
Isolation mode
One test will no longer destroy environment of other tests - now you can run run each with full isolation.
This is especially helpful in big projects (such as mine), when an end-to-end setup is not easy to tear down, and one failed test can cause all subsequent tests to have damaged execution environment and thus fail.
Technical details: If this mode is enabled, a hot restart will be performed after each attempt of each test.
Abundant information for errors
When something is not working, it tries to print out as much information as possible to help you quickly debug.
For example, suppose you provide a finder
that accidentally matches two widgets. Then, convenient_test will tell you the location of each widget, as well as a chain of render objects. Together with the screenshots/videos it provides, the finder problem can be fixed easily.
<sup><sub>Remark: The GIFs are outdated and does not show this feature.</sub></sup>
Enhanced golden
This is a standalone feature, i.e. you can use it without using convenient_test.
Golden overview GUI
It is a simple GUI to list and check all golden differences, with interactivity like pan and zoom. I personally use it daily as follows: After changing a feature, I run all regression tests. Then, if some tests report a golden change, I look at this page to check whether the change is intended or a bug.
To enter this page, tap the "golden" button at the top of homepage.
Golden utilities
- Allow a few pixels to be different: Flutter's builtin golden test requires every pixel to be exactly the same. I often see cases when the golden screenshot is the "same" from human eyes, but is marked as failed because of non-perceptable differences. This feature allows you to configure amount of toleration to avoid that. (
GoldenConfig.maxToleration
,GoldenConfig.greaterThanToleration
) - Crop a part of widget / Screenshot widgets above your widget: For example, suppose you have a button with a hint bubble, implemented by flutter_portal or
Overlay
. Then, if you golden byfind.byKey(yourButton)
, you will not screen the hint bubble as well. By using thiscropBbox
feature, you can take a screenshot of the whole screen, and crop the parts (the button) you want. (GoldenConfig.cropBbox
) - Forbid updating local files: Useful when your file is generated from elsewhere (e.g. another line of code), and you never want to update it even with
--update-goldens
. (GoldenConfig.allowUpdate
) - When a golden assertion fails, its image details will be also shown in the manager panel.
Quickstart example
goldenFileComparator = EnhancedLocalFileComparator.configFromCurrent(); // setup
// just like the old way, except that we call `EnhancedLocalFileComparator.createUri` with extra configurations
final config = GoldenConfig(...);
await expectLater(whatever, matchesGoldenFile(EnhancedLocalFileComparator.createUri('something.png', config)));
<!--[TODO screenshot]-->
Utilities
- Enter text without replacing (
enterTextWithoutReplace
): The standardenterText
replaces the old content, making it inconvenient if we want to append some text - which is the normal behavior of humans. - Pump with run async (
runAsyncAndPumpUntil
,pumpAndSettleWithRunAsync
, etc): If your code does something that needs "real" async, a simplepumpAndSettle
will not work, so this function is created.
integration_test
is still there
You can still use everything from integration_test
, mockito
, test
, etc. This package is not a reinvented wheel, and has exposed the underlying integration_test
to you.
If you want to migrate to this package from raw integration_test
, have a look at Getting Started section below.
Flaky tests awareness
Flaky is flaky, and we are aware of it. It is neither failed nor success, so you will not be alarmed by false positives, and not completely ignore it.
<!--[TODO screenshot: Several tests, one failed, one flaky, one success]-->CI / headless mode
packages/convenient_test_manager_dart
runs without GUI
and only produce log data and exit code, making it suitable to be run in a CI.
Simply use convenient_test_manager_dart
instead of convenient_test_manager
.
See Run the manager
and Getting started.
# in one shell
convenient_test_manager_dart
# in another shell, run your worker app
flutter run integration_test/main_test.dart --host-vmservice-port 9753 --disable-service-auth-codes --dart-define "CONVENIENT_TEST_APP_CODE_DIR=$PWD"
Reports
If you want to examine the details with GUI, just open the generated artifact in the GUI using Load Report
button.
By default the manager will save reports to a temporary directory. Typically:
- macOS:
$HOME/Library/Caches/ConvenientTest
- Linux:
/tmp/ConvenientTest
- Windows:
%LocalAppData%\Temp\ConvenientTest
Use --report-save-path /some/path
(or --dart-define "CONVENIENT_MANAGER_REPORT_SAVE_PATH=/some/path"
) to change this.
Format
The reports are exported using protobuf.
See the schema at convenient_test.proto
.
Run single test/group
Tap "Run" icon button after each test or group to re-run only that test/group, without running anything else. All within seconds - no need to recompile like what the original test package did.
Visually see target regions
Useful when replaying videos and screenshots
Mark
s have colored borders- Gestures have visual feedbacks
Raw logs
Tap "Raw Logs" in the right panel to see raw outputs of a test.
Header panel
Following is a brief description of the functionalities of each button in the header panel.
- Run All: Run all tests below
- Halt: Stop the currently running worker
- Interactive Mode: Enter "interactive mode" (described above)
- Reload Info: Reload test metadata such as the group names and test names
- Reconnect VM: Connect to the Dart VM of the worker app
- Load Report: Load and display a report generated when
SaveReport
toggle is enabled or run within CI/headless mode - Golden Diff Page: Enter the golden diff page (described above)
Following are about the toggles:
- Isolation: Whether to enable "isolation mode" (described above)
- UpdateGoldens: Whether to update the golden files on the disk, similar to
--update-goldens
flag in a traditional Flutter test - Retry: When a test fails, whether to retry (and mark as flaky), or directly mark it failed
- SaveReport: Whether to save logs to a report file, such that it can be visualized later via "Load Report"
- Hover: When hovering the action log panel, whether highlight the hovered entry and immediately show related information
- AutoJump: Whether to jump to newly received action entries
- AutoExpand: Whether to expand and collapse automatically for the newly received test and action entries
Tutorial: Run examples
- Clone this repository and enter the
packages/convenient_test/example
folder. - Run the example app (e.g. using iOS simulator) via:
Can also be run via VSCode or Android Studio with similar commands.flutter run integration_test/main_test.dart --host-vmservice-port 9753 --disable-service-auth-codes --dart-define "CONVENIENT_TEST_APP_CODE_DIR=$PWD"
- Run the GUI located in
packages/convenient_test_manager
. (See details in section below) - Enjoy the GUI!
Getting started
- (Optional but recommended) Navigate to the flutter home directory and apply this patch:
cd $FLUTTER_HOME && curl https://raw.githubusercontent.com/fzyzcjy/flutter_convenient_test/master/doc/patches/flutter/get-running-async-tasks.diff | git apply
- While this is not strictly necessary, it is recommended to improve scenarios when running multiple async tasks. see #337 for more information
- Navigate to your project dir and run
flutter pub add convenient_test dev:convenient_test_dev
- or manually update your
pubspec.yaml
to includeconvenient_test: ^1.0.0
in thedependencies
section, andconvenient_test_dev: ^1.0.0
in thedev_dependencies
section. As normal, we need toflutter pub get
.
- or manually update your
- Create
integration_test/main_test.dart
file in your app. Fill it likevoid main() => convenientTestMain(MyConvenientTestSlot(), () { ... the normal test code you write });
. See the examplemain_test.dart
for demonstration. - Run your app (e.g. using iOS simulator) via
flutter run integration_test/main_test.dart --host-vmservice-port 9753 --disable-service-auth-codes --dart-define "CONVENIENT_TEST_APP_CODE_DIR=$PWD"
. Can also be run via VSCode or Android Studio with similar commands.- for use with the Android emulator, add
--dart-define CONVENIENT_TEST_MANAGER_HOST=10.0.2.2
- for use with the Android emulator, add
- Run the Convenient Test Manager (See details in section below)
Run the manager
packages/convenient_test_manager
is a flutter GUI app that allows running and inspecting individual tests. Use this for development and debugging.packages/convenient_test_manager_dart
is a dart CLI app that runs all tests and terminates with a corresponding exit code. Use this for CI and automated testing. See CI / headless mode
Method 1: Download precompiled binary
See Releases for binaries.
Additionally, there are precompiled binaries for each commit, at the "artifacts" section in the CI page
(download only available for logged in GitHub users).
For example, go to the most recent CI run, download manager_gui_macos
artifact, unzip it, and open it.
Method 2: flutter run
/ flutter build
The GUI and CLI managers are just flutter and dart projects as well, respectively. So they can be run and built in the same way one would with any other flutter or dart project:
Check the requirements for developing flutter desktop apps for macOS, Linux, and Windows. Then:
git clone https://github.com/fzyzcjy/flutter_convenient_test.git
cd flutter_convenient_test/packages/convenient_test_manager
# compile executable. platform = linux | macos | windows
flutter build <platform>
# or just run directly
flutter run -d <platform>
For CLI manager:
cd flutter_convenient_test/packages/convenient_test_manager
dart compile exe bin/convenient_test_manager_dart.dart
# or just run directly
dart run
Method 3: dart pub global
(only CLI manager)
Set up once (re-run to update):
dart pub global activate --source git https://github.com/fzyzcjy/flutter_convenient_test.git --git-path packages/convenient_test_manager_dart
[!TIP] Add
--git-ref v1.x.x
todart pub global activate
to check out a specific release.
Run using:
dart pub global run convenient_test_manager_dart
Miscellaneous
Configuration
There are a few ways to configure the manager:
- From
$HOME/.config/convenient_test.json
. - From various environment variables.
- From command line arguments (when run via
convenient_test_manager_dart
command line) - Change configurations via GUI (e.g. switches at right-top)
For all options that are configurable and all environment variable names, please see
consts.dart
static_config.dart
global_config_store.dart
only manager related options
(By doing so you can always see the up-to-date information and there is no possibility to see outdated doc.)
If you are trying to use this package with Android Virtual Emulator (AVD), you may need to set kConvenientTestManagerHost
to 10.0.2.2
(add --dart-define CONVENIENT_TEST_MANAGER_HOST=10.0.2.2
) because of its network topology. Please see issue #253 for more details.
Limitations
Video replaying
feature is not yet implemented in Android (but feel free to PR!).
Q&A
pumpWithRunAsyncUntil
timed out
One thing to check is whether there are infinite animations. For example, setting EditableText.debugDeterministicCursor = true
can avoid cursor blinking forever.
Contributors
For detailed PRs (excluding myself), please see this link.
<!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section --> <!-- ALL-CONTRIBUTORS-BADGE:END --> <!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --> <!-- prettier-ignore-start --> <!-- markdownlint-disable --> <table> <tbody> <tr> <td align="center" valign="top" width="14.28%"><a href="https://github.com/Rohithgilla12"><img src="https://avatars.githubusercontent.com/u/19389850?v=4?s=100" width="100px;" alt="Rohith Gilla"/><br /><sub><b>Rohith Gilla</b></sub></a><br /><a href="https://github.com/fzyzcjy/flutter_convenient_test/commits?author=Rohithgilla12" title="Code">💻</a></td> <td align="center" valign="top" width="14.28%"><a href="https://mehrankordi.com"><img src="https://avatars.githubusercontent.com/u/1547111?v=4?s=100" width="100px;" alt="Mehran Kordi"/><br /><sub><b>Mehran Kordi</b></sub></a><br /><a href="https://github.com/fzyzcjy/flutter_convenient_test/commits?author=MCord" title="Code">💻</a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/ethancadoo"><img src="https://avatars.githubusercontent.com/u/100008135?v=4?s=100" width="100px;" alt="ethancadoo"/><br /><sub><b>ethancadoo</b></sub></a><br /><a href="https://github.com/fzyzcjy/flutter_convenient_test/commits?author=ethancadoo" title="Code">💻</a> <a href="https://github.com/fzyzcjy/flutter_convenient_test/commits?author=ethancadoo" title="Documentation">📖</a></td> <td align="center" valign="top" width="14.28%"><a href="https://vhanda.in"><img src="https://avatars.githubusercontent.com/u/426467?v=4?s=100" width="100px;" alt="Vishesh Handa"/><br /><sub><b>Vishesh Handa</b></sub></a><br /><a href="https://github.com/fzyzcjy/flutter_convenient_test/commits?author=vHanda" title="Code">💻</a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/ostk0069"><img src="https://avatars.githubusercontent.com/u/27538852?v=4?s=100" width="100px;" alt="Takuma Osada"/><br /><sub><b>Takuma Osada</b></sub></a><br /><a href="https://github.com/fzyzcjy/flutter_convenient_test/commits?author=ostk0069" title="Code">💻</a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/ronba"><img src="https://avatars.githubusercontent.com/u/10216856?v=4?s=100" width="100px;" alt="ronba"/><br /><sub><b>ronba</b></sub></a><br /><a href="https://github.com/fzyzcjy/flutter_convenient_test/commits?author=ronba" title="Code">💻</a></td> <td align="center" valign="top" width="14.28%"><a href="https://pacia.tech"><img src="https://avatars.githubusercontent.com/u/40357511?v=4?s=100" width="100px;" alt="Bartek Pacia"/><br /><sub><b>Bartek Pacia</b></sub></a><br /><a href="https://github.com/fzyzcjy/flutter_convenient_test/commits?author=bartekpacia" title="Code">💻</a></td> </tr> <tr> <td align="center" valign="top" width="14.28%"><a href="https://blog.alexv525.com"><img src="https://avatars.githubusercontent.com/u/15884415?v=4?s=100" width="100px;" alt="Alex Li"/><br /><sub><b>Alex Li</b></sub></a><br /><a href="https://github.com/fzyzcjy/flutter_convenient_test/commits?author=AlexV525" title="Code">💻</a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/getBoolean"><img src="https://avatars.githubusercontent.com/u/19920697?v=4?s=100" width="100px;" alt="getBoolean"/><br /><sub><b>getBoolean</b></sub></a><br /><a href="https://github.com/fzyzcjy/flutter_convenient_test/commits?author=getBoolean" title="Code">💻</a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/gilice"><img src="https://avatars.githubusercontent.com/u/104317939?v=4?s=100" width="100px;" alt="gilice"/><br /><sub><b>gilice</b></sub></a><br /><a href="https://github.com/fzyzcjy/flutter_convenient_test/commits?author=gilice" title="Code">💻</a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/madmini"><img src="https://avatars.githubusercontent.com/u/26089559?v=4?s=100" width="100px;" alt="Max Jakobitsch"/><br /><sub><b>Max Jakobitsch</b></sub></a><br /><a href="https://github.com/fzyzcjy/flutter_convenient_test/commits?author=madmini" title="Code">💻</a> <a href="https://github.com/fzyzcjy/flutter_convenient_test/commits?author=madmini" title="Documentation">📖</a></td> <td align="center" valign="top" width="14.28%"><a href="https://keybase.io/jmatth"><img src="https://avatars.githubusercontent.com/u/1316184?v=4?s=100" width="100px;" alt="Josh Matthews"/><br /><sub><b>Josh Matthews</b></sub></a><br /><a href="https://github.com/fzyzcjy/flutter_convenient_test/commits?author=jmatth" title="Code">💻</a></td> <td align="center" valign="top" width="14.28%"><a href="https://gnunicorn.org"><img src="https://avatars.githubusercontent.com/u/40496?v=4?s=100" width="100px;" alt="Benjamin Kampmann"/><br /><sub><b>Benjamin Kampmann</b></sub></a><br /><a href="https://github.com/fzyzcjy/flutter_convenient_test/commits?author=gnunicorn" title="Code">💻</a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/meltzow"><img src="https://avatars.githubusercontent.com/u/1238001?v=4?s=100" width="100px;" alt="Mario Meltzow"/><br /><sub><b>Mario Meltzow</b></sub></a><br /><a href="https://github.com/fzyzcjy/flutter_convenient_test/commits?author=meltzow" title="Code">💻</a></td> </tr> <tr> <td align="center" valign="top" width="14.28%"><a href="https://github.com/lunakoan"><img src="https://avatars.githubusercontent.com/u/119916943?v=4?s=100" width="100px;" alt="Luna"/><br /><sub><b>Luna</b></sub></a><br /><a href="https://github.com/fzyzcjy/flutter_convenient_test/commits?author=lunakoan" title="Code">💻</a></td> </tr> </tbody> </table> <!-- markdownlint-restore --> <!-- prettier-ignore-end --> <!-- ALL-CONTRIBUTORS-LIST:END -->Thanks for testing frameworks in JavaScript, especially Cypress
, for giving inspirations!