mirror of
https://github.com/Kong/insomnia
synced 2024-11-07 22:30:15 +00:00
[INS-2051] Reorganize smoke tests README (#5264)
* [INS-2051] Reorganize smoke tests README * Add images to smoke-tests readme * Split CLI part of doc, rm notes * Reorganize README a bit * edit intro * playwright badge
This commit is contained in:
Filipe Freire
GitHub
parent
0b30be3f6b
commit
45845f7fe3
58
packages/insomnia-smoke-test/CLI.md
Normal file
58
packages/insomnia-smoke-test/CLI.md
Normal file
@ -0,0 +1,58 @@
|
||||
# CLI
|
||||
|
||||
## Run CLI Smoke Tests
|
||||
|
||||
```shell
|
||||
# Package the Inso CLI binaries
|
||||
npm run inso-package
|
||||
|
||||
# Run CLI tests
|
||||
npm run test:smoke:cli
|
||||
```
|
||||
|
||||
## Debugging CLI tests using watcher
|
||||
|
||||
This is helpful for debugging failing api tests and changing the send-request abstraction
|
||||
|
||||
From project root, in separate terminals:
|
||||
|
||||
```sh
|
||||
# start smoke test api
|
||||
npm run serve --prefix packages/insomnia-smoke-test
|
||||
|
||||
# build send-request
|
||||
npm run build:sr --prefix packages/insomnia
|
||||
|
||||
# watch inso
|
||||
npm run start --prefix packages/insomnia-inso
|
||||
|
||||
# run api test
|
||||
$PWD/packages/insomnia-inso/bin/inso run test "Echo Test Suite" --src $PWD/packages/insomnia-smoke-test/fixtures/inso-nedb --env Dev --verbose
|
||||
```
|
||||
|
||||
## How to update the `inso-nedb` fixtures
|
||||
|
||||
Run Insomnia with `INSOMNIA_DATA_PATH` environment variable set to `fixtures/inso-nedb`, e.g.:
|
||||
|
||||
```bash
|
||||
INSOMNIA_DATA_PATH=packages/insomnia-smoke-test/fixtures/inso-nedb /Applications/Insomnia.app/Contents/MacOS/Insomnia
|
||||
```
|
||||
|
||||
Relaunch the app one more time, so that Insomnia compacts the database.
|
||||
|
||||
The `.gitignore` file will explicitly ignore certain database files, to keep the directory size down and avoid prevent sensitive data leaks.
|
||||
|
||||
## How to run inso with the `inso-nedb` fixture locally?
|
||||
|
||||
Set the `--src` argument pointed to `packages/insomnia-smoke-test/fixtures/inso-nedb`:
|
||||
|
||||
```bash
|
||||
# if installed globally
|
||||
inso --src <INSO_NEDB_PATH>
|
||||
|
||||
# using the package bin
|
||||
./packages/insomnia-inso/bin/inso --src <INSO_NEDB_PATH>
|
||||
|
||||
# using a binary
|
||||
./packages/insomnia-inso/binaries/insomnia-inso --src <INSO_NEDB_PATH>
|
||||
```
|
||||
@ -1,50 +1,95 @@
|
||||
# Insomnia Smoke Tests
|
||||
|
||||
This project contains the smoke testing suite for Insomnia and Inso.
|
||||
[](https://github.com/microsoft/playwright)
|
||||
|
||||
Tests for the Electron app are written using [Playwright](https://github.com/microsoft/playwright) while tests for the CLI use [execa](https://github.com/sindresorhus/execa).
|
||||
This project contains the smoke testing suite for Insomnia App.
|
||||
|
||||
## Structure
|
||||
> To find more about Inso CLI smoke tests, check [this document](CLI.md).
|
||||
|
||||
| Folder | Purpose |
|
||||
| ------------ | --------------------------------- |
|
||||
| `/cli` | tests for inso |
|
||||
| `/tests` | tests for Insomnia |
|
||||
| `/playwright`| test helpers |
|
||||
| `/server` | Express server used by the tests |
|
||||
| `/fixtures` | data used by tests and the server |
|
||||
- [Insomnia Smoke Tests](#insomnia-smoke-tests)
|
||||
- [Quick-start](#quick-start)
|
||||
- [Debugging and Developing Tests locally](#debugging-and-developing-tests-locally)
|
||||
- [Playwright VS Code extension](#playwright-vs-code-extension)
|
||||
- [Playwright Inspector](#playwright-inspector)
|
||||
- [Playwright Trace viewer](#playwright-trace-viewer)
|
||||
- [Additional Log levels](#additional-log-levels)
|
||||
- [Reproducing CI Failures](#reproducing-ci-failures)
|
||||
- [Getting traces from CI](#getting-traces-from-ci)
|
||||
- [Build and package methods](#build-and-package-methods)
|
||||
|
||||
## Run Insomnia smoke tests
|
||||
## Quick-start
|
||||
|
||||
### Development method
|
||||
Prerequisites:
|
||||
|
||||
In one terminal run the watcher
|
||||
- Clone the project
|
||||
- Run `npm run bootstrap`
|
||||
|
||||
To run all tests:
|
||||
|
||||
- In one terminal run: `npm run watch:app`
|
||||
- In another terminal run: `npm run test:smoke:dev`
|
||||
|
||||
To run single tests:
|
||||
|
||||
- Filter by the file or test title, e.g. `npm run test:smoke:dev -- oauth`
|
||||
|
||||
## Debugging and Developing Tests locally
|
||||
|
||||
### Playwright VS Code extension
|
||||
|
||||
In order to run/debug tests directly from VS Code:
|
||||
|
||||
- Install the [Playwright extension](https://marketplace.visualstudio.com/items?itemName=ms-playwright.playwright).
|
||||
- With the extension installed, run on terminal `npm run watch:app`.
|
||||
|
||||
You can trigger tests from the `Testing` tab, or within the test files clicking the run button.
|
||||
|
||||
|
||||
|
||||
If no tests appear, you may need to run "Refresh playwright tests". This can be done from the command palette, or by using the button at the top of the `Testing` tab.
|
||||
|
||||
|
||||
|
||||
### Playwright Inspector
|
||||
|
||||
You can step through tests with playwright inspector: `PWDEBUG=1 npm run test:smoke:dev`
|
||||
|
||||
This is also useful to help create new tests.
|
||||
|
||||
|
||||
|
||||
### Playwright Trace viewer
|
||||
|
||||
We generate [Playwright Traces](https://playwright.dev/docs/trace-viewer) when tests run. These can be used to debug local and CI test failures.
|
||||
|
||||
|
||||
|
||||
To open a local trace viewer for a given test output, run:
|
||||
|
||||
```shell
|
||||
npm run watch:app # Run watcher
|
||||
# Example:
|
||||
npx playwright show-trace packages/insomnia-smoke-test/screenshots/app-can-send-requests/trace.zip
|
||||
```
|
||||
|
||||
In a second terminal run/debug/step through smoke tests
|
||||
Alternatively you can upload this trace to [trace.playwright.dev](https://trace.playwright.dev/).
|
||||
|
||||
```shell
|
||||
# Run tests
|
||||
npm run test:smoke:dev
|
||||
### Additional Log levels
|
||||
|
||||
# Run individual tests
|
||||
npm run test:smoke:dev -- oauth
|
||||
You can enable additional logging to help you debug tests:
|
||||
|
||||
# Debug tests with playwright logs
|
||||
DEBUG=pw:api npm run test:smoke:dev
|
||||
- Playwright logs: `DEBUG=pw:api npm run test:smoke:dev`
|
||||
- Insomnia console logs: `DEBUG=pw:browser npm run test:smoke:dev`
|
||||
- WebServer console logs: `DEBUG=pw:WebServer npm run test:smoke:dev`
|
||||
|
||||
# Debug tests with insomnia console logs
|
||||
DEBUG=pw:browser npm run test:smoke:dev
|
||||
## Reproducing CI Failures
|
||||
|
||||
# Debug tests with webserver console logs
|
||||
DEBUG=pw:WebServer npm run test:smoke:dev
|
||||
### Getting traces from CI
|
||||
|
||||
# Step through tests with playwright inspector
|
||||
PWDEBUG=1 npm run test:smoke:dev
|
||||
```
|
||||
Traces from CI execution can be found in the failed CI job's artifacts.
|
||||
|
||||
|
||||
|
||||
After downloading the artifacts, these can be extracted and loaded up into the [Trace viewer](#playwright-trace-viewer).
|
||||
|
||||
### Build and package methods
|
||||
|
||||
@ -52,7 +97,7 @@ It's possible to run the smoke tests for:
|
||||
|
||||
- A `build`, the JS bundle that is loaded into an electron client
|
||||
- A `package`, the executable binary (e.g. `.dmg` or `.exe`)
|
||||
|
||||
|
||||
For `build`:
|
||||
|
||||
```shell
|
||||
@ -74,117 +119,3 @@ npm run test:smoke:package
|
||||
```
|
||||
|
||||
Each of the above commands will automatically run the Express server, so you do not need to take any extra steps.
|
||||
|
||||
### Show trace viewer
|
||||
|
||||
Each time you run a playwright test a trace.zip will be generated, this is useful for debugging CI. [docs](https://playwright.dev/docs/trace-viewer)
|
||||
|
||||
```shell
|
||||
# Open a trace viewer for a given test output
|
||||
npx playwright show-trace packages/insomnia-smoke-test/screenshots/app-can-send-requests/trace.zip
|
||||
# Alternatively you can upload this trace to https://trace.playwright.dev/
|
||||
```
|
||||
|
||||
### Using the Playwright VS Code extension
|
||||
|
||||
You can also use the [playwright extension](https://marketplace.visualstudio.com/items?itemName=ms-playwright.playwright) to run and debug tests directly from VS Code.
|
||||
|
||||
With the extension installed, in a terminal, run the watcher
|
||||
|
||||
```shell
|
||||
npm run watch:app
|
||||
```
|
||||
|
||||
You can now run and debug playwright tests, either from the `Testing` tab, or by going into one of the test files and clicking the run button.
|
||||
If no tests appear, you may need to run "Refresh playwright tests". This can be done from the command palette, or by using the button at the top of the `Testing` tab.
|
||||
|
||||
## Run Inso CLI smoke tests
|
||||
|
||||
```shell
|
||||
# Package the Inso CLI binaries
|
||||
npm run inso-package
|
||||
|
||||
# Run CLI tests
|
||||
npm run test:smoke:cli
|
||||
```
|
||||
|
||||
### Write Inso CLI smoke tests
|
||||
|
||||
In order to run CLI tests for development, open two terminal tabs in `packages/insomnia-smoke-test`:
|
||||
|
||||
```shell
|
||||
# In the first tab, serve the Express API
|
||||
npm run serve
|
||||
|
||||
# In the second tab, run the Inso CLI tests
|
||||
npm run cli
|
||||
```
|
||||
|
||||
This will allow you to write and monitor the server separately from each test, speeding up the development cycle.
|
||||
|
||||
### Debug Inso CLI api test in watch mode
|
||||
|
||||
This is helpful for debugging failing api tests and changing the send-request abstraction
|
||||
|
||||
From project root, in seperate terminals:
|
||||
|
||||
```sh
|
||||
# start smoke test api
|
||||
npm run serve --prefix packages/insomnia-smoke-test
|
||||
|
||||
# build send-request
|
||||
npm run build:sr --prefix packages/insomnia
|
||||
|
||||
# watch inso
|
||||
npm run start --prefix packages/insomnia-inso
|
||||
|
||||
# run api test
|
||||
$PWD/packages/insomnia-inso/bin/inso run test "Echo Test Suite" --src $PWD/packages/insomnia-smoke-test/fixtures/inso-nedb --env Dev --verbose
|
||||
```
|
||||
|
||||
## General guidelines
|
||||
|
||||
### Data
|
||||
|
||||
Individual tests will automatically run against a clean Insomnia data directory to keep data isolated.
|
||||
|
||||
### Dependencies
|
||||
|
||||
A test should not depend on any external services unless absolutely necessary. If a particular endpoint is required (eg. for authentication or a specific content type), implement a new endpoint in `/server`.
|
||||
|
||||
## Working with fixtures
|
||||
|
||||
### How to update the inso-nedb fixture
|
||||
|
||||
In order to update the inso-nedb fixutre you need to launch Insomnia using the inso-nedb directory. To do this, set the INSOMNIA_DATA_PATH environment variable and launch from the command line.
|
||||
|
||||
#### MacOS
|
||||
|
||||
```bash
|
||||
INSOMNIA_DATA_PATH=packages/insomnia-smoke-test/fixtures/inso-nedb /Applications/Insomnia.app/Contents/MacOS/Insomnia
|
||||
```
|
||||
|
||||
#### Linux
|
||||
|
||||
TODO
|
||||
|
||||
#### Windows
|
||||
|
||||
TODO
|
||||
|
||||
After making your changes, be sure to relaunch the app one more time from the command line, so that Insomnia compacts the database. The .gitignore file will explicity ignore certain database files, to keep the directory size down and avoid prevent data leakage in case you are signed in to the app when launching with this database.
|
||||
|
||||
### How to run inso with the inso-nedb fixture locally?
|
||||
|
||||
Set the `--src packages/insomnia-smoke-test/fixtures/inso-nedb` flag
|
||||
|
||||
```bash
|
||||
# if installed globally
|
||||
inso --src ...
|
||||
|
||||
# using the package bin
|
||||
./packages/insomnia-inso/bin/inso --src ...
|
||||
|
||||
# using a binary
|
||||
./packages/insomnia-inso/binaries/insomnia-inso --src ...
|
||||
```
|
||||
|
||||
BIN
packages/insomnia-smoke-test/docs/imgs/artifacts.png
Normal file
BIN
packages/insomnia-smoke-test/docs/imgs/artifacts.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 84 KiB |
BIN
packages/insomnia-smoke-test/docs/imgs/editor.png
Normal file
BIN
packages/insomnia-smoke-test/docs/imgs/editor.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 665 KiB |
BIN
packages/insomnia-smoke-test/docs/imgs/playwright-inspector.jpg
Normal file
BIN
packages/insomnia-smoke-test/docs/imgs/playwright-inspector.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 173 KiB |
BIN
packages/insomnia-smoke-test/docs/imgs/playwright-trace.jpg
Normal file
BIN
packages/insomnia-smoke-test/docs/imgs/playwright-trace.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 255 KiB |
BIN
packages/insomnia-smoke-test/docs/imgs/refresh.png
Normal file
BIN
packages/insomnia-smoke-test/docs/imgs/refresh.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 61 KiB |
Loading…
Reference in New Issue
Block a user