How to Write Instructions
At walrus.ai, tests can be written in plain English. The English syntax means walrus tests are easy to write, and very flexible. However, with such a broad syntax, sometimes instructions can be ambiguous. This documentation exists to help remove that ambiguity.
In this documentation, you will find:
- The rules that we use to interpret instructions
- Specific examples of how we use those rules
- Tips and tricks for writing resilient walrus tests.
It must be unambiguous how a given instruction should be completed given the context clues on the page.
If your instruction provided requires clicking on drop-downs, hovering over collapsable menus or going to different parts on the website not shown on the present page, you must specify in instructions. We will only execute test instructions based on provided instruction and what exists on present page
instructions:- 'Click top left menu > Select Prime Video in drop-down menu'
- We will pass this part of the test if 'Prime Video' is found under the drop-down menu
instructions:- 'Select Prime Video'
- We will fail this test if 'Prime Video' is not visible on the present page and requires clicking drop-down menu
Instructions may be written to reference a specific element, or simply to complete some action
Specific Instruction: Specifying an element to interact with
instructions:- 'Click the Continue Button'
- We will pass this test if the page presents a button that contains 'Continue'
- We will fail this test if the page does not present a button that contains 'Continue'. If the page presents a 'Proceed' button, we will still fail this test due to the fact the instruction specifies a 'Continue' button
Generic Instruction: Not specifying specific elements
- We will pass test if a button that allows them to continue is presented, such as 'Proceed' or 'Submit'. This instruction iteration would be a great alternative to avoid the failure in the above example
- We will fail test if there is no button that allows them to continue
- If a modal pops up, we can click out of modal to dismiss it
- If the modal does not dismiss after being clicked-out-of, we will accept / take the affirmative action on the modal in order to continue with the test
- How to prevent false failures: Determine if any modals that pop up in the test environment can be dismissed when clicking out of the modal. If not, specify in test how to continue or dismiss the modal.
- If you would only like certain fields filled out, specify which fields
instructions:- 'Fill in Your Name and Email'
Will be completed as:
- If no fields are specified, we will fulfill all presented fields
instructions:- 'Fill in fields'
Will be completed as:
- If auto-fill suggestions load in an input field, we will select the first suggestion after filling in the field
Instructions:- 'Add 1 market street, san francisco for the address'
- If we get captcha'd during the test, we will complete the captcha and continue with the test
We will click any variant of sign in or sign up (log in, create account, sign in, etc.) both before and after entering credentials
If there are multiple cues on the page that makes it difficult to understand which action to take, we will fail the test
instructions:- 'Check that the Toggle is on'
We will fail this test because there multiple toggles that affect notifications presented on the page.
If the instructions were re-written as:
instructions:- 'Check that the "Subscriptions" Notification toggle is on'
On the current page, there is only one toggled labeled as 'Subscriptions'. By iterating the instruction to have only one interpretation and can be completed on the presented page, we will pass this test
- If there are multiple login methods, specify which method
- If email/PW login is specified, the credentials must be specified
- If test requires to login with Google, that must be specified + which Google account
- If you are instructing us to take an action and there are multiple ways we can take it (ie. different buttons that say the same thing), we will choose one.
Once we have successfully completed a step within the instructions, we will move on to the next step. We will not make any assumptions or use domain knowledge to guess which assertions to take after the action
Extraneous error messages does not dictate the success of the instruction step if the action itself was successfully completed
instructions:- 'Click on the What Have You Become? Video'
Current page after completing first step:
Although an error message has appeared, we will still pass this part of the test because we were able to successfully click on the video link
We will only interact with specified elements within an instruction, then consider the instruction complete
instructions:- 'Under Accounts & Lists, click Start Here'- 'Verify you receive an email verification link'- 'Click on the link in the email'- 'Verify login is successful'
We will fail this test at instruction 2, as there were no instructions on how to receive an email verification link. As we fail tests, we will send back failure messages with the reason for failure. In this example, we would send an error message such as, "No email received or specified. Did you mean to add instructions on filling out the signup form?"
During tests, we can assert or act based on data from previous instructions in order to complete the test execution. If we should assert on data during the test that was not entered by us during the test execution, you should note to reference it later in the test.
instructions:- 'Register with an alias of email@example.com'- 'Go to Users'- 'Verify the newly registered email address loads in the list'
Since we registered the email in a previous instruction, we can assert on that information
If the test requires referencing data or value not inputed during the test, you must note that reference in the instructions
instructions:- 'Search for a flight from New York to San Francisco'- 'Note the price of the first flight option'- 'Search for a flight from Seattle to San Francisco'- 'Verify the first flight option is cheaper than the previously searched flight'
We will pass this test because we are able to reference data outputted during the test execution
If the instructions were provided as:
instructions:- 'Search for a flight from New York to San Francisco'- 'Search for a flight from Seattle to San Francisco'- 'Verify the first flight option is cheaper than the previous searched flight'
We will fail this test because there were no instructions to reference/note the price of the first flight during a previous step
Instructions can be written out with the use of quotations when certain text must be matched exactly as is (case-insensitive). The quoted text will be referenced as contained within a string on the page. Quotations can be single, double, or backticks.
instructions:- 'Click on the "sign-up" button'
While there is a button that allows us to sign-up, we will fail this test because the provided instructions quoted sign-up, meaning the test execution is looking for the exact match.
Alternatively, if the instructions were provided as:
instructions:- 'Click on the "create" button'
We will pass this test because the word "create" is contained within the button, even though it is next to another word