How to Write Instructions
At walrus.ai, tests can be written in plain English. The English syntax means walrus tests are easy to write, and very flexible. However, with such a broad syntax, sometimes instructions can be ambiguous. This documentation exists to help remove that ambiguity.
In this documentation, you will find:
- The rules that we use to interpret instructions
- Specific examples of how we use those rules
- Tips and tricks for writing resilient walrus tests.
Rules for resolving ambiguity
It must be unambiguous how a given instruction should be completed given the context clues on the page.
Context cues must be visible, or specified in the instructions
If your instruction provided requires clicking on drop-downs, hovering over collapsable menus or going to different parts on the website not shown on the present page, you must specify in instructions. We will only execute test instructions based on provided instruction and what exists on present page
Good Instruction
instructions:- 'Click top left menu > Select Prime Video in drop-down menu'
- We will pass this part of the test if 'Prime Video' is found under the drop-down menu
Bad Instruction
instructions:- 'Select Prime Video'
- We will fail this test if 'Prime Video' is not visible on the present page and requires clicking drop-down menu
Specific vs Generic context instruction
Instructions may be written to reference a specific element, or simply to complete some action
Specific Instruction: Specifying an element to interact with
instructions:- 'Click the Continue Button'
- We will pass this test if the page presents a button that contains 'Continue'
- We will fail this test if the page does not present a button that contains 'Continue'. If the page presents a 'Proceed' button, we will still fail this test due to the fact the instruction specifies a 'Continue' button
Generic Instruction: Not specifying specific elements
instructions:- 'Continue'
- We will pass test if a button that allows them to continue is presented, such as 'Proceed' or 'Submit'. This instruction iteration would be a great alternative to avoid the failure in the above example
- We will fail test if there is no button that allows them to continue
Exceptions
1. Dismissing modals
- If a modal pops up, we can click out of modal to dismiss it
- If the modal does not dismiss after being clicked-out-of, we will accept / take the affirmative action on the modal in order to continue with the test
- How to prevent false failures: Determine if any modals that pop up in the test environment can be dismissed when clicking out of the modal. If not, specify in test how to continue or dismiss the modal.
2. Filling in Forms
- If you would only like certain fields filled out, specify which fields
instructions:- 'Fill in Your Name and Email'
Will be completed as:
- If no fields are specified, we will fulfill all presented fields
instructions:- 'Fill in fields'
Will be completed as:
- If auto-fill suggestions load in an input field, we will select the first suggestion after filling in the field
Instructions:- 'Add 1 market street, san francisco for the address'
3. Captchas
- If we get captcha'd during the test, we will complete the captcha and continue with the test
4. Log-in Flows
We will click any variant of sign in or sign up (log in, create account, sign in, etc.) both before and after entering credentials
Conflicting context cues
If there are multiple cues on the page that makes it difficult to understand which action to take, we will fail the test
Provided instructions:
instructions:- 'Check that the Toggle is on'
Current page:
We will fail this test because there multiple toggles that affect notifications presented on the page.
If the instructions were re-written as:
instructions:- 'Check that the "Subscriptions" Notification toggle is on'
On the current page, there is only one toggled labeled as 'Subscriptions'. By iterating the instruction to have only one interpretation and can be completed on the presented page, we will pass this test
Login methods
- If there are multiple login methods, specify which method
- If email/PW login is specified, the credentials must be specified
- If test requires to login with Google, that must be specified + which Google account
Exceptions
- If you are instructing us to take an action and there are multiple ways we can take it (ie. different buttons that say the same thing), we will choose one.
How we move through from one instruction to the next
When we complete the instruction as written, we move on to the next instruction
Once we have successfully completed a step within the instructions, we will move on to the next step. We will not make any assumptions or use domain knowledge to guess which assertions to take after the action
Extraneous error messages
Extraneous error messages does not dictate the success of the instruction step if the action itself was successfully completed
Provided instructions:
instructions:- 'Click on the What Have You Become? Video'
Current page after completing first step:
Although an error message has appeared, we will still pass this part of the test because we were able to successfully click on the video link
Interacting with specific elements
We will only interact with specified elements within an instruction, then consider the instruction complete
Provided instructions:
instructions:- 'Under Accounts & Lists, click Start Here'- 'Verify you receive an email verification link'- 'Click on the link in the email'- 'Verify login is successful'
We will fail this test at instruction 2, as there were no instructions on how to receive an email verification link. As we fail tests, we will send back failure messages with the reason for failure. In this example, we would send an error message such as, "No email received or specified. Did you mean to add instructions on filling out the signup form?"
How we reference previous instructions
During tests, we can assert or act based on data from previous instructions in order to complete the test execution. If we should assert on data during the test that was not entered by us during the test execution, you should note to reference it later in the test.
instructions:- 'Register with an alias of winona@walrus.services'- 'Go to Users'- 'Verify the newly registered email address loads in the list'
Since we registered the email in a previous instruction, we can assert on that information
Noting values to use in a later instruction
If the test requires referencing data or value not inputed during the test, you must note that reference in the instructions
Passing Test
instructions:- 'Search for a flight from New York to San Francisco'- 'Note the price of the first flight option'- 'Search for a flight from Seattle to San Francisco'- 'Verify the first flight option is cheaper than the previously searched flight'
We will pass this test because we are able to reference data outputted during the test execution
Failing Test
If the instructions were provided as:
instructions:- 'Search for a flight from New York to San Francisco'- 'Search for a flight from Seattle to San Francisco'- 'Verify the first flight option is cheaper than the previous searched flight'
We will fail this test because there were no instructions to reference/note the price of the first flight during a previous step
How we interpret the use of quotation marks
Instructions can be written out with the use of quotations when certain text must be matched exactly as is (case-insensitive). The quoted text will be referenced as contained within a string on the page. Quotations can be single, double, or backticks.
instructions:- 'Click on the "sign-up" button'
While there is a button that allows us to sign-up, we will fail this test because the provided instructions quoted sign-up, meaning the test execution is looking for the exact match.
Alternatively, if the instructions were provided as:
instructions:- 'Click on the "create" button'
We will pass this test because the word "create" is contained within the button, even though it is next to another word