How to Write Instructions

At walrus.ai, tests can be written in plain English. The English syntax means walrus tests are easy to write, and very flexible. However, with such a broad syntax, sometimes instructions can be ambiguous. This documentation exists to help remove that ambiguity.

In this documentation, you will find:

1. The rules that we use to interpret instructions
2. Specific examples of how we use those rules
3. Tips and tricks for writing resilient walrus tests.

Rules for resolving ambiguity

It must be unambiguous how a given instruction should be completed given the context clues on the page.

Context cues must be visible, or specified in the instructions

If your instruction provided requires clicking on drop-downs, hovering over collapsable menus or going to different parts on the website not shown on the present page, you must specify in instructions. We will only execute test instructions based on provided instruction and what exists on present page

Good Instruction

instructions:
  - 'Click top left menu > Select Prime Video in drop-down menu'

• We will pass this part of the test if 'Prime Video' is found under the drop-down menu

Bad Instruction

instructions:
  - 'Select Prime Video'

• We will fail this test if 'Prime Video' is not visible on the present page and requires clicking drop-down menu

Specific vs Generic context instruction

Instructions may be written to reference a specific element, or simply to complete some action

Specific Instruction: Specifying an element to interact with

instructions:
  - 'Click the Continue Button'

• We will pass this test if the page presents a button that contains 'Continue'
• We will fail this test if the page does not present a button that contains 'Continue'. If the page presents a 'Proceed' button, we will still fail this test due to the fact the instruction specifies a 'Continue' button

Generic Instruction: Not specifying specific elements

instructions:
  - 'Continue'

• We will pass test if a button that allows them to continue is presented, such as 'Proceed' or 'Submit'. This instruction iteration would be a great alternative to avoid the failure in the above example
• We will fail test if there is no button that allows them to continue

Exceptions

1. Dismissing modals

1. If a modal pops up, we can click out of modal to dismiss it
2. If the modal does not dismiss after being clicked-out-of, we will accept / take the affirmative action on the modal in order to continue with the test
3. How to prevent false failures: Determine if any modals that pop up in the test environment can be dismissed when clicking out of the modal. If not, specify in test how to continue or dismiss the modal.

2. Filling in Forms

1. If you would only like certain fields filled out, specify which fields

instructions:
  - 'Fill in Your Name and Email'

Will be completed as:

2. If no fields are specified, we will fulfill all presented fields

instructions:
  - 'Fill in fields'

Will be completed as:

3. If auto-fill suggestions load in an input field, we will select the first suggestion after filling in the field

Instructions:
  - 'Add 1 market street, san francisco for the address'

3. Captchas

1. If we get captcha'd during the test, we will complete the captcha and continue with the test

4. Log-in Flows

We will click any variant of sign in or sign up (log in, create account, sign in, etc.) both before and after entering credentials

Conflicting context cues

If there are multiple cues on the page that makes it difficult to understand which action to take, we will fail the test

Provided instructions:

instructions:
  - 'Check that the Toggle is on'

Current page:

We will fail this test because there multiple toggles that affect notifications presented on the page.

If the instructions were re-written as:

instructions:
  - 'Check that the "Subscriptions" Notification toggle is on'

On the current page, there is only one toggled labeled as 'Subscriptions'. By iterating the instruction to have only one interpretation and can be completed on the presented page, we will pass this test

Login methods

• If there are multiple login methods, specify which method
• If email/PW login is specified, the credentials must be specified
• If test requires to login with Google, that must be specified + which Google account

Exceptions

1. If you are instructing us to take an action and there are multiple ways we can take it (ie. different buttons that say the same thing), we will choose one.

How we move through from one instruction to the next

When we complete the instruction as written, we move on to the next instruction

Once we have successfully completed a step within the instructions, we will move on to the next step. We will not make any assumptions or use domain knowledge to guess which assertions to take after the action

Extraneous error messages

Extraneous error messages does not dictate the success of the instruction step if the action itself was successfully completed

Provided instructions:

instructions:
  - 'Click on the What Have You Become? Video'

Current page after completing first step:

Although an error message has appeared, we will still pass this part of the test because we were able to successfully click on the video link

Interacting with specific elements

We will only interact with specified elements within an instruction, then consider the instruction complete

Provided instructions:

instructions:
  - 'Under Accounts & Lists, click Start Here'
    - 'Verify you receive an email verification link'
    - 'Click on the link in the email'
    - 'Verify login is successful'

We will fail this test at instruction 2, as there were no instructions on how to receive an email verification link. As we fail tests, we will send back failure messages with the reason for failure. In this example, we would send an error message such as, "No email received or specified. Did you mean to add instructions on filling out the signup form?"

How we reference previous instructions

During tests, we can assert or act based on data from previous instructions in order to complete the test execution. If we should assert on data during the test that was not entered by us during the test execution, you should note to reference it later in the test.

instructions:
    - 'Register with an alias of winona@walrus.services'
    - 'Go to Users'
    - 'Verify the newly registered email address loads in the list'

Since we registered the email in a previous instruction, we can assert on that information

Noting values to use in a later instruction

If the test requires referencing data or value not inputed during the test, you must note that reference in the instructions

Passing Test

instructions:
    - 'Search for a flight from New York to San Francisco'
    - 'Note the price of the first flight option'
    - 'Search for a flight from Seattle to San Francisco'
    - 'Verify the first flight option is cheaper than the previously searched flight'

We will pass this test because we are able to reference data outputted during the test execution

Failing Test

If the instructions were provided as:

instructions:
    - 'Search for a flight from New York to San Francisco'
    - 'Search for a flight from Seattle to San Francisco'
    - 'Verify the first flight option is cheaper than the previous searched flight'

We will fail this test because there were no instructions to reference/note the price of the first flight during a previous step

How we interpret the use of quotation marks

Instructions can be written out with the use of quotations when certain text must be matched exactly as is (case-insensitive). The quoted text will be referenced as contained within a string on the page. Quotations can be single, double, or backticks.

instructions:
    - 'Click on the "sign-up" button'

While there is a button that allows us to sign-up, we will fail this test because the provided instructions quoted sign-up, meaning the test execution is looking for the exact match.

Alternatively, if the instructions were provided as:

instructions:
    - 'Click on the "create" button'

We will pass this test because the word "create" is contained within the button, even though it is next to another word