Relational Selectors in AskUI

February 13, 2024

Closely stacked amplifiers, cassettes, old TVs, CDs and records.

In this tutorial, we will explain how all the different relational selectors work in askui, so you get a thorough understanding of them.  

Relational selectors can be difficult and sometimes confusing to work with at first. Sometimes it is even not clear why a specific element will not be selected or why the selector does not seem to work.  

After reading this tutorial you will know how to use the full power of all the relational selectors. Additionally, you learn what pitfalls you can fall into and how to avoid them in the future.

In this article we’ll discuss the following relational selectors:

above()
below()
contains()
in()
leftOf()
rightOf()
nearestTo()

Prerequisites

You have AskUI installed (Windows, Linux, macOS)

You can access the Selectorshub practice page

above()

When you want to click on a textfield and it is above an element, like for example a button with the text Submit. You can do it with above(). The following code snippet moves the mouse to the textfield above the Submit-button:

await aui
  .click()
  .textfield()
  .above()
  .button()
  .withText('Submit')
  .exec();

Moving mouse to textfield above the submit button.

below()

When you want to select a textfield you can do so by finding the correct label, which is often above the textfield. The following code snippet moves the mouse to the textfield below the text Mobile Number:

await aui
  .moveMouseTo()
  .textfield()
  .below()
  .text()
  .withText('Mobile Number')
  .exec();

Moving mouse to textfield below text Mobile Number.

contains()

For selecting an element, that contains another element, contains() is the right candidate. It is especially useful if you want to select a textfield with a placeholder text inside it. The text inside the textfield is annotated as an element itself.

If you have problems with selecting a specific element, always run annotate() to create a screenshot of all the annotations or use annotateInteractively() to see if you need to use contains().

The following snippet moves the mouse to a textfield based on its placeholder text First Crush which is contained in the textfield:

await aui
  .moveMouseTo()
  .textfield()
  .contains()
  .text()
  .withText('First Crush')
  .exec();

Moving mouse to the textield that contains the text First Crush.

in()

When you want to target an element that is inside another element you can use in().

--------------------
|     outerEl      |
|  --------------  |
|  |  innerEl   |  |
|  --------------  |
|                  |
--------------------

The following code snippet moves the mouse pointer to the text of the first textfield AskUI found:

await aui
  .moveMouseTo()
  .text()
  .in()
  .textfield()
  .exec();

leftOf() and rightOf()

If you want to select an element based on its location left or right of another element you have to use leftOf() or rightOf() respectively.

💡 If you do not specify another filter like withText() then you will get the nearest element. Otherwise, askui retrieves the nearest element that matches the filter!

await aui
  .moveMouseTo()
  .text()
  .leftOf()
  .text()
  .withText('Denmark')
  .exec();

Moving mouse to text left of text Denmark.

await aui
  .moveMouseTo()
  .text()
  .rightOf()
  .text()
  .withExactText('Joe Root')
  .exec();

Moving mouse to text right of exact text Joe Root.

nearestTo()

Filtering with the nearestTo() filter will return the element nearest to another element. This is useful when the direction is not clear on where to search. Especially responsive designs are prone to wrap elements into a new line where leftOf() and rightOf() would fail.

await aui
  .moveMouseTo()
  .textfield()
  .nearestTo()
  .text()
  .withTextRegex('User Em*')
  .exec();

Moving mouse to textfield nearest to text with RegEx User Em*.

Additional Considerations About the Distance Metric

The distance is not entirely based on physical distance. If you use an additional filter like withText() it optimizes to find the best match. It also considers special cases, for example, modal dialogs. Therefore the element selected by askui might sometimes be wrong from a user's point of view.

You can use moveMouseTo() like in the following example to see what element a command targets.

await aui
  .moveMouseTo()
  .textfield()
  .above()
  .button()
  .withText('Submit')
  .exec()

Conclusion

Relational selectors are a powerful tool to create robust automation. However, sometimes they are tricky to master.

This tutorial equipped you with the knowledge to use their full power in your automation.

If you have a recurring or persisting issue, don’t hesitate to ask the community for help!

Johannes Dienst