Relational Selectors: Intersection Area

Relational selectors like above() or leftOf() help you to detect elements related to another element.

But there are some edge cases like textfields with not enough space between them that lead to a lot of frustration.

AskUI now supports an optional parameter intersection_area for the relational selectors above(), below(), rightOf() and leftOf() that gives you control over HOW elements are detected.

This blog post explains the different options with examples and illustrations.

Prerequisites

• AskUI installed in version >= 0.20.2

The Example: Untargetable Elements

Let's look at an example that illustrates the different use cases:

Up until now, relational selectors used the area in the direction of the element which was determined by the edges to find elements. You can see an example for the instruction text().rightOf().button() in the next picture, where Text0 and Text1 will be detected:

This seemed like the behavior that a user would expect from a relational selector. However, you could experience the following challenges:

• How do I target Text0 only when the bounding box of Text1 overlaps with the bounding box of Text0?
• How do I even target Text2 and Text3? Hint: You could not!
• How do I target Text0 only in this example?

The Solution: Parameter intersection_area

This is why we implemented an optional parameter intersection_area that gives you more control over the area elements will be related to.

It can take the following values:

• element_center_line - considered above/below/left/right of the other element if element's bounding box intersects with a horizontal line passing through the center of the other element
• element_edge_area - considered above/below/left/right of the other element if element's bounding box intersects with an area between the top and the bottom edge of the other element
• display_edge_area - considered above/below/left/right of the other element no matter where it is placed vertically on the screen (y-axis)

You have to specify the area after the index parameter. Please find a few usage examples below and also check the API docs:

// Select first text in the edge area above button
...text().leftOf(0, 'element_edge_area').button()

// Select the third text in the display edge area
...text().leftOf(2, 'display_edge_area').button()

// Select the second text in the center line from button
...text().leftOf(1, 'element_center_line').button()

Intersection: Horizontal Line

If you set intersection_area to element_center_line the elements that are touched by the line starting from the center of the element will be detected. In the following image, Text1 will not be detected because of this.

This is useful in edge cases where Text1 is unintentionally detected because the bounding box of Text1 overlaps with the edge area of the button.

For example, adjacent textfields that do not have enough spacing between them are usually prone to this problem.

Intersection: Area

This default value element_edge_area was used until the update and will be used if you do not specify an intersection_area. It is a sensible default that is the correct behavior in most cases.

Intersection: Display

If you set the intersection_area to display_edge_area everything, and everything means everything in the area spanned vertically and horizontally, on the screen in the direction of the relational selector will be detected.

Note: You have to experiment with the index as the order of the detected elements may not be obvious or logical.

Conclusion

This update to relational selectors was a needed one. It gets rid of some unsolvable edge cases like overlapping bounding boxes. It also provides fine-grained control over selecting elements relationally.

If you have questions or need support, do not hesitate to join our Community.