This tutorial demonstrates different methods to handle text elements with askui. askui offers four different methods:
- containsText()
- withExactText()
- withText()
- withTextRegex()
Requirements
For a convenient demonstration, we will use a Flutter web demo provided by Flutter.
Basic Text Matching
The simplest way to interact with a text element is to use withText(). Go to the demo app page and run the code below:
await aui
.moveMouseTo()
.text()
.withText('matrial')
.exec();
await aui
.mouseLeftClick()
.exec();
withText() tries to find a text that matches the whole sequence. In most test cases, you will want to stick to this method, as it supports Fuzzy Matching and tolerates misspelled text. 💡Note that the above example code has two typos. The text matrial doesn't match the text in the demo app, which is Material, although askui will find the text element that roughly matches the text on the screen.
Match a Substring within a Text
Even though the method withText() is handy and quite reliable, you might face a test case where you know only a fraction of the text element that you want to interact with. In such case, containsText() is the method you want to use:
await aui
.moveMouseTo()
.text()
.containsText('Bottom')
.exec();
await aui
.mouseLeftClick()
.exec();
Be aware that even if the method containsText() supports Fuzzy Matching, it won't match the whole sequence by just a few characters. Try to use this code with the given demo app:
// this will fail
await aui
.moveMouseTo()
.text()
.containsText('Bottm')
.exec();You will notice that askui fails to match the given text Bottm, whereas this code will work:
// this will succeed
await aui
.moveMouseTo()
.text()
.containsText('Bottm appbar')
.exec();
// this will also succeed
await aui
.moveMouseTo()
.text()
.containsText('Bottom')
.exec();The biggest difference between withText() and containsText() is whether it matches the text as a whole sequence or not. Matching many text elements with a repeating affix could be a practical use case for the containsText().
It is recommended to experiment enough with these methods to find a better option that suits your specific case since it's not guaranteed that the given text can be fuzzy-matched with target texts.
Match the Exact Text
If you already know what text you are looking for, or if there are too many similar text elements, we could use the method withExactText().
Go to Material->Data tables from the demo app's main page. You will see a table with different foods with nutrition factors for each.
Let's say we want to click on the items with exactly 25.0 gm of Fat. In our demo app, only the Doughnut is the matching item. Run this code and see how withText() matches the text:
// Use this helper function to calculate the centroid of the detected elements.
function getCentroid(element: any): any {
let x = (element.bndbox.xmax - element.bndbox.xmin) / 2.0 + element. bndbox.xmin;
let y = (element.bndbox.ymax - element.bndbox.ymin) / 2.0 + element.bndbox.ymin;
return { x: x, y: y };
}
// Find all the text elements that match '25.0'
const elts = await aui
.get()
.text()
.withText('25.0')
.exec();
// Then, iterate through the found elements and click on them
for(let i=0; i<elts.length; ++i) {
const centroid = getCentroid(elts[i]);
await aui
.moveMouse(Math.round(centroid.x), Math.round(centroid.y))
.exec();
await aui
.mouseLeftClick()
.exec();
}
You will see that AskUI clicks not only the 25.0 but also the 26.0 due to the Fuzzy Matching. The result of this test code may differ in your case, because of the different screen resolutions and the rendered size of the demo app.
It will give you a clear idea of where you will need to use the method withExactText() instead of withText(). Try to run the same code after replacing the withText() with withExactText():
// Find all the text elements that match 25.0' exactly
const elts = await aui
.get()
.text()
.withExactText('25.0')
.exec();
// Then, iterate through the found elements and click on them
for(let i=0; i<elts.length; ++i) {
const centroid = getCentroid(elts[i]);
await aui
.moveMouse(Math.round(centroid.x), Math.round(centroid.y))
.exec();
await aui
.mouseLeftClick()
.exec();
}
Match Text with Regular Expression
The method withTextRegex() supports Regular Expression to match any text most flexibly. Although it might be tricky to use regex due to its esoteric appearance, it is maybe one of the best solutions when it comes to character matching.
On the same page of the demo app, let's say that we want to click on the items whose Calorie is between 300 and 500 (cal>=300 && cal<500). Since regex doesn't support numeric comparison, we will try to match the digits in a sequence:
// Find all the text that matches the expression
const cals = await aui
.get()
.text()
.withTextRegex('[3-4][0-9]{2}')
.exec();
// Then, iterate through the found elements and click on them
for(let i=0; i<cals.length; ++i) {
const coord = getCentroid(cals[i]);
console.log(cals[i].text, coord);
await aui
.moveMouse(Math.round(coord.x), Math.round(coord.y))
.exec();
await aui
.mouseLeftClick()
.exec();
}The regular expression [3-4][0-9]{2} means,
- [3-4]: Match one character between 3 and 4.
- [0-9]: Match one character between 0 and 9.
- {2}: Repeat the previous expression ([0-9]) two times.
As the result, it will try to match every text that has a sequence starting with the digit 3 or 4, and then has any two digits in a row afterward.
Conclusion
When using askui for automated tests, text elements are playing a big role, since they typically appear more distinctively than other elements such as icons or textfields. Hence, knowing the benefits of using different text filters can become critical in scaffolding a robust test suite.
If you got any issues by following this tutorial, or if you want to share your askui use case, you are welcome to join our community!
