Get Started with the AI Element

Are you a heavy user of AskUI's Custom Elements but are not happy with the workflow?

Using external snipping tools and managing all your custom elements can become tedious.

The AskUI AI Element offers a built-in solution within the AskUI Development Environment (ADE) inside the askui-shell.

In this tutorial you will learn how to create AI elements with a custom overlay. You will also learn how to manage and delete AI elements inside the askui-shell.

Prerequisites

• Installed AskUI with the installer in version >= 24.7.1: Windows, macOS, Linux

Activate the AI Element Commands

First active your ADE by running the following the command in a terminal of your choice.

askui-shell

Then activate the AI element commands by activating the Experimental Commands:

AskUI-ImportExperimentalCommands

Create an AI Element

The most basic way to create an AI Element ist to run the following command:

AskUI-NewAIElement

This will bring up an overlay where you can select one or multiple AI elements by drawing a rectangle or polygon around them. For a detailed explanation you can check the Snipping Workflow docs.

You close the overlay by pressing ESC on your keyboard.

Then you will be prompted in the ADE if you want to preview the AI Element and what name it should get. Your output will look something like this.

# Output
Taking screenshots for project:
  WorkspaceId: <workspaceId>
Finished taking of 2 screenshots.
Do you want to preview screenshot #1? [y/n]: y
Screenshot #1 requires additional metadata.
  Enter a name for the screenshot 1: <name1>
Screenshot #1:
  Name:     <name1>
  File:     <askuiUserDirectory>\SnippingTool\AIElement\<workspaceId>\{26342074-5B2A-4695-09ED-41EE47992546}.png
  Metadata: <askuiUserDirectory>\SnippingTool\AIElement\<workspaceId>\{26342074-5B2A-4695-09ED-41EE47992546}.json
Do you want to preview screenshot #2? [y/n]: n
Screenshot #2 requires additional metadata.
  Enter a name for the screenshot 2: <name2>
Screenshot #2:
  Name:     <name2>
  File:     <askuiUserDirectory>\SnippingTool\AIElement\<workspaceId>\{644A00B8-1E27-7ADF-7C68-1751672A5931}.png
  Metadata: <askuiUserDirectory>\SnippingTool\AIElement\<workspaceId>\{644A00B8-1E27-7ADF-7C68-1751672A5931}.json

Here is a video that shows the whole workflow.

Optional Parameters

To speedup and customize the workflow you can provide some optional parameters:

• Name (Optional): The name of the screenshot file. If defined, it indicates that only one element is being snipped. If not defined, you will be prompted to provide a name later. For snipping multiple elements, you must provide a name for each one.
• WorkspaceId (Optional): By default, the command will use the Workspace ID specified in the settings file. If neither is set, an error will be thrown.
• AlwaysPreview (Optional): Automatically opens the preview without prompting. Can not be used together with the NoPreview parameter.
• NoPreview (Optional): Skips the preview and does not open it. *Can not be used with the AlwaysPreview parameter.
• OneShot (Optional): The snipping ends after the first successful creation of an AI Element.

Manage AI Elements

When you are handling only a bunch of AI elements you might remember their names. But once you have to manage dozens of them you want some help. Let's explain which commands are available for you.

Show and Filter a List of All AI Elements

You can get a list of all AI elements by running AskUI-ShowAIElement. If you only want to filter for a specific element, you can use the -NameFilter parameter as shown in the following example:

AskUI-ShowAIElement -NameFilter "hand-icon"

# Output
'1' AI Element was found.
AI Element #1:
  Name:     hand-icon
  NodeJS Code Snippet: aiElement('hand-icon')
  NodeJS Code Example: await aui.click().aiElement('hand-icon').exec()
  Image File:     /Users/joe/.askui/SnippingTool/AIElement/d928f70b-7ff0-4cea-b9f6-c6522a2eb492/{CEFCF578-9EE2-58A0-DFFF-A387FFF7DE30}.png

Display an AI Element

Recognizing an element just by its name, may be tricky 🙃. But we have another command for that: AskUI-OpenAIElement. Again use the parameter NameFilter to specify which element to show.

AskUI-OpenAIElement -NameFilter "hand-icon"

Remove an AI Element

Last but not least, not all AI elements are made to be eternal. Use the command AskUI-RemoveAIElement to remove AI elements you do not need any more.

It accepts the following parameters:

• NameFilter (Mandatory): Specifies a name filter for the AI element to be removed.
• NoPreview (Optional): Skips the preview of the filtered AI elements before removal.
• AutoApproval (Optional): Automatically approves the deletion without prompting for confirmation.

AskUI-RemoveAIElement -NameFilter "hand-icon" -AutoApproval

Conclusion

The AI element makes dealing with custom elements a lot easier and faster. No need to use external snipping tools and workflows any more. Also, creating the AskUI instruction to target the element gets created for you to copy into your AskUI workflow 🥳.