Dylan Pierce shared an interesting approach to smoke testing at our meetup: Fill out a contact form with the help of Computer Vision & Large Language Models. Especially the vision part was impressive but it lacked one specific feature: Interaction with the form. How can you be sure your form works if you do not try to fill it out and send it?

This is where I thought an AskUI integration into the Pipedream workflow could be useful. AskUI uses visual selectors instead of code selectors and can interact like a human with a form like that. Giving us a real smoke test!

In this blog, I will describe how I integrated AskUI into a Pipedream workflow to benefit for visual selection and user interaction.

Prerequisites

• AskUI Controller installed and configured on a remote accessible system like Gitpod or a cloud machine. You can use our Try-Out-Repository or install on your own system (Windows, Linux, macOS)

• Pipedream account and a workflow. Follow their Introduction

What Will We Build? Aka the Use Case

Dylan Pierce showed an awesome use case with Pipedream and Puppeteer, where he implemented a smoke test with the help of AI without writing any selectors by himself. I highly recommend to watch the recording:

Our Use Case: Smoke Test with Visual Selectors

Dylans use case involved querying a Large Language/Multimodal Model to implement the smoke test. We will modify this a little bit to use the visual selectors from AskUI, which do not rely on the specific UI-technology but identify elements through their appearance with an AI vision model.

Here are the steps we will implement:

• Trigger the smoke test once per day
• Perform the smoke test with AskUI on a remote system
• Send an email if the smoke test was successful or not

1. Add a Trigger

The first thing Pipedream wants us to add is a trigger. We add a Schedule trigger that will run our Workflow everyday at 9:00 AM.

2. Prepare a Custom Code Action

AskUI does not have an action in Pipedream. So we will make use of the AskUI node package and run a custom code action. For this we a a _Code action. In this action we will fill out the simple authentication from https://authenticationtest.com/simpleFormAuth/. If we see the success page we will send

We have to do the following steps:

• Add a property uiControllerUrl, so we do not have to hardcode it into the code
• Store our credentials in environment variables
• Import UiControlClient from the AskUI node package
• Configure the AskUI UiControlClient to use the credentials and the uiControllerUrl

The first thing we want to add to our custom code is the uiControllerUrl. Click Refresh fields so the Configure tab shows up at the start of the action. Here is the relevant code snippet.

...
export default defineComponent({

  props: {
    uiControllerUrl: { type: "string" }
  },

  async run({ steps, $ }) {
...

Then head over to environment variables and add your workspaceId and accessToken there. You obtained those by following the prerequisites instructions above. You can now setup the connection to the AskUI Controller over the UiControlClient like this. Notice how easy it is to use arbitrary node-packages in Pipedream? I only needed to import UiControlClient and it just worked 🥳.

Note: The following code also contains the teardown.

import { UiControlClient } from 'askui';

...
  async run({ steps, $ }) {
    const result = {};

    const aui = await UiControlClient.build({
      credentials: {
        workspaceId: process.env.workspaceId,
        token: process.env.token,
      },
      uiControllerUrl: this.uiControllerUrl
    });
    await aui.connect();

    // AskUI Workflow will be added here

    aui.disconnect();

    return result;
  },
})

3. Write the AskUI Workflow

When you look at our example form, we want to fill out, you notice that we have to do the following things:

• Write simpleForm@authenticationtest.com into the textfield E-Mail Address
• Write pa$$w0rd into the next textfield
• Click the button Log In
• Validate the success

The less inference we invoke the faster the AskUI workflow will execute. Everything that prompts AskUI to search for an element on the screen invokes inference. So let us try to invoke the inference only once by finding the first textfield for the E-Mail Address. Then we will use keypresses to navigate the form. This is the code to achieve this:

// This line only works with the Gitpod setup shown in the next section
// Select the browser url textfield for your use case with the appropriate instruction!
await aui.typeIn('https://authenticationtest.com/simpleFormAuth/')
         .textfield()
         .contains()
         .text()
         .containsText('docs.askui')
         .exec();
await aui.pressKey('enter').exec();

// Fill out the form
await aui.typeIn('simpleForm@authenticationtest.com')
         .textfield()
         .contains()
         .text('E-Mail Address')
         .exec();
await aui.pressKey('tab').exec();
await aui.type('pa$$w0rd').exec();
await aui.pressKey('tab').exec();
await aui.pressKey('enter').exec();

// Check if the the login succeeded: Login Success is shown on the page
// Pass result to next step
try {
  await aui.expect().text('Login Success').exists().exec();
  result.success = "Smoke Test successful!";
} catch(error) {
  result.success = "Smoke Test failed!";
}

aui.disconnect();

return result;

Send an Email

Doing a smoke test without reporting about its success state would not help us. So we will just send us an email with the Send Yourself an Email action.

As subject we choose Smoke Test State and for the text we reference our success variable we returned in our action above with {{steps.code.$return_value.success}}.

Gitpod As Remote Machine

If you do not have a remote machine ready-to-go you can use a service like Gitpod. We have a prepared Try-Out-Repository which comes with AskUI already setup and a VNC to observe the AskUI workflow. Start the repository in Gitpod over the Open in Gitpod-button and let it finish the predefined AskUI workflow. When it reached the AskUI Docs (docs.askui.com) maximize the browser window in the Simple Browser tab.

Switch to the TERMINAL tab and start the AskUI-Controller with the following command:

./node_modules/askui/dist/release/latest/linux/askui-ui-controller.AppImage

Also make sure that you expose the port to the AskUI Controller (open lock icon). Head to the PORTS tab and make the port 6769 public. Then copy the URL and add it as the property uiControllerUrl in the Pipedream action.

Conclusion

Building a smoke test with Pipedream and AskUI was a practical use case to see how both tools integrate. The simplicity of Pipedream and its ability to integrate JavaScript code and Node packages was helpful. With that AskUI could be setup seamlessly inside an action and connected to an external AskUI Controller.