Use Selenium and AskUI Together

When you automate browser applications there is no way around an automation tool as Selenium. It is optimized for browser interaction and executes blazingly fast.

However sometimes you need to leave the confines of your browser to interact with another application or your operating system. This is simply not possible with Selenium alone.

AskUI shines in this regard as it is not limited to a browser with its visual selectors and real input commands like keypresses and mouse movement.

In this post you will learn to setup Selenium together with AskUI to define workflows that can leave the browser.

Prerequisites

• AskUI installed and configured on your system (Windows, Linux, macOS)

Install Selenium

The first thing you have to do is to install the selenium-webdriver. And since we are using TypeScript, let's install the type definitions also:

npm install --save-dev selenium-webdriver @types/selenium-webdriver

Configure Selenium

With Selenium ready to go, you have to head over to the file helpers/askui-helper.ts, configure the Selenium Webdriver there. Here is how the the content of the file should look like:

import { UiControlClient, UiController } from 'askui';
import { AskUIAllureStepReporter } from '@askui/askui-reporters';

// Server for controlling the operating system
let uiController: UiController;

// Client is necessary to use the askui API
let aui: UiControlClient;

// 1. Selenium setup
import { Builder, Browser, WebDriver } from 'selenium-webdriver';
let seleniumDriver: WebDriver;

jest.setTimeout(60 * 1000 * 60);

beforeAll(async () => {

  // 2. Selenium setup
  seleniumDriver = await new Builder().forBrowser(Browser.CHROME).build();

  uiController = new UiController({
    /**
      * Select the display you want to run your tests on, display 0 is your main display;
      * ignore if you have only one display
      */
    display: 0,
  });

  await uiController.start();

  aui = await UiControlClient.build({
    credentials: {
      workspaceId: '<your-workspace-id>',
      token: '<your-access-token>',
    },
    reporter: new AskUIAllureStepReporter(),
  });

  await aui.connect();
});

...

afterAll(async () => {
  // 3. Selenium setup
  await seleniumDriver.quit();

  aui.disconnect();

  await uiController.stop();
});

export { aui };

// 4. Selenium setup
export { seleniumDriver };

1. Import the necessary classes to initialize a WebDriver for the Chrome browser
2. Initialize the WebDriver for Chrome browser
3. Make sure the browser gets closed after execution
4. Export the WebDriver for use in a AskUI Workflow

Write a Sample AskUI Workflow

With the WebDriver set up, hop over to the file my-first-askui-test-suite.test.ts and import it there alongside a few more classes you need to write Selenium code. Add this to the start of the file:

import { seleniumDriver } from './helpers/askui-helper';
import { By, Key, until } from 'selenium-webdriver';

Now you can implement a very simple workflow:

1. Maximize the browser window
2. Open the website https://google.com
3. Dismiss the cookie popup by clicking the Reject all button
4. Search for webdriver and wait for the search results to appear
5. Then with AskUI: Click on the button with text Images to get to the image search
6. Then refine your search by clicking on architecture
7. Move your mouse to the image where a Chrome and a Firefox logo in it

describe('jest with askui', () => {
  it('should open askui.com in Chrome with selenium', async () => {

    // Selenium
    await seleniumDriver.manage().window().maximize();
    await seleniumDriver.get('https://www.google.com');
    await seleniumDriver.findElement(By.xpath('//button/div[text()="Reject all"]')).click();
    await seleniumDriver.findElement(By.name('q')).sendKeys('webdriver', Key.RETURN);
    await seleniumDriver.wait(until.titleIs('webdriver - Google Search'), 1000);
    await seleniumDriver.sleep(1000);

    // AskUI
    await aui.mouseLeftClick().exec();
    await aui.click().button().withText('Images').exec();
    await aui.click().text('architecture').exec();
    await aui.moveMouseTo().element().matching('image with a firefox and chrome logo in it').exec()

    await seleniumDriver.sleep(5000);
  });
});

Testrun

AskUI-RunProject

The following gif shows the execution of the whole workflow:

Conclusion

Integrating AskUI with a framework like Selenium comes down to configuring Selenium Webdriver in Jest. If you are solely automating in a web browser Selenium is a solid choice. Once you also want to manipulate a desktop app or interact with your operating system AskUI can take over.

Combining Selenium with AskUI gives you all the options to automate workflows over multiple applications.