Awesome

TagUI

Why This | Set Up | To Use | Cheat Sheet | Developers | Credits | Video

TagUI is a command-line tool for digital process automation (RPA)

• automate Chrome in visible / invisible mode
• visual automation of websites and desktop
• write in 20+ human languages & JavaScript
• Chrome extension for recording web actions
• Python & R integrations for machine learning

Why This

The goal of UI (user interface) automation is to reproduce cognitive interactions that you have with websites or your desktop, so that your computer can do it for you, base on your schedule or conditions. TagUI helps you rapidly automate repetitive or time-critical tasks - use cases include digital process automation, data acquisition and testing apps.

Read on for more info or jump right to the flow samples section to see examples of TagUI automation in natural-language-like syntax, which makes automation easy to maintain. This is a full-feature and free open-source tool, so there's nothing to upgrade to or any paid subscription. To feedback suggestions or bugs, raise an issue on GitHub.

Key Strengths

• cross-platform, works on Windows, macOS, Linux
• $0 to use, under Apache 2.0 open-source license
• easy to use, thus rapid development & deployment
• easy for IT policies to deploy, simply unzip and run
• modifiable source code for tailor-made solutions

Neutral Differences

• scripts written in 21 human languages, not flowchart
• JavaScript for advanced scripting instead of C# / VB
• visual and OCR based automation for desktop apps

Key Weaknesses

• lack of enterprise audit, control, dashboard, reporting
• lack of SLA or 24/7 support team for incident-handling
• lack of large development teams (30 to 100+ people)
• lack of user / developer base grown over many years
• lack of consultancies / partners distribution network

HOW IT WORKS

TagUI converts your intentions from over 20 human languages to lines of working JavaScript code that perform UI automation. Under the hood, it uses Chrome DevTools Protocol, SikuliX, CasperJS, PhantomJS & SlimerJS.

// sample TagUI flow to log into Typeform and download a survey report
https://www.typeform.com
click login
type username as user@gmail.com
type password as 12345678
click btnlogin
download https://admin.typeform.com/xxx to report.csv

// besides web element identifiers, images of the elements can be used
click login_button.png
type username_box.png as user@gmail.com

// in addition (x,y) coordinates of user-interface elements can be used
click (1200,200)
type (800,400) as user@gmail.com

• TagUI will instantly convert the automation flow above into JavaScript code and automatically perform the steps to download a report. Conditions can also be written in natural language for making decisions or checking the webpage. This makes it easy for rapid prototyping, deployment and maintenance of UI automation, whether you are a developer or not. The language engine supports over 20 languages and can be modified or extended easily by users.

• There is automatic waiting for UI elements to appear + error-checking + nesting of JavaScript code blocks. TagUI also supports visual automation of website and desktop through built-in integration with SikuliX. Instead of using element identifiers, images can be used to identify user interface elements to interact with. TagUI has native integrations with Python and R for big data and machine learning capabilities.

• The automation flow can be triggered from scheduling, command line (in natural language), API URL, email etc. Everything happens headlessly in the background without seeing any web browser, so that you can continue using the computer uninterrupted. Running on a visible web browser is also supported, using Chrome or Firefox.

• If you know JavaScript and want to be more expressive, you can even use JavaScript directly in the flow. If not, you will still enjoy friendly but powerful features such as repositories to store your reusable objects, flexible datatables for batch automation, and a Chrome extension which creates automation flows by recording your actions. For rapid prototyping, there's an interactive live mode for trying out TagUI steps or JavaScript code in real-time.

• REST API or OS command calls can be made with a single line. For auditing and personal tracking, there is a report option which can track all your run results in a csv spreadsheet, complete with execution logs in html format. Not forgetting the option to run automation flows hosted online or auto-upload run results online for sharing.

Set Up

TagUI is in v5.11 - it unzips and runs directly on Windows, macOS, Linux (link to release notes)

PACKAGED INSTALLATION

• TagUI is easy to use right away, in most environments all dependencies are already packaged in
• To use visual automation on desktop or browser, be sure to have OpenJDK v8 (64-bit) or later

Recommended locations to unzip to

• Windows - c:\ or desktop
• macOS - your desktop
• Linux - /home/your_userid

Avoid spaces in the folder path as some components of TagUI don't work well with spaces in folder and file names. Optionally, configure web browser settings in tagui_config.txt, such as browser resolution, step timeout of 10s etc.

Tip - to try cutting edge version with the latest features, download master.zip to overwrite your existing packaged installation. Be sure to manually select and move the folders & files inside master.zip's TagUI-master/src folder to replace your existing tagui/src folder, some OSes will delete existing target folders that are missing from source folder.

MANUAL INSTALLATION

1. PhantomJS (headless scriptable web browser) - http://phantomjs.org
2. CasperJS (navigation/testing for PhantomJS) - http://casperjs.org
3. SlimerJS (scriptable web browser for Firefox) - https://slimerjs.org
4. TagUI (general purpose UI automation tool) - https://git.io/vMCTZ
5. SikuliX (required for visual automation features) - http://sikulix.com
6. PHP (only required for manual Windows setup) - http://windows.php.net

Tip - take note to put TagUI to a folder path without spaces (some dependencies have issue with that). for manual Windows setup, 1. set SLIMERJS_EXECUTABLE env variable to point to slimerjs.bat, 2. put GNU utilities (cut / gawk / grep / head / sort / tail / tee), curl ssl in tagui\src\unx, 3. add phantomjs\bin, casperjs\bin, php folders to path

To Use

COMMAND LINE

tagui flow_filename option(s) for Windows and ./tagui flow_filename option(s) for macOS/Linux

• Flow filename can be a local file or the URL of an online file
• Filename can have no extension, .txt or .js or .tagui extension
• Type tagui without parameters to see its version and options

By default, log files (.log .raw .js) are created after running. To switch off, put an empty file tagui_no_logging in tagui/src folder. If your command prompt or terminal font size is too small, you can set it to much larger font sizes for easier reading. Below example runs a script to search on Yahoo website and capture screenshots of the results.

Windows - unzip the tagui folder to c:\. Open command prompt with Start Menu -> type run -> type cmd and enter

c:
cd c:\tagui\src
tagui samples\1_yahoo chrome

macOS - unzip the tagui folder to your desktop. Open terminal from Apps -> Utilities -> Terminal and enter commands

cd /Users/your_userid/Desktop/tagui/src
./tagui samples/1_yahoo chrome

Linux - unzip the tagui folder to a convenient folder on your laptop for eg /home/your_userid and enter commands

cd /home/your_userid/tagui/src
./tagui samples/1_yahoo chrome

If the script works successfully, you will notice 5 .png files - congratulations, you have run your first TagUI script!

Troubleshoot OS-specific exceptions

• For Windows computers, if you see 'MSVCR110.dll is missing' error, install this from Microsoft website (choose vcredist_x86.exe) - this file is required to run the Windows PHP engine packaged with TagUI. Some IT policies restrict TagUI from writing to c:\tagui and working properly, in that case please unzip to user desktop folder.
• For newer macOS versions, if you get a 'dyld: Library not loaded' error, install OpenSSL in this way. macOS Catalina update has introduced tighter security controls, see solutions for the PhantomJS and Java popups.
• For some flavours of Linux (Ubuntu for example) which do not have PHP pre-installed, google how to install PHP accordingly (eg Ubuntu, apt-get install php). Most Linux distributions would already come with PHP.
• If you Ctrl+C to break a TagUI automation, you can use tagui/src/end_processes command (for macOS/Linux) or end_processes.cmd (for Windows) to kill any dead processes of TagUI integrations (Chrome, SikuliX, Python etc)

Troubleshoot browser-specific exceptions

• Google Chrome browser has to be installed by user before trying to automate Chrome browser. TagUI creates a separate Chrome browser user profile for use in automation, it is stored in tagui\src\chrome\tagui_user_profile.
• Make sure TagUI's Chrome browser is set to 100% zoom. As TagUI mimics the user mouse-clicks at the (x,y) coordinates of web elements, using a different zoom level will cause clicks to be triggered at wrong locations.
• For Microsoft Edge or Internet Explorer, TagUI can only automate them using visual automation, keyboard and mouse automation. There is no backend integration to automate through web element identifiers like Chrome or Firefox.
• For Firefox automation, download an older version here (SlimerJS doesn't work with Firefox v60 onwards).

More details on TagUI command-line usage

After each automation run, a .log file will be created to store output of the execution, a .js file is the generated JavaScript file, a .raw is the expanded flow after reading in any module sub-scripts that are called in that flow.

These files are created by default for user reference purpose and can be helpful in debugging or troubleshooting the automation flow. To switch off creation of these files, put an empty file tagui_no_logging in tagui/src folder.

To make tagui command accessible to run from any directory

• For MacOS/Linux, use ln -sf /full_path/tagui/src/tagui /usr/local/bin/tagui to create symbolic link
• For Windows, add your tagui/src folder to path, then tagui will be accessible from any folder

To do that on Windows, create a .cmd or .bat file with contents like the following, which goes to the directory where you want to run the automation, and run tagui command on the file with your specified options. Double-clicking the .cmd or .bat file will start automation. You can also associate .tagui files directly to be opened by tagui\src\tagui.cmd command.

@echo off
c:
cd c:\folder
tagui filename quiet speed chrome

To do that on macOS / Linux, create a file with contents like the following, which goes to the directory where you want to run the automation, and run tagui command on the file with your specified options. You will need to use the command chmod 700 on the file to give it execute permissions, so that it can be run by double-clicking on it.

cd /Users/username/folder
tagui filename quiet speed chrome

BY SCHEDULING

To schedule an automation flow with crontab (for macOS/Linux), for example at 8am daily

0 8 * * * /full_path/tagui/src/tagui flow_filename option(s)

For Windows, use Task Scheduler (search schedule from Start Menu) or something like Z-Cron

CHROME EXTENSION

Download from Chrome Web Store to use TagUI Chrome web browser extension for recording automation flows. TagUI Chrome extension records steps such as page navigation, clicking web elements and entering information.

After installing the Chrome extenstion, to start recording automation flows

1. Go to the website URL you want to start the automation at
2. Click the TagUI icon, followed by Start button
3. Carry out the steps you want to automate
4. Click the TagUI icon, followed by Stop button
5. Click Export button to view generated TagUI script

While recording the steps, you can right-click to bring up a menu for steps such as showing the element identifier. The recording isn't foolproof (for example, the underlying recording engine cannot capture frames, popup windows or tab key input). It's meant to simplify flow creation with some edits, instead of typing everything manually. See this video for an example of recording a sequence of steps, editing for adjustments and playing back the automation.

NATIVE LANGUAGES

To run TagUI flows in human languages or output flow execution in other languages (see demo run)

1. set your default flow language with tagui_language variable in tagui_config.txt
2. write automation flow in human language base on language definition .csv files
3. optionally set tagui_language in flow to any other languages as output language

Tip - as Windows Unicode support isn't as straightforward as macOS/Linux, doing this in Windows may require changing system locale, using chcp command, and selecting a font to display native language correctly (more info)

TagUI language engine supports over 20 languages and can be modified or extended easily by users to improve accuracy or add more languages. The languages are Bengali, Chinese, English, French, German, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Polish, Portuguese, Romanian, Russian, Serbian, Spanish, Tagalog, Tamil, Thai, Vietnamese. This starting set is partly chosen base on the list of most commonly used languages, partly from the countries around where I'm from (Singapore), and partly from countries with a lot of developers.

If your native language isn't in the above list, you can also automate building a new human language definition by using this language build automation flow (src/languages/build) that self-builds the vocabulary set using Google Translate. To do that, update build.csv with the languages that you want to build and run tagui build using chrome in src/languages folder. Use quiet option to hide the verbose automation output. The generated files are named as their 2-character language codes to prevent overwriting existing language definitions by accident. To use the generated .csv files, rename them to their full language names. See full list of languages possible to be generated by Google Translate.

Most of the language definitions are automatically self-built using Google Translate (except english.csv, chinese.csv, japanese.csv, german.csv, french.csv which have been vetted). Native language users can update the language definition csv themselves and are invited to submit PRs with correct words to be used. Some languages are very different from English structure (for eg written from right to left) and would be impossible to use correctly in TagUI.

VISUAL AUTOMATION

TagUI has built-in integration with SikuliX (base on OpenCV) to allow visually identifying the web elements and desktop UI (user interface) elements for interaction by providing their images or (x,y) coordinates. Applicable steps are click, hover, type, select, read, show, save, snap, keyboard, mouse. To use visual automation, OpenJDK v8 (64-bit) or later is needed.

1. Check that OpenJDK (64-bit) is installed (entering java -version returns your Java version)
2. After Java is installed, you will have to restart your command prompt or terminal to use it
3. On Windows, set display magnification to recommended %, if it doesn't work then 100%
4. On Windows, if TagUI just hangs there, see if it's due to this issue and try the solution
5. On macOS, if can't find image on screen, may be due to how the image was captured
6. On Linux, requires installing and setting up dependencies by following this guide

Tip - the first time visual automation is run, the SikuliX engine will need to initialise Jython. Run again to use.

To use visual automation, simply specify an image (in .png or .bmp format) to visually look for in place of the element identifier. Relative paths are supported for image filenames (eg pc.png, images/button.bmp). Note that the element that corresponds to the image must be visible on the screen for visual automation to succeed. If it's blocked by another window for example, the automation will be unable to find the element.

Alternatively, you can specify the (x,y) coordinates of the element that you want to interact with.

To type onto the screen instead of a particular element, use keyboard text or keyboard [modifiers]text (examples). To do a snapshot or an OCR of the whole screen, use page.png or page.bmp as the element identifier for steps snap / read. The usual helper functions visible() / present() can also be used to check whether an image is visible on the screen.

The keyboard and mouse steps, as well as helper functions mouse_xy(), mouse_x(), mouse_y(), can be used to do complex UI interactions. A screen (real or Xvfb) is needed for visual automation. Tesseract OCR (optical character recognition) is used for visually retrieving text. Also, by using vision step, you can send custom SikuliX commands to do things that are not covered by TagUI.

Transparency (0% opacity) is supported in .png images, for eg using an image of an UI element with transparent background to enable clicking on an UI element that appears on different backgrounds on different occasions.

Another example is an image of the window or frame (PDF viewer, MS Word, textbox etc) with the center content of the image set as transparent. This allows using read, show, save, snap steps to perform OCR and save snapshots for application windows, containers, frames, textboxes with varying content. Also for these steps, (x1,y1)-(x2,y2) can be used as the identifier to define the region of interest on the screen to perform OCR or capture snapshot.

PYTHON INTEGRATION

TagUI has built-in integration with Python (works out of the box for both v2 & v3) - a programming language with many popular frameworks for big data and machine learning. The py step can be used to run commands in Python and retrieve the output of those commands. To use Python integration in TagUI, first download Python for your OS. macOS and Linux normally come pre-installed with Python. Make sure that python command is accessible from command prompt.

In your automation flow, use the py step followed by the Python commands to be executed, separated by ;. You can then use the print command in Python to output the result you want to be accessible in your automation flow as py_result variable. If the result is JSON data, the JSON object py_json will be created for easy access to JSON data elements. If not, py_json will be null.

// using py step to denote Python code, and getting back output from py_result
py a=1;b=2
py c=a+b
py print(c)
echo py_result

// alternatively, you can use py begin and py finish to denote a Python code block
// indentation of Python code is also supported, for example in conditions or loops
py begin
a=1;b=2
c=a+b
print(c)
py finish
echo py_result

// an example of passing dynamically generated variables to Python integration
phone = 1234567
name = 'donald duck'
py_step('phone = ' + phone)
py_step('name = "' + name + '"')

py print(name)
echo py_result
py print(phone)
echo py_result

Tip - you can also use the execfile() command in Python v2.X to run Python scripts. Or use exec(open('filename').read()) in Python v3.X to run Python scripts

R INTEGRATION

TagUI has built-in integration with R - an open-source software environment for statistical computing and graphics. R can be used for big data and machine learning. The r step can be used to run commands in R and retrieve the output of those commands. To use R integration in TagUI, first download R software for your OS. Make sure that Rscript command is accessible from your command prompt (added to path or symbolically linked).

In your automation flow, use the r step followed by the R commands to be executed, separated by ;. You can then use the cat() command in R to output the result you want to be accessible in your automation flow as r_result variable. If the result is JSON data, the JSON object r_json will be created for easy access to JSON data elements. If not, r_json will be null.

// using r step to denote R code, and getting back output from r_result
r a=1;b=2
r c=a+b
r cat(c)
echo r_result

// alternatively, you can use r begin and r finish to denote a R code block
r begin
a=1;b=2
c=a+b
cat(c)
r finish
echo r_result

// an example of passing dynamically generated variables to R integration
phone = 1234567
name = 'donald duck'
r_step('phone = ' + phone)
r_step('name = "' + name + '"')

r cat(name)
echo r_result
r cat(phone)
echo r_result

Tip - you can also use the source() command in R to run R scripts

CLI ASSISTANT

TagUI scripts are already in natural-language-like syntax to convert to JavaScript code. What's even better is having natural-language-like syntax on the command line. Instead of typing tagui download_bank_report june creditcard to run the automation flow download_bank_report with parameters june creditcard, you can type erina download my june creditcard bank report. For a demo of the CLI (command-line interface) assistant in action, see this video.

The commands erina (macOS/Linux) and erina.cmd (Windows) can be renamed to any other names you like. The commands can be set up in the same way as the tagui / tagui.cmd above to be accessible from any folder. The command basically interprets this general syntax erina single-word-action fillers options/parameters fillers single-or-multi-word-context to call run the corresponding automation flow action_context with options/parameters.

Also, adding using chrome / using headless / using firefox at the end will let it run using the respective browsers. The default location where automation flows are searched for is in tagui/flows folder and can be set in tagui_helper.php. Filler words (is, are, was, were, my, me) are ignored as they don't convey useful info. More details here.

TAGUI WRITER, SCREENSHOTER & EDITOR

TagUI Writer is a Windows app created by Arnaud Degardin / @adegard which makes it easy to write TagUI scripts. By pressing Ctrl + Left-click, a popup menu will appear with the list of TagUI steps for you to paste into your text editor. Arnaud also created a ScreenShoter app which makes it easy to capture snapshots for TagUI visual automation. Lastly, TagUI Editor allows you to edit and run TagUI scripts via AutoHotKey. To download, click here.

TAGUI FOR PYTHON

TagUI for Python is a Python package created by TagUI's creator Ken Soh / @kensoh. It brings the digital process automation capabilities of TagUI directly to the Python environment through a simple, expressive and powerful API. To install, pip install tagui. For its Python API and architecture, refer to the project homepage. At merely ~1k lines of code, the package is built on TagUI and its architecture is based on integration with TagUI's live mode.

FLOW SAMPLES

TagUI includes the following automation flow samples (tagui/src/samples folder)

Cheat Sheet

AUTOMATION WORKFLOW

• What happens behind the scenes when you run an automation flow (TagUI architecture diagram)

FIND XPATH OF WEB ELEMENT

• In Chrome browser, right-click on the element, click Inspect, right-click on HTML code block, then

STEPS DESCRIPTION

• TagUI will auto-wait for an UI element to appear and interacts with it as soon as it appears
• Element identifier can be auto-recorded using TagUI Chrome extension, or found from web browser
• TagUI auto-selects provided identifier in this order - xpath, css, id, name, class, title, aria-label, text(), href

Tip - to use variables where text is expected, `variable` can be used. XPath is an expressive way to identify web elements. If you know xpath and use xpath for element identifier, use double quotes for text //*[@title="Login"]

For headless and visible Chrome, file downloads can be done using normal webpage interaction or by specifying the URL as a navigation flow step. For Firefox and PhantomJS, the download and receive step can be used. Note that on Windows, snap step requires display magnification to be set at 100% to work properly.

As TagUI default execution context is local, to run javascript on webpage dom (eg document.querySelector) use dom step. Set dom_json variable to pass a variable for use in dom step. Or dom_json = {tmp_number: phone, tmp_text: name} to pass multiple variables for use in dom step (dom_json.tmp_number and dom_json.tmp_text).

CONDITIONS EXAMPLES

• Write text in quotation marks (either " or ' works) to differentiate text from variable names
• { and } block is required after for loop (while is deprecated - no auto-wait, hangs CasperJS)
• Use for loop to repeat blocks of steps (nested loops, conditions, break, continue supported)

Use { and } step to define step/code blocks for powerful repetitive automation with for loop. The usual break and continue commands can be used on next line after an if condition, to break out of the immediate loop or continue to the next iteration. To loop 'indefinitely' use for n from 1 to infinity, where infinity is a pre-defined variable at 1024.

When using contain / equal, you can write with or without s behind. You can use if present('element') to check if the element exists, before doing the step on next line. Other useful functions include visible('element'), count('element'), url(), title(), text(), timer(), which can be used in conditions and steps such as check or echo.

HELPER FUNCTIONS

• Below are helper functions which can be used in your steps or code like a variable
• You can define your own JS functions in tagui_local.js / tagui_global.js (see eg)
• TagUI will look for a local function file tagui_local.js in the same folder of flow
• TagUI will also look for a global function file tagui_global.js in tagui/src folder
• Local definitions take precedence over global definitions (same for repositories)

csv_row() is useful for organising lots of data gathered during automation to be written into csv output file -

// example of using csv_row() function to save data as CSV file
read name_element to name
read price_element to price
read details_element to details
item_info = [name, price, details]
write csv_row(item_info) to product_list.csv

clipboard() is useful for accessing text on the clipboard or putting text there (for eg to paste text quickly) -

// example of using clipboard() to retrieve text contents from PDF file
dclick pdf_document.png
wait 3 seconds
keyboard [ctrl]a
keyboard [ctrl]c
text_contents = clipboard()

// example of using clipboard('text') to type text quickly by pasting it
text_to_type = 'long string of text to be entered into UI element'
click my_life_story_textbox
js clipboard(text_to_type) or clipboard(text_to_type);
keyboard [ctrl]v

mouse_xy(), mouse_x(), mouse_y() require visual automation and can be used to interact with UI (user-interface) elements on the screen by specifying their (x,y) coordinates, for eg below clicking to the right of an UI element by 200 pixels -

// example of using mouse_xy(), mouse_x(), mouse_y() functions
hover element.png
echo mouse_xy()
x = mouse_x() + 200
y = mouse_y()
click (`x`,`y`)

MODULES

• You can reuse and combine automation scripts to automate complex flows
• Sub-scripts can be multiple levels deep and be of any filename or extension
• To call sub-scripts, use tagui step followed by absolute or relative filename
• For example, tagui login_crm or tagui crm.login or tagui outlook\sendmail
• Variables can be directly used or modified in parent script and sub-scripts

The tagui step works by expanding content of a sub-script into the parent script, at the line where tagui step is used to call the sub-script. Thus variables that are accessible from the parent flow file will also be accessible from the sub-script. A .raw file will be created to store expanded contents of the automation flow (useful for checking error messages).

REPOSITORIES

• Repository csv files have 2 columns, for example below (headers up to you to name)
• TagUI will look for a local repository file tagui_local.csv in the same folder of flow
• TagUI will also look for a global repository file tagui_global.csv in tagui/src folder
• Using `object` in your flow replaces it with its definition (which can contain objects)
• For example, `type email` becomes type user-email-textbox as user@gmail.com

DATATABLES

• When running TagUI, specify the csv file to use, eg tagui flow_filename trade_data.csv
• TagUI loops through each row to run automation using the data from different rows
• Using `column_name` in your flow replaces it with the value for that iteration
• Eg, echo 'TESTCASE - `testname`' in your flow shows TESTCASE - Trade USDSGD
• echo '`[iteration]`' can be used in your flow to show the current iteration number

Tip - use speed option to skip the 3-second delay between iterations (and skip restarting of Chrome browser)

AUDIT & REPORTING

• when report option is used, TagUI will track run results to tagui/src/tagui_report.csv
• this can be useful for personal tracking, auditing purpose, or measuring time savings
• html logs will also be generated to store the full execution sequences and outcomes

Developers Reference

TagUI excels in automating user-interface interactions. It's designed to make prototyping, deployment and maintenance of UI automation easier by minimizing iteration time for each phase. Originally developed by a frustrated test automation engineer, to maximize automation code output with minimal coding efforts when automating web interactions.

API

Automation flows can also be triggered via API URL. TagUI has an API service and runner for managing a queue of incoming requests via API (for macOS/Linux). To set up, add a crontab entry on your server with the desired frequency to check and process incoming service requests. Below will check every 15 minutes and run pending flows in the queue. If there's an automation in progress, TagUI will wait for the next check instead of concurrently starting a new run.

0,15,30,45 * * * * /full_path_on_your_server/tagui_crontab

To call an automation flow from your application or web browser, use below API syntax. Custom input(s) supported. The default folder to look for flow_filename would be in tagui/src folder. Automation flows can also be triggered from emails using the API. For email integration, check out Tmail. It's an open-source mailbot to act on incoming emails or perform mass emailing; it also delivers emails by API. Emails with run-time variables can be sent directly from your flow with a single line (see flow sample 6C_datatables). For sending SMS, you can use api step to send through SMS-sending services, or host your own from your webserver using services like Skebby (see this example from @adegard). If you have data transformation in your process pipeline check out TLE, which can help with converting data.

your_website_url/tagui_service.php?SETTINGS="flow_filename option(s)"

Besides integrating with web applications, TagUI can be extended to integrate with hardware (eg Arduino or Raspberry Pi) for physical world interactions or machine learning service providers for AI decision-making ability. Input parameters can be sent to an automation flow to be used as variables p1 to p9. Output parameters from an automation flow can be sent to your Arduino or application API URL (see samples 3_github and 6C_datatables).

For making outgoing API calls in your automation flow, to feed data somewhere or send emails etc, use the api step followed by full URL (including parameters) of the API call. Raw response from API will be saved in api_result variable. If the API response is JSON data, the variable api_json will be created for easy access to JSON data elements. If not, api_json will be null. For example, api_json.parent_element.child_element retrieves value of child_element.

api_config = {method:'PUT', header:['Header1: value1','Header2: value2'], body:{'id':123,'pwd':'abc'}};

For advanced API calls, you can set above api_config variable which defaults as {method:'GET', header:[], body:{}}. Besides GET, you can use other methods such as POST, PUT, DELETE etc. You can define multiple headers in the format 'Header_name: header_value' and provide a payload body for PUT requests for example. You can set it as above before using the api step, or set using api_config.method = 'PUT'; and api_config.header[0] = 'Header1: value1'; etc. api_config.body will be automatically converted to JSON format for sending to API endpoint.

In addition, macOS/Linux/Windows commands can be executed using run step. This can be used to call other apps or services hosted locally on the OS, instead of being hosted through webservices. To use run step, specify the full command including parameters. Execution result will be stored in run_result, which can be used in your flow.

CHROME

TagUI has built-in integration with Chrome web browser to run web automation in visible or headless mode. It uses a websocket connection to directly communicate automation JavaScript code and information to Chrome.

To develop new custom methods for Chrome integration, see this TagUI issue and tagui_header.js for examples of websocket calls from TagUI to Chrome (via Chrome DevTools Protocol). The function chrome_step(method, params) sends message to Chrome and returns the response. You will see examples from simple websocket calls such as getting webpage title to stacked ones such as handling of frame or popup window. To tweak how TagUI launches / kills Chrome and the integration PHP process, see TagUI runner script for macOS/Linux or Windows.

Probably the best way to see the websocket communication in action is to enter TagUI live mode (add live step in your automation flow), then tail -f tagui_chrome.log in another terminal to see the Chrome DevTools Protocol messages going to and fro as you enter TagUI steps or JavaScript code. If you are running on Windows, you can click on the PHP process window directly to see the messages.

At run-time TagUI will start a PHP thread in the background to manage the integration with Chrome for concurrent communication. The Textalk PHP websocket is used as it's super-light and most importantly, it works even without any update for 2 years. The normal approach to integrate with Chrome is through chrome-remote-interface project or tools such as Puppeteer and friends. However, that approach introduces Node.js dependency which means users without a Node.js development environment cannot run TagUI with Chrome.

In order to retain TagUI unzip and run functionality, the approach of launching a separate PHP thread is chosen. Since the TagUI natural language interpreter is already written in PHP, there is no new dependency. Also, doing websocket communication within the single-threaded JavaScript environment used by CasperJS is not possible as it involves a redesign of fundamental CasperJS methods such as casper.exists to support async/await or use JavaScript promises which are not yet supported by CasperJS (for compatibility with latest PhantomJS).

Like chrome-remote-interface, TagUI communicates with Chrome through Chrome DevTools Protocol. The protocol is primarily designed for debugging the web browser instead of web automation, so many methods are still in experimental status. However, the API is stable enough for TagUI steps to work with Chrome. When chrome or headless option is used, TagUI replaces CasperJS methods it uses with custom methods to talk to Chrome instead of PhantomJS. When firefox option is used or by default, TagUI doesn't invoke custom methods and PHP process.

TESTING

The check step allows easy testing of conditions. When the test option is used, test assertions are automatically performed for check steps. This lets CasperJS framework output a XUnit XML file, which can be used for CI/CD integration with tools such as Jenkins. Below are examples of check steps, more examples in sample flow 7_testing.

// check whether the element search-buttons exists
check present('search-buttons') | 'search button exists' | 'search button does not exist'

// check whether the search button text is correct
read search-buttons to button
check button equals to 'Search' | 'search button text is correct - ' button | 'search button text is wrong - ' button

// check that the page has the text My Portfolio
check text() contains 'My Portfolio' | 'page text contains My Portfolio' | 'page text does not contain My Portfolio'

// check that number of header menu items are more than or equals to 6
check count('uh-tb-') more than or equals to 6 | 'header menu items >= 6' | 'header menu items < 6'

CasperJS test scripts are inherently different in structure and syntax from its usual automation scripts. With the test option, TagUI automatically sets up your automation flow to work as a test script. TagUI allows you to reuse the same flow for testing or automation by running it with or without the test option. More info about CasperJS testing framework.

Note that CasperJS is not yet supporting Chrome, below additional examples using CasperJS built-in test assertions won't work when chrome or headless option is used. TagUI smart selector can still be accessed using tx('selector'). As TagUI allows you to write JavaScript / CasperJS code directly within the automation flow, advanced testing or coding techniques that can be implemented in CasperJS should work directly within your flow.

test.assertTextExists('ABOUT','Check for ABOUT text');
test.assertSelectorHasText(tx('header'), 'Interface automation','Check for phrase in header element');

In the event you get an error saying that it cannot understand the step for your JavaScript line, raise an issue or modify the source code (tagui_parse.php is where interpretation of natural language to CasperJS JavaScript code takes place). Alternatively, use step js to explicitly declare that whatever follows on that line is JavaScript code and ensure that the line is not treated as a TagUI step.

FILES

SECURITY

Below architecture diagram for TagUI will be used to elaborate on the topic of security -

As TagUI and the foundation it's built on is open-source software, it means users can read the source code of TagUI and all its dependencies to check if there is a security flaw or malicious code. This is an advantage compared to using commercial software that is closed-source, as users cannot see what is the code behind the software.

Following are links to the source code for TagUI and its open-source dependencies. You can dig through the source code for the other open-source dependencies below, or make the fair assumption that security issues would have been spotted by users and fixed, as these projects are mature and have large user bases.

• TagUI - https://github.com/kelaberetiv/TagUI
• SikuliX - https://github.com/RaiMan/SikuliX1
• CasperJS - https://github.com/casperjs/casperjs
• PhantomJS - https://github.com/ariya/phantomjs
• SlimerJS - https://github.com/laurentj/slimerjs
• Python - https://github.com/python/cpython
• R - https://github.com/wch/r-source
• PHP - https://github.com/php/php-src

Following are more comments specific to TagUI for the topic of security -

• By design, TagUI does not send outgoing web traffic or send outgoing data, other than what the user is automating on, for example visiting a website. However, there is an upload option (off by default) when used, TagUI will go to an open-source service Hastebin that allows uploading of information to share via URL. This sends outgoing data of the automation execution log to that website and returns an URL to user so that user can easily share the output of the automation with colleagues or friends.

• TagUI allows running of automation files whether they are stored locally as a file on your laptop or computer, or if the file is an online URL. This means users should be cautious to run automation from online URLs unless they are sure that the online URL is a safe source and not doing something destructive. This can be checked by accessing the URL using a web-browser to see the automation steps performed in the online automation file.

• TagUI comes with a web service, that can be used to trigger automation flows if TagUI is installed on a web-server. More details in Developers Reference section under API. The design minimizes security risk by not forcing users to turn on exec() for their PHP configuration. This means execution of malicious commands cannot be injected through the web-service. Instead, the web-service works by using crontab to periodically check for automation requests. Webservice execution of online flows is now disabled by default so that malicious users cannot trigger execution of a malicious flow hosted online on the webserver.

• TagUI has very powerful features, including the ability to run command line programs, Python code, R code, API calls, and ability to make changes to the local file systems. These features can become security risks if a malicious automation flow file is executed. Imagine TagUI as a tool, such as fire. In evil hands it can be used to do evil things at scale. In good hands, it can be used to solve pain points related to processes that are repetitive and huge in volume.

• As TagUI can replicate what a normal human user can do on the computer, TagUI is restricted and compliant to the same security policies as the normal human user would be subjected to. For example, having to use a hardware token in order to access the laptop, or user password in order to login to the computer. Or having a 2FA hardware token in order to access confidential information from a web or enterprise application.

Credits

• TagUI - Ken Soh from Singapore / @kensoh
• SikuliX - Raimund Hocke from Germany / @RaiMan
• CasperJS - Nicolas Perriault from France / @n1k0
• PhantomJS - Ariya Hidayat from Indonesia / @ariya
• SlimerJS - Laurent Jouanneau from France / @laurentj

License

TagUI is open-source software released under Apache 2.0 license