UIAutomation tests for iOS apps are a great way to test an app’s user interface and interaction in a way that traditional unit tests for iOS apps cannot. That said, building them right takes a considerable investment in time. Invariably, you’re building upon the primitives that Apple has provided with their UIA JavaScript classes.
Running UIAutomation scripts through the Instruments GUI can be tedious; especially when you’re in a heavy develop-run-test workflow. Fortunately, Apple has given us a way to run the Automation Instrument through the command line. Apple’s documentation on Automating UI Testing is however, brief.
The section of Apple’s documentation that concerns us is entitled Executing an Automation Instrument Script from the Command Line. Unfortunately, at the time of this writing, the information in that section of Apple’s documentation still had errors when iOS8 went public.1.
You might have read that the -w <deviceID> switch was optional if you wanted to run in the simulator.
That was likely true in the past, but no longer with Xcode 6.0.1. You now need to provide the symbolic name of the simulator you wish to run your app in.
Additionally, the documentation suggests that you can find the Automation Instrument template at the following outdated path:
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/Library/ Instruments/PlugIns/AutomationInstrument.bundle/Contents/Resources/Automation.tracetemplate
With Xcode 6.0.1, the path to the Automation Instrument template is now:
/Applications/Xcode.app/Contents/Applications/Instruments.app/Contents/PlugIns/ AutomationInstrument.xrplugin/Contents/Resources/Automation.tracetemplate
While these paths have been known to change with various Xcode releases, what’s really tedious with command line invocation of UIAutomation tests is the fact that you have to provide the path to the app in the simulator. That path has a UDID for the simulator in it, and a GUID for the application itself. Presumably this mimics some of the internal organization used on the devices themselves.
In any event, when you reconfigure your defined simulators in the updated iOS Simulator app, those simulator UDIDs are going to change. When you clean out your build folders with a sledgehammer and restart fresh, those GUIDs for your apps are likely going to change too. Hard coding references to such in your test script runner would be annoying.
It’s with that motivation that I wrote a couple of scripts to encapsulate all of this minutia, and abstract it for us developers who understand how it works, but don’t want to get lost in the weeds every time something changes.
UI Automation Runner
The ui_automation_runner.sh script I developed can be found in this UI Automation Runner project. The project’s README file covers the basics.
The benefit to using such a script is that it is focused on one thing: running your automation tests. It doesn’t pretend to build or install your apps. There are so many more complex tools that run UIAutomation scripts and do a whole lot more at the same time. I was hard pressed to find something nimble with the singular purpose of just running a single UIAutomation test.
When paths for the Automation Instrument change with a future Xcode release, or the default home for simulators changes, you can go ahead and edit these two lines in the ui_automation_runner.sh script, assuming you know what you’re doing. And if you beat me to it, send me a pull request and I’ll update it for everyone.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# ---------- DO NOT EDIT ANYTHING BELOW THIS LINE, UNLESS YOU KNOW WHAT YOU'RE DOING ----------- # ===== GLOBAL CONSTANTS ===== # This is where Xcode installs simulators. Accurate as at Xcode 6.0.1. BASE_SIMULATORS_PATH="$HOME/Library/Developer/CoreSimulator/Devices" # UIAutomation Instruments Template location. Accurate as at Xcode 6.0.1. INSTRUMENTS_AUTOMATION_TEMPLATE_PATH="/Applications/Xcode.app/Contents/Applications/Instruments.app/Contents"\ "/PlugIns/AutomationInstrument.xrplugin/Contents/Resources/Automation.tracetemplate" # DO NOT CHANGE INDENTATION! |
You should construct your UIAutomation tests so that the one test file you pass to the command line for invocation, ends up running your entire JavaScript test suite.
To use this project, you’ll actually leave the ui_automation_runner.sh script file untouched. Instead, you’ll change the settings in the companion run_tests.sh script that then invokes the ui_automation_runner.sh script which in turn, intelligently invokes Apple’s own instruments command.
Here’s the script that you need to customize. As you can see, it’s reasonably clear where you provide the details for your specific app, simulator, test file and paths.2
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 |
#!/bin/bash # run_tests.sh # Created by Sohail A. # @idStar # This script is a companion to the script ui_automation_runner.sh, which is designed not to be edited. # In this very script however, you'll actually modify some variables below to specify values for # what you would like run, where to find your test, where to dump results, etc. # ===== DEVICES AND SIMULATORS ===== # Define all of your simulators and devices here. # Create descriptive variables to hold the UDIDs of your devices and the custom names of your simulators. # If you're new to Xcode 6's custom simulators, read the following for more context: # https://developer.apple.com/library/ios/documentation/IDEs/Conceptual/iOS_Simulator_Guide/GettingStartedwithiOSStimulator/GettingStartedwithiOSStimulator.html # Specifically, the section entitled, "Change the Simulated Device and iOS Version". In a nutshell, you can to choose # which simulators are defined, and what they're called. The ui-automation-runner.sh script will use these names # to find the right simulator to launch. We recommend creating a variable for each of your defined simulators here, # so you can switch between them easily when providing a value to the 'SIMULATOR_NAME_OR_DEVICE_UDID' down below. device_udid_iPhone5s_iOS8="f04d9cefeccf5013e3ce89db955e68d6ce1551c4" # Not my real device UDID, but you get the idea simulator_iPhone6Plus_iOS8="iPhone 6 Plus" simulator_iPad_Air_iOS8="iPad Air" simulator_iPhone5s_iOS7_1="iPhone 5s (iOS7.1)" # ===== REACHING THE TEST RUNNER ===== # You may choose to keep this app specific script co-located with your app, and your copy of the # ui_automation_runner script in some other more generic location. Whatever you choose, that needs to be reflected # as an absolute or relative path to the current file: AUTOMATION_RUNNER_SCRIPT_PATH="./ui_automation_runner.sh" # ===== YOUR APP AND TEST FILE SETTINGS ===== # The name of your app. Use your Xcode project name. This is not necessarily the icon display name visible in Springboard. # Leave off the ".app" extension so that you can reference the same app name when switching between device and simulator. TEST_APP_NAME="ACMERoadRunnerRadar" # Set which simulator or device you want Instruments Automation to run with: SIMULATOR_NAME_OR_DEVICE_UDID=${simulator_iPhone5s_iOS7_1} # The directory in which we can find the test file you'll specify below: JAVASCRIPT_TEST_FILES_DIRECTORY="$HOME/Developer/clients/acme/roadrunnerradar/ACMERoadRunnerRadarAutomationTests/" # The JavaScript test file you'd like to run. For a suite of tests, have this file simply import and # execute other JavaScript tests, so that you can conceivably run a full suite of tests with one command: JAVASCRIPT_TEST_FILE="TestRunner.js" # The directory into which the instruments command line tool should dump its verbose output: TEST_RESULTS_OUTPUT_PATH="$HOME/Developer/clients/acme/roadrunnerradar/ACMERoadRunnerRadarAutomationTests/TestRuns/" # ---------- DO NOT EDIT ANYTHING BELOW THIS LINE, UNLESS YOU KNOW WHAT YOU'RE DOING ----------- "$AUTOMATION_RUNNER_SCRIPT_PATH" \ "$SIMULATOR_NAME_OR_DEVICE_UDID" \ "$TEST_APP_NAME" \ "$JAVASCRIPT_TEST_FILE" \ "$JAVASCRIPT_TEST_FILES_DIRECTORY" \ "$TEST_RESULTS_OUTPUT_PATH" |
Here’s what a sample invocation looks like:3
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
sohail@103:~/Developer/clients/acme/roadrunnerradar/ACMERoadRunnerRadarAutomationTests master$ run_tests.sh Instruments UIAutomation command invoked: instruments -w 'iPhone 5s' \ -t '/Applications/Xcode.app/Contents/Applications/Instruments.app/Contents/PlugIns/AutomationInstrument.xrplugin/Contents/Resources/Automation.tracetemplate' \ '/Users/sohail/Library/Developer/CoreSimulator/Devices/7232A640-A9D2-4626-A2AD-37AFFF706718/data/Containers/Bundle/Application/E71B915E-051D-4BEF-9083-34416D02EC91/RoadRunnerRadar.app' \ -e UIASCRIPT '/Users/sohail/Developer/clients/acme/roadrunnerradar/ACMERoadRunnerRadarAutomationTests/TestRunner.js' \ -e UIARESULTSPATH '/Users/sohail/Developer/clients/acme/roadrunnerradar/ACMERoadRunnerRadarAutomationTests/TestResults/' 2014-09-20 21:15:21 +0000 Start: Slingshot Apparatus Install . . . |
- Apple’s documentation was timestamped September 17, 2014. Of course, I’ve submitted feedback to Apple for each of the documentation errors I cite in this article. [↩]
- If you’re serious about reading this file line by line, you’d find it easier to read on GitHub itself. [↩]
- I’ve truncated the output after the first test logs that it started. The output can get quite verbose. [↩]