At my day job I’ve been tasked with improving our webapp testing strategy. This need came about after a bug was introduced in one of our webapps and not picked up by our current testing process. The bug was related to inconsistencies in the implementation of the Javascript date API between iOS Safari and Android Chrome browsers.
We currently employ a (pretty loose) TDD workflow for our Backbone webapps. Writing our specs with Jasmine and using Grunt as our task runner. This is all integrated into our Jenkins CI. The Jasmin specs are all run though an instance of Phantom in Jenkins via the (one of many) grunt task(s) when we push our code up and release though Jenkins. The build either passes or fails.
From investigating all the various options and frameworks, all signs pointed to Karma. Karma would give us:
- Seamless integration into our current work flow
- Immediate feedback – you write a test for a new piece of functionality, hit save and it immediately tells you it’s failed. Write the functionality, hit save and it tells you right away if it passes your test.
- Testing on multiple real devices
It was a fairly painless process to set Karma up using the official docs and get my tests running on the browsers on my dev environment (Safari, Chrome, Canary, Firefox, Phantom) and there are plenty of resources online for this so I won’t go into this here. What I did have issues with, especially with the lack of resources online, was running my specs on mobile devices.
I came across an iOS launcher plugin that had looked promising to get me going with iOS Safari but it’s not been touched by the developer in a couple of years and after a bit of experimentation discovered that doesn’t appear to work with node versions > 0.10. At the time of writing, I’m working with 0.12.
More research led me to Selenium, a browser automator. From there, I came across Appium which uses the same WebDriver protocol as Selenium but also enables you to automate mobile browsers. Download the Appium desktop app here
To interact with Appium, there is the Karma webdriver launcher (npm install karma-webdriver-launcher).
The idea is pretty simple (but the config is a little fiddly):
- Make sure you have xcode with iOS simulator installed and the Android Development Studio with at least 1 AVD (android virtual device) set up.
- Fire up the Appium desktop app to start an Appium server on the machine with the virtual devices (a mac for iOS testing and windows/linux/mac for Android testing). This can be the same machine as you run your Karma instance or a machine accessible by it.
- Configure your karma.conf.js file to let it know where your virtual devices are
- Run your tests and Karma will pipe the tests out to the Appium server with the WebDriver API. Appium will then fire up the emulators and run the tests in the specified browsers.
karma.conf.js
Karma will launch the browers to localhost:9876 (or whatever other port you set it to) so if running your Karma instance on a different machine to your test machine, you will want to let your emulated browsers where to go to run the tests. Obviously localhost on the test machine is of no use if localhost there doesn’t have your Karma instance. You can simply do this by specifying the hostname in your karma.conf.js. ie:
hostname: '192.168.10.126', |
You will need to declare your custom launchers for your emulators too. In my case, this looks like this:
customLaunchers: { 'iOS-Safari' : { base: 'WebDriver', platformName: "iOS", deviceName: 'iPhone 5', config: webdriverConfig, browserName: 'Safari', }, 'Android' : { base: 'WebDriver', platformName: "Android", deviceName: 'Android', config: webdriverConfig, browserName: 'Browser', } }, |
webdriverConfig just contains the IP and port of the Appium server. In my case:
var webdriverConfig = { hostname: '192.168.10.126', port: 4723 } |
Here is my complete karma.conf.js:
// Karma configuration // Generated on Thu Aug 06 2015 11:58:03 GMT+0100 (BST) var webdriverConfig = { hostname: '192.168.10.126', port: 4723 } module.exports = function(config) { config.set({ // base path that will be used to resolve all patterns (eg. files, exclude) basePath: '', browserNoActivityTimeout: 1000000, // frameworks to use // available frameworks: https://npmjs.org/browse/keyword/karma-adapter frameworks: ['jasmine', 'requirejs'], // list of files / patterns to load in the browser files: [ 'test/test-main.js', {pattern: 'environment-config.js', included: false}, {pattern: 'environment-config.json', included: false}, {pattern: 'js/**/*.*', included: false}, {pattern: 'test/jasmine/spec/**/*.spec.js', included: false}, ], junitReporter: { outputDir: 'reports/karma', outputFile: undefined, suite: '' }, // list of files to exclude exclude: [ ], // preprocess matching files before serving them to the browser // available preprocessors: https://npmjs.org/browse/keyword/karma-preprocessor preprocessors: { }, // test results reporter to use // possible values: 'dots', 'progress' // available reporters: https://npmjs.org/browse/keyword/karma-reporter reporters: ['progress', 'junit'], // web server port port: 9876, hostname: '192.168.10.126', //points to accessible Karma server // enable / disable colors in the output (reporters and logs) colors: true, // level of logging // possible values: config.LOG_DISABLE || config.LOG_ERROR || config.LOG_WARN || config.LOG_INFO || config.LOG_DEBUG logLevel: config.LOG_INFO, // enable / disable watching file and executing tests whenever any file changes autoWatch: true, // start these browsers // available browser launchers: https://npmjs.org/browse/keyword/karma-launcher browsers: ['iOS-Safari', 'Chrome'], customLaunchers: { 'iOS-Safari' : { base: 'WebDriver', platformName: "iOS", deviceName: 'iPhone 5', config: webdriverConfig, browserName: 'Safari', }, 'Android' : { base: 'WebDriver', platformName: "Android", deviceName: 'Android', config: webdriverConfig, browserName: 'Browser', } }, // Continuous Integration mode // if true, Karma captures browsers, runs the tests and exits singleRun: true }) } |
Appium Android Config
You can get the device name and platform version of your installed AVDs with the command:
~/Library/Android/sdk/tools/android list avd |
This will give you something that looks like this:
Nothing changed in advanced settings except you must put in your Android SDK path so Appium knows where to find the SDK.
Appium iOS Config
Appium General Config
The server address here is the address and port of the Appium instance (note in my example, Appium and Karma are running on the same machine under different ports but as mentioned above, you can have the Appium server running on an entirely different machine as long as it is accessible by the Karma service)
Next steps
I’ve not yet solved the running of iOS and Android tests simultaneously on the same machine but I am currently trying to solve this by setting up a Linux VM on the test machine to run the Android simulator so I can pipe to 2 machines effectively and test on both at the same time.
Also to tackle is running the tests on Chrome on the Android simulator. Currently I’m only testing on the “stock” browser. Desktop Chrome tests will most likely cover me in every single instance, though for completeness I want to get this solved.
Hey, you need to spin up 2 separate appium servers to run tests in parallel. They need to have different ports and preferably bootstrap ports as well, so e.g.
appium -a 0.0.0.0 -p 4730 -bp 5000
appium -a 0.0.0.0 -p 4750 -bp 5100
Note that in Appium 1.5 you’ll be able to run mutliple sessions without having to start new appium server session 🙂
Thanks for the note 🙂 Looking forward to the release of 1.5. Any idea on timescales?
beta soon (couple weeks tops), actual release – supposedly soon after!
Thank you for this article, it helped me getting started on running karma tests on ios and android 🙂
Is it possible to run unit tests on actual devices? If Karma can’t launch the browser on the phone, is there a way to connect to a Karma daemon from an iPhone or Android phone to do this?