Running Karma on iOS and Android Simulators with Appium

At my day job I’ve been tasked with improving our webapp testing strategy. This need came about after a bug was introduced in one of our webapps and not picked up by our current testing process. The bug was related to inconsistencies in the implementation of the Javascript date API between iOS Safari and Android Chrome browsers.

We currently employ a (pretty loose) TDD workflow for our Backbone webapps. Writing our specs with Jasmine and using Grunt as our task runner. This is all integrated into our Jenkins CI. The Jasmin specs are all run though an instance of Phantom in Jenkins via the (one of many) grunt task(s) when we push our code up and release though Jenkins. The build either passes or fails.

From investigating all the various options and frameworks, all signs pointed to Karma. Karma would give us:

• Seamless integration into our current work flow
• Immediate feedback – you write a test for a new piece of functionality, hit save and it immediately tells you it’s failed. Write the functionality, hit save and it tells you right away if it passes your test.
• Testing on multiple real devices

It was a fairly painless process to set Karma up using the official docs and get my tests running on the browsers on my dev environment (Safari, Chrome, Canary, Firefox, Phantom) and there are plenty of resources online for this so I won’t go into this here. What I did have issues with, especially with the lack of resources online, was running my specs on mobile devices.

I came across an iOS launcher plugin that had looked promising to get me going with iOS Safari but it’s not been touched by the developer in a couple of years and after a bit of experimentation discovered that doesn’t appear to work with node versions > 0.10. At the time of writing, I’m working with 0.12.

More research led me to Selenium, a browser automator. From there, I came across Appium which uses the same WebDriver protocol as Selenium but also enables you to automate mobile browsers. Download the Appium desktop app here

To interact with Appium, there is the Karma webdriver launcher (npm install karma-webdriver-launcher).

The idea is pretty simple (but the config is a little fiddly):

• Make sure you have xcode with iOS simulator installed and the Android Development Studio with at least 1 AVD (android virtual device) set up.
• Fire up the Appium desktop app to start an Appium server on the machine with the virtual devices (a mac for iOS testing and windows/linux/mac for Android testing). This can be the same machine as you run your Karma instance or a machine accessible by it.
• Configure your karma.conf.js file to let it know where your virtual devices are
• Run your tests and Karma will pipe the tests out to the Appium server with the WebDriver API. Appium will then fire up the emulators and run the tests in the specified browsers.

karma.conf.js

Karma will launch the browers to localhost:9876 (or whatever other port you set it to) so if running your Karma instance on a different machine to your test machine, you will want to let your emulated browsers where to go to run the tests. Obviously localhost on the test machine is of no use if localhost there doesn’t have your Karma instance. You can simply do this by specifying the hostname in your karma.conf.js. ie:

hostname: '192.168.10.126',

You will need to declare your custom launchers for your emulators too. In my case, this looks like this:

customLaunchers: {
    'iOS-Safari' : {
        base: 'WebDriver',
        platformName: "iOS",
        deviceName: 'iPhone 5',
        config: webdriverConfig,
        browserName: 'Safari',
    },
    'Android' : {
        base: 'WebDriver',
        platformName: "Android",
        deviceName: 'Android',
        config: webdriverConfig,
        browserName: 'Browser',
    }
},

webdriverConfig just contains the IP and port of the Appium server. In my case:

var webdriverConfig = {
    hostname: '192.168.10.126',
    port: 4723
}

Here is my complete karma.conf.js:

// Karma configuration
// Generated on Thu Aug 06 2015 11:58:03 GMT+0100 (BST)
var webdriverConfig = {
    hostname: '192.168.10.126',
    port: 4723
}
module.exports = function(config) {
 
  config.set({
 
    // base path that will be used to resolve all patterns (eg. files, exclude)
    basePath: '',
 
    browserNoActivityTimeout: 1000000,
 
    // frameworks to use
    // available frameworks: https://npmjs.org/browse/keyword/karma-adapter
    frameworks: ['jasmine', 'requirejs'],
 
    // list of files / patterns to load in the browser
    files: [
      'test/test-main.js',
      {pattern: 'environment-config.js', included: false},
      {pattern: 'environment-config.json', included: false},
      {pattern: 'js/**/*.*', included: false},
      {pattern: 'test/jasmine/spec/**/*.spec.js', included: false},
    ],
 
    junitReporter: {
        outputDir: 'reports/karma',
        outputFile: undefined,
        suite: ''
    },
 
    // list of files to exclude
    exclude: [
    ],
 
    // preprocess matching files before serving them to the browser
    // available preprocessors: https://npmjs.org/browse/keyword/karma-preprocessor
    preprocessors: {
    },
 
    // test results reporter to use
    // possible values: 'dots', 'progress'
    // available reporters: https://npmjs.org/browse/keyword/karma-reporter
    reporters: ['progress', 'junit'],
 
    // web server port
    port: 9876,
 
    hostname: '192.168.10.126', //points to accessible Karma server
 
    // enable / disable colors in the output (reporters and logs)
    colors: true,
 
    // level of logging
    // possible values: config.LOG_DISABLE || config.LOG_ERROR || config.LOG_WARN || config.LOG_INFO || config.LOG_DEBUG
    logLevel: config.LOG_INFO,
 
    // enable / disable watching file and executing tests whenever any file changes
    autoWatch: true,
 
    // start these browsers
    // available browser launchers: https://npmjs.org/browse/keyword/karma-launcher
    browsers: ['iOS-Safari', 'Chrome'],
 
    customLaunchers: {
        'iOS-Safari' : {
            base: 'WebDriver',
            platformName: "iOS",
            deviceName: 'iPhone 5',
            config: webdriverConfig,
            browserName: 'Safari',
        },
        'Android' : {
            base: 'WebDriver',
            platformName: "Android",
            deviceName: 'Android',
            config: webdriverConfig,
            browserName: 'Browser',
        }
    },
 
    // Continuous Integration mode
    // if true, Karma captures browsers, runs the tests and exits
    singleRun: true
 
  })
}

Appium Android Config

You can get the device name and platform version of your installed AVDs with the command:

~/Library/Android/sdk/tools/android list avd

This will give you something that looks like this:

Nothing changed in advanced settings except you must put in your Android SDK path so Appium knows where to find the SDK.

Appium iOS Config

Appium General Config

The server address here is the address and port of the Appium instance (note in my example, Appium and Karma are running on the same machine under different ports but as mentioned above, you can have the Appium server running on an entirely different machine as long as it is accessible by the Karma service)

Next steps

I’ve not yet solved the running of iOS and Android tests simultaneously on the same machine but I am currently trying to solve this by setting up a Linux VM on the test machine to run the Android simulator so I can pipe to 2 machines effectively and test on both at the same time.

Also to tackle is running the tests on Chrome on the Android simulator. Currently I’m only testing on the “stock” browser. Desktop Chrome tests will most likely cover me in every single instance, though for completeness I want to get this solved.