Selenium Demystified

The Selenium Story

In 2004, Jason Huggins developed an early version of Selenium-RC as an internal tool for ThoughtWorks. Selenium-IDE was originally created by Shinya Kasatani and donated to the Selenium project in 2006. Google got involed in 2009 and is largely responsible for what became WebDriver. Then in 2012 a W3C standard draft was pusblished, solidifying WebDriver’s place as the defacto standard. Today Selenium is widely used in the QA community. Just hop on any employment board and see how many QA Automation positions mention Selenium.

Requirements

You understand basic HTML and JavaScript for the Selenium IDE content.

For the WebDriver content you’ll need a basic understanding of Java or other supported language.

Selenium IDE

The Selenium IDE is a Firefox add-on that allows the user to record, click, and play back actions in the browser. If you have ever used macros in Microsoft’s Office products or AutoIT this will be familiar. These tests are saved in HTML files in a format referred to as Selenese. The test playback is done via JavaScript [source]. So when you have Command = open and Target = http://timothycope.com

Capture1

it is executed almost like you opened up the Developer Tools Console (F12) and in the console tab ran window.location.href = 'http://timothycope.com';

Capture2

Selenium Tests

Test steps are comprised of three things; command, target, and value.

Capture3

  • A command is what tells Selenium what to do. Selenium commands come in three ‘flavors’;
    • Actions are commands that generally manipulate the state of the application. They do things like “click this link” and “select that option”. If an Action fails, or has an error, the execution of the current test is stopped.
      • Many Actions can be called with the “AndWait” suffix, e.g. “clickAndWait”. This suffix tells Selenium that the action will cause the browser to make a call to the server, and that Selenium should wait for a new page to load.
    • Accessors examine the state of the application and store the results in variables, e.g. “storeTitle”. They are also used to automatically generate Assertions.
    • Assertions are like Accessors, but they verify that the state of the application conforms to what is expected. Examples include “make sure the page title is X” and “verify that this checkbox is checked”.
  • The target is a element locator for the html element. There are a number of supported selectors;
    • By ID – This is the most efficient and preferred way to locate an element. Common pitfalls that UI developers make is having non-unique id’s on a page or auto-generating the id, both should be avoided. A class on an html element is more appropriate than an auto-generated id.
    • By Class – “Class” in this case refers to the attribute on the DOM element. Often in practical use there are many DOM elements with the same class name, thus finding multiple elements becomes the more practical option over finding the first element.
    • By Tag Name – The DOM Tag Name of the element.
    • By Name – Find the input element with matching name attribute.
    • By Link Text – Find the link element with matching visible text.
    • By Partial Link Text – Find the link element with partial matching visible text.
    • By CSS -Like the name implies it is a locator strategy by css. Native browser support is used by default, so please refer to w3c css selectors for a list of generally available css selectors. If a browser does not have native support for css queries, then Sizzle is used. IE 6,7 and FF3.0 currently use Sizzle as the css query engine.
      • Beware that not all browsers were created equal, some css that might work in one version may not work in another.
    • By XPATH – At a high level, WebDriver uses a browser’s native XPath capabilities wherever possible. On those browsers that don’t have native XPath support, we have provided our own implementation. This can lead to some unexpected behaviour unless you are aware of the differences in the various xpath engines.
    • Using JavaScript – You can execute arbitrary javascript to find an element and as long as you return a DOM Element, it will be automatically converted to a WebElement object.
  • The value is used for certain commands. Official documentation is currently lacking in respect to a list of all API endpoints and their support with various frameworks. Here is a list of all actions, though.

IDE to WebDriver

You can “write” your tests by using Selenium IDE to record your actions. Then you just need to export to your language of choice. You may find that some of your tests are not exported how you might expect. Some libraries are not fully fleshed out so you’ll get a comment in the exported coded saying that the Command is unsupported. When this happens you’ll need to code something language specific to suit your needs.

After you export the test from IDE to WebDriver, your code will now be interpreted by WebDriver and executed on the browser.

flow-chart

It is more likely that you will build a suite of tests and have them executed by something like JUnit or NUnit. You could even run on multiple machines using something like Selenium Grid2, too. Many CI tools, like Jenkins, can be used to kick off the tests automatically with any new code changes.

Selenium WebDriver

The Selenium WebDriver is an application  that works as a middle man between your automation code and your browser. This application called be called from many languages and frameworks using the WebDriver API. The level of support you get differs based on the programming language, operating system, and browser you use. Know that Selenium is developed  in Java + Windows + Firefox. There is limited support for Linux and OSX. Additional browsers are supported by extensions. Headless browsers, like HtmlUnit, allow a UI-less WebView to be used as a web browser.

Software QA Automation in an Agency Setting

Background

I have worked in “enterprise” settings most of my career. In these settings automation handled a full regression and was costly to maintain. In this article I will point out a few lessons learned from working in an agency setting. This will be scoped to web applications.

Definitions

  • Enterprise (setting) – Typically an app or two with a very long life cycle. Most often, a bug introduced could mean lost revenue.
  • Agency (setting) – Applications are developed and handed over to the client when development is complete.
  • Concurrent (testing model) – ‘Regression’ automated tests are ran after a new build. At the same time, or shortly after, you would test new functionality. A ticket is not complete until the new functionality is added to the automated regression plan.

Considerations

  1. Does the automation require heavy maintenance?
    • In development a page can change with every commit.
  2. How easy does your automation lend itself to the scrum methodology?
    • If your automation package requires a hand off longer than a couple hours, then it is likely not set up in an intuitive way.
    • Can anyone contribute at any time?
      • Multiple testers on a given project ought to be able to make changes to test cases as needed.
      • The answer to this question can be reliant on a couple factors;
        • Do you use source control?
        • Is your automation framework available to the entire team?
  3. What value does the automation provide?
    • If the manual regression test takes longer than a day and needs to be done more than once a month, then you are likely investing in automation.

Providing Cost Benefit

In an agency setting we bill time+materials. If it takes me an hour to test a new function, it might take an hour and a half more to make a solid automated test (I always ballpark initial automation as “time to test + 50%”). So already we are at 2.5 hours. This isn’t a big deal, as the regression test provides value in saving regression hours for subsequent builds. Additional cost is incurred with each build that breaks your test. As your regression grows, so to does your required maintenance. A way to minimize this cost is to use a simple framework. QA automation specialist have their choice from many frameworks and those frameworks typically offer more than you really need. Selenium is one such tool. It provides a bevy of commands that mostly go unused and requires many tools to actually use. For Selenium you work in a fragmented system. You could develop your test using Selenium IDE and export to your language of choice. You then fix any issues with your exported code and use something like jUnit to run the tests. If you want to schedule your test and email results, then you’ll have to write that.

I am a fan of using a paid product, called Telerik Test Studio. Telerik allows us to make, edit, run, schedule, and report on tests quite easily. Its downfall is that it is only available on Windows and tests only in IE, Firefox, Chrome, and WPF. Most shops end up using Selenium as its free and can be extended to many devices via third-party plugins. Microsoft Test Manager uses the Selenium driver for Chrome, too.

I use my automation suite to primarily test MVP flows and happy paths. I will include some negative tests, of course, but most of this should have already been addressed in manual testing so these tests provide little value outside of a “sanity” check. “But it doesn’t cover Safari or mobile”, you might contend. I have found mobile device emulators to be unreliable and outside of Appium there isn’t a one size-fits-all solution. I do most of my initial, manual testing in Safari as my primary machine is a Mac. I also have a pile of mobile devices and an intern. If something works in Telerik but not on those devices, it’s usually a device specific issue anyway.

The Value of Automation

As noted before, the automation provides a quick way to verify previous functionality in new builds. The argument is easily won that it is necessary for quicker and more reliable QA. I have never had an employer refuse to purchase an automation solution. My argument was presented in showing how long it takes to create/run test in given frameworks. The cost, compared to Selenium, is often paid for in saved development hours. In the agency setting, I am always thinking of the fairness to the client. Should the client have to pay for extra maintenance hours because you chose the wrong framework? The answer is no. That means that maintenance needs to be as simple and quick as possible. The simplicity would allow any team member the ability to update the test case(s). The speed in which issues are resolved is related to your testing framework.

Other Facets of Test Automation

In addition to automated regression tests, I rely on SortSite. This tool saves a lot of time checking for these things manually and helps ensure a quality product.

Covered by SortSite

I also use OWASP’s ZAP scanner for light security testing. I covers 6 of the Top 10 common vulnerabilities. For further testing, you may want to invest in Acunetix.

Covered by ZAP

A Note on CI

When your dev team finds out you have automation, they’ll likely ask if it can be included as part of the build process. This comes from the love of unit tests included in the project. This is certainly feasible, but not without issues. It is almost guaranteed that some tests will fail. The new build might have changed an element’s id or the xpath has changed. For that reason I don’t include automated regression in the build process. I can just run it when I need to, which is often right after a build or deployment.

Setup Selenium, NUnit, and Visual Studio

Update

This article is still relevant but a more comprehensive guide exists at http://timothycope.com/c-selenium-automation. This new guide also covers Page Object Design Pattern (POM) and Selenium Server (Grid).

Background

Selenium automates browsers. That’s it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should!) also be automated as well.
http://docs.seleniumhq.org/

NUnit is a unit-testing framework for all .Net languages. Initially ported from JUnit, the current production release, version 2.6, is the seventh major release of this xUnit based unit testing tool for Microsoft .NET. It is written entirely in C# and has been completely redesigned to take advantage of many .NET language features, for example custom attributes and other reflection related capabilities. NUnit brings xUnit to all .NET languages.
http://www.nunit.org/

Install NUnit

You will need NUnit to execute your automated tests. Download the win MSI file and run it. Select a complete installation when prompted.

Setup Visual Studio

  1. Open Visual Studio
  2. Click File, click New, and select Project…
  3. In the left navigation pane, expand Templates, then Visual C#, and then select Class Library in the center window pane.
  4. In Package Manager Console, type Install-Package Selenium.WebDriverBackedSelenium
  5. In Package Manager Console, type Install-Package Selenium.Support
  6. In Package Manager Console, type Install-Package NUnit

Creating Your First Test

I renamed Class1.cs to AutomatedTest01.cs and that is reflected in the code below.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using OpenQA.Selenium;
using OpenQA.Selenium.IE;
using NUnit.Framework;
using OpenQA.Selenium.Firefox;

namespace SeleniumClassLibrary
{
    [TestFixture]
    public class AutomatedTest01
    {

        IWebDriver driver;

        [TestFixtureSetUp] // http://www.nunit.org/index.php?p=testFixture&r=2.5
        public void TestSetUp()
        {
            // Set the IWebDriver to use FireFox
            driver = new FirefoxDriver();

            // Set the timeout to 30 seconds
            driver.Manage().Timeouts().ImplicitlyWait(new TimeSpan(0, 0, 30));

        }

        [Test] // http://www.nunit.org/index.php?p=test&r=2.5
        public void NavigateToGoogle()
        {
            // Navigate to URL
            driver.Navigate().GoToUrl("http://www.google.com");

            // Assert that the IWebDriver window's title is "Google"
            Assert.AreEqual("Google", driver.Title);
        }

        [TestFixtureTearDown] // http://www.nunit.org/index.php?p=fixtureTeardown&r=2.5
        public void FixtureTearDown()
        {
            // Closes all windows associated with the IWebDriver
            driver.Quit();
        }
    }
}

Save and build the project.

Run the Automated Test

  1. Open NUnit
  2. Click File and select Open Project…
  3. Find the .dll from your Visual Studio Project
  4. Click Run in the center window pane

Importing Code from Selenium IDE for Firefox

You can download the Firefox add-on that allows you to record-and-click basic webpage tasks.

  1. In Selenium IDE:
    1. Click File, select Export Test Case As…, and select C# / NUnit / WebDriver
  2. You can copy+paste the code into a new .cs file in your project or import the cs file
  3. Save and Build your project

Note: You may have to make some manual changes to the code to have it playback correctly in NUnit.

using System;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading;
using NUnit.Framework;
using OpenQA.Selenium;
using OpenQA.Selenium.Firefox;
using OpenQA.Selenium.Support.UI;

namespace SeleniumClassLibrary
{
    // Genereated by Selenuim IDE for Firefox
    [TestFixture]
    public class AutomatedTest02
    {

        private IWebDriver driver;
        private StringBuilder verificationErrors;
        private string baseURL;
        private bool acceptNextAlert = true;

        [SetUp]
        public void SetupTest()
        {
            driver = new FirefoxDriver();
            baseURL = "http://timothycope.com/";
            verificationErrors = new StringBuilder();
        }

        [TearDown]
        public void TeardownTest()
        {
            try
            {
                driver.Quit();
            }
            catch (Exception)
            {
                // Ignore errors if unable to close the browser
            }
            Assert.AreEqual("", verificationErrors.ToString());
        }

        [Test]
        public void The01Test()
        {
            driver.Navigate().GoToUrl(baseURL + "/");
            Thread.Sleep(2500);
            driver.FindElement(By.LinkText("ABOUT TIMOTHY COPE")).Click();
            Thread.Sleep(2500);
            for (int second = 0; ; second++)
            {
                if (second >= 60) Assert.Fail("timeout");
                try
                {
                    if (IsElementPresent(By.XPath("//img[@alt='logo']"))) break;
                }
                catch (Exception)
                { }
                Thread.Sleep(1000);
            }
        }
        private bool IsElementPresent(By by)
        {
            try
            {
                driver.FindElement(by);
                return true;
            }
            catch (NoSuchElementException)
            {
                return false;
            }
        }

        private bool IsAlertPresent()
        {
            try
            {
                driver.SwitchTo().Alert();
                return true;
            }
            catch (NoAlertPresentException)
            {
                return false;
            }
        }

        private string CloseAlertAndGetItsText()
        {
            try
            {
                IAlert alert = driver.SwitchTo().Alert();
                string alertText = alert.Text;
                if (acceptNextAlert)
                {
                    alert.Accept();
                }
                else
                {
                    alert.Dismiss();
                }
                return alertText;
            }
            finally
            {
                acceptNextAlert = true;
            }
        }
    }
}

Solution on GitHub

You can download the project I made from GitHub:
https://github.com/kirbycope/SeleniumClassLibrary

Automated Software Testing Tools

About Automated Software Testing

In software testing, test automation is the use of special software (separate from the software being tested) to control the execution of tests and the comparison of actual outcomes with predicted outcomes. Test automation can automate some repetitive but necessary tasks in a formalized testing process already in place, or add additional testing that would be difficult to perform manually.
http://en.wikipedia.org/wiki/Test_automation

Picking the Right Tools for the Job

There is no one-size-fits-all solution. You will need to take into consideration:

  • End-users device; Desktop and/or mobile?
  • (Web Applications) Web browsers
  • Technology used to make the tested application (ASP, WinForms, etc.)
  • Size of QA Team
  • Size of QA Automation Team
  • Price

What makes an automation tool good? Here are my thoughts:

  • Usability
    • Buttons and options are tidy and make sense as to their grouping
    • Programs flows like a book, left-to-right and top-to-bottom
  • Ease of Use
    • Beginners can record-and-click to create tests
    • Advanced users can write custom code
  • Productivity
    • A high purchase price is offset after a few months of test automation
    • Compare how long it takes to create and then modify a test in each tool
  • Comments
    • Each test, step, custom code has a description and/or comment

Automation Software

There are many tools in the wild but I can only speak as to the ones I have used.

These tools are good for scripting.

  • AutoIT
    • AutoIt v3 is a freeware BASIC-like scripting language designed for automating the Windows GUI and general scripting.
    • Example: I made a list of URLs to browse via IE to test a toolbar for memory leaks. Monitored the toolbar process via PerfMon.
  • AutoHotKey
    • Fast scriptable desktop automation with hotkeys
    • Example: I made a script that simulated a key press every 10 seconds.
  • Chromium browser automation
    • Extension for automating chromium browser, Create project -> Record -> Edit Automation -> Manage -> Play
    • Example: I recorded myself trying to log in to a site. Played back 10 times to see if the account got locked or if the time between requests was extended after each bad attempt.
  • PowerShell
    • Comes with most Windows OS distributions
    • Supports .Net framework
    • Example: I wrote a script that copies the latest build from the build machine to each test environment and installed that latest build. The scripts also sent out an email when it successfully ran a warm-up test, telling QA the build was ready to be tested.

These tools are good for testing:

  • Selenium
    • Made by eBay using java tools
    • Can run test in JUnit or NUnit
    • Limited browser support
    • Free
  • Telerik Test Studio
    • Supports most modern browsers
    • Supports Silverlight and WPF applications
    • Framework is available for free
    • Integrates with TFS
  • Microsoft Test Manager
    • Part of Visual Studio (Test Professional, Premium or higher)
    • Same browser support of Selenium
    • Test plan and test case repository

Recommendations

For scripting, I prefer PowerShell. It is fully supported by Microsoft and comes pre-installed on most Windows OS deployments.

For testing, I prefer Test Studio. It has quarterly releases by Telerik and hits all the right notes in what I consider a good tool.

Why Telerik Test Studio?

  • Usability
    •  Layout is clear and familiar (follows MS Office lead)
  • Ease of Use
    • Beginners can record-and-click to create new tests. They can also edit tests, like modify the find logic of an html element, rather easily.
    • Advanced users can write custom code in VB or C#.
  • One environment
    • In Selenium you need separate programs to create, edit, and playback test.
    • In Test Manager you need to have Visual Studio Pro (or higher) to convert test cases from recorded actions to automated tests.
  • Compatibility
    • Browsers: Internet Explorer, Chrome, Firefox, Safari
    • Application Platforms: Silverlight and WPF
  • Productivity
    • Tests and created and modified quickly.
  • Comments
    • Each Test and Test List has a description. Each Test Step can be commented or have its description altered.

Was there a close second?

Microsoft Test Manager is evolving with each iteration and could come to surpass HP’s Quality Center for enterprise solutions. It could overtake Telerik if it lets the Test Professional edition write automated tests and those tests are in a separate solution for the application.

Strengths

  • Test plans and test cases are saved as part of the project
  • Manual testing has great features
    • Sidebar to step through test cases and mark as pass or fail
    • Take photos or videos of testing
    • Attach bugs in TFS

Weaknesses

  • Fragmented IDE requirements
    • Must have Test Professional to make tests but must have Visual Studio Professional (or higher) to convert tests to automation
  • Limited browser support
  • High initial cost

Best Practices

  •  Try not to rely on existing data
    • Your data will typically be wiped when a new build is deployed or when a vm is snapshot reverted
  • Check for dependencies
    • Do not fail an automated test that deletes a record if the record cannot be found.
      • This is a setup issue. The functionality is not being tested, only it’s error handling.
  • One failed test should not invalidate the rest of testing
    • Because the homepage test fails does not mean you can’t test other pages
  • Do not leave artifacts
    • Added a record? Why not modify it and then delete it to?
      • This will test CRUD operations
  • Try not to rely on known IDs
    • Data can be  entered in asynchronously or tests ran out of order
    • Instead try to look up the information first, see Check for dependencies (above)
  • Make test independent
    • Test 1: Login
    • Test 2: Login and Add Record
      • A test list that run Test 1 then Test 2 will run the login code twice.
        • This will add time to the overall test time
        • This will allow you to run the test individually after a test list fails
  • Use modular test and code
    • Never repeat code, instead write a module or function to handle things like logging on

Security Software

OWASP offers an incredible security tool, called Zed Attack Proxy. Beginners can simply enter the URL to their web application and click go.

Excel Macros Primer

About Excel Macros

The macro code language for most Office programs, including Excel, is Visual Basic for Applications (VBA). You may have recorded macros in Excel by stepping through actions that the program saves for you. When you record a macro, Excel records the VBA code describing your actions in a module attached to the workbook.
http://office.microsoft.com/en-us/training/introduction-to-terms-RZ001150634.aspx?section=3

Downloading Excel Macros

Try Office 365 Home
Office 365 gives you the power and flexibility to get things done from virtually anywhere.
http://office.microsoft.com/en-us/try/

Starting Excel Macros

IMPORTANT: This feature isn’t available in Office on a Windows RT PC.
Excel provides several ways to run a macro. A macro is an action or a set of actions that you can use to automate tasks.
http://office.microsoft.com/en-us/excel-help/run-a-macro-HA102748985.aspx

  1. Open Excel
  2. Do one of the following:
    1. Click File, click new, select New Workbook
    2. Click File, click open, select your macro-enabled workbook
  3. Click the View tab
  4. The Macros button should appear on the right side of the Ribbon(top menu bar).

Hello World!

Recording

  1. Click the Macros button on the View tab
  2. Select Record Macro…
  3. Click OK
  4. Type Hello World! in cell A1
  5. Click the Macros button again and select Stop Recording

Editing

  1. Click the Macros button on the View tab
  2. Select View Macros
  3. Select your macro from the list
  4. Click the Edit button

Your code should look something like this:

Sub Macro1()
'
' Macro1 Macro
'

'
    Range("A1").Select
    ActiveCell.FormulaR1C1 = "Hello World!"

End Sub
"Hello World!"

Playback

  1. Clear the contents of cell A1
  2. Click the Macros button on the View tab
  3. Select View Macros
  4. Select your macro from the list
  5. Click the Run button
"Hello World!"

Variables

You often have to store values when you perform calculations with Visual Basic. For example, you might want to calculate several values, compare them, and perform different operations on them, depending on the result of the comparison. You have to retain the values if you want to compare them.
http://msdn.microsoft.com/en-us/library/cd6hcy37.aspx

Sub Macro1()
 Dim message As String
 message = "Hello World!"
 Range("A1").FormulaR1C1 = message
End Sub
"Hello World!"

Functions

All executable code must be inside a procedure. Each procedure, in turn, is declared within a class, a structure, or a module that is referred to as the containing class, structure, or module.
To return a value to the calling code, use a Function procedure; otherwise, use a Sub procedure.
http://msdn.microsoft.com/en-us/library/sect4ck6.aspx

Sub Macro1()
    ' Call the function
    writeMessage
End Sub

Sub writeMessage()
    Dim message As String
        message = "Hello World!"
    Range("A1").FormulaR1C1 = message
End Sub
"Hello World!"

Passing Variables to Functions

In this example, the function writeMessage will accept a value and declare it as a string.

Sub Macro1()
    writeMessage ("Hello World!")
End Sub

Sub writeMessage(message As String)
    Range("A1").FormulaR1C1 = message
End Sub
"Hello World!"

Error Handling

In Microsoft Excel, you can use the On Error statement in a Microsoft Visual Basic for Applications macro (Sub procedure) to trap errors and direct procedure flow to the error-handling statements in a procedure. By using error handling, you make your macros and your application easier to use by intercepting run-time errors before the user sees them.
http://support.microsoft.com/kb/141571

Sub Macro1()
    ' Run the Error handler "ErrHandler" when an error occurs.
    On Error GoTo Errhandler

    writeMessage ("Hello World!")

    ' Disable the error handler.
      On Error GoTo 0

    ' Exit the macro so that the error handler is not executed.
      Exit Sub

Errhandler:
      ' If an error occurs, display a message and end the macro.
      MsgBox "An error has occurred. The macro will end."

End Sub

Sub writeMessage(message As String)
    Range("A1").FormulaR1C1 = message
End Sub
"Hello World!"

Further Reading

Get in the Loop with Excel Macros

  • Microsoft’s training course about Excel Macros

Excel VBA Tutorial

  • Excel Easy’s VBA (Visual Basic for Applications) guide