Automated Software Testing Tools

About Automated Software Testing

In software testing, test automation is the use of special software (separate from the software being tested) to control the execution of tests and the comparison of actual outcomes with predicted outcomes. Test automation can automate some repetitive but necessary tasks in a formalized testing process already in place, or add additional testing that would be difficult to perform manually.
http://en.wikipedia.org/wiki/Test_automation

Picking the Right Tools for the Job

There is no one-size-fits-all solution. You will need to take into consideration:

  • End-users device; Desktop and/or mobile?
  • (Web Applications) Web browsers
  • Technology used to make the tested application (ASP, WinForms, etc.)
  • Size of QA Team
  • Size of QA Automation Team
  • Price

What makes an automation tool good? Here are my thoughts:

  • Usability
    • Buttons and options are tidy and make sense as to their grouping
    • Programs flows like a book, left-to-right and top-to-bottom
  • Ease of Use
    • Beginners can record-and-click to create tests
    • Advanced users can write custom code
  • Productivity
    • A high purchase price is offset after a few months of test automation
    • Compare how long it takes to create and then modify a test in each tool
  • Comments
    • Each test, step, custom code has a description and/or comment

Automation Software

There are many tools in the wild but I can only speak as to the ones I have used.

These tools are good for scripting.

  • AutoIT
    • AutoIt v3 is a freeware BASIC-like scripting language designed for automating the Windows GUI and general scripting.
    • Example: I made a list of URLs to browse via IE to test a toolbar for memory leaks. Monitored the toolbar process via PerfMon.
  • AutoHotKey
    • Fast scriptable desktop automation with hotkeys
    • Example: I made a script that simulated a key press every 10 seconds.
  • Chromium browser automation
    • Extension for automating chromium browser, Create project -> Record -> Edit Automation -> Manage -> Play
    • Example: I recorded myself trying to log in to a site. Played back 10 times to see if the account got locked or if the time between requests was extended after each bad attempt.
  • PowerShell
    • Comes with most Windows OS distributions
    • Supports .Net framework
    • Example: I wrote a script that copies the latest build from the build machine to each test environment and installed that latest build. The scripts also sent out an email when it successfully ran a warm-up test, telling QA the build was ready to be tested.

These tools are good for testing:

  • Selenium
    • Made by eBay using java tools
    • Can run test in JUnit or NUnit
    • Limited browser support
    • Free
  • Telerik Test Studio
    • Supports most modern browsers
    • Supports Silverlight and WPF applications
    • Framework is available for free
    • Integrates with TFS
  • Microsoft Test Manager
    • Part of Visual Studio (Test Professional, Premium or higher)
    • Same browser support of Selenium
    • Test plan and test case repository

Recommendations

For scripting, I prefer PowerShell. It is fully supported by Microsoft and comes pre-installed on most Windows OS deployments.

For testing, I prefer Test Studio. It has quarterly releases by Telerik and hits all the right notes in what I consider a good tool.

Why Telerik Test Studio?

  • Usability
    •  Layout is clear and familiar (follows MS Office lead)
  • Ease of Use
    • Beginners can record-and-click to create new tests. They can also edit tests, like modify the find logic of an html element, rather easily.
    • Advanced users can write custom code in VB or C#.
  • One environment
    • In Selenium you need separate programs to create, edit, and playback test.
    • In Test Manager you need to have Visual Studio Pro (or higher) to convert test cases from recorded actions to automated tests.
  • Compatibility
    • Browsers: Internet Explorer, Chrome, Firefox, Safari
    • Application Platforms: Silverlight and WPF
  • Productivity
    • Tests and created and modified quickly.
  • Comments
    • Each Test and Test List has a description. Each Test Step can be commented or have its description altered.

Was there a close second?

Microsoft Test Manager is evolving with each iteration and could come to surpass HP’s Quality Center for enterprise solutions. It could overtake Telerik if it lets the Test Professional edition write automated tests and those tests are in a separate solution for the application.

Strengths

  • Test plans and test cases are saved as part of the project
  • Manual testing has great features
    • Sidebar to step through test cases and mark as pass or fail
    • Take photos or videos of testing
    • Attach bugs in TFS

Weaknesses

  • Fragmented IDE requirements
    • Must have Test Professional to make tests but must have Visual Studio Professional (or higher) to convert tests to automation
  • Limited browser support
  • High initial cost

Best Practices

  •  Try not to rely on existing data
    • Your data will typically be wiped when a new build is deployed or when a vm is snapshot reverted
  • Check for dependencies
    • Do not fail an automated test that deletes a record if the record cannot be found.
      • This is a setup issue. The functionality is not being tested, only it’s error handling.
  • One failed test should not invalidate the rest of testing
    • Because the homepage test fails does not mean you can’t test other pages
  • Do not leave artifacts
    • Added a record? Why not modify it and then delete it to?
      • This will test CRUD operations
  • Try not to rely on known IDs
    • Data can be  entered in asynchronously or tests ran out of order
    • Instead try to look up the information first, see Check for dependencies (above)
  • Make test independent
    • Test 1: Login
    • Test 2: Login and Add Record
      • A test list that run Test 1 then Test 2 will run the login code twice.
        • This will add time to the overall test time
        • This will allow you to run the test individually after a test list fails
  • Use modular test and code
    • Never repeat code, instead write a module or function to handle things like logging on

Security Software

OWASP offers an incredible security tool, called Zed Attack Proxy. Beginners can simply enter the URL to their web application and click go.