Those of us who grew up watching The Jetsons are still waiting for our flying cars and robot servants to make life easier.  The thought of highly sophisticated robots driven by robust artificial intelligence programming sure sounded like a great idea.  The reality is that today’s robots play an integral part in modern manufacturing, but not quite in the way the cartoons of decades past had promised.  They do a great job performing precise but repetitive tasks under the watchful eye of human counterparts, but can’t just be left alone to run your entire operation.  For me, the same holds true for automated software testing.  The idea of software tools that test other software sounds very promising, but it’s important to keep a few things in mind when deciding if automated testing is right for your software project:

  • Automated testing isn’t a “cure all” for your testing challenges.
  • It’s not a fit for every project.
  • Be sure you understand the limitations.
  • Be clear about your goals and what you expect to get out of it, especially your ROI goals.

Just as today’s manufacturing robots are good for some tasks like welding the frame on an automobile, but not suited for installing the radio and dashboard components, it’s important to understand the limitations of automated software testing.  There are some instances where exercising repetitive tasks under the watchful eye of a skilled QA staff member can be very beneficial, but that won’t always cover your testing needs.  There are still many cases where traditional manual software testing will yield faster results and provide broader test coverage.

Automated testing yields the greatest benefit on mature products.

For the most part, automated testing relies on recording testing steps and playing them back to ensure the software behaves the same way with each subsequent playback.  When the software is unchanged, this often allows for the “push button” execution that we all want where you just set the process in motion with a few mouse clicks and wait for the results.  When the software UI is still in flux, however, time has to be spent adjusting the scripts to accommodate the changes.  This means the greatest benefit with automated testing comes only after the user interface and workflow have been locked-down.  Depending on your project method, this may be quite late in the lifecycle.  The greatest efficiency comes with mature products where the UI is unlikely to change.

Paying someone to write and re-write automated scripts can be a money pit for startup projects.

Startup projects that evolve as the lifecycle progresses are often not a good fit for automated testing.  If you are constantly tweaking the interface, or open to the idea of fundamental changes based on feedback, it may not be worth the investment to pay someone to write scripts and keep revising them as the application changes.  I’ve personally seen projects where the ROI on automated testing was never achieved because the scripts couldn’t be updated as fast as the interface was changing. For projects that haven’t yet reached maturity, manual software testing would have been a better use of resources until the interface stabilized.

Don’t forget to leave the happy path.

Those of us in QA often talk about the “happy path” in testing.  The happy path is the workflow that the business analysts and User Experience designers envisioned the users following when using the application.  The happy path is often the one that’s scripted in automated testing, right down to deciding on a mouse click versus using the keyboard to select something.  We all know, however, that defects and unexpected behavior are often found when we test scenarios that wander off the happy path.  End users have an uncanny ability to discover alternate routes off the happy path, and can find way to use the application that you never anticipated.  Since automated testing always follows the same recorded paths, it’s important to understand that  automated testing won’t find the type of behavior that you get from manual software testing.

So which is better, manual or automated testing?  The answer is: both. Each has its place in your SDLC (software development life cycle), and each has its pros and cons. To choose what is best for your project entails understanding what you hope to achieve, and being realistic about how mature your product is, and how likely it is to change. Even if your project can benefit from automated testing, there are some aspects that can only be addressed by an experienced manual software testing team. Your challenge is to find a balance, like the automotive industry, in combining automation with traditional hands-on attention to your product. Have questions about software testing or need help to choose? Contact us today, we’re happy to help!