1. Tools Support for Testing: Introduction
As seen in the previous sections, there are many tasks and activities that need to be performed during the testing process. In order to assist in making the testing process easier to perform and manage, many different types of tools or software applications have been developed and used for a wide variety of testing tasks. Some of them have been developed inhouse by an organization’s own software development team and testing department. Others have been developed by software houses to sell to organizations that perform testing. This section discusses the potential benefits of using tools. We will see that success with tools is not guaranteed even if the most expensive tool is acquired. There are also risks in using tools even though we will not mention any specific tool by name. But we will describe the most commonly used types of test tools and concludes with a process for introducing a tool into a test organization.
We can see various test tools used for one or more activities that support testing. These include tools that are directly used in testing, such as test execution tools and test data preparation tools. Tools that help to manage requirements, test cases, test procedures, automated test scripts, test results, test data and defects, and for reporting and monitoring test execution tools that are used for investigation and evaluation. For example, tools that monitor file activity for an application. Any tool that assists in testing a spreadsheet or email application are also a test tool. In this meaning, test tools can have one or more of the following purposes, depending on the context improve the efficiency of test activities by automating repetitive tasks or tasks that require significant resources when done manually. For example, test execution, migration testing improve the efficiency of test activities by supporting manual test activities throughout the testing process. Improve the quality of test activities by allowing for more consistent testing and a higher level of defect vbility. Automate activities that cannot be executed manually, for example, largescale performance testing increase reliability of testing, for example, by automating large data comparisons or simulating behavior.
2. Test Tools Classification: Introduction
Tools can be classified based on several criteria, such as purpose, pricing, license model, for example, commercial or open source, and technology. Used tools are classified in this syllabus according to the test activities that they support. Some tools clearly support one or mainly one activity. Others may support more than one activity but are classified under the activity with which they are most closely associated. Tools from a single provider, especially those that have been designed to work together, may be provided as an integrated suite and bundled into one package. ISTQB classified the testing tool into the following categories tool support for management of testing and tests tool support for static testing tool support for testing, design and implementation tool support for test, execution and logging tool support for performance and monitoring tool support for specific testing needs.
First, notice that in the syllabus that all the previous points are of type K one, which means you only need to remember them. You only need to know that they exist. Second, I want you to think carefully about each title. It says tool support for whatever tool support, not tool that will do whatever. This means that those tools are helper tools, not necessarily tools that will do the actual job. So for example, tool support for management of testing are tools that will be used during the management of testing activity and not necessarily our tools that will do the actual management of testing for us. So an email application can be considered as tool support for management of testing and tests and so on. Some tools offer support that is typically more appropriate for developers. Tools that are used during component and integration testing. For example, such tools are marked with D next to its name or title. There are something extra that the Ice UKB mentioned in its foundation curriculum, so I will share it here with you. A tool that measures some aspect of software may have unexpected side effects on that software.
For example, a tool that measures timing for non functional performance testing needs to interact very closely with the software in order to measure it. A performance tool will set a start time and a stop time for a given transaction in order to measure the response time, for example. But the act of taking that measurement, for example, storing the time at those two points, could actually make the whole transaction take slightly longer than it would do if the tool wasn’t measuring the response time. Of course, the extra time is very small, but it is still there, it still exists.
This effect is called the probe effect, and tools suffer from this effect are called intrusive tools. We will learn now about many types of tools. We will not mention any tool by name. Even if I needed to mention any tool by name, then it’s only for clarification purposes, you don’t need to memorize it. Therefore, we only care about the kind of tools questions in the exam related to this part are asking about what does the tool do? Main feature, which category does it belong to? Which step in the testing process does it use in? And finally, who uses it? So I will try to highlight those points when I talk about each tool.
3. Test Tools Classification: Part 1
This broad category contains any tool that provides support for managing the tested process over the entire software development lifecycle. Think of most of those tools as databases where everyone is using those tools either by adding content to the database or by getting reports from those databases to help taking decisions to direct the testing activity tools are under this category are Test management tools. Everyone is using the test management tools, but those tools are mainly used by test managers and test leaders, mainly to create reports that would help them manage the testing activities and make decisions. Such tools are also called ALM, which stands for Application Lifecycle Management Tools. We also have under this category requirements Management Tools. Tools that help managing the requirements are also tools that can be very helpful in managing the testing process. Such tools help to trace tests to requirements and requirements tests. Everyone is using the requirement management tools, but particularly system analysts. Defect Management Tools We have talked about defect management tools in a previous section. Defects are used heavily during the implementation and the execution process and used by Everyone, but mainly are used by testers and developers. Configuration management Tools and also we have talked about configuration management tools before, and configuration tools are used by Everyone and Last.
Continuous Integration Tools continuous integration is built on other software development best practices, including automated testing, version control, build automation, and automated deployments. Each one of these practices of continuous integration has its own collection of tools, and as you see, it’s mainly used by developers. If we can talk more about test management tools, we can say that test management tools often need to interface with other tools or spreadsheets for various reasons, including to reduce useful information in a format that fits the needs. Of the organization to maintain consistent flexibility to requirements in a requirement Management Tool to link with test object version information in the Configuration Management Tool. This is particularly important to consider when using an integrated tool, for example Application Lifecycle Management, or ALM as we have said before, which includes a test management module and possibly a defect management system as well as other modules, for example, project, schedule and budget information that are used by different groups within an organization. Another category of tools are tool support for static testing. Static testing tools are associated with the activities and benefits described in the static testing section we have talked about before. Examples of such tools include tools that support reviews.
We have discussed the review process and different review types in a previous section. Review process support tools are any tools that can be used to help or support the review process. We also have static analysis tools. This is mainly or normally used by developers as part of the development and component testing process prior to dynamic testing. The input to those tools is usually the source code, and the tool will list all the possible defects it finds in the code without running it. Another category for tool support for test design and implementation. Test design tools aid in the creation of maintainable worker products in test design and implementation, including test cases, test procedures and test data. Examples of such tools include test design tools, modelbased testing tools, test data preparation tools, acceptance testdriven development at DDD and behavior driven Development BDD Tools and last test driven Development TDD tools, which are used by developers. Let’s talk a little bit about each one of them. Test Design Tools Test design tools are used to generate automatically test cases with inputs and expected results, or at least test inputs only. There is a common term in the testing industry called test Oracle. Oracle here has nothing to do with the famous database provider Oracle. Oracle is an English word that means computerized, robotic or automatic.
Therefore, a test oracle means any computerized means to help generate test cases or test inputs. We expect to provide such tools with some kind of input. For example, the requirements document, the design document, the graphical user interface or the database model and the output from those tools would be some sort of ready made test cases or test inputs. Imagine if we have a document about the graphical user interface of a system and the fields in each screen. I guess we can create a tool which can take such document as an input and the output will be distributed for the number of fields. We would have a test case for the above range, a second district case for within range, and a third district case for the below range using equivalence partitioning test design technique. If ever messages are stored in this document as well, then the expected result for each test case can be specified as well. Just as like creating test cases automatically, we can have tools that automatically create that test data and all the test results for us. Needless to mention, we may end up with the problem of having too many tests and then we would need to find a way of identifying the most important tests to run. Second, model Based Testing Tools as we have mentioned before, model based testing is a technique which is used by the team of testers to check the runtime behavior of a software under test against the predictions made by a formal specification of model. Model based testing of MBT tools enable a functional specification to be captured in the form of a model such as an activity diagram.
This task is generally performed by a system designer. The MBT tool interprets the model in order to create test cases specifications which can then be saved in a test management tool and or executed by a test execution tool. If you need to learn more about model based testing, you can refer to IST KB MBT Foundation Level Model Based Testing syllabus. Third category under the tool support for design and implementation is Test Data Preparation tools. Now, imagine if you want to have a test case where you need to test adding a million rows of data to a table in a database which already holds millions of records of data as a stress testing test case, would you be able to create the data needed manually? No way. Tested data preparation tools enable data to be selected from an existing database or created, generated, manipulated, and edited for views in tests. The most sophisticated tools can deal with a range of files and database formats. In some cases, tools that support test design and implementation may also support test execution and logging, or provide their outputs directly to other tools that support test execution and logging. Next, we will talk about Acceptance Test Driven Development ETDD and Behavior Driven Development BDD tools behavior Driven Development and Acceptance test Driven Development involve generating test conditions and test cases from user stories and acceptance criteria period to coding. They also verify, validate and detect defects in the user stories and acceptance criteria. You can learn more about those techniques in the Isqb Foundation Level Agile Testing extension. Last Test Driven Development TDD Tools test Driven Development, or TDD, is an approach with the exact reverse of traditional development. In this approach, testing is done first and then the code is written. As you can see, tools that support TDD are mainly used by.
4. Test Tools Classification: Part 2
Tools support for Test Execution and Logging Many tools exist to support and enhance test execution and logging activities. All the types of tools mentioned under this category are used in the execution part of the testing process. Examples of these tools include test execution tools, for example, to run with duration tests cover edge tools, for example requirements coverage or code coverage. Test harnesses and unit test Frameworks let’s talk about each of them in a little bit of detail. Test Execution Tools When people mention testing tool, Test Automation Tool is the one that comes to mind. Test Automation Tool or Test Execution tool is a tool that can run tests for you automatically. So instead of executing a test procedure manually by hand, we would write the test procedure in a way that the computer will understand and the computer can execute it automatically. To write the test procedure in a way that the computer will understand would mean that we might need a specific language to write the testable procedure, which in this case will be called test script. So a test script is a testable seizure written using a scripting language such as scripts would need to store the inputs that will be used by the script and maybe store the expected outcome as well. If the tool can compare the expected result versus the actual result, then the tool should usually provide a test log for each test run with the result of the test execution and test comparison. If the tool doesn’t store the expected outcome of the script and therefore cannot compare the actual outcome of the test versus the expected result, then we could call the tool semi automatically. In this case, we would need to store the actual outcome of the test execution for further investigation.
They can also be used to record tests and usually support a scripting language or Guid based configuration for parameterization of data and other customizations in the test. Although they are commonly referred to as testing tools, they are actually best used for regression testing, so they could be referred to as regulation testing tools rather than testing tools. A test execution tool most often runs tests that have already been run before. One of the most significant benefits of using this type of tool is that whenever an existing system is a change, for example, for a defect fix or an enhancement, all of the tests that were run before could potentially be run again to make sure that the changes have not disturbed the existing system by introducing or revealing a defect. We will talk more about those types of tools in a separate lecture. Coverage Measurement Tools We have learned how to measure test coverage in a previous section, but do we actually do it manually? Thankfully not. Coverage tools are here for the risk queue and they are mainly used by developers. Last, we have test Harnesses unit test framework tools also used by developers. Remember the steps and drivers we mentioned before when we talked about unit testing and integration testing. Well, the steps and drivers are types of test harnesses. Test harness is any code written by the developer to help testing the code. There are also unit test frameworks that would help the developer to write such test code tools. Support for Performance Measurement and Dynamic Analysis Performance measurement and dynamic analysis tools are essential in supporting performance and load testing activities as these activities cannot effectively be done manually.
Examples of these tools include performance testing tools which are concerned with measuring the system performance to see whether or not the system will stand up at different circumstances. Under the umbrella of performance testing, there are different kind of testing, for example load testing, volume testing and stress testing. Also, we have monitoring tools. I want you to think of monitoring tools as task manager under the Windows operating system. It can show you measures and statistics about your software. How does your software use the CPU, memory networks and other system resources? I always advise or recommend to my sisters to keep the task manager open and visible all the time while testing dynamic analysis tools which are used by developers. We have mentioned before static testing tools and they are tools that could find defects in the software while it’s not running. And now we are talking about dynamic testing tools. So from its name, what do you think it do? Correct dynamic analysis tools are tools that can find effects in the software while it is running. Last, we have tools of both for specialized testing needs. In addition to tools that support the general testing process, there are many other tools that support more specific testing issues. Examples of these tools include data quality assessment, data conversion and migration usability testing accessibility testing localization testing which is converting the software to a specific language to be local in some country security testing portability testing for example, testing software across multiple supported platforms.
5. Special Considerations for Test Execution Tools
In order to have a smooth and successful implementation. There are a number of things that should be considered when selecting and integrating test execution tools into an organization. As we mentioned before, test execution tools test the software using automated test scripts. This type of tools often require significant effort in order to attract achieve significant benefits. Test execution tools are used by testers. Therefore, any tester who wishes to use a scripting test execution tool will need to use programming skills to create and modify those scripts. First, let’s talk about capture and playback test Execution Tools the idea behind Capture Playback Tools is that while you are running your manual tests, the tester assembly turns on the capture feature of the tool. The tool would record everything the tester do his mouse clicks, keyboard clicks, mouse movements, everything. When done, the tester can save what was recorded to play it later. When played back, the recorded actions, all the testosterone’s actions will exactly be repeated.
Mouse clicks, keyboard clicks, mouse movements, everything. So if the software and its surrounding environment is exactly the same as when we recorded our test, then the playback will be some sort of repeating the test again and again as many times as you want. But all you have to do is just to hit play. A capture script is a linear representation with specific data and actions as part of each script. The concept of such type of test execution tools breaks down when you try to replay the captured tests and the software, or the surrounding environment of the software. It changed it from its original state. Imagine if there was a button at a specific location on the screen and you clicked that button.
The tool would save the horizontal and vertical location of your mouse down click if for any reason the button moved its location. Then when you try to lay back the script, your mouse down click won’t actually hit the button at all and you would need to either rerecord the script or fix the script. Also, the script may rely heavily on the circumstances, state and context of the system at the time the script was recorded. For example, the script may have used the file in a specific folder. If that file moved, or if the folder moved, then the test will break. The test input information is hard coded, meaning that the test input data is embedded inside the individual script for each test. So it will always use the same data every time you run the script again. Any of these things can be overcome by modifying the scripts, but the effort would be more complicated than just re recording the script again. Also, this approach doesn’t scale to large number of test scripts.
Capture playback tools can be of great benefit during exploratory testing. Remember exploratory testing, one of the experience based test case design techniques? In exploratory testing, the tester blaze with the software until she hits a bug. Sometimes it would be very hard for the tester to remember the steps that we’d used the bug. But if we were using the capture tool, we would have saved a list of the tester’s actions, which she can play back to reduce the bug so we can document and report the defect. The latest generation of these tools which take advantage of smart image capturing technology has increased the usefulness of this class of tools. Although the generative scripts, it still require ongoing maintenance as the system’s user interface evolves over time. Second type of test execution tools are data driven test execution tools. Another type of test exemption tools, which is of advanced capability over captured scripts is using a datadriven approach.
A datadriven testing approach separates out the test inputs the data and usually store it into an external file, let’s say a spreadsheet. So with know how to coded data, our test script will be more generic that can read the input data and execute the same test script with different data every time. Testers who are not familiar with the scripting language can then create the tested data for those redefined scripts. A more advanced data driven technique is where instead of using datablazed in a spreadsheet, the data is generated using algorithms based on configurable parameters at one time and supplied to the application. For example, a tool may use an algorithm which generates a random data or ID or password or something of that kind.
The last type we will talk about is key driven test execution tools. Key driven scripts give significantly more benefits than the previous two types we’ve talked about. In a keyword driven testing approach, the spreadsheet contains keywords describing the actions to be taken, also called the action words besides the test data, testers, even if they are not familiar with the scripting language, can then define tests using the keywords which can be tailored to the application being tested. Further details and examples of data driven and keyword driven testing approaches are given in the ISTQB Advanced level test automation engineer Syllabus. The above approaches require someone to have expertise in the scripting language, testers developers, or specialists in test automation. Regardless of the scripting technique used, the expected results for each test need to be compared to actual results from the test, either dynamically while the test is running, or stored for later post execution comparison. Automation testing became a trend these days, so it’s a skill that you might consider to acquire if you haven’t already done so.
6. Benefits and Risks of Test Automation
There are potential benefits and opportunities with the use of tools in testing, but there are also risks. This is particularly true for test execution tools, which is often referred to as test automation. Benefits potential benefits of using tools to support test execution include repetitive work is reduced. Imagine, for example, if you had to run the same test case tens of times to prove that it’s already not working before you create a defectory board against it. It’s boring and wastes time. Such a task, it can be done by a tool, would be a great benefit to reduce repetitive work, greater consistency and reputability. Using the same example, there’s a big chance you might make a mistake if you had to run the same test case tens of times. Such mistakes will make your work not consistent and not repeatable. A tool will exactly reproduce what it did before, so each time it’s run, the result is consistent. Objective Assessment if you ask a tester how much coverage you have achieved, they might tell you the subjective opinion or tell the numbers that you wanted to hear.
Using the tool to measure coverage, for example, will always give you an objective assessment. Ease of access to information about tests or testing. We mentioned in test monitoring that we will be collecting lots of data. This data needs to be stored, communicated and maintained. We will surely need tools to do that. Risks although there are many benefits that can be achieved using tools to support the testing activities, but many organizations have not achieved the benefits they expected. Istkbull seems to warn us of the risks of using a tool by listing more than ten plus risks. After all, it’s a huge investment and if it didn’t work, it will be a waste of money, time and effort. Simply purchasing a tool is no guarantee of success and achieving benefits, just as buying a gym machine at home doesn’t guarantee that you will be better if you don’t know how to use it. Risks of using tools include unrealistic expectations for the tool. Unrealistic expectations may be one of the greatest twists to success with tools. I have seen companies that thought that purchasing a testing automation tool would make them get rid of the testers altogether. This is very unrealistic and will never happen. Companies should have clear objectives for what the tool can do and that those objectives are realistic. Underestimating effort for the initial introduction having purchased a tool, you will need to train some people to use the tool. There could be some resistance from some people. There will be technical problems to overcome. Underestimating effort to achieve significant benefits. There’s a learning curve to anything you learn. Remember when you first used Microsoft Word, for example, or Excel? When was that?
Like ten years ago or maybe more. Can you claim that you are a professional user of Microsoft Word? Most of us cannot claim that. Most of us used just a few features of the software, leaving hundreds more that we are not aware of. For example, I remember I asked my assistance once to send an email to 500 users. A couple of days later I asked her if she sends an email. She said there’s like 300 remaining. I taught her then about the mailing functionality in Microsoft Word, where we linked Word with the Excel sheet that has the names and emails of the users and Voila. The email was sent to 500 people in just five minutes. Effort to maintain the test assets insufficient planning for maintenance of the assets that the toolboard uses is a strong contributor to tools being dumped. For example, people forget that they might need to update the test scripts that will be used by the test automation tools every time there’s a change in the software or the environment used.
Over reliance on the tool. If you buy the smartest phone, you still need to know how to use it. It’s the smartness comes from how good are you in using it? Same thing about testing tools. The issue can help, but it doesn’t replace the intelligence needed to know how best to use it and how to evaluate current and future uses of the tool. The tool will not understand the context or the domain of the application under test as much as you do. Neglecting interoperability between critical tools also, the interrelationship between the tools is important. For example, if you purchased a design specification tool that doesn’t work with the test execution tool, then you would need some effort to convert the output of the design specification tool to something that the test execution tool can understand. That’s an extra effort. We also have risks such as vendor going out of business, poor vendor for support, and inability to support a new platform.
Now imagine that the vendor where you bought your tool form went out of business or decided not to support a specific new platform. All of these are very bad at customer service. You would be stuck with whatever tool you have now in hand for some time. Neglecting version control same thing if you cannot upgrade your tool to a new version, you would miss new features and bug fixes in that version. Risk of suspension of open source or free tool project if the tool is an open source or a free tool and it’s being decided to suspend that tool, you would also be stuck and it would take a huge effort to move to another tool. I remember we had to spend days moving our data when Microsoft decided to suspend source Safe, which is a configuration management tool. And last, there may be no clear ownership of the tool, for example, for monitoring, updates and so on.
7. Effective Use of Tools
Main principles for Tool Selection we have seen from the previous lecture that there are many risks in using tools to support the testing activity. That’s why we need to be very careful when introducing a tool to our company to avoid falling into any of the mini traps we mentioned before. To get an idea what we are trying to achieve here, imagine if I give the smartest phone in the market to Illiterate Man. Would he benefit from it either? So it doesn’t matter how good the tool is, but we still need to be ready for it. The place to start when introducing a tool into an organization is not with the tool itself, it’s with the organization. In order for a tool to provide benefit, it must match a need within the organization and solve that need in a way that both effective and efficient. The tool should help to build on the strengths of the organization and addresses its weaknesses. The organization needs to be ready for the changes that will come with the new tool. If the current testing practices are not good and the organization is not mature enough, then it’s generally more cost effective to improve the testing practices rather than to try to find tools to support poor practices.
Automating chaos just gives faster chaos. Of course, we can sometimes improve our own processes in parallel with introducing a tool to support those practices, and we can pick up some good ideas for improvement from the ways that the tools work. However, be aware that the tool should not take the lead, but should provide support to what your organization defines. The main considerations in selecting a tool for an organization include assessment of organizational maturity, strengths and weaknesses, and identification of opportunities for an improved test bosses supported by tools. For example, you should ask yourself will we be ready to change to adopt the new tool? Number two identification of opportunities for an improved test process supported by tool understanding of the technologies used by the test object or test objects in order to select a tool that is compatible with the technology. The build and continuous integration tools already in use within the organization in order to ensure tool compatibility and integration.
Evaluation against clear requirements and objective criteria, you should ask yourself do we have clear requirements and objectives for the new tool? Do we know exactly what we need to achieve? Number six consideration of whether or not the tool is available for a free trial period and for how long evaluation of the vendor, including training, support and commercial aspect or support for non commercial, for example, open source tools. Identification of internal requirements for coaching and monitoring in the use of the tool. Do we know how we will coach our employees to use the tool? Do we know how we will mentor the usage of the tool to make sure it meets its objectives? Number nine evaluation of training needs considering the current test teams test automation skills we should ask ourselves do we have a training plan to train our employees to use the tool? Number ten consideration of bonds and cons of various licensing models for example, commercial or open source estimation of cost benefit ratio based on a concrete business case with the benefit outweigh the cost of the tool? Can we have a proof of concept by using a test tool during the evaluation phase to establish whether it performs effectively with the software under test and within the current infrastructure or to identify changes needed to the infrastructure to effectively use the tool or not? Pilot projects for introducing a tool into an organization once a tool is purchased and that’s very important to understand once the tool is already purchased, we should gradually introduce the tool to the different teams in the organization and it should start with a pilot project. A pilot project means using the tool on a very small scale with sufficient time to explore different ways of using the tool. So a pilot project should have the following objectives gaining indepth knowledge about the tool understanding both its strengths and weaknesses evaluating how the tool fits the existing processes and practices and determining what would need to change deciding on standard ways of using, managing, storing and maintaining the tool and the test assets. For example, deciding on naming conventions for files and tests selecting coding standards creating libraries and defining the modularity of test suites assessing whether the benefits will be achieved at a reasonable cost understanding the metrics that you wish the tool to collect and report and configuring the tool to ensure these metrics can be captured and rebooted. Last, we will talk about success factors for tools. As we have said, success is not guaranteed or automatic when implementing a testing tool, but many organizations have succeeded. Here are some factors for evaluation implementation, deployment and ongoing support of tools within an organization that have contributed to success.
Incremental roll out after the pilot to the rest of the organization. Adopting and improving processes testware and tool artifacts to get the best fit and balance between them and the use of the tool. Providing advocate training, coaching and monitoring for new users, defining and communicating guidelines for the use of the tool based on what we learned in the pilot project, implementing a way to gather usage information from the actual use, and implementing a continuous improvement mechanism. On how to use the tool monitoring tool use and benefits achieved providing support for the test team for a given tool gathering lessons learned from all teams. It’s also important to ensure that the tool is still technically and organizationally integrated into the software development lifecycle, which may involve separate organizations responsible for operations and or third party sublives. Thank you.
