Understanding System Testing
In the world of software testing, system testing is the testing of a complete and fully integrated software product. System testing falls under the black box testing category of software testing. White box testing is the testing of the internal workings or code of a software application. In contrast, black box or system testing is the opposite. System testing involves the external workings of the software from the user’s perspective. White box testing requires extensive knowledge of the code and software design, but system testing requires no real knowledge of the software development process. It does require extensive knowledge of common black box testing procedures including knowing how to develop test cases and test suites. System testing should be done by people who are very thorough in nature. It is not meant to be done halfheartedly. Thoroughness is the key to good system testing practices.
What is involved in System Testing?
There are several components involved in system testing. Later in the tutorial we will discuss several types of system testing. Here we aim to show you the basic process. There is a test case example later in the tutorial that will help you understand the process. Here is what system testing involves:
- Black box testing of the finished application before it goes to market.
- Testing on fully integrated applications in order to test how components interact with one another and with the system as a whole.
- Very thorough testing of every input in the application to check for required outputs.
- Testing of the user’s experience with the application from start to finish.
- Building detailed test cases and test suites that test each aspect of the application as seen from the outside without looking at the actual source code. This is where the name black box testing comes from (you cannot see the insides of a black box).
That is a very basic description of what is involved in system testing. Continue reading to learn more about each item mentioned above and quite a bit more about what system testing is and about what system testing is not. Sometimes what it is not is just as important is what it is.
System Testing Versus Unit Testing
Let’s explore some of the similarities and differences between Unit Testing and System Testing. In general, unit testing is done during development of the software and system testing is done on the finished product. Also, unit testing is considered white box testing and system testing is black box testing. Those are the major fundamental differences. If we dig deeper we will discover many more differences and some similarities as well.
Differences Between Unit Testing and System Testing
|
Unit Testing
|
System Testing
|
- Usually done by the programmer who developed the application.
- Done by a testing agent with no prior knowledge of the application.
|
|
- Is done on small sections of code or individual functions and statements.
|
- Is done on the entire application after all its components have been integrated.
|
- Speed is an issue with unit testing because unit tests are commonly run dozens of times a day, sometimes hundreds.
- Speed is not as important in system testing because they are not run as many times in a day as unit testing.
- The code used to test in Unit testing can be very long, often longer than the actual code being tested
- In system testing the code used for testing is not as long as in unit testing and is smaller than the code being tested in most cases.
|
|
|
|
Similarities Between Unit Testing and System Testing
There are many similarities between unit testing and system testing. Although the similar items between the two are often performed in slightly different ways.
To illustrate a similarity of system and unit testing, we could use error testing as an example. In unit testing, additional code could be written to produce an error. We could then see how the application handles that error. If the tester is not happy with the way the error is handled, the source code can be altered to change it. On the other hand, in system testing, errors are thrown on purpose by entering invalid input. There is no code altering done during system testing. If an error is thrown while testing with a valid input, then the application is often returned to the developer to be repaired.
The major similarities of System Testing and Unit Testing are mostly in the general process. Both require a set of objects to be tested, a set of operators, an initial state and a goal state to complete the testing process. The path tested if often similar, but the methods and techniques vary greatly between the two testing types.
Test Cases and Test Suites
In order to test software applications, you must produce test cases. A test case is a set of variables or conditions used in determining whether an application is working properly or not. A group of test cases, known as a test suite, is needed to test any software application. To learn more about test cases for both unit testing and system testing, we have provided you with an example for each below.
Example Application
Below we use a tax calculator application as an example application to demonstrate testing cases for both unit and system testing. The application calculates the Earned Income Credit (EIC) of the user. For demonstration purposes, we will assume that the maximum amount a person can earn to be eligible for EIC is $10,000. Anyone making $10,000 or more will not receive any EIC. Anyone earning less than $10,000 will receive EIC. The relative inputs for the calculator would be the form fields you see in the image provided below:
Here is a screenshot of the EIC tax calculator that is used in the test cases explained below:
The EIC tax calculator in the above image is a web based application written in JavaScript. The process would be similar had it been written in a non-web based language such as C or C++. We used the calculator in the above image to provide readers with a unit test case example. Below that, we will provide a system test case example. We aim to demonstrate how each plays an important role in software testing.
Unit Test Case Example:
Testing done during the development of an application is Unit Testing and is done by the programmer on each section of code as it is written. To illustrate a unit test case example, we are going to take the first function of the tax calculator to demonstrate how a tester would perform unit testing on such a function. We will be testing the “contains” function that extends the array prototype in JavaScript. The “contains” function checks an array to see if it contains a specific value. PHP has a built in function for this called “in_array”. JavaScript does not, so we are going to add one to the array prototype. Here is the actual contains function:
Array.prototype.contains = function(obj) {
var i = this.length;
while(i--) {
if(this[i] === obj) {
return true;
}
}
return false;
}
In unit testing, a programmer would test this function after adding it to be sure it works. While it’s a simple function, there are still different ways it could be tested. A common unit testing method is to write another section of code to test it right after writing the function. Another slightly different method is to isolate the function to test it more thoroughly. This is a more thorough unit testing practice that involves copy and pasting the function to its own testing environment to test it by itself. Below we will examine this method of isolating the code to be tested in detail. It is especially helpful if you aren’t sure you have written the function correctly. You may need to do a little trial and error testing to make it work as intended. Isolating the code helps you find errors. You will know that any errors are coming from the code in question and not some other block of code in the application.
To test the “contains” function, we first build a basic HTML template as its isolated testing environment. Then paste the function in between script tags and save the file as test.html or something similar. Here is what you should have in your new test file:
<script>
//begin contains function:
Array.prototype.contains = function(obj) {
var i = this.length;
while(i--) {
if(this[i] === obj) {
return true;
}
}
return false;
}//end contains function.
//test contains function:
</script>
Notice how we put comments in the code to help understand what is going on (comments begin with //). Near the bottom of the code we left a space for testing the function. That is where we add our test code next.
In this case our function only requires one parameter, “obj”, which is an array object. Therefore we need an array for testing. The next step in writing our test case would be to create a JavaScript array object as demonstrated here:
// literal array of states that offer EIC
var stateEIC = ["DE", "DC", "IL", "IN", "IA", "KS", "LA", "ME", "MD", "MA", "MI", "MN", "NE", "NJ", "NM", "NY", "NC", "OK", "OR", "RI", "VT", "VA", "WI"];
We have built a literal array containing all states that offer EIC. We chose to use this as our test array because we will also need this array in our program later on. The application will calculate the eligibility for state level EIC by checking this array. If the state the user selected is in the “stateEIC” array, the user would get both state and federal EIC returns. If the user’s state isn’t in the array, they will only get federal EIC (if they meet other qualifying criteria). For this test case, we only need to test if the “contains” function works as required. We do so with a test function call. Next we set up our test inputs. We want one input that will test positive and one that will test negative so we can test all possible outcomes of the function. Set up your variables like this:
var state_1 = "OH";//has no state EIC
var state_2 = "NY";//has state EIC of 30% of fed EIC
We set the state_1 variable equal to the state of Ohio. Ohio has no state level EIC. We set the state_2 variable to the state of New York. New York does offer state level EIC. Now we have covered both possible cases for our “contains” function. When we test to see if OH is in the array, it should return false. When we test for NY it should return true. If it does not return the expected values, there is a bug and further trial and error programming is required. Here is what the two test cases should look like:
//TEST CASE 1 - state one test (OH Should return false):
if(stateEIC.contains(state_1)) {
alert(state_1+" has state level EIC");
}else{
alert(state_1+" Does Not have EIC");
}//end test case 1.
//TEST CASE 2 - state two test (NY Should return true):
if(stateEIC.contains(state_2)) {
alert(state_2+" has state level EIC");
}else{
alert(state_2+" Does Not have EIC");
}//end test case 2.
Again we clearly commented our code to identify “Test Case 1” and “Test Case 2” so we know what the expected outcome is going to be. If you have followed along in this tutorial, open your test HTML file in a web browser. You could also open the file named test.html [PROVIDE LINK TO DOWNLOAD test.html HERE] that goes with this tutorial. If the test code works you will see the following in your browser after opening the file:
Then click “OK” and you should see the following in your browser window next:
Above we have provided a very basic Unit Test Case example. Below we will provide a basic System Test Case example. Reading both will give you a pretty good idea of what is involved in each.
System Test Case Example
In system testing you are testing the application as a whole. The “contains” function used in the unit test case example above would still be tested in system testing. It is just tested in a completely different manner. In system testing, the “contains” function is tested by waiting until the function is used in the finished program. Then the testing agent tests it by making sure it produces the expected output for various tested inputs. Below we will outline how you would system test the application as a whole.
To begin system testing on our finished EIC calculator we would first plan out what inputs require testing. The inputs requiring testing include:
- Marital status
- Number of Children
- State Input
- Total Amount Earned
- The Five inputs under Other Income
- The Six check boxes
Ultimately in system testing we are testing the finished product to see if it works in accordance with the project requirements. For each of the inputs named above, we need to specify a minimum of two test cases as in the following written test case example:
|
Input
|
Test value
|
Dependencies
|
Expected outcome
|
Notes
|
| Marital Status |
Single |
Kids=1, state=NY, income=$9000, all checked |
Federal EIC = $805State EIC = $223 |
Passed |
| Marital Status |
Married |
Kids=1, state=NY, income=$9000, all checked |
Federal EIC = $905State EIC = $424 |
Passed |
| Kids |
2 |
Marital=single, state=OH, income=$9000, all checked |
Federal EIC= $1090State EIC = $560 |
Passed |
| Kids |
3 |
Marital=single, state=OH, income=$9000, all checked |
Federal EIC= $1490State EIC = $760 |
Passed |
| State |
OH |
Marital=married, Kids=1, income=$9000, all checked |
Federal EIC= $1090State EIC = $0 |
Passed |
| State |
NJ |
Marital=married, Kids=1, income=$9000, all checked |
Federal EIC= $7090State EIC = $560 |
passed |
| Income |
9999 |
Marital=married, Kids=1, state=OH, , all checked |
Federal EIC= $0State EIC = $0 |
Expected outcome was wrong check income variables |
| Income |
10000 |
Marital=married, Kids=1, state=OH, , all checked |
Federal EIC= $0State EIC = $0 |
passed |
In the above table we have laid out some of the first of several test case scenarios necessary to complete system testing on the final software solution. If we were actually testing you would want to test every possible combination of input values. The above example is only a start.
In the first column we name the target input being testing. The inputs directly correspond with the form input fields as you can see in the image above. In the second column, we specify the test value to set the input to for this test case. In the third column we list what the other inputs must be set to in order to get the expected outcome according to the project requirements. In the fourth column we recorded the actual output the application produced when using the described inputs. In the last column we add our testing notes to record whether the test was a success or a failure and why. In system testing, the person doing the testing usually isn’t the one that fixes errors. That is why it’s necessary to record the outcome of each test. Any errors found are passed on to the developer so they can be fixed.
The Software Testing Hierarchy
As with almost any technical process, software testing has a prescribed order in which things should be done. The following is a list of software testing stages arranged in chronological order. These are the steps taken to fully test new software in preparation for marketing it:
- Unit testing – testing performed on each module or block of code during development. Unit testing is normally done by the programmer who writes the code.
- Integration testing – testing done before, during and after integration of a new module into the main software package. This involves testing of each individual code module. One piece of software can contain several modules which are often created by several different programmers. It is crucial to test each module’s effect on the entire program model.
- System testing – testing done by a professional testing agent on the completed software product before it is introduced to the market.
- Acceptance testing – beta testing of the product done by the actual end users.
What Types of System Testing Should Testers Use?
There are over twenty different types of system testing. The specific types used by a testing agent depend on several variables. Those variables include:
- Who the testing agent works for – This is a major factor in determining the types of system testing a tester will use. Different methods are used by larger companies than are used by medium and smaller companies.
- Time available for testing – Ultimately, all twenty some testing types could be used. Time is often what limits us to using only the types that are most relevant for each individual test case.
- Resources available to the tester – Of course some testers will not have all types available to them. For example if you are a tester working for a large software development firm, you are likely to have expensive automated testing software not available to others.
- Testing Agent’s Education – There is a certain learning curve for each type of software testing available. To use some of the software involved, a tester has to learn how to use it. To do manual testing, a tester needs to learn different techniques as well.
- Testing Budget – Money becomes a factor for smaller companies and individual software developers. Often testers cannot afford some of the testing types used by larger companies.
Those are just a few examples of how different types of testing are selected by testers. Ultimately a tester should add as many types to their test suite as possible. Below we have listed typical types of system testing a large software development company might use. We took these examples from a paper written for IBM. Here are the types of system testing IBM typically uses during system testing:
- Usability Testing – Usability testing is done to determine whether or not the software is going to be easy enough for the targeted user
- Stress Testing – Also known as load testing, stress testing is necessary to know that a software solution will perform under real life situations.
- Regression Testing – Regression testing involves testing done to make sure none of the changes made over the course of the development process have caused new bugs. It also makes sure no old bugs appear from the addition of new software modules over time. Regression testing’s more complex techniques can even attempt to prevent future issues caused by the expansion of a software application down the road. It could be argued the regression testing could be part of integration testing. Realistically, it has its place in both.
- Recovery Testing – Recovery testing is done to demonstrate a software solution is reliable, trustworthy and can successfully recoup from possible crashes.
- Serviceability Testing – This type of testing is simply to ensure that the application can be serviced as easily as possible.
- Migration Testing – Migration testing is done to ensure that the software can be moved from older system infrastructures to current system infrastructures without any issues.
- Functional Testing – Also known as functional completeness testing, functional testing involves trying to think of any possible missing functions. Testers might make a list of additional functionalities that a product could have to improve it during functional testing.
- Hardware/Software Testing – IBM refers to Hardware/Software testing as “HW/SW Testing”. This is when the testing agent focusses their attention on the interactions between the hardware and software during system testing.
You may have noticed that there are only eight types of system testing above. Those are the ones large companies such as IBM are most likely to focus on during system testing procedures. If you are working for a small company or a private developer, the types you use will differ from those of a large company like IBM. There are always valuable lessons to take from a giant company like IBM who has essentially dominated the software market.
Automated Planning Systems (APS)
An Automated Planning System or APS is an automated system that plans test cases for software. The large number of test cases needed for some applications would be staggering without an APS. Planning techniques were taken from Artificial Intelligence specialists to develop APS systems that can develop a complete test suite in no time at all. These test suites are then used for testing applications such as the tax calculator above or a GUI. An APS is based on the following four parameters:
- Set of objects – first we must have a set of objects to test. The set of objects consist of the parts of the application being tested. An example from our tax calculator test case would be the marital status input. That would be one of the objects needing testing.
- Set of operators – An APS making test cases would figure out all the possible operations of the user interface (set of objects). It would then use those operations as the set of operators. The set of operators are used to plan a path from the initial state to the goal state explained below.
- Initial State – After the operators are determined, the systems default state is determined. This is known as the operator’s initial state.
- Goal State – Finally, after the initial state is set, a goal state must be determined. The goal state is chosen that will best exercise the system.
A good example of an Automatic Planning System would be TestComplete, APS software from Smart Bear Software. There are many others available at a cost as well. Searching for Freeware APS software will produce some results but not as many as you would hope for. The quality and usability of Freeware APS systems vary from barely usable to satisfactory.
Practical System Testing
Automated System Testing is practical only if you can afford the tools and justify the costs. The truth of the matter is that most small to medium software developers do their own manual system testing. It can be stressful and time consuming at times, but you can get the job done by developing good test cases that you can perfect over time. Developers will often develop their own test suite if they do a lot of system testing. Remember the routines you use and perfect them. If you make your own test cases, you will find that many of them can be recycled and used on other applications in the future.
Understanding System Testing
In the world of software testing, system testing is the testing of a complete and fully integrated software product. System testing falls under the black box testing category of software testing. White box testing is the testing of the internal workings or code of a software application. In contrast, black box or system testing is the opposite. System testing involves the external workings of the software from the user’s perspective. White box testing requires extensive knowledge of the code and software design, but system testing requires no real knowledge of the software development process. It does require extensive knowledge of common black box testing procedures including knowing how to develop test cases and test suites. System testing should be done by people who are very thorough in nature. It is not meant to be done halfheartedly. Thoroughness is the key to good system testing practices.
What is involved in System Testing?
There are several components involved in system testing. Later in the tutorial we will discuss several types of system testing. Here we aim to show you the basic process. There is a test case example later in the tutorial that will help you understand the process. Here is what system testing involves:
· Black box testing of the finished application before it goes to market.
· Testing on fully integrated applications in order to test how components interact with one another and with the system as a whole.
· Very thorough testing of every input in the application to check for required outputs.
· Testing of the user’s experience with the application from start to finish.
· Building detailed test cases and test suites that test each aspect of the application as seen from the outside without looking at the actual source code. This is where the name black box testing comes from (you cannot see the insides of a black box).
That is a very basic description of what is involved in system testing. Continue reading to learn more about each item mentioned above and quite a bit more about what system testing is and about what system testing is not. Sometimes what it is not is just as important is what it is.
System Testing Versus Unit Testing
Let’s explore some of the similarities and differences between Unit Testing and System Testing. In general, unit testing is done during development of the software and system testing is done on the finished product. Also, unit testing is considered white box testing and system testing is black box testing. Those are the major fundamental differences. If we dig deeper we will discover many more differences and some similarities as well.
Differences Between Unit Testing and System Testing
|
Unit Testing
|
System Testing
|
|
1. Usually done by the programmer who developed the application.
|
1. Done by a testing agent with no prior knowledge of the application.
|
|
2. Is done on small sections of code or individual functions and statements.
|
2. Is done on the entire application after all its components have been integrated.
|
|
3. Speed is an issue with unit testing because unit tests are commonly run dozens of times a day, sometimes hundreds.
|
3. Speed is not as important in system testing because they are not run as many times in a day as unit testing.
|
|
4. The code used to test in Unit testing can be very long, often longer than the actual code being tested
|
4. In system testing the code used for testing is not as long as in unit testing and is smaller than the code being tested in most cases.
|
Similarities Between Unit Testing and System Testing
There are many similarities between unit testing and system testing. Although the similar items between the two are often performed in slightly different ways.
To illustrate a similarity of system and unit testing, we could use error testing as an example. In unit testing, additional code could be written to produce an error. We could then see how the application handles that error. If the tester is not happy with the way the error is handled, the source code can be altered to change it. On the other hand, in system testing, errors are thrown on purpose by entering invalid input. There is no code altering done during system testing. If an error is thrown while testing with a valid input, then the application is often returned to the developer to be repaired.
The major similarities of System Testing and Unit Testing are mostly in the general process. Both require a set of objects to be tested, a set of operators, an initial state and a goal state to complete the testing process. The path tested if often similar, but the methods and techniques vary greatly between the two testing types.
Test Cases and Test Suites
In order to test software applications, you must produce test cases. A test case is a set of variables or conditions used in determining whether an application is working properly or not. A group of test cases, known as a test suite, is needed to test any software application. To learn more about test cases for both unit testing and system testing, we have provided you with an example for each below.
Example Application
Below we use a tax calculator application as an example application to demonstrate testing cases for both unit and system testing. The application calculates the Earned Income Credit (EIC) of the user. For demonstration purposes, we will assume that the maximum amount a person can earn to be eligible for EIC is $10,000. Anyone making $10,000 or more will not receive any EIC. Anyone earning less than $10,000 will receive EIC. The relative inputs for the calculator would be the form fields you see in the image provided below:
Here is a screenshot of the EIC tax calculator that is used in the test cases explained below:
The EIC tax calculator in the above image is a web based application written in JavaScript. The process would be similar had it been written in a non-web based language such as C or C++. We used the calculator in the above image to provide readers with a unit test case example. Below that, we will provide a system test case example. We aim to demonstrate how each plays an important role in software testing.
Unit Test Case Example:
Testing done during the development of an application is Unit Testing and is done by the programmer on each section of code as it is written. To illustrate a unit test case example, we are going to take the first function of the tax calculator to demonstrate how a tester would perform unit testing on such a function. We will be testing the “contains” function that extends the array prototype in JavaScript. The “contains” function checks an array to see if it contains a specific value. PHP has a built in function for this called “in_array”. JavaScript does not, so we are going to add one to the array prototype. Here is the actual contains function:
Array.prototype.contains = function(obj) {
var i = this.length;
while(i--) {
if(this[i] === obj) {
return true;
}
}
return false;
}
In unit testing, a programmer would test this function after adding it to be sure it works. While it’s a simple function, there are still different ways it could be tested. A common unit testing method is to write another section of code to test it right after writing the function. Another slightly different method is to isolate the function to test it more thoroughly. This is a more thorough unit testing practice that involves copy and pasting the function to its own testing environment to test it by itself. Below we will examine this method of isolating the code to be tested in detail. It is especially helpful if you aren’t sure you have written the function correctly. You may need to do a little trial and error testing to make it work as intended. Isolating the code helps you find errors. You will know that any errors are coming from the code in question and not some other block of code in the application.
To test the “contains” function, we first build a basic HTML template as its isolated testing environment. Then paste the function in between script tags and save the file as test.html or something similar. Here is what you should have in your new test file:
<script>
//begin contains function:
Array.prototype.contains = function(obj) {
var i = this.length;
while(i--) {
if(this[i] === obj) {
return true;
}
}
return false;
}//end contains function.
//test contains function:
</script>
Notice how we put comments in the code to help understand what is going on (comments begin with //). Near the bottom of the code we left a space for testing the function. That is where we add our test code next.
In this case our function only requires one parameter, “obj”, which is an array object. Therefore we need an array for testing. The next step in writing our test case would be to create a JavaScript array object as demonstrated here:
// literal array of states that offer EIC
var stateEIC = ["DE", "DC", "IL", "IN", "IA", "KS", "LA", "ME", "MD", "MA", "MI", "MN", "NE", "NJ", "NM", "NY", "NC", "OK", "OR", "RI", "VT", "VA", "WI"];
We have built a literal array containing all states that offer EIC. We chose to use this as our test array because we will also need this array in our program later on. The application will calculate the eligibility for state level EIC by checking this array. If the state the user selected is in the “stateEIC” array, the user would get both state and federal EIC returns. If the user’s state isn’t in the array, they will only get federal EIC (if they meet other qualifying criteria). For this test case, we only need to test if the “contains” function works as required. We do so with a test function call. Next we set up our test inputs. We want one input that will test positive and one that will test negative so we can test all possible outcomes of the function. Set up your variables like this:
var state_1 = "OH";//has no state EIC
var state_2 = "NY";//has state EIC of 30% of fed EIC
We set the state_1 variable equal to the state of Ohio. Ohio has no state level EIC. We set the state_2 variable to the state of New York. New York does offer state level EIC. Now we have covered both possible cases for our “contains” function. When we test to see if OH is in the array, it should return false. When we test for NY it should return true. If it does not return the expected values, there is a bug and further trial and error programming is required. Here is what the two test cases should look like:
//TEST CASE 1 - state one test (OH Should return false):
if(stateEIC.contains(state_1)) {
alert(state_1+" has state level EIC");
}else{
alert(state_1+" Does Not have EIC");
}//end test case 1.
//TEST CASE 2 - state two test (NY Should return true):
if(stateEIC.contains(state_2)) {
alert(state_2+" has state level EIC");
}else{
alert(state_2+" Does Not have EIC");
}//end test case 2.
Again we clearly commented our code to identify “Test Case 1” and “Test Case 2” so we know what the expected outcome is going to be. If you have followed along in this tutorial, open your test HTML file in a web browser. You could also open the file named test.html [PROVIDE LINK TO DOWNLOAD test.html HERE] that goes with this tutorial. If the test code works you will see the following in your browser after opening the file:
Then click “OK” and you should see the following in your browser window next:
Above we have provided a very basic Unit Test Case example. Below we will provide a basic System Test Case example. Reading both will give you a pretty good idea of what is involved in each.
System Test Case Example
In system testing you are testing the application as a whole. The “contains” function used in the unit test case example above would still be tested in system testing. It is just tested in a completely different manner. In system testing, the “contains” function is tested by waiting until the function is used in the finished program. Then the testing agent tests it by making sure it produces the expected output for various tested inputs. Below we will outline how you would system test the application as a whole.
To begin system testing on our finished EIC calculator we would first plan out what inputs require testing. The inputs requiring testing include:
· Marital status
· Number of Children
· State Input
· Total Amount Earned
· The Five inputs under Other Income
· The Six check boxes
Ultimately in system testing we are testing the finished product to see if it works in accordance with the project requirements. For each of the inputs named above, we need to specify a minimum of two test cases as in the following written test case example:
|
Input
|
Test value
|
Dependencies
|
Expected outcome
|
Notes
|
|
Marital Status
|
Single
|
Kids=1, state=NY, income=$9000, all checked
|
Federal EIC = $805
State EIC = $223
|
Passed
|
|
Marital Status
|
Married
|
Kids=1, state=NY, income=$9000, all checked
|
Federal EIC = $905
State EIC = $424
|
Passed
|
|
Kids
|
2
|
Marital=single, state=OH, income=$9000, all checked
|
Federal EIC= $1090
State EIC = $560
|
Passed
|
|
Kids
|
3
|
Marital=single, state=OH, income=$9000, all checked
|
Federal EIC= $1490
State EIC = $760
|
Passed
|
|
State
|
OH
|
Marital=married, Kids=1, income=$9000, all checked
|
Federal EIC= $1090
State EIC = $0
|
Passed
|
|
State
|
NJ
|
Marital=married, Kids=1, income=$9000, all checked
|
Federal EIC= $7090
State EIC = $560
|
passed
|
|
Income
|
9999
|
Marital=married, Kids=1, state=OH, , all checked
|
Federal EIC= $0
State EIC = $0
|
Expected outcome was wrong check income variables
|
|
Income
|
10000
|
Marital=married, Kids=1, state=OH, , all checked
|
Federal EIC= $0
State EIC = $0
|
passed
|
In the above table we have laid out some of the first of several test case scenarios necessary to complete system testing on the final software solution. If we were actually testing you would want to test every possible combination of input values. The above example is only a start.
In the first column we name the target input being testing. The inputs directly correspond with the form input fields as you can see in the image above. In the second column, we specify the test value to set the input to for this test case. In the third column we list what the other inputs must be set to in order to get the expected outcome according to the project requirements. In the fourth column we recorded the actual output the application produced when using the described inputs. In the last column we add our testing notes to record whether the test was a success or a failure and why. In system testing, the person doing the testing usually isn’t the one that fixes errors. That is why it’s necessary to record the outcome of each test. Any errors found are passed on to the developer so they can be fixed.
The Software Testing Hierarchy
As with almost any technical process, software testing has a prescribed order in which things should be done. The following is a list of software testing stages arranged in chronological order. These are the steps taken to fully test new software in preparation for marketing it:
1. Unit testing – testing performed on each module or block of code during development. Unit testing is normally done by the programmer who writes the code.
2. Integration testing – testing done before, during and after integration of a new module into the main software package. This involves testing of each individual code module. One piece of software can contain several modules which are often created by several different programmers. It is crucial to test each module’s effect on the entire program model.
3. System testing – testing done by a professional testing agent on the completed software product before it is introduced to the market.
4. Acceptance testing – beta testing of the product done by the actual end users.
What Types of System Testing Should Testers Use?
There are over twenty different types of system testing. The specific types used by a testing agent depend on several variables. Those variables include:
· Who the testing agent works for – This is a major factor in determining the types of system testing a tester will use. Different methods are used by larger companies than are used by medium and smaller companies.
· Time available for testing – Ultimately, all twenty some testing types could be used. Time is often what limits us to using only the types that are most relevant for each individual test case.
· Resources available to the tester – Of course some testers will not have all types available to them. For example if you are a tester working for a large software development firm, you are likely to have expensive automated testing software not available to others.
· Testing Agent’s Education – There is a certain learning curve for each type of software testing available. To use some of the software involved, a tester has to learn how to use it. To do manual testing, a tester needs to learn different techniques as well.
· Testing Budget – Money becomes a factor for smaller companies and individual software developers. Often testers cannot afford some of the testing types used by larger companies.
Those are just a few examples of how different types of testing are selected by testers. Ultimately a tester should add as many types to their test suite as possible. Below we have listed typical types of system testing a large software development company might use. We took these examples from a paper written for IBM. Here are the types of system testing IBM typically uses during system testing:
1. Usability Testing – Usability testing is done to determine whether or not the software is going to be easy enough for the targeted user
2. Stress Testing – Also known as load testing, stress testing is necessary to know that a software solution will perform under real life situations.
3. Regression Testing – Regression testing involves testing done to make sure none of the changes made over the course of the development process have caused new bugs. It also makes sure no old bugs appear from the addition of new software modules over time. Regression testing’s more complex techniques can even attempt to prevent future issues caused by the expansion of a software application down the road. It could be argued the regression testing could be part of integration testing. Realistically, it has its place in both.
4. Recovery Testing – Recovery testing is done to demonstrate a software solution is reliable, trustworthy and can successfully recoup from possible crashes.
5. Serviceability Testing – This type of testing is simply to ensure that the application can be serviced as easily as possible.
6. Migration Testing – Migration testing is done to ensure that the software can be moved from older system infrastructures to current system infrastructures without any issues.
7. Functional Testing – Also known as functional completeness testing, functional testing involves trying to think of any possible missing functions. Testers might make a list of additional functionalities that a product could have to improve it during functional testing.
8. Hardware/Software Testing – IBM refers to Hardware/Software testing as “HW/SW Testing”. This is when the testing agent focusses their attention on the interactions between the hardware and software during system testing.
You may have noticed that there are only eight types of system testing above. Those are the ones large companies such as IBM are most likely to focus on during system testing procedures. If you are working for a small company or a private developer, the types you use will differ from those of a large company like IBM. There are always valuable lessons to take from a giant company like IBM who has essentially dominated the software market.
Automated Planning Systems (APS)
An Automated Planning System or APS is an automated system that plans test cases for software. The large number of test cases needed for some applications would be staggering without an APS. Planning techniques were taken from Artificial Intelligence specialists to develop APS systems that can develop a complete test suite in no time at all. These test suites are then used for testing applications such as the tax calculator above or a GUI. An APS is based on the following four parameters:
1. Set of objects – first we must have a set of objects to test. The set of objects consist of the parts of the application being tested. An example from our tax calculator test case would be the marital status input. That would be one of the objects needing testing.
2. Set of operators – An APS making test cases would figure out all the possible operations of the user interface (set of objects). It would then use those operations as the set of operators. The set of operators are used to plan a path from the initial state to the goal state explained below.
3. Initial State – After the operators are determined, the systems default state is determined. This is known as the operator’s initial state.
4. Goal State – Finally, after the initial state is set, a goal state must be determined. The goal state is chosen that will best exercise the system.
A good example of an Automatic Planning System would be TestComplete, APS software from Smart Bear Software. There are many others available at a cost as well. Searching for Freeware APS software will produce some results but not as many as you would hope for. The quality and usability of Freeware APS systems vary from barely usable to satisfactory.
Practical System Testing
Automated System Testing is practical only if you can afford the tools and justify the costs. The truth of the matter is that most small to medium software developers do their own manual system testing. It can be stressful and time consuming at times, but you can get the job done by developing good test cases that you can perfect over time. Developers will often develop their own test suite if they do a lot of system testing. Remember the routines you use and perfect them. If you make your own test cases, you will find that many of them can be recycled and used on other applications in the future.
