How to select which TestNG suites to run in Maven Surefire plugin

If you want to select a different TestNG test suite .xml files to run test from you can do the following in the pom.xml file:


             <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.13</version>
                <configuration>
                    <forkMode>always</forkMode>
                    <systemPropertyVariables>
                        <reporter.debug>false</reporter.debug>                      
                    </systemPropertyVariables>
                    <suiteXmlFiles>
                        <suiteXmlFile>${suiteXmlFile}</suiteXmlFile>
                    </suiteXmlFiles>
                </configuration>
            </plugin>

You can however add a specific test suite file to a Maven property:

 <properties>
     <suiteXmlFile>testng.xml</suiteXmlFile>
 </properties>

To run the tests from a particular TestNG test suite file you can run the following command from terminal:

mvn clean test -DsuiteXmlFile=custom.xml

TestNG – Advanced Groups Selection

TestNG has only the functionality to run tests depending on what groups you include or exclude in the .xml testNG suite file.
If you want to run only tests that are present in two or more groups (intersection of groups) you’ll have to use the beanshell functionality that TestNG supports in the test suite .xml file.

The TestNG default including and excluding groups functionality:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="my test suite" parallel="false">

        <test name="all tests" preserve-order="true">
        <groups>
            <define name="all">
                <include name="group1"/>
                <include name="group2"/>
            </define>
            <run>
                <include name="all" />
                <exclude name="group3" />
            </run>
        </groups>

        <classes>
            <class name="com.tests.MyTest" />
        </classes>

    </test>

Lets say that you have a test file that is part of more groups, for example:

    @Test(groups = {"group1","group2","group3"})
    public void myTest(){
       ...
    }

In order to run tests that are part of an intersection of groups you must use the TestNG BeanShell functionality.
Here you can find some details in the TestNG Documentation – Advanced Groups Selection section from http://testng.org/doc/documentation-main.html#beanshell.

In order to use Maven and pass parameters you can modify the suite file to look something like:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="my test suite" parallel="false">
    <test name="my tests" preserve-order="true">
        <method-selectors>
            <method-selector>
                <script language="beanshell"><![CDATA[
                runTest = false;
                groupIn = System.getProperty("includeGroups");
                groupNotIn = System.getProperty("excludeGroups");

                if ((groupIn == null) || (groupIn == "")) {
                    runTest = false;
                } else {
                    StringTokenizer groupsTagList = new StringTokenizer(groupIn, ",");
                    runTest = true;

                    while (groupsTagList.hasMoreTokens()) {
                      if (!(groups.containsKey(groupsTagList.nextToken())))
                        runTest = false;
                    }

                    if (runTest && (groupNotIn != null) && (groupNotIn != "")) {
                        StringTokenizer groupsNotInList = new StringTokenizer(groupNotIn, ",");

                        while (groupsNotInList.hasMoreTokens()) {
                           if ((groups.containsKey(groupsNotInList.nextToken())))
                             runTest = false;
                        }
                     }
                }
                return runTest;
                ]]>
                </script>
            </method-selector>
        </method-selectors>
        <classes>
            <class name="com.tests.MyTest" />
        </classes>

</test>
</suite> 


The Maven command to run you specific test that is part of the two groups is the following:

mvn clean test -DincludeGroups=group1,group2 

TestNG, Maven, Surefire plugin – Run Specific Group of Tests

Today I played a little bit with the TestNG groups, running unit tests with Maven SureFire Plugin and TestN. The things discovered are listed below:

If you want to run TestNG tests with Maven commands you can do the following:

  • In order to run the test from the defined group myfirstgroup you have to add the groups parameter to the @Test annotation:

@Test(groups = "myfirstgroup")
public void firstTest(){
.....
}

@Test(groups = "myfirstgroup")
public void secondTest(){
.....
}

  • Inside the testng.xml file the following code should be added:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="smoke" parallel="false">
    <test name="smoke" preserve-order="true">
        <groups>
            <define name="all">
                <include name="myfirstgroup"/>
            </define>
            <run>
                <include name="all" />
                <exclude name="failed" />
            </run>
        </groups>

        <classes>
            <class name="com.qa.tests.MyClass" />           
        </classes>
    </test>
</suite>

  • The pom.xml file should look like:
<plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.12</version>
                <configuration>
                    <forkMode>always</forkMode>
                    <systemProperties>
                        <property>
                            <name>reporter.debug</name>
                            <value>false</value>
                        </property>
                    </systemProperties>
                    <suiteXmlFiles>
                        <suiteXmlFile>${suiteXmlFile}</suiteXmlFile>
                    </suiteXmlFiles>
                </configuration>
            </plugin>
  • Maven run syntax from Terminal or Continuous Integration tool:
mvn clean test -DsuiteXMLfile=testNG.xml -Dgroups=myfirstgroup

Best Testing Automation Tools Today

There are a lot automation tools on Internet, some of them are free, some of them cost a lot of money. It depends what you are trying to test: UI, API, unit, integration, security, performance, load, memory leak testing.

There are suitable tools for almost each type of testing.

UI: Selenium WebDriver – (open source)

API: (REST, SOAP) jMeter, SoapUI (open source)

Unit testing: jUnit, TestNG (JAVA), QUnit(JS) etc

Performance: jMeter (open source)

Security: OWASP – tools and documentation

More details on request.

Please comment on the missing ones for a full list, thank you.

JMeter Testing Tool – Setup Win and Mac

JMeter is an open source Java application, can be downloaded from http://jmeter.apache.org/. JMeter is used for performance and load testing, however you can use it very easy for functional testing to make HTTP requests, and test REST APIs.

Steps to run JMeter:

  • download JMeter binaries from the download page (you can download the sources and build locally the JMeter project)
  • extract the .zip file
  • in order to be able to double-click .jmx files (JMeter files) you have to: add the /bin path of JMeter to Path variables, and create a new .bat file containing: jmeter -t %1 (on Win)
  • open a Terminal to /bin folder and run ./jmeter.sh for Mac or add jmeter to the variables of the system and then type jmeter in a terminal
  • to run JMeter from command-line you should go from terminal to the JMeter’s bin directory and then type in the following:
  • java -jar ApacheJMeter.jar -n -t <testfile>

With JMeter testing tool you can create performance tests, load tests, memory-leak tests, stress tests but working with the number of threads and loops for each test. We’ll take a look into each type of test in the next articles.

Representational State Transfer (REST)

Representational State Transfer (REST) is a software architecture style.

Its key points are that its stateless which is good for the Web service’s performance and for the simplicity of the implementation; uses resources that can be accessed through a directory-like URI structure; uses HTTP methods to make CRUD (create, read, update, delete) operations in order to manipulate resources:

  • POST – create a resource
  • PUT – update a resource
  • GET – retrieve a resource
  • DELETE – delete a resource or a set of resources

Content-Type can be set to XML or JSON, also you can implement both of them.

Good details can be found here:

http://broadcast.oreilly.com/2011/06/the-good-the-bad-the-ugly-of-rest-apis.html

http://en.wikipedia.org/wiki/Representational_state_transfer

https://www.ibm.com/developerworks/webservices/library/ws-restful/

 

Methods for measuring page load time for a web application

Methods for automatically measuring page load time

There are several ways in which a QA engineer could create automated tests that measure the page load time for a web application.

I will mention all the methods for doing this and later describe in detail each of them:

  • Option number 1: using Selenium WebDriver and the Navigation Timing API
  • Pros: – one can get a lot of information like the unload, redirect, app cache, DNS, request, response, processing time, total rendition time
    – it replaces the JavaScript code developed in house that was used for doing these kind of measurements

    Cons: – not supported by all the browsers, so does not help in every situation

  • Option number 2: using the Selenium WebDriver and the Firebug net extension
  • Pros: – can give also the number of requests made for all the resources that were retrieved

    Cons: – works only with Firefox

  • Option number 3: using Phantom JS and JUnit
  • Pros: – easy to integrate with in house made automated test frameworks
    – less time for test execution as it does not involve WebDriver
    Cons: – not sure if it returns all the requests that are made each time

  • Option number 4: using Selenium WebDriver and Browser mob proxy
  • Pros: – the results provided by this option are very close to what one can see in the Firefox net console
    Cons: – a little complicated to set up in an automated environment

  • Option number 5: using Webpagetest
  • Pros: – easy to use
    Cons: – not easily to integrate in CI

Benefits for measuring the page load time
There quite a few usecases that could benefit from such an approach.

  • continuous integration that could provide very fast feedback regarding the regressions from the latest commits
  • comparison between a web app with a some of the features enabled/disabled (for ex. analytics features that add additional calls to each normal call)
  • monitoring web applications in the combination with Amazon EC2 instance machines for example
  • useful information regarding the theoretical perceived load time of the web app (not exactly as RUM but gives the developers some idea)

[twitter-follow screen_name=’cristivazzolla’]