How to optimize testng and seleniums tests

Question

For my internship, I have to use TestNG and selenium to test the web application. But I have a problem, sometimes selenium or the Browser does not work for some random reason, so the working test is marked as "failure". To avoid this, I can use the annotation @Test(invocationCount = 4, successPercentage = 25), and then, if the test passes once, the test will be marked as "Success", this is good, but the problem is that this solution multiplies the testing time by 4, it is not very effective.

What I can do to reduce the testing time is to write some rule "if the test failed, repeat this test (and only if the test did not work), and if it worked the second, third or fourth time, then mark this test as" successful "Therefore, I can avoid these random errors. But I did not find how to write this rule, I saw that we can add a listener, so we have a method called" onTestFailure", so I can do something when the test failed fail, but I don’t know how to repeat the test.

I also found testng-failed.xml where all failed tests were saved, so we can run this xml file to run these tests again, but this will remove the report from the previous first run, but I want the tests to be marked as “successful” if the second launch is successful. (I have included testNG / selenium in Jenkins, so I have a graph with all the tests, so this method is not very adapted, but this method does not multiply the test time by 4, and this is what I want)

So, if you have any clue how to do this, it would be very nice.

Answer 1

IRetryAnalyzer, Listener , .

IRetryAnalyzer:

public class RetryAnalyzer implements IRetryAnalyzer  { 
private int count = 0; 
// this number is actually twice the number
// of retry attempts will allow due to the retry
// method being called twice for each retry
private int maxCount = 6; 
protected Logger log;
private static Logger testbaseLog;

static {
    PropertyConfigurator.configure("test-config/log4j.properties");
    testbaseLog = Logger.getLogger("testbase.testng");
}

public RetryAnalyzer()
{
    testbaseLog.trace( " ModeledRetryAnalyzer constructor " + this.getClass().getName() );
    log = Logger.getLogger("transcript.test");
}

@Override 
public boolean retry(ITestResult result) { 
    testbaseLog.trace("running retry logic for  '" 
            + result.getName() 
            + "' on class " + this.getClass().getName() );
        if(count < maxCount) {                     
                count++;                                    
                return true; 
        } 
        return false; 
}
}

RetryListener:

public class RetryTestListener extends TestListenerAdapter  {
private int count = 0; 
private int maxCount = 3; 

@Override
public void onTestFailure(ITestResult result) {     
    Logger log = Logger.getLogger("transcript.test");
    Reporter.setCurrentTestResult(result);

    if(result.getMethod().getRetryAnalyzer().retry(result)) {    
        count++;
        result.setStatus(ITestResult.SKIP);
        log.warn("Error in " + result.getName() + " with status " 
                + result.getStatus()+ " Retrying " + count + " of 3 times");
        log.info("Setting test run attempt status to Skipped");                 
    } 
    else
    {
        count = 0;
        log.error("Retry limit exceeded for " + result.getName());
    }       

    Reporter.setCurrentTestResult(null);
}

@Override
public void onTestSuccess(ITestResult result)
{
    count = 0;
}

TestNG , , , . , , , , , :

private IResultMap removeIncorrectlyFailedTests(ITestContext test)
{     
  List<ITestNGMethod> failsToRemove = new ArrayList<ITestNGMethod>();
  IResultMap returnValue = test.getFailedTests();

  for(ITestResult result : test.getFailedTests().getAllResults())
  {
    long failedResultTime = result.getEndMillis();          

    for(ITestResult resultToCheck : test.getSkippedTests().getAllResults())
    {
        if(failedResultTime == resultToCheck.getEndMillis())
        {
            failsToRemove.add(resultToCheck.getMethod());
            break;
        }
    }

    for(ITestResult resultToCheck : test.getPassedTests().getAllResults())
    {
        if(failedResultTime == resultToCheck.getEndMillis())
        {
            failsToRemove.add(resultToCheck.getMethod());
            break;
        }
    }           
  }

  for(ITestNGMethod method : failsToRemove)
  {
      returnValue.removeResult(method);
  }  

  return returnValue;
}

, , , .addListener, retryAnalyzer @Test.

Answer 2

onTestFailure. TestNG , . onTestFailure. 2. , .

private final Map rerunCountForTesCase = new HashMap();
    @Override
    public boolean retry(ITestResult result) {
                // here i have unique test case IDs for each of test method. 
                // So using utility method to get it. You can use method calss+name combination instead of testcaseid like me
        String executingTestCaseID = ReportingUtilities.getTestCaseId(result);
        if(rerunCountForTesCase.containsKey(executingTestCaseID))
        {
            count = rerunCountForTesCase.get(executingTestCaseID);
        }
        else
        {
            rerunCountForTesCase.put(executingTestCaseID, 0);
        }
        if (count 0)
                {
                    logInfo(tcID,"Sleeping for "+timeInSecs+ " secs before rerunning the testcase");
                    Thread.sleep(timeInSecs * CommonFwBase.SHORTWAIT );
                }
            } catch (InterruptedException e) {
                logError(null, e.getMessage());

            }

            rerunCountForTesCase.put(executingTestCaseID, ++count);
            return true;
        }
        return false;

    }

- onTestFailure. , . , , ( , ). , . , , , .

        m_ptests = suiteTestContext.getPassedTests();
        m_ftests = suiteTestContext.getFailedTests();
        m_stests = suiteTestContext.getSkippedTests();

        List<ITestNGMethod> methodsToRemove = new ArrayList<ITestNGMethod>();

        for(ITestResult failed_result : m_ftests.getAllResults())
        {
            String failed_testName = failed_result.getMethod().getMethodName();
            String failingTest_className = failed_result.getClass().getName();
            for(ITestResult passed_result : m_ptests.getAllResults())
            {
                String passing_testName = passed_result.getMethod().getMethodName();
                String passingTest_className = failed_result.getClass().getName();
                if(failed_testName.equals(passing_testName) &&  
                        passingTest_className.equals(failingTest_className) ))
                {
                    if(passed_result.getEndMillis() > failed_result.getEndMillis())
                    {

                        methodsToRemove.add(failed_result.getMethod());
                        break;
                    }

                }
            }
        }

        // remove the test that passed on retry
        for(ITestNGMethod failedMethodToRemove : methodsToRemove)
        {
            m_ftests.removeResult(failedMethodToRemove);
        }

, .

Answer 3

- "" . "quick'n'dirty" . for-loop try..catch , Throwable ( ). , .

:

@Test
public void testA() throws Throwable {
    Throwable err = null;
    for (int i=0; i<4; i++) {
        try {
            testImplementationA();
            return;
        } catch (Throwable e) {
            err = e;
            continue;
        }
    }
    throw err;
}

private void testImplementationA()
{
    // put test code here
}

, , , . , , , , . / Jenkins , .