Challenges

One of leading European telecommunication operators was developing a new-generation STB on Android platform, having necessity to run functional testing in parallel with product development. Each weekly release needed full functional testing, with bugs immediately reported to Redmine for further investigation. Due to significant application changes between releases, manual testing was a necessity for early stages of the development with a goal of having fully automated test suite once the project enters a stable phase. Regression tracking between releases and having real-time overview of testing results were some of usual requirements accompanying this kind of project.

Summary

Requirements

  • Full functional testing of Android STB
  • Weekly release cycle require  automatic testing
  • Agile project management
  • Redmine integration
  • Regression and real-time reporting

ACHIEVEMENTS

  • Test suite of 1274 tests created
  • Achieved 80% of test automation
  • Test execution reduction time 50%
  • Full integration of QA and R&D processes
  • Quick regression check

Solution

Considering the given requirements, Intent+ end-to-end test and requirement management solution was a natural choice for the test platform. Intent+ provides seamless combination of automatic and manual tests, integration with automatic testing platform RT-Executor, as well as integration with Redmine.

In early stages of the project, our experienced QA team was creating test suites very quickly, by reusing rich RT-RK test base. The first phase of test automation was creation of stress test suite. As product had entered a stable development phase, conversion of manual test cases into automatic was started and coverage of 80% was reached soon.

During the whole lifetime of the project, the testing process was consistent – the full set of manual 1300+ functional tests was run once a week, while rack of 3 RT- Executor test stations was performing nightly runs of fully automatic regression test suite. Results of both automatic and manual tests were cross-compared to verify outcomes of both test types, and to detect bugs that appeared only occasionally. Customer was defining tasks, tracking execution process, and bugs in real time.

Process overview
Process overview

 

test setup graph
Test Setup
test-time-reduction
Test execution time reduction