When to use performace tests? Go to top

Performance tests are suitable to use in cases when we need to verify and measure some response time in the UI of the DUT (Device Under Test). Typically, we simulate a user performing certain action and then analyze the device output, detecting at which moment the expected output appears. For example, we can use this type of testing to measure the time needed for the DUT to display some menu item, or the time needed to hide that menu item. This type of testing isn’t limited to menu item measurements – it can be used for booting tests, zapping tests, etc.
 

Typical test sequence Go to top

Figure 1. Typical steps in performance tests

As shown in the Figure 1, a typical performance test case is performed in 3 steps. We will focus only to related parts of the test script to this scenario that we are explaining; the rest of the code has already been explained in the Anatomy of a test script section. A detailed explanation of the API functions used in this example can be seen in the API section.

 

1. Put the device to the desired state Go to top

As a first step, we need to put the device to the appropriate state.

## Send RC command to DUT to bring DUT to initial state
# ([INIT] macro is defined in RemoteController.ini file)
TEST_CREATION_API.send_ir_rc_command("[INIT]")

Macro [INIT] used here for navigation is defined in the RemoteController.ini configuration file. It represents a RC sequence of different RC keys to be sent to the DUT in order to navigate to the desired state. Every RC key used in macros is represented by its name (names are created using the RC Capture application), number of times that RC key should be pressed and the time delay in ms after every press. This macro used in the example above has a value:

[INIT] "INIT 1 1000"
 

2. Execute action Go to top

The next step is to execute the desired command for which the output time will be measured. This step is also done using the Remote Controller and the appropriate macro.

## Send RC command to DUT to open the Main menu
# RC Command should have no extra delay in order to measure the time required to open menu
# after pressing Home RC command
TEST_CREATION_API.send_ir_rc_command("[HOME_0]")

Note here that the delay field in the command should be 0, so the value of the macro is:

[HOME_0] "HOME 1 0"
 

3. Start the measurement algorithm and make the result decision Go to top

After the command is executed, the measuring should be started, until the output changes to the appropriate value:

## Constant multiplier used for conversion from seconds to milliseconds
MS_MULTIPLIER = 1000
## Time until comparing is finished (in seconds)
WAIT_TIMEOUT = 10
## Convert start time to milliseconds
start_time_in_ms = int(time.time() * MS_MULTIPLIER)

## Check if grabbed picture is same as referent one
# Macro [HOME_TOP] is defined in RT-AV101.ini file,
# and it is used to monitor precise region in the picture
# that will be the same as that region on referent picture.
if (TEST_CREATION_API.wait_for_picture("home_ref.bmp", "[HOME_TOP]", WAIT_TIMEOUT)):
    ## Get time(in milliseconds) required to open main menu
    duration = int(time.time() * MS_MULTIPLIER) - start_time_in_ms

    ## Check if measured duration for opening menu is lower then max expected value
    if (duration < MAX_ALLOWED_TIME):
        TEST_CREATION_API.write_log_to_file("Duration for opening home menu is: "+str(duration))
        test_result = "PASS"
    else:
        TEST_CREATION_API.write_log_to_file("Duration for opening home menu is to long: "+str(duration))
else:
    ## Write message to log file if comparing of pictures returns False and continue execution
    TEST_CREATION_API.write_log_to_file("Home menu is not displayed correctly")

Note here that the function wait_for_picture() receives the [HOME_TOP] macro as an input argument, which is defined in the RT-AV101.ini configuration file. That macro is used to monitor a precise region in the picture that will be the same as that region on the referent picture. It has the following format:

[HOME_TOP] "compare_left x_start" "compare_top y_start" "compare_right x_end" "compare_bottom y_end"

For the exact coordinates that should be defined here, the tool called Reference Creator can be used.

Note that almost the same test case can be used to measure the time needed for the shown menu to disappear, which can be accomplished just by changing the navigation controls (close the menu instead of opening it).

 

Limitation of this approach Go to top

The main limitation of this approach is that the detection precision depends on the region of interest size and dynamics. If the region of interest has dynamic background, and the region of interest is small, the detection efficiency decreases.

 

Test implementation in Python Go to top

For your reference, a full example is available in the source code below:

## @package Wait for picture
# The test is performed by entering main menu and measuring
# time required to open menu.
#
# In this type of test cases, referent picture (picture that we are waiting for to change)
# is sent to RT-AV101 grabber device and the device waits until the picture is removed or presented on,
# depending on the functionality that is desired.
#
# test_name = Wait for picture
# ID = Path of test case in Intent+
# copyright = 2014 RT-RK. All rights reserved.

## This module provides various time-related functions
import time

## This module contains common definitions used through various tests for Black Box testing.
import TEST_CREATION_API

## Time until comparing is finished (in seconds)
WAIT_TIMEOUT = 10
## Max allowed time to open main menu (in milliseconds)
MAX_ALLOWED_TIME = 2000
## Constant multiplier used for conversion from seconds to milliseconds
MS_MULTIPLIER = 1000

## Test steps execution
def runTest():
    ## Use Try Except block to catch errors detected during execution
    try:
        ## Set initial result of test execution to FAIL
        test_result = "FAIL"

        ## Initialize grabber device
        # (enable communication between PC and grabber device through LAN interface)
        TEST_CREATION_API.initialize_grabber()

        ## Start grabbing video on video interface (HDMI1)
        # Preview is turned on by default
        # (to turn off the preview set second parameter of function to False)
        TEST_CREATION_API.grabber_start_video_source(TEST_CREATION_API.VideoInterface.HDMI1)

        ## Send RC command to DUT to bring DUT to initial state
        # ([INIT] macro is defined in RemoteController.ini file)
        TEST_CREATION_API.send_ir_rc_command("[INIT]")

        ## Send RC command to DUT to open the Main menu
        # RC Command should have no extra delay in order to measure the time required to open menu
        # after pressing Home RC command
        TEST_CREATION_API.send_ir_rc_command("[HOME_0]")

        ## Convert start time to milliseconds
        start_time_in_ms = int(time.time() * MS_MULTIPLIER)

        ## Check if grabbed picture is same as referent one
        # Macro [HOME_TOP] is defined in RT-AV101.ini file,
        # and it is used to monitor precise region in the picture
        # that will be the same as that region on referent picture.
        if (TEST_CREATION_API.wait_for_picture("home_ref.bmp", "[HOME_TOP]", WAIT_TIMEOUT)):
            ## Get time(in milliseconds) required to open main menu
            duration = int(time.time() * MS_MULTIPLIER) - start_time_in_ms

            ## Check if measured duration for opening menu is lower then max expected value
            if (duration < MAX_ALLOWED_TIME):
                TEST_CREATION_API.write_log_to_file("Duration for opening home menu is: "+str(duration))
                test_result = "PASS"
            else:
                TEST_CREATION_API.write_log_to_file("Duration for opening home menu is to long: "+str(duration))
        else:
            ## Write message to log file if comparing of pictures returns False and continue execution
            TEST_CREATION_API.write_log_to_file("Home menu is not displayed correctly")
    ## If exception is detected,
    # test result variable is set to INCONCLUSIVE and log of error is written to log file
    except Exception as error:
        ## Set test result variable to INCONCLUSIVE if exception is caught
        test_result = "INCONCLUSIVE"

        ## Write exception message to log file
        TEST_CREATION_API.write_log_to_file(error)

    ## Update test result with current state of test_result variable
    TEST_CREATION_API.update_test_result(test_result)

    ## Return DUT to initial state
    TEST_CREATION_API.send_ir_rc_command("[INIT]")

    ## Stop grabbing video on HDMI1 interface
    TEST_CREATION_API.grabber_stop_video_source()

    ## Deinitialize grabber device
    TEST_CREATION_API.deinitialize_grabber()