When to use reference video analysis? Go to top

referent-video-picture
Figure 1. Referent video file

Reference video analysis is suitable to use in cases when we need to verify that some content is properly decoded. One example of such a case is checking if the DUT (Device Under Test) can properly decode stream with specific characteristics. One example of such use-case is shown in the Figure 1. Custom made static referent stream is presented to the DUT and the content on its output is analysed with the picture quality metrics analysis.

 

Typical test sequence Go to top

referent-video-steps
Figure 2. Typical steps in reference video based test

As shown in the Figure 2, a typical test case which uses referent video is performed in 5 steps. We will focus only to related parts of the test script for this scenario that we are explaining; the rest of the code has already been explained in the Anatomy of a test script section. A detailed explanation of the API functions used in this example can be seen in the API section.

In the first step, we need to prepare and make available a reference stream to the DUT. This can be done by simply copying the stream to a USB or to an already available stream server, or by using a signal modulator, and that case will be explained here. In the second step, we navigate the DUT to the desired state (do a channel search or play some referent content from the media player application). When in desired state, we use grabber to capture the DUT output video to a video file. In the forth step, we decide which video comparation algorithms should be used. In the last step, we analyse the output of the video comparison algorithm and decide on the outcome of the test.

 

1. Play stream Go to top

The most common way of testing the proper decoding content of the DUT output is by using a Stream Modulator phisical device that feeds the DUT with the test signal. The appropriate command in the form of macro needs to be sent to the Stream Modulator, and that macro command is different from one physical device to another (there are different kinds of Stream Modulators available on the market, they all have different parameters that need to be set in order to feed the test signal to the DUT). The format of the command can be found here.

In the first step, we  issue a command to the Stream Modulator to start to generate input signal for DUT:

## Activate stream player device and play stream which contains static video image
# [STREAM_WITH_STATIC_VIDEO] macro is defined in StreamPlayer.ini file
TEST_CREATION_API.stream_play("[STREAM_WITH_STATIC_VIDEO]")
 

2. Navigate Go to top

In the second step, we navigate the DUT to the desired state (menu option):

## Perform DVBT scan
TEST_CREATION_API.send_ir_rc_command("[DVBT_SCAN]")
TEST_CREATION_API.send_ir_rc_command("[EXIT_DVBT_SCAN]")

## Zapping time in digital systems isn't a constant category,
# but there are always some values defined as max zapping duration,
# time in which the zapping will be completed. Here we define max zapping time in seconds.
MAX_ZAP_TIME_DURATION = 6

## Tune to scanned channel
TEST_CREATION_API.send_ir_rc_command("[CH_500]")
time.sleep(MAX_ZAP_TIME_DURATION)

Macros [DVBT_SCAN], [EXIT_DVBT_SCAN], and [CH_500] used here for the navigation are defined in the RemoteController.ini configuration file. They represent an RC sequence of different RC keys to be sent to the DUT in order to navigate to the desired state. Every RC key used in macros is represented by its name (the names are created by using the RC Capture application), number of times that RC key should be pressed and the time delay in ms after every press. Example of one RC Key:

"HOME 1 2000"
 

3. Capture Go to top

When in the desired state, we use grabber to capture the DUT output video to a video file. The file will be recorded in the MPEG format and stored in the current test execution folder.

## Record video with default duration of recording (5 seconds)
# If required custom video duration, second parameter should be added as time in milliseconds
TEST_CREATION_API.record_video("video")
 

4. Video analysis Go to top

Here we decide which video comparation algorithms should be used. Details about available video analysis algorithms are available in the Video analysis section.

## Instance of PQMAnalyse type
pqm_analyse = TEST_CREATION_API.PQMAnalyse()

## Set what algorithms should be checked while analyzing given video file with PQM.
# Attributes are set to false by default.
pqm_analyse.black_screen_activ = True
pqm_analyse.blocking_activ = True
pqm_analyse.ringing_activ = True
pqm_analyse.freezing_activ = True
pqm_analyse.blurring_activ = True
pqm_analyse.packet_loss_activ = True
# Name of the video file that will be analysed by PQM. If no extension is provided,
# PQM assumes that the extension is .mpg,
# otherwise it will search for the file with the extension that you have provided.
pqm_analyse.file_name = "video"

## Analyse recorded video
TEST_CREATION_API.pqm_analysis(pqm_analyse)
 

5. Test result decision Go to top

As a last step, we analyse the output of the video comparison algorithm and decide on the outcome of the test. In this case, because we are using a referent stream with the static content, the algorithm should detect freezing, and other artefacts shouldn’t be detected in order to mark this test as passed.

## Check if errors are detected in recorded video, 
# here the picture on the output should be frozen,
# because the input signal has static image showing.
if (pqm_analyse.black_screen_detected == TEST_CREATION_API.AlgorythmResult.NOT_DETECTED and \
        pqm_analyse.blocking_detected == TEST_CREATION_API.AlgorythmResult.NOT_DETECTED and \
        pqm_analyse.ringing_detected == TEST_CREATION_API.AlgorythmResult.NOT_DETECTED and \
        pqm_analyse.freezing_detected == TEST_CREATION_API.AlgorythmResult.DETECTED and \
        pqm_analyse.blurring_detected == TEST_CREATION_API.AlgorythmResult.NOT_DETECTED and \
        pqm_analyse.packet_loss_detected == TEST_CREATION_API.AlgorythmResult.NOT_DETECTED):
    ## Set test result variable to PASS if there is no errors were detected
    test_result = "PASS"
else:
    ## Write message to log file if errors were detected
    TEST_CREATION_API.write_log_to_file("PQM analysis fails")
 

Limitation of this approach Go to top

The main limitation is that the recording length can’t exceed 100 seconds. Other limitation is that for every decoding comparation content, a custom static referent video needs to be created, and that is proven to be a time consuming task. 

 

Test implementation in Python Go to top

For your reference, a full example is available in the source code below:

## @package PQM analysis
# The test is performed by tuning to any channel and observing
# if video is decoded without artefacts.
#
# PQM.dll contains set of functions and variables for controlling PQM algorithms.
# This device contains set of algorithms which are used for video or image analysing.
# Implemented algorithms are detecting black screen, blocking, blurring, packet loss,
# ringing, jitter and freezing.
#
# test_name = PQM analysis
# ID = Path of test case in Intent+
# copyright = 2014 RT-RK. All rights reserved.

## This module provides various time-related functions
import time

## This module contains common definitions used through various tests for Black Box testing.
import TEST_CREATION_API

## Zapping time in digital systems isn't a constant category,
# but there are always some values defined as max zapping duration,
# time in which the zapping will be completed. Here we define max zapping time in seconds.
MAX_ZAP_TIME_DURATION = 6

## Test steps execution
def runTest():
    ## Use Try Except block to catch errors detected during execution
    try:
        ## Set initial result of test execution to FAIL
        test_result = "FAIL"

        ## Initialize grabber device
        # (enable communication between PC and grabber device through LAN interface)
        TEST_CREATION_API.initialize_grabber()

        ## Start grabbing video on video interface (HDMI1)
        # Preview is turned on by default
        # (to turn off the preview set second parameter of function to False)
        TEST_CREATION_API.grabber_start_video_source(TEST_CREATION_API.VideoInterface.HDMI1)

        ## Activate stream player device and play stream which contains static video image
        # [STREAM_WITH_STATIC_VIDEO] macro is defined in StreamPlayer.ini file
        TEST_CREATION_API.stream_play("[STREAM_WITH_STATIC_VIDEO]")

        ## Send RC command to DUT to bring DUT to initial state
        # ([INIT] macro is defined in RemoteController.ini file)
        TEST_CREATION_API.send_ir_rc_command("[INIT]")

        ## Perform DVBT scan
        TEST_CREATION_API.send_ir_rc_command("[DVBT_SCAN]")
        TEST_CREATION_API.send_ir_rc_command("[EXIT_DVBT_SCAN]")

        ## Tune to scanned channel
        TEST_CREATION_API.send_ir_rc_command("[CH_500]")
        time.sleep(MAX_ZAP_TIME_DURATION)

        ## Record video with default duration of recording (5 seconds)
        # If required custom video duration, second parameter should be added as time in milliseconds
        TEST_CREATION_API.record_video("video")

        ## Instance of PQMAnalyse type
        pqm_analyse = TEST_CREATION_API.PQMAnalyse()

        ## Set what algorithms should be checked while analyzing given video file with PQM.
        # Attributes are set to false by default.
        pqm_analyse.black_screen_activ = True
        pqm_analyse.blocking_activ = True
        pqm_analyse.ringing_activ = True
        pqm_analyse.freezing_activ = True
        pqm_analyse.blurring_activ = True
        pqm_analyse.packet_loss_activ = True
        # Name of the video file that will be analysed by PQM. If no extension is provided,
        # PQM assumes that the extension is .mpg,
        # otherwise it will search for the file with the extension that you have provided.
        pqm_analyse.file_name = "video"

        ## Analyse recorded video
        TEST_CREATION_API.pqm_analysis(pqm_analyse)

        ## Check if errors are detected in recorded video, 
        # here the picture on the output should be frozen,
        # because the input signal has static image showing.
        if (pqm_analyse.black_screen_detected == TEST_CREATION_API.AlgorythmResult.NOT_DETECTED and \
                pqm_analyse.blocking_detected == TEST_CREATION_API.AlgorythmResult.NOT_DETECTED and \
                pqm_analyse.ringing_detected == TEST_CREATION_API.AlgorythmResult.NOT_DETECTED and \
                pqm_analyse.freezing_detected == TEST_CREATION_API.AlgorythmResult.DETECTED and \
                pqm_analyse.blurring_detected == TEST_CREATION_API.AlgorythmResult.NOT_DETECTED and \
                pqm_analyse.packet_loss_detected == TEST_CREATION_API.AlgorythmResult.NOT_DETECTED):
            ## Set test result variable to PASS if there is no errors were detected
            test_result = "PASS"
        else:
            ## Write message to log file if errors were detected
            TEST_CREATION_API.write_log_to_file("PQM analysis fails")

    ## If exception is detected,
    # test result variable is set to INCONCLUSIVE and log of error is written to log file.
    except Exception as error:
        ## Set test result variable to INCONCLUSIVE if exception is caught
        test_result = "INCONCLUSIVE"

        ## Write exception message to log file
        TEST_CREATION_API.write_log_to_file(error)

    ## Update test result with current state of test_result variable
    TEST_CREATION_API.update_test_result(test_result)

    ## Return DUT to initial state
    TEST_CREATION_API.send_ir_rc_command("[INIT]")

    ## Stop grabbing video on HDMI1 interface
    TEST_CREATION_API.grabber_stop_video_source()

    ## Stop stream player device
    TEST_CREATION_API.stream_stop()

    ## Deinitialize grabber device
    TEST_CREATION_API.deinitialize_grabber()