Configurations & macros
Configurations and macros are used to provide common data for devices and scripts that are used throughout group of test cases (test plan). In this section, we will introduce you to concepts of configurations and macros.
Configurations Go to top
Introduction Go to top
Configuration (Testing Environment) is a software and hardware setup on which the testing team is going to perform the testing of the Device Under Test (DUT). This setup consists of the physical setup which includes hardware components, and logical setup that includes Audio/Video algorithms, commands protocol etc. More informations regarding general principles of BBT Testing System and how it works, can be read in the Getting started with automated testing section.
Configurations should be chosen according to the requirements of the test cases or in other words, what should be tested on the DUT and how. According to the previous statement, a tester should make a list of all DUT interfaces that need to be tested and how will they be handled:
- which inputs on the DUT will be used and which devices will feed them
- which outputs on the DUT will be observed and which grabber devices will be used
- how will the DUT be controlled
- which algorithms will be used to make an assessment of the response of the DUT
Example Go to top
Typical configuration contains the following devices:
- RT-AV100 – The Grabber Device which captures DUT Audio/Video output and sends it to the PC to be analyzed using algorithms for Audio/Video analysis
- RT-IR001 – The Remote Controller which places the DUT into a predefined test condition by navigation through its menus.
- Dektec DTU-215 – The Stream Modulator with the task of feeding the DUT with the test signal
- Picture Algorithm – for offline analysis of pictures grabbed from the DUT
- Audio Algorithm – for offline analysis of audio data grabbed from the DUT
- Serial interface – for purpose of sending RS-232 commands to the DUT
Here is an example of the test environment configuration file based on the configuration described above:
[interface] alias = COM1 name = SERIAL port = 1 baud = 115200 //baud = 4800 - for Marmitek parity = N length = 8 stop = 0 //SerialGeneric.dll contains the set of functions and variables for controlling //communication between BBT system and any device connected to the appropriate //serial port. This device driver enables user to set communication parameters //(such as timeout limit, length of received data, predefined 'end of transmission’ //word), communicate over serial port and to define file for logging received data). [device] alias = SerialGeneric name = SERIALGENERIC interface = COM1 [interface] alias = Socket name = SOCKET protocol = RAW port = 5001 address = 192.168.227.31 //Stream player if DekTec is on the local machine [device] alias = LocalStreamPlayer name = DTA107 config = StreamPlayer.ini Path = D:\Streams\ //Stream player if DekTec is on the remote machine [device] alias = RemoteStreamPlayer target = Socket name = DTA DeviceSN = 4215002996 //Stream player UT100B - DVB-T modulator //UT100B.dll contains set of functions and variables for controlling //ITE-UT100B/C device. T100B/C is a digital video USB device which //works as DVB-T modulator. It generates a modulated RF signal in a [50 MHz – 950 MHz] //and [1200 MHz – 1350 MHz] frequency ranges with 1KHz step size. //Device is powered and communicated with pc via USB bus. [device] alias = StreamPlayer name = UT100B config = StreamPlayer_UT100B.ini Path = \\streamsrv\streams\Bytel\ //USBRemoteController.dll contains set of functions and variables for controlling //USB infrared remote controller emulator. [device] alias = RemoteController name = REMOTECONTROLLER idtype = 0 id = RT-IR001 config = RemoteController.ini cfgfile = RemoteController.rcc gap = 450 //PictureBlockCompare.dll contains set of functions and variables for controlling //PictureBlockCompare device. PictureBlockCompare device is not a physical device //which appears in the system, but an implementation of the picture compare //algorithm which is used for calculating picture quality. The algorithm performs //comparison of two pictures (grabbed and referent) as well as comparison of //regions of interest. [device] alias = PictureAlgorithm name = PICTUREBLOCKCOMPARE config = PictureAlgorithm.ini //Also you can set some of the device parameters inside the configuration file: //reffile_path = D:\Referent_Files_VSTV_SW_version_1_11\ //pixels_to_shift = 3 //BlockHeight = 16 //BlockWidth = 16 //BlockHeight = 40 //BlockWidth = 50 //Manual.dll contains set of functions and variables for controlling Manual device. //Manual is not a device which physically exists in the system, it is only a dialog //used in semi-automatic testing when it is needed to set the test case execution //result by the tester. [device] alias = Manual name = MANUAL //RT-AV101.dll contains set of functions and variables for controlling RT-AV101 //Capture device. RT-AV101 is a device which performs real-time capturing of audio and //video content from the STB and its transfer to PC for further analysis. Device is //communicating with PC through network interface (Ethernet). Network connection has //to be 1Gbps in order to transfer images in real-time and Maximum Transmission Unit //(MTU) should be set to 1500. [device] alias = RT-AV101 name = RT-AV101 config = RT-AV101.ini single_grab = 1 //RegionExtractor.dll contains set of functions and variables for controlling //device for selection of region of interest on test picture. RegionExtractor device is //not a physical device which appears in the system, but an implementation of the //method for defining of the part of the picture which will not be analyzed during //picture comparison. //Defining of above mentioned regions of interest will be performed by masking of //all parts of the picture which are irrelevant for specific test procedure. [device] alias = RegionExtractor name = REGIONEXTRACTOR //FFT.dll contains set of functions and variables for controlling FFT (Fast //Fourier Transform) algorithm. This algorithm compares two audio files where //sample rate, sample size and number of channels in compared files must be //the same. [device] alias = AudioAlgorithm name = FFT config = AudioAlgorithm.ini log_file = log_fft.txt threshold = 30 //AudioAbsence.dll contains set of functions and variables for controlling device for //detecting audio absence. AudioAbsence device is not a physical device which appears //in the system, but an implementation of the algorithm which detects audio level in a //wav file. [device] alias = AudioAbsence name = AUDIOABSENCE config = AudioAbsence.ini threshold = -50 //Excel.dll contains set of functions and variables for reading and writing data from //Excel cells. [device] alias = EXCEL name = EXCEL //PSNRBlock.dll contains set of functions and variables for controlling PSNRBlock //device. PSNRBlock device is not a physical device which appears in the system, //but an implementation of the PSNR algorithm which is used for calculating picture //quality. [device] alias = PSNR name = PSNRBLOCK config = PictureAlgorithm.ini //OCR.dll contains set of functions and variables for controlling algorithm for optical //character recognition (OCR). OCR is an algorithm for conversion of captured //pictures of typewritten or printed text into computer-readable text. [device] alias = OCR name = OCR config = OCR.ini [device] alias = StringCompare name = LCSW //Marmitek device used for hard reboot of DUT //[device] //alias = PowerSwitch //name = POWERSWITCH //interface = COM1 //USB relay device used for hard reboot of DUT [device] alias = USBRelay name = USBSPDIFMUX idtype = 0 id = RT-RELAY config = USBRelay.ini //When test plan is loaded (by default) Relay will be turned on. //If you want to be turned off uncomment next line //chid = 0 //Serial relay device used for hard reboot of DUT [device] alias = Relay name = RELAY_SIMPLE interface = COM1 //PQM.dll contains set of functions and variables for controlling PQM algorithms. //This device contains set of algorithms which are used for video or image analyzing. //Implemented algorithms are detecting black screen, blocking, blurring, packet loss, //ringing, jitter and freezing. [device] alias = PQM name = PQM //AQM.dll contains set of functions and variables for controlling AQM algorithms. //AQM device is used audio analyzing and implemented algorithms that can be used //for this purpose are audio absence, clicking, clipping and freezing. [device] alias = AQM name = AQM
With this configuration, basic functional and stress test cases can be performed. Depending on the test cases requirements and in house hardware equipment, they can be replaced with another devices which would perform the same activity.
Anatomy of Configuration file Go to top
Declaration of each device begins with one of the two keywords:
- [interface] – used to define communication interface for device from the test configuration
- [device] – used to define device (physical or logical) for test configuration
When [interface] or [device] has been defined, the following two fields are mandatory:
- alias – defines the identification of a device which will be referred from the test case script. This name is arbitrary and can contain up to 20 characters (spaces are not allowed). For example, if an alias is defined as the PictureAlgorithm (as from the example configuration above), it will be used from the Python script like this:
device.handler("PictureAlgorithm", "SET", "refpicture", "Example.bmp")
More informations about device.handler function can be found in document on this location.
- name – represents a predefined name for each device. Handling of devices (physical and logical) is realized through separate libraries. Each library exports device name and this name can not be changed. Predefined name for each device can be found inside the documentation for a specific device. All the documentation is located inside the folder where the RT-Executor has been installed (Documentation/Test creation manual)
After the device has been defined, the following parameters are optional:
- cfgfile – configuration file for the device. This device is specific for each device implementation inside its own library. For example, the USBREMOTECONTROLLER (from the example configuration above) cfgfile is set to the “RemoteController.rcc” file name, which holds learned RC codes for the DUT RC controller, generated by the RC Capture application
- config – macro file for the device. More information about macro files is located in the Macros section of this document
- interface – used to define communication interface (SERIAL or SOCKET) for a specific device. From the example configuration above, it can be seen, that first communication interface (SERIAL) has been defined (with alias COM1), and than that device has been assigned to the SerialGeneric device
- device variables – each device has a group of variables, by means of which the functionality of that device is implemented. Each variable has a default value. To change this value at the test plan loading point, user can add it to the configuration file. For the configuration example above, port, baud, parity, length and stop are such examples for the SERIAL device. More information about the list of variables for devices can be found inside the folder with the documentation (Documentation/Test creation manual)
Macros Go to top
Commands to devices can be issued in two ways:
- Direct commanding
- Macro usage
Direct commanding approach is based on configuring and controlling of the device from the test script exclusively, where all parameters and commands are executed in separate lines. Following example shows this approach on the RemoteController device:
# Send specified RC key # MENU - key which we are sending # 1 - how many times are we going to send this key in a row # 500 – delay in ms after the last sending of the key # 1 – for how long this key should be sent (this is the feature when long key press is required) device.handler("Remote*", "SET", "\”MENU 1 500 1”\", "") # Example of long key press on POWER button (~3s) device.handler("Remote*", "SET", "\”POWER 1 100 70”\", "")
Advantages: This kind of device controlling allows direct modification of all device parameters inside the test script itself, speeds up test creation process (especially of small groups of tests) and allows easy and quick modification of a single test case.
Disadvantages: When configuring and controlling devices from the test itself, such tests often become less readable as well as too large and very complicated for maintenance.
Macro usage (macro file usage) is approach where user can use previously prepared macro definitions (set of commands which will be sent to the device) stored in the macro file. The following example shows this approach on the RemoteController device:
# Send one or set of keys via USB RC emulator # [ENTER_PIN] is set of keys which will be sent # # macro example: # [ENTER_PIN] “4 4 300” “OK 1 500” device.handler("RemoteController*", "SET", "[ENTER_PIN]", "")
Here is the example of the macro file for the RemoteController (RemoteController.ini):
[POWER_ON_OFF] “POWER 1 300” [ENTER_PIN] “4 4 300" "OK 1 500” # “4 4 300” : RC emulator will send key “4” four times in a row # and after the last sending delay will be 300ms; between # consecutive sending delay is defined with variable “gap” # (default 300ms) # “OK 1 500” : RC emulator will send key “OK” once and after that # delay is 500ms [FACTORY_RESET] "MENU 1 2000" "LEFT 1 500" "DOWN 1 500" "OK 1 20000" [MEDIA_BROWSER_PHOTOS] "USB 1 2000" "LEFT 2 500" "OK 1 500" "DOWN 1 500" "OK 1 500" [EXIT_MEDIA_BROWSER] "STOP 1 1000" "BACK 2 1000" [SORT_CHANNEL] "MENU 1 2000" "DOWN 1 500" "DOWN 2 500" [CHANGE_ASPECT_RATIO_4_3] "MENU 1 2000" "RIGHT 1 500" "DOWN 1 500" "RIGHT 1 3000" "OK 1 4000"
Advantages: This kind of device controlling makes test cases shorter and clearer (more readable), particularly when macros are intuitively named ([FACTORY_RESET], [ENTER_MEDIA_SUBMENU]…). Also, this approach allows easy adjustment of large number of test cases without changing of test scripts by simple modification of macro definitions (modifications will affect all test scripts where particular macro is used).
Disadvantages: The lack of parametrization causes the increase of the number of macros, especially during continuously repeated actions (e.g. channel search on different frequencies).
Described mechanism of macros can be applied on all other types of devices. The syntax of writing macro lines inside macro file follows:
[MACRO_NAME] “variable1 value1 delay1” “variable2 value2 delay2” “variable3 value3 delay3” …
- MACRO_NAME is name of macro
- variable1, variable2 … variableN are variables for specific device
- value1, value2 … valueN are values to which variables are initialized
- delay1, delay2 … delayN are delays after execution of macro (in miliseconds)