Getting started with automatic tests
On this page, we will provide you with the overview of automated testing using our BBT platform. We will explain general principles of the Black Box Testing methodology, show you how such methodology is typically used in practice based on our BBT framework and which components are involved in automated testing. Here we also explain organization of tests into test plans and how test plans are loaded and executed. With this knowledge in hand, you will be ready to proceed further with the hardware and software set-up and run your first automated test!
About BBT Testing System Go to top
BBT platform is based on the Black Box Testing methodology (Figure 1). This type of testing ignores internal mechanisms of the system or component being observed and focuses specifically on the outputs generated as the system response to specific inputs and conditions of test execution. Test designer selects valid input and determines the correct output.
BBT platform is a framework for automated testing of multimedia devices. BBT platform consists of software and hardware components offering a wide range of possibilities for testing of integrated DTV systems, digital satellite and terrestrial receivers (STB), DVD, Blu-ray players etc. Platform utilizes Python scripting language for test automation, providing ways to control all components of a test system through simple Python API.
Typical BBT Test setup Go to top
Considering general Black Box Testing ideas shown in Figure 1, we will present you with a typical test-setup based on our BBT platform. As you can see in Figure 2, each general item from Figure 1 has its representation in our system, with addition of BBT test station which controls all components of the test system. Test station executes automatic test scripts which are written in Python scripting language. Python test scripts have access to each hardware or software component of the system through platform API functions. Central software component of the test station is RT-Executor, application through which test scripts are loaded and executed and which provides drivers needed to establish control of physical hardware from Python script.
BBT test station may control various signal sources in order to provide known inputs to DUT. In example use-case, we may use stream modulator such as Dektec DTU-215 to provide known test input to STB being tested. The list of supported signal sources is provided in supported devices section.
As Black Box Testing methodology assumes that we control DUT by using only its external controls, we follow the same approach here. Typical STB/DTV device is controlled via remote controller and therefore BBT platform provides several programmable remote control units, such as RT-IR001 and RT-IR016U (see section Remote Controller device). Issuing RCU commands to DUT is fully supported in Python, with the possibility to create complex navigation patterns using macros.
Having DUT set into expected state and providing known input, the next step is capturing DUT output. For DUT such as STB, typical output is HDMI. RT-RK provides its own audio/video grabber device RT-AV100, which is a perfect match for testing purposes. Furthermore, BBT platform provides support for 3rd party devices such as Black Magic grabber device.
The outcome of a test is determined by analyzing captured DUT output. For this purpose, BBT platform provides various audio/video/picture analysis algorithms (treated as Logical Devices) which may be used to analyze various aspects of captured audio/video/picture. For example, OCR device may be used to recognize text on part of the captured image, Picture Compare device may be used to match a part of captured image with the referent image, Audio Presence Algorithm may be used to detect presence of audio in a captured audio content.
BBT test execution procedure Go to top
Flow chart of test execution procedure used in BBT Test System is illustrated on Figure 1. As figure shows, it is not as simple as loading and executing single test case, due to the fact that each test case requires proper environment in which it is executed. This environment assumes devices needed for test which are properly initialized.
Loading of test plan is actually a preparation phase for the test case execution. This operation includes several important operations:
- Obtaining the list of test cases from the server
- Loading the list of devices used in a test plan
- Initialization of loaded devices
At the end of this step, test execution environment if fully initialized and ready for the test execution.
Load Test Case is a phase where main test script file and all other files and configurations relevant for testing (Audio/Video referent files, region masks etc.) are loaded from the server and stored locally to the test station.
Execute Test Case is a main part of a test procedure where main script file is loaded and executed. Script file is written in Python programming language as a series of steps which define test scenario. More details how to write Python script file can be found in Test Scripting section.
Store Results To Database stage is executed after previous test case script has been successfully executed. In this stage, main test case result (Pass, Fail or Inconclusive) will be stored to the central database together with all intermediate results obtained during test case execution.
Generate Reports stage is performed after all test cases from loaded test plan are executed and their results written to the database. It’s important to mention that all the results are stored in the centralized database so user can create different kinds of reports, depending on which reports should be presented.
Get started! Go to top
Now that you are introduced to the basic principles of the Black Box Testing and corresponding components of the BBT platform, it is time to setup an environment needed to run your first test script using the BBT platform. Please follow these steps: