OpentestOtherSrvLinuxDevtest

= What's linux-devtest = It is a python script to run automated tests on Linux using Opentest framework.

= How to install it = Download latest opentest_installer*.tar.gz from http://arago-project.org/files/releases/opentest/ untar it, run it and then: Select options 2 (STAF) and 8 (Command Line Tools) Select options 2 (STAF), 4 (TMC), 5 (VATF), 6 (TEE), 7 (BEE) and 8 (Command Line Tools)
 * If you plan to run tests using TI's board farm
 * If you plan to run tests on your own desktop

NOTE: Depending on your linux distribution, you may also need to install python module 'argparse'.

There are many files that contain setup configuration. The installer will preset most values during the installation process, however the most important configuration parameters are documented here:

Defines Test Master Controller to send test requests to. Location: config/site_info.xml  xml element Default: 158.218.108.109 (which is TI Germantown TMC server address)
 * Test Master Controller (TMC) IP address

Defines ftp server directory url where local files (i.e. kernel, dtb, etc.) should be copy to. This URL should be accessible from TMC defined above. Location: config/site_info.xml  xml element Default: ftp://anonymous:anonymous@gtautoftp.gt.design.ti.com/anonymous/opentest/
 * Local FTP Server URL

Configuration parameters applicable to installation case 2) only (running tests on your own desktop):  Describes the equipment (DUT and test) available in the VATF Test Execution Engine  and its properties  Location: /bench/
 * VATF TEE's bench file

Root directory where NFS tar balls (if provided) can be untared and exported to DUT Location: VATF bench file
 * NFS export root directory

= How to use it = linux-devtest.py allows users to load a specified evm type with specified software images and then run a set of tests. The tool supports 3 different types of test executions as described in the following sections:

Executing ad-hoc commands
linux-devtest.py -s Use this option to run any arbitrary commands or even a shell script that can reside in your host pc.

Setting EVM type
linux-devtest.py ... -hw omap5-evm,linux The -hw syntax is evm,capabilities where capabilities is underscore-separated list of strings

Setting SW images
linux-devtest.py ... -p /my/path/MLO -b /my/path/uboot -k /my/path/uImage -d /my/path/dtb -n 1.2.3.4:/my/nfs/root All SW images are optional. If you don't specify them, the test script assumes that by default they are available in the MMC card. It is possible to tweak the default behavior by defining arguments using --advance-params option. For more information see advanced-params

Forcing execution on a specific Test Execution Engine (TEE)
linux-devtest users have different options regarding which TEE host is used to run a test. If you don't know or you are not sure about what a TEE is, please see Opentest Architecture overview The different options to select a TEE are: 1) Let Opentest use any TEE that is capable of running the test request. In this case, Opentest will select a capable idle TEE, and if all capable TEEs are busy, it will put the request in the shorter queue. This is the default behavior when one does not pass -a or -f (more about these is coming below) parameters 2) Use only TEEs that belong to a given farm. The farm are collection of boards dedicated to test automation purposes and located on the same site. At the moment, we have farms at TI Germantown (called tigt_farm) and at TI Dallas (called tid_farm). Users may want to choose a farm close to them for cases where software images are large or when user want the board to use a NFS server hosted by the users PC. To request test execution on a given farm use -f option as shown below: linux-devtest.py ... -f tigt_farm 3) Use a specific TEE. For instance, users who owns TEEs may prefer to run the tests on their TEEs. To force execution on a particular TEE use the -a option. The assignment string pass to -a is " on ", where TEE can be either just the TEE type or the type@id and Machine is the IP address. For example: ./linux-devtest.py ... -a "vatf on 192.168.1.10"

Executing ad-hoc Test Examples
1) Running arbitrary commands ./linux-devtest.py -hw omap5-evm,linux -a "vatf on 192.168.1.10" --advanced-params var_use_default_env=yes -s "echo 'hello world'; echo 'goodbye world'" 2) Running shell script from host machine ./linux-devtest.py -hw omap5-evm,linux -a "vatf on 192.168.1.10" --advanced-params var_use_default_env=yes -s /home/a0850405/tmp/mytest.sh

Executing LTP-DDT Tests
linux-devtest.py -t  Use this option to run your own LTP-DDT test plans.

Setting EVM type
linux-devtest.py ... -hw beaglebone The -hw syntax in this case just identifies the platform name.

Setting SW images
See setting SW images

Creating your own LTP-DDT testplan
1) Update default tests. This will clone ltp-ddt project if required. ./linux-devtest.py -U 2) Create your own test plan cp tests/default.py tests/mytests.py 3) Modify tests_to_run list in tests/mytests.py to suit your needs by copying entries from tests_available list into it. Each entry in tests_available lists points to a ltp-ddt test scenario (aka test suite). The syntax is: : :

Executing LTP-DDT Tests Examples
1) Running mytests testplan using specified kernel, devicetree blob and network filesystem. Execution is forced to tee vatf@2 on machine 158.218.103.10 ./linux-devtest.py beaglebone -n 192.168.1.10:/home/a0850405/NFS_exports/linux/6c0214dbe934fc0c6c2b8dcd2245d31e -a "vatf@2 on 192.168.1.10" -k /mnt/gtautoftp/tmp/carlos/nightly-images/ti-linux-3.8-rc5/uImage-beaglebone.bin -d /mnt/gtautoftp/tmp/carlos/nightly-images/ti-linux-3.8-rc5/uImage-beaglebone.dtb -w -t mytests 2) Running default ltp-ddt tests (i.e. no -t | -T | -s options) on am335x-evm ./linux-devtest.py -hw am335x-evm -a "vatf@4 on 192.168.1.10" -n 192.168.1.10:/home/a0850405/NFS_exports/linux/arago-test/autofs/4bd521fbc31c8e728ac5d778b0648fcd

Executing Testlink Testplans
linux-devtest.py -T  Use this option to run Testlink testplans.

Setting EVM type
linux-devtest.py ... -hw beaglebone The -hw syntax in this case just identifies the platform name.

Setting SW images
See setting SW images

Executing Testlink Tests Examples
1) Running Testlink lcpd_functional_sanity_tests that it is part of linux_psp2 project ./linux-devtest.py -hw am335x-evm -n 192.168.1.10:/home/a0850405/NFS_exports/linux/6c0214dbe934fc0c6c2b8dcd2245d31e -a "vatf@4 on 192.168.1.10" -k /mnt/gtautoftp/tmp/carlos/nightly-images/ti-linux-3.8-rc5/uImage-beaglebone.bin -d /mnt/gtautoftp/tmp/carlos/nightly-images/ti-linux-3.8-rc5/uImage-beaglebone.dtb -T linux_psp2:lcpd_functional_sanity_tests

See arguments list for more information about how to get the list of projects and testplans available in Testlink

Detailed arguments list
usage: linux-devtest.py [-h] [-hw HW] [-p PRI_BOOT] [-b SEC_BOOT] [-k KERNEL] [-m KERNEL_MODULES] [-d DTB] [-r RAMFS] [-n NFS] [-s SCRIPT | -t TESTS | -T TESTPLAN | --list-projects | --list-testplans LIST_TESTPLANS | --list-builds LIST_BUILDS] [-w] [-a ASSIGNED_TO_TEE] [-c] [--advanced-params ADVANCED_PARAMS] [-R REPORT] [-U] [-f FARM] Run ltp-ddt tests using Opentest optional arguments: -h, --help           show this help message and exit -hw HW               DUT type and optional capabilities. i.e.                       beaglebone,linux -p PRI_BOOT, --pri-boot PRI_BOOT Primary bootloader. Can be local file or http/ftp url -b SEC_BOOT, --sec-boot SEC_BOOT Secondary bootloader (i.e. u-boot). Can be local file or http/ftp url -k KERNEL, --kernel KERNEL kernel image to load on DUT. Can be local file or                       http/ftp url -m KERNEL_MODULES, --kernel-modules KERNEL_MODULES kernel modules to install in the FS. Can be local file or http/ftp url -d DTB, --dtb DTB    device tree blob to pass to kernel. Can be local file or http/ftp url -r RAMFS, --ramfs RAMFS RAMFS image to load on DUT (i.e. rootfs.ext2.gz). Can be local file or http/ftp url -n NFS, --nfs NFS    Use NFS. Value can be local file, http/ftp url or nfs url (i.e. server:/path/to/nfs/root). Recent filesystem tar ball is always available at http://gtautoftp.gt.de                       sign.ti.com/anonymous/linux/latest/arago-test- image.tar.gz -s SCRIPT, --script SCRIPT Shell test script to run in the DUT -t TESTS, --tests TESTS File with list of ltp-ddt tests to run -T TESTPLAN, --testplan TESTPLAN Run Testlink named testplan on named testproject optionally using named build. Syntax:' : [:buildname]' --list-projects      Get list of Testlink test projects --list-testplans LIST_TESTPLANS Get list of Testlink test plans for named project ID. Value must be valid testlink project ID. Used --list- projects to get valid project IDs --list-builds LIST_BUILDS Get list of Testlink test builds for named testplan ID. Value must be valid testlink testplan ID. Used --list-testplans to get valid testplan IDs -w, --distribute-workload Allow running test on multiple DUTs in parallel -a ASSIGNED_TO_TEE, --assigned-to-tee ASSIGNED_TO_TEE Force tests to run on this Test Execution Engine -c, --force-test-scripts-clone Clone latest TEE test scripts --advanced-params ADVANCED_PARAMS extra tweaks for advanced users. Separate multiple values with ~ -R REPORT, --report REPORT Get Testlink testplan results in this format -U, --update-default-tests Get latest version of test cases from ltp-ddt -f FARM, --farm FARM Force test to run in a board at this farm (tigt_farm,                        tid_farm, tii_farm). To run on any farm then just use 'farm'

--advanced-params

 * var_use_default_env=1 means Boot using default environment (i.e. env default -a -f; boot )
 * var_use_default_env=2 means Do not touch uboot env, just power cycle and let it go

= how to create new tests (ltp-ddt) = The easier way to create new ltp-ddt tests is to run tests using nfs and then to build/install your local ltp-ddt into this filesystem. One way to implement this strategy is to follow this approach: 1) Get recent arago-test-image filesystem installed on your PC by running hello world (default) test ./linux-devtest -hw omap5-evm -t default -n http://gtautoftp.gt.design.ti.com/anonymous/linux/latest/arago-test-image.tar.gz

2) Make sure ltp-ddt sources are up to date (in case previous step did not clone ltp-ddt) ./linux-devtest.py -U

3) Build Kernel, modules and headers and install them in the filesystem cd make ARCH=arm CROSS_COMPILE=arm-arago-linux-gnueabi- uImage modules sudo make INSTALL_MOD_PATH= modules_install  sudo make INSTALL_HDR_PATH= /usr headers_install

4) Build and install latest ltp-ddt sources cd  make CROSS_COMPILE=arm-arago-linux-gnueabi- KERNEL_INC= /include KERNEL_USR_INC= /usr/include SKIP_IDCHECK=1 PLATFORM=omap5-evm sudo make DESTDIR= /opt/myltp SKIP_IDCHECK=1 PLATFORM=omap5-evm install
 * 1) Change cross-compiler, kernel include and kernel usr include and platform accordingly in the command below
 * 2) KERNEL_INC should point to you kernel sources include directory
 * 3) KERNEL_USR_INC should point to the filesystem/usr/include directory
 * 1) Finally install ltp-ddt in the filesystem

= how to contribute tests back to ltp-ddt = After Setting up your ltp-ddt sources you can create patches against ltp-ddt and submit your contributions to opentest@arago-project.org. You may want to check the README-DDT file in the ltp-ddt directory for more information about how to create new tests.

= More information =
 * Check linux-devtest's README file
 * Check README-DDT file in ltp-ddt directory
 * Ask for help at opentest@arago-project.org