OpentestHowTo

= How to setup new TEE =

Install and Configure optional debian packages
Set proxy settings (i.e. ~/.bashrc and ubuntu network proxy) Set git-proxy.sh (see below) sudo apt-get install nfs-kernel-server sudo apt-get install iperf sudo apt-get install git sudo apt-get install u-boot-tools (required for mkimage) Make sure /etc/exports is exporting NFS mount point, e.g. /home/tigtfarm01/NFS_exports *(rw,nohide,insecure,no_subtree_check,async,no_root_squash) restart service, e.g. sudo service nfs-kernel-server restart sudo apt-get install cifs-utils (previously known as smbfs) sudo apt-get install openssh-server sudo apt-get install tftpd-hpa and set /tftpboot dir in /etc/default/tftpd-hpa sudo apt-get install isc-dhcp-server (if need to provide dhcp service on private network) It is recommended to isolate DUTs from corporate NW. The TEE host PC must have at least 2 ethernet interfaces, one will be connected to the corporate nw and the other to the private test nw. TEE will use corporate iface to register itself with Opentest TMC TEE will run nfs-server, tftp and dhcp services on private iface only. Configure: /etc/default/isc-dhcp-server /etc/dhcp/dhcpd.conf. After configuring the dhcp server setting for the private network, make sure set this option with global scope "one-lease-per-client true;". This ensures that only one address is assigned per MAC which reduces the risk of running out of IP addresses. Might need rule to restart tftp service after iface is up. For example add following to / etc/network/if-up.d/dut-services: restart tftpd-hpa restart nfs-kernel-server restart isc-dhcp-server mount gtautoftp and gtsnowball Download & Run latest opentest installer from http://arago-project.org/files/releases/opentest/ IMPORTANT: Make sure you add '-x 1' to the downwardTranslator.xsl command element to force a reboot on first failure. * Sample proxy env vars: http_proxy=http://webproxy.ext.ti.com:80 ftp_proxy=http://webproxy.ext.ti.com:80 GIT_PROXY_COMMAND=/home/tigtfarm01/git-proxy.sh https_proxy=http://webproxy.ext.ti.com:80 * Sample git-proxy.sh #! /bin/bash PROXYHOST=webproxy.ext.ti.com PROXYPORT=80 exec env corkscrew ${PROXYHOST} ${PROXYPORT} $*

Create udev rules to have fix names to tty nodes associated with each equipment
1) Create udev rules file. e.g. sudo vim /etc/udev/rules.d/opentest.rules 2) For each equipment on which permanent rule is desired do the following: udevadm info --attribute-walk --name /dev/tty Determine unique attribute for the equipment, for example serial number add rule to opentest.rules file. eg: SUBSYSTEM=="tty" ATTRS{serial}=="ST168498" SYMLINK+="vatf@am335x-evm It is recommended to use the tee name as the symlink name for DUT serial ports.  Warning: Some boards like am335x-sk exposes two interfaces with the same serial number because there is only one ftdi chip in the evm,  so on those you can not use serial attribute. One possibility is to use KERNELS atribute which is unique and won't change as long as you don't  change the USB port where the board connects to the host pc,  For a good article on udev rules, check http://www.reactivated.net/writing_udev_rules.html

Setting up IP forwarding and NAT to access Internet from private NW
*** Enable IP forwarding *** $ sudo sysctl -w net.ipv4.ip_forward=1 *** Setup iptables *** $ sudo iptables --flush $ sudo iptables --table nat --flush $ sudo iptables --delete-chain $ sudo iptables --table nat --delete-chain # adjust eth2 below as appropriate. This is the interface with Internet connectivity $ sudo iptables --table nat --append POSTROUTING --out-interface eth2 -j MASQUERADE # adjust eth0 below as appropriate. This is the interface on the private LAN $ sudo iptables --append FORWARD --in-interface eth0 -j ACCEPT $ sudo sh -c 'iptables-save > /etc/iptables.conf' Add the following command to /etc/rc.local iptables-restore < /etc/iptables.conf

Setup DNS servers
* Open /etc/dhcp/dhcpd.conf and setup following entries: option domain-name "ti.com"; option domain-name-servers 192.0.2.2, 192.0.2.3; * Restart dhcp server: sudo service isc-dhcp-server restart Note: http_proxy must be defined in the DUT environment. By default wget uses proxy if environment variable is defined, so when accessing internal URLs it is required to disabled proxy by passing '--proxy off' option in wget command.

Disable WattsUpPro
sudo vim /etc/UPower/UPower.conf Change: EnableWattsUpPro=true To: EnableWattsUpPro=false

Restart upower daemon
* Get process ID, e.g. ps -ef | grep -i power * Kill daemon, e.g. sudo kill -9  * Daemon should start automatically after few seconds, if not you can start it, e.g. sudo /usr/lib/upower/upowerd &

Installing ubuntu 12.04 with SoftwareRAID RAID 1 (You need 2 Hard Drive, preferably same size)
1) Create CD with ubuntu alternate image (e.g. ubuntu12.04.3-alternate-amd64.iso) 2) Create 32GB (logical), and rest-of-space (primary) partitions on each drive. Make sure to set 'use as physical volumne for RAID' on all the partitions Also make sure you set boot flag on primary partition 3) Configure software RAID 4) Create MD device 5) Choose RAID1 and select 2 swap partitions 6) Repeat 4) and 5) for 2 primary partitions 7) Now go to RAID1 device #0. Press enter and set use as 'swap area' 8) Now go to RAID1 device #1. Press enter and set use as 'ext4 journal file system' 9) Compleye ubuntu installation Sample video at http://www.youtube.com/watch?v=z84oBqOxsD0

Installing ubuntu 12.04 with FakeRAID RAID 1 (You need 2 Hard Drives, preferably same size)
1) Create CD with ubuntu alternate image (e.g. ubuntu12.04.3-alternate-amd64.iso) 2) Following the system's motherboard documentation, use bios to setup the hard drive in RAID mode 3) Using the BIOS option ROM utility, create the RAID 1 arrays with the two HD 4) Boot the system with ubuntu CD created in step 1 5) Select "yes" when asked if you want the installer to activate the RAID 1 array 7) Select "yes" when asked if you want to write the changes to disk 8) Select "use complete disk for installation" when asked where to install 9) Enter the required information for system name, user name, password, etc when asked 10)IMPORTANT: Before the installer performs the last step, it will displayed a message saying that it did not detect another OS    and it will ask if it should go ahead and install the GRUB (the bootloader) to the array. Select "No", and when prompt for the install location     enter the dev node of the array, typically /dev/dm-0 (if not sure what the dev node is do Ctrl+F1 to get to a shell and do ls -ltr /dev/mapper/, and use the target of the link, then Ctrl+F7 to return to the installer), then enter. 11) When the installer is done it will ask you to remove the disk and reboot

Configuring static ip address Ubuntu 12.04 (Optional, it is better to use dhcp which is the default configuration)
NOTE: This section is not required in the majority of the cases, the prefer method is to use dhcp for the system to obtain an ip address. This section is here for completeness, and for the special cases in which a static ip address is required. 1) Open Network Connections. Dash home -> type network -> click on "Network Connections" Application. 2) Select the network interface you which to configure and click the "Edit" button. 3) Click on the IPv4 settings tab 4) Select "Manual" from the drop-down box of the "Method:" field 5) Click the "Add" button on the adresses section 6) Enter the IP address, netowork mask and gateway on the editable sections of the address just created 7) Enter the DNS server's adresses (comma separated) in the "DNS servers:" text field. For example 1.2.3.4, 1.2.3.5  8) Enter the comma separated list of search domains (if any) in the "Search Domains:" text field. For example gt.design.ti.com, am.dhcp.ti.com, telogy.design.ti.com, sanb.design.ti.com, toro.design.ti.com, dal.design.ti.com 9) Change the fqdn (fully qualified domain name) so that it matches your new name in /etc/hosts. 10) Reset your machine to make sure the changes take effect. 11) Contact IT so that they can set the address to in use and add the system to their dns servers.

Send notifications when something happens SoftwareRAID
1) Setup email service * sudo dpkg-reconfigure exim4-config   - select mail sent by smarthost, no local mail   - leave defaults for most question except for 'IP address or host name of the outgoing smarthost':     Use: smtp.mail.ti.com 2) Configure email address sudo vim /etc/mdadm/mdadm.conf and then set/change MAILADDR lcpd_systest@list.ti.com 3) restart the PC to make sure mdadm contains right info

Send notifications when something happens FakeRAID
1) Setup email service, sudo apt-get install exim4 * sudo dpkg-reconfigure exim4-config   - select mail sent by smarthost, no local mail   - leave defaults for most question except for 'IP address or host name of the outgoing smarthost':     Use: smtp.mail.ti.com 2) Create a bash script in the root file system (i.e sudo vim /sbin/raid_status.sh) with the following content: #!/bin/sh # check raid status and email if not ok   STATUS=`/sbin/dmraid -s | grep "status"` if [ "$STATUS" != "status : ok" ] then /sbin/dmraid -s | mail -s "RAID ERROR ON `hostname`: $STATUS" lcpd_systest@list.ti.com fi

3) Change the mode of the script to 755, i.e. "sudo chmod 755 /sbin/raid_status.sh" 4) setup a cron job for the root user. Run "sudo crontab -e" and add the following line to the crontab 0 1 * * * . For example to run script /sbin/raid_status.sh every day at 1 am   0 1 * * * /sbin/raid_status.sh

Test notifications when something happens mdadm (SoftwareRAID)
sudo mdadm --detail /dev/md1 (or whatever md node you want to simulate a failure on) sudo mdadm --manage --set-faulty /dev/md1 /dev/sda2 # EMAIL notification should go out at this point # Now remove and add back the node so things go back to normal. # It will take a bit for the recovery process to complete sudo mdadm /dev/md1 -r /dev/sda2 sudo mdadm /dev/md1 -a /dev/sda2

Recovering malfunctioning HD with fakeRAID
1) Shutdown the system, replace the bad HD and reboot 2) Use OROM to add the new HD to the RAID array 3) The Volume in OROM will show a Rebuild status 4) Exit OROM and let the system boot. 5) Use the dmraid command to rebuild "sudo dmraid -R " where  is the name of the raid array created.       To obtain the name of the raid set run "sudo dmraid -s", and copy the value of the name: field that is printed out, i.e. isw_....  6) watch “dmsetup status” for the ratio to be 1.

Upgrading from 12.04 LTS to 14.04 LTS fakeRAID
1) When the "New Ubuntu release xx.xx LTS" alert appears in the Update manager click the "Upgrade" button 2) The system will take some time to download and upgrade the software and eventually an error message will appear stating that there was an error installing GRUB. At this point select continue without installing but DO NOT REBOOT THE SYSTEM AFTER THE UPGRADE COMPLETES. 3) Open a terminal console and do "ls -l /dev/mapper". In the listing you will noticed that the link to ../dm-0 is missing which is why the GRUB installation failed  4) Change directory to /dev/mapper 5) Create a symbolic link to ../dm-0, with "sudo ln -sf ../dm-0 ", where can be obtained with "sudo dmraid -s", and using the value of the name: field that is printed out, i.e. isw_..... Which also happens  to be the base name of the other links found for the RAID array in /dev/mapper  6) Run sudo apt-get purge grub-pc 7) Run sudo apt-get install grub-pc:      a) When asked to remove the files under /boot/grub select no       b) When the grub configuration menu appears select the options containing dm-0 for the  install location  8) Once GRUB is installed and configure, run "sudo update-grub" 9) Reboot the system 10) if a message with grub appears after reboot, open a terminal window and run "sudo update-grub" again. 11) A problem with cups may appear after reboot, select send report and this message should not appear again

Installing a debian based farm system
This sections covers a debian installation streamlined for standalone/farm systems (systems with very little use interaction), where the desktop environment has been removed. In other words, system setup as specified in this section will not provide the user with a desktop/GUI based environment. Before proceeding make sure the system has network connectivity Create an installer CD with http://gtautoftp.gt.design.ti.com/anonymous/releases/debian-8.2-amd64-CD-1.iso Select the "Install" option from the Main menu

Once the installation process has begun it will ask several question regarding host name, root/user accounts, passwords, etc.  when asked where to install select "Manual" and setup a SoftwareRaid installation as explained in the previous Software Raid sections After the installer has finished it will reboot the system, remove the disk and login using the user account created. Once logged in perform the following configuration steps: 1) Change the access mode to 777 of the /tftpboot and /usr/local/staf/data   2) Change ownership recursively (chown -R option) of /usr/local/vatf to .staff 3) Update vatf to the latest version by: cd /usr/local/vatf; git reset --hard HEAD; git fetch; git rebase      (git reset --hard HEAD is done in case there were some changes specific to this installation that are no longer required)     4) Configure the network interfaces by editing /etc/network/interfaces

5) Configure private subnets, tftp server, exported nfs folders, email, udev rules, raid notifications and any other desired feature as explained in the previous sections   6) Edit the .bashrc file and add the proxy configuration by adding the appropriate export statements 7) Add the /sbin folder to the PATH by adding "export PATH=$PATH:/sbin" to .bashrc

8) To install TEEs or BEEs run the opentest installer

= How-To Install Opentest Client tools =
 * Find a Ubuntu system (10.04 or later should be fine)
 * Download and untar latest opentest_installer*.tar.gz (recommended 13.09 or later) from http://arago-project.org/files/releases/opentest/
 * Run the installer (e.g. ./install_opentest.sh) and select option 8 (Command Line Tools). This option will also install the STAF utility.  For installation details see the list below:
 * The installer will install Java if required
 * Then it will install STAF (you can leave all the default install values)
 * Then it will install opentest_client components. You can leave default values for most options, however subnets must be entered. If you are in the US we recommend entering at least following 3 subnets (separated by spaces): 158.218.*.* 10.218.*.* 128.247.*.*
 * When prompted for the Tools installation menu select option 1 to Install All tools.
 * When prompted where to install the files you can enter . to install in the current directory. You should see this directory in the next prompt.
 * Finally you can select option 9 to quit.

= How-To Start STAF = $ staf local help Usage: STAF [-verbose]    if [ $# = 0 ] then STAF_INSTANCE_NAME=STAF else if [ $1 != "start" ] then STAF_INSTANCE_NAME=$1 else # Ignore "start" STAF instance name STAF_INSTANCE_NAME=STAF fi fi export PATH LD_LIBRARY_PATH CLASSPATH STAFCONVDIR STAF_INSTANCE_NAME $ ./startSTAFProc.sh $ nohup: appending output to `/home/charliebrown/nohup.out' $ cat /home/charliebrown/nohup.out Machine         : charliebrown Machine nickname : charliebrown Startup time    : 20130905-18:06:02
 * Check if STAF is running
 * If you don't get an error while running staf local help command, then STAF is running and you are good to go. However if you do get an error then you need to open /usr/local/staf/STAFEnv.sh and verify that STAF_INSTANCE_NAME is set to a value as shown below
 * Source the /usr/local/staf/STAFEnv.sh script to properly initialize your environment for STAF (assuming that STAF was installed in the default location).
 * . /usr/local/staf/STAFEnv.sh
 * Finally run /usr/local/staf/startSTAFProc.sh to start STAF. Cat nohup.out file to check correct initialization
 * You should now be able to run the staf local help command and verify your STAF installation.

= How-To easily Run a boot test on multiple boards = ./CI.rb -m CI_sample_boottest.rb --machine-adapter-type none --input-adapter-type none --build-name chtest --do-not-publish-results
 * Make sure you have already install Opentest Client tools
 * Also verify that STAF is running
 * Go to the location where you installed Opentest client and modify :kernel, :dtb and :modules files path in CI_sample_boottest.rb file
 * Run following command

= How-To Run your own tests on the boards Farm = # Simple test to boot and echo 'hello world' upon booting. ./linux-devtest.py -s "echo 'hello world'"
 * Make sure you have already install Opentest Client tools
 * Also verify that STAF is running
 * Run linux-devtest.py -s from the location where you installed Opentest client tools. You will find this script in the client directory.


 * Please check linux-devtest.py help for more details

= How-To Run LTP-DDT tests on the boards Farm = ./linux-devtest.py -t 
 * Make sure you have already install Opentest Client tools
 * Also verify that STAF is running
 * Run linux-devtest.py -t from the location where you installed Opentest client tools


 * Please check linux-devtest.py help for more details

Creating your own LTP-DDT testplan
./linux-devtest.py -U cp tests/default.py tests/mytests.py Each entry in tests_available lists points to a ltp-ddt test scenario (aka test suite). The syntax is: : :
 * Update default tests. This will clone ltp-ddt project if required.
 * Create your own test plan
 * Modify tests_to_run list in tests/mytests.py to suit your needs by copying entries from tests_available list into it.

How to create new LTP-DDT tests


LTP-DDT expects certain test binaries to be installed in the filesystem. If you use a tisdk filesystem then all the required test tools have been already installed in the filesystem. One can get a recent tisdk rootfs image from http://lcpd.dal.design.ti.com/core-sdk/nightly/deploy/images/ Select appropriate tisdk-rootfs-image-*.tar.gz
 * Export NFS filesystem in your host.

./linux-devtest.py -U
 * Make sure ltp-ddt sources are up to date. This step will clone latest ltp-ddt project into ltp-ddt directory

cd make ARCH=arm CROSS_COMPILE=... zImage modules sudo make INSTALL_MOD_PATH= modules_install sudo make INSTALL_HDR_PATH= /usr headers_install
 * Build Kernel, modules and headers and install them in the filesystem

cd . # This is the directory created by Step 2 above make CROSS_COMPILE=arm-linux-gnueabihf- KERNEL_INC= /include KERNEL_USR_INC= /usr/include SKIP_IDCHECK=1 PLATFORM=omap5-evm sudo make DESTDIR= /opt/myltp SKIP_IDCHECK=1 PLATFORM=omap5-evm install
 * Build and install latest LTP-DDT sources
 * 1) Change cross-compiler, kernel include and kernel usr include and platform accordingly in the command below
 * 2) KERNEL_INC should point to you kernel sources include directory
 * 3) KERNEL_USR_INC should point to the filesystem/usr/include directory
 * 1) Finally install ltp-ddt in the filesystem

how to contribute tests back to ltp-ddt
Please submit your LTP-DDT patches to opentest@arago-project.org. You may want to check the README-DDT file in the ltp-ddt directory for more information about how to create new tests.

= How-To Run Testlink tests plans on the boards Farm = ./linux-devtest.py -T 
 * Use this option to run a Testlink test plan. If you use this option results will be stored in the Testlink server.
 * Make sure you have already install Opentest Client tools
 * Also verify that STAF is running
 * Run linux-devtest.py -T from the location where you installed Opentest client tools

The Testlink testplan name syntax is:  : 

To get the list of available projects, use following command: ./linux-devtest.py --list-projects

To get the list of available testplans on a project, use --list-testplans  where project ID can be obtained from --list-projects command. For example: ./linux-devtest.py --list-testplans 1140921 '''The above command will return the list of testplans for LCPD linux. Project linux_psp2(id=1140921) hosts LCPD's linux test plans.'''

Please note that --list-testplans uses project ID, while linux-devtest.py -T uses project and testplans names.


 * Please check linux-devtest.py help for more details

= linux-devtest.py arguments help =

command help
./linux-devtest.py -h
 * 1) use -h flag to show command help

-hw argument
The -hw argument selects the type of hardware (aka DUT) to run the tests on. It is also possible to specify capabilities or requirements that selected DUT must have. For instance, capabilities could be used to ask for a board that has an usb mass storage device attached to it. The argument syntax is -hw evm,capabilities where capabilities is an optional underscore-separated list of strings Valid evm names are the same used by OE/Yocto: am335x-evm, omap5-evm, beaglebone, dra7xx-evm, etc. For valid capabilities see Capabilities cheat sheet linux-devtest.py ... -hw omap5-evm,linux

-p, -b, -k, -m, -d, -r, -n Software arguments
Set software images (i.e. bootloader, kernel, filesystem, etc.) to used for the test. It is not mandatory to provide all of them. For example if bootloader (-p and -b) are not provided then the system will try to boot the DUT using whatever bootloader image is already installed on it. Most software image values can take either a local path or a http/ftp url. Use ./linux-devtest.py -h for more details

-u, --user-bins argument
Executable or tarball of executables that should be installed on FS. This is useful for cases where user wants to run a binary that is not available in the default filesystem The value provided can be either a local file or a public http/ftp url linux-devtest.py ... -u /home/tmp/myapp

-s, --script argument
Use this option to run any arbitrary commands or a shell script in the DUT. The syntax is either a string of semicolon-separated commands (e.g. "echo 'hello'; echo 'bye'") or a path to a shell script. In either case the test will pass if the return value of the commands or script is zero. linux-devtest.py ... -s /home/tmp/mytest.sh

-t, --tests argument
Use this option to run your own ltp-ddt test plan. Please note that -s, -t and -T are mutually exclusive options linux-devtest.py ... -t mytests To create your own ltp-ddt test plan follow this steps: ./linux-devtest.py -U cp tests/default.py tests/mytests.py Each entry in tests_available lists points to a ltp-ddt test scenario (aka test suite). The syntax is: : :
 * Update default tests. This will clone ltp-ddt project if required.
 * Create a copy of default test plan
 * Modify tests_to_run list in tests/mytests.py to suit your needs by copying entries from tests_available list into it.

-T, --testplan argument
The Testlink Testplan name syntax is: :. For example: ./linux-devtest.py -T linux_psp2:ch_sandbox
 * Use this option to run a Testlink test plan. If you use this option results will be stored in the Testlink server.

./linux-devtest.py --list-projects
 * To get the list of available projects, use following command:

./linux-devtest.py --list-testplans 1140921 '''The above command will return the list of testplans for LCPD linux. linux_psp2 is the project name that host LCPD's linux test plans.'''
 * To get the list of available testplans on a project, use --list-testplans  where project ID can be obtained from --list-projects command. For example:

Please note that --list-testplans uses project ID, while linux-devtest.py -T uses project and testplans names.

-f, --farm argument
forces usage of a board in a farm. -f tigt_farm -f tid_farm

--advanced-params
advanced-params are variables that advanced users can set to tweak execution. At the moment the only advanced-params supported by linux-devtest.py is var_use_default_env. Usually linux-devtest.py will try to boot the DUT using provided software images (i.e. kernel, filesystem, etc.), but advanced users may change that default behavior.
 * var_use_default_env=1 means Boot using default environment (i.e. env default -a -f; boot )
 * var_use_default_env=2 means Do not touch uboot env, just power cycle and let it go

Errors
$ sudo apt-get install python-argparse
 * If you get ImportError: No module named argparse while running linux-devtest.py then you need to install python-argparse

= How to change the default limits in the TMC (Ubuntu Server) = The default configuration should be good enough to handle the socket and thread demands in most components in the system. However, for high load scenarios, these default settings might not be enough for the number of requests received in the TMC. In this scenarios the following steps may be taken to avoid errors and crashes: #Opentest tweaks # Increase number of incoming connections net.core.somaxconn = 65535 # Increase number of incoming connections backlog net.core.netdev_max_backlog = 65536 net.ipv4.ip_local_port_range = 15000  61000 net.ipv4.tcp_fin_timeout = 25 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_tw_reuse = 1 #Increase the number of allowed threads and open files kernel.threads-max = 999999 kernel.pid_max = 999999 vm.max_map_count = 1999999 fs.file-max = 999999 * hard nofile 999999 * soft nofile 999999 * hard stack 102400 * soft stack 102400 SERVICE TMCDISPATCHER LIBRARY JSTAF EXECUTE /opt/opentest/Dispatcher.jar OPTION "JVMName=DISPATCHERJVM" OPTION "J2=-Xms12g" OPTION "J2=-Xmx12g"
 * Edit /etc/sysctl.conf and add the following:
 * Edit /etc/security/limits.conf and add the following:
 * When registering the TMC services with staf, do it on a separate JVM and specify the heap size, i.e the registration command/STAF.cfg should look something like this:
 * Once these changes are made reboot the system and restart the TMC