This page describes the available regression tests, how to set them up if you have the data, and the length of time to run each test. The page also describes the current weekly schedule of regression tests for Sphinx 4. The schedule was based upon the times of the tests and the time constraints in which the tests must be run.

These tests were designed to verify performance regression. They run automatically in machines located at Carnegie Mellon. The tests use data released by the LDC. The advantage is that the data are well known in the speech community. The disadvantage is that the data are licensed, and not everyone has access to them.

There are plans to create unit regression tests that could be run by developers just before checking in code. These would run quickly, providing a fast test that things did not break. They would use open source data also, so anyone could run the tests.

Overview

The regression test main script does a fresh download of the code from the Sphinx-4 repository (currently, a svn repository at http://sourceforge.net). The script runs tests, and raw result numbers are stored at a cvs repository at sourceforge.net. The script also creates HTML reports (//cf.// tests running on filbert) and sends email reports to the cmusphinx-results mailing list. Check the main mailing list page for the archive or to subscribe/unsubscribe.

Installing the Tests

Required software

The tests run automatically as a cron job. Therefore, the system that runs the tests needs to have the following easily available (//e.g.// in the system path):

• cron

• svn

• cvs

• rsync

• bash

• awk

• javac, version 1.6 at least

Storing results

The test results are stored in files kept in a CVS repository at sourceforge.net. They are kept in CVS rather than SVN to avoid sending a “commit” message every time the regression test scripts update the results. There are steps that have to be done manually.

First, get the CVS data.

env CVS_RSH=ssh cvs -z3 -d:ext:USERNAME@cmusphinx.cvs.sourceforge.net:/cvsroot/cmusphinx checkout regressionResults

About once a year, clean the main regression.log file from old results. For example, for year 2010, you do the following.

# cleanup.sh
grep '|2010-' regression.log > regression.2010.log
grep -v '|2010-' regression.log > regression.temp
cat regression.header regression.system regression.temp > regression.log
rm regression.temp
cvs add regression.2010.log
cvs co -m "update files" 

If the machine you are using for tests is not already in the regression.log file, you will have to update both regression.log and regression.system (or only the latter if it is time for the annual cleanup). You will have to add a line contaning, in this order, as detailed in regression.header:

1. the string “system” literally
2. machine name
3. number of CPUS
4. cacheSize (in kbytes)
5. clock (MHz)
6. memory (Mbytes)
7. architecture
8. OS

For example, this line was added for the machine filbert:

system|filbert|8|4096|2660|15904|x86_64|Linux|

In Linux, the information about CPU speed, memory, etc can be found in either /proc/meminfo or /proc/cpuinfo.

Data

The tests assume that the data (audio, acoustic models) used are available under /lab, and the environment variable $SF_ROOT points to the root of a working copy of the sphinx4 code.

At CMU, the data are available from the robust account at ~robust/lab. Create the link:

ln -s ~robust/lab /lab

Final steps

Create the variable SF_ROOT pointing to the working copy of the repository. If the Sphinx-4 working copy is located at ~/SourceForge, add this to your ~/.profile file, or create it if it does not exist:

export SF_ROOT=${HOME}/SourceForge

With these in place, install the crontab below. Beware that cron uses bash regardless of your choice of shell.

crontab regression_crontab

# regression_crontab
MAILTO=cmusphinx-results@lists.sourceforge.net
50 18 * * * (. $HOME/.profile ; cd $SF_ROOT/sphinx4/tests/regression; svn -q up .; ./regressionTest nightly batch)
35 23 * * 0 (. $HOME/.profile ; cd $SF_ROOT/sphinx4/tests/regression; ./regressionTest sunday batch)
35 23 * * 1 (. $HOME/.profile ; cd $SF_ROOT/sphinx4/tests/regression; ./regressionTest monday batch)
35 23 * * 2 (. $HOME/.profile ; cd $SF_ROOT/sphinx4/tests/regression; ./regressionTest tuesday batch)
35 23 * * 3 (. $HOME/.profile ; cd $SF_ROOT/sphinx4/tests/regression; ./regressionTest wednesday batch)
35 23 * * 4 (. $HOME/.profile ; cd $SF_ROOT/sphinx4/tests/regression; ./regressionTest thursday batch)
35 23 * * 5 (. $HOME/.profile ; cd $SF_ROOT/sphinx4/tests/regression; ./regressionTest friday batch)
35 23 * * 6 (. $HOME/.profile ; cd $SF_ROOT/sphinx4/tests/regression; ./regressionTest saturday batch)
50 23 * * 0 (. $HOME/.profile ; cd $SF_ROOT/sphinx4/tests/regression; ./regressionTest async0 batch)
50 23 * * 1 (. $HOME/.profile ; cd $SF_ROOT/sphinx4/tests/regression; ./regressionTest async1 batch)
50 23 * * 2 (. $HOME/.profile ; cd $SF_ROOT/sphinx4/tests/regression; ./regressionTest async2 batch)
50 23 * * 3 (. $HOME/.profile ; cd $SF_ROOT/sphinx4/tests/regression; ./regressionTest async3 batch)
50 23 * * 4 (. $HOME/.profile ; cd $SF_ROOT/sphinx4/tests/regression; ./regressionTest async4 batch)
50 23 * * 5 (. $HOME/.profile ; cd $SF_ROOT/sphinx4/tests/regression; ./regressionTest async5 batch)
50 23 * * 6 (. $HOME/.profile ; cd $SF_ROOT/sphinx4/tests/regression; ./regressionTest async6 batch)
30 17 * * * (. $HOME/.profile ; cd $SF_ROOT/sphinx4/scripts; svn -q up .; ./updateS4Javadocs.sh)
05 02 * * * (. $HOME/.profile ; cd $SF_ROOT/web; svn -q up .; $SF_ROOT/web/script/nightlyBuild.sh)
05 06 * * * (. $HOME/.profile ; cd $SF_ROOT/web; svn -q up .; $SF_ROOT/web/script/update_sf.sh)
00 03 * * * (. $HOME/.profile ; cd $SF_ROOT/web; svn -q up .; $SF_ROOT/web/script/sfbackup.sh)

Regression Test Times

This chart shows the available tests and the approximate time to run each test.

Some test notes

• trigram tests are going away in favor of trigram_fst tests
• Each test has a ‘quick’ version that takes 1/5 as long as the full test
• There is a flaw in the fst/SimpleLinguist implementation that yields very large heaps for the rm1_bigram_fst and rm1_trigram_fst tests. Once this flaw if corrected, the rm1 fst tests can be incorporated into the weekly schedule

Test naming conventions

Full test names are built by concatenating the test name and the language model. Some examples are:

• an4_words_wordlist
• rm1_flat_unigram_quick
• an4_spelling_trigram_fst
• tidigits_wordlist_quick

Note that this is a minor modification to the current naming scheme. Previously, some tests had no language model listed (an4, ti46). The regression.log will be updated to reflect this change for all old tests.

Test schedule

Tests are run every night, on multiple machines and operating systems. Tests start no earlier than 8PM eastern time, and should run no later than 6AM the following morning. This allows for 10 hours of test time per machine per day. Saturday and Sunday tests can run between the hours of 6AM and 8PM as well.

Standard Test

There is a ‘standard test’ set which is run every night on all machines. It consists of the following test:

Weekly test schedule

By day:

By test:

• Note: Once the RM1 tests have been optimized to run in a reasonable amount of time, they will be added to the set of standard tests.

Test Machines

Historical Test Machines