Cram: It's test time

Cram is a functional testing framework for command line applications based on Mercurial's unified test format.

Cram tests look like snippets of interactive shell sessions. Cram runs each command and compares the command output in the test with the command's actual output.

Here's a snippet from Cram's own test suite:

The $PYTHON environment variable should be set when running this test
from Python.

  $ [ -n "$PYTHON" ] || PYTHON="`which python`"
  $ [ -n "$PYTHONPATH" ] || PYTHONPATH="$TESTDIR/.." && export PYTHONPATH
  $ if [ -n "$COVERAGE" ]; then
  >   coverage erase
  >   alias cram="`which coverage` run --branch -a $TESTDIR/../scripts/cram"
  > else
  >   alias cram="$PYTHON $TESTDIR/../scripts/cram"
  > fi
  $ command -v md5 > /dev/null || alias md5=md5sum

Usage:

  $ cram -h
  [Uu]sage: cram \[OPTIONS\] TESTS\.\.\. (re)

  [Oo]ptions: (re)
    -h, --help          show this help message and exit
    -V, --version       show version information and exit
    -q, --quiet         don't print diffs
    -v, --verbose       show filenames and test status
    -i, --interactive   interactively merge changed test output
    -y, --yes           answer yes to all questions
    -n, --no            answer no to all questions
    -E, --preserve-env  don't reset common environment variables
    --keep-tmpdir       keep temporary directories
    --shell=PATH        shell to use for running tests (default: /bin/sh)
    --indent=NUM        number of spaces to use for indentation (default: 2)

The format in a nutshell:

Download

Installation

You can use pip to install Cram:

$ pip install cram

Or you can install Cram using GNU make:

$ wget https://bitheap.org/cram/cram-0.6.tar.gz
$ tar zxvf cram-0.6.tar.gz
$ cd cram-0.6
$ make install

Usage

Cram will print a dot for each passing test. If a test fails, a unified context diff is printed showing the test's expected output and the actual output. Skipped tests (empty tests and tests that exit with return code 80) are marked with s instead of a dot.

For example, if we run Cram on its own example tests:

.s.!
--- /home/brodie/src/cram/examples/fail.t
+++ /home/brodie/src/cram/examples/fail.t.err
@@ -3,21 +3,22 @@
   $ echo 1
   1
   $ echo 1
-  2
+  1
   $ echo 1
   1

 Invalid regex:

   $ echo 1
-  +++ (re)
+  1

 Offset regular expression:

   $ printf 'foo\nbar\nbaz\n\n1\nA\n@\n'
   foo
+  bar
   baz

   \d (re)
   [A-Z] (re)
-  #
+  @
s.
# Ran 6 tests, 2 skipped, 1 failed.

Cram will also write the test with its actual output to examples/fail.t.err.

When you're first writing a test, you might just write the commands and run the test to see what happens. If you run Cram with -i or --interactive, you'll be prompted to merge the actual output back into the test. This makes it easy to quickly prototype new tests.

You can specify a default set of options by creating a .cramrc file. For example:

[cram]
verbose = True
indent = 4

Is the same as invoking Cram with --verbose and --indent=4.

To change what configuration file Cram loads, you can set the CRAMRC environment variable. You can also specify command line options in the CRAM environment variable.

Note that the following environment variables are reset before tests are run:

Cram also provides the following environment variables to tests:

Also note that care should be taken with commands that close the test shell's stdin. For example, if you're trying to invoke ssh in a test, try adding the -n option to prevent it from closing stdin.

News

Version 0.6 (Aug. 1, 2013)

Version 0.5 (Jan. 8, 2011)

Version 0.4 (Sep. 28, 2010)

Version 0.3 (Sep. 20, 2010)

Version 0.2 (Sep. 19, 2010)

Version 0.1 (Sep. 19, 2010)

Development

Download the official development repository using Mercurial:

hg clone https://bitbucket.org/brodie/cram

Or Git:

git clone https://github.com/brodie/cram.git

Test Cram using Cram:

make tests

Get a test coverage report using coverageppy:

make coverage

Visit Bitbucket or GitHub if you'd like to fork the project, watch for new changes, or report issues.