Cram: It's test time
Cram is a functional testing framework for command line applications based on Mercurial's unified test format.
Cram tests look like snippets of interactive shell sessions. Cram runs each command and compares the command output in the test with the command's actual output.
Here's a snippet from Cram's own test suite:
The $PYTHON environment variable should be set when running this test from Python. $ [ -n "$PYTHON" ] || PYTHON="`which python`" $ [ -n "$PYTHONPATH" ] || PYTHONPATH="$TESTDIR/.." && export PYTHONPATH $ if [ -n "$COVERAGE" ]; then > coverage erase > alias cram="`which coverage` run --branch -a $TESTDIR/../scripts/cram" > else > alias cram="$PYTHON $TESTDIR/../scripts/cram" > fi $ command -v md5 > /dev/null || alias md5=md5sum Usage: $ cram -h [Uu]sage: cram \[OPTIONS\] TESTS\.\.\. (re) [Oo]ptions: (re) -h, --help show this help message and exit -V, --version show version information and exit -q, --quiet don't print diffs -v, --verbose show filenames and test status -i, --interactive interactively merge changed test output -y, --yes answer yes to all questions -n, --no answer no to all questions -E, --preserve-env don't reset common environment variables --keep-tmpdir keep temporary directories --shell=PATH shell to use for running tests (default: /bin/sh) --indent=NUM number of spaces to use for indentation (default: 2)
The format in a nutshell:
Cram tests use the
Lines beginning with two spaces, a dollar sign, and a space are run in the shell.
Lines beginning with two spaces, a greater than sign, and a space allow multi-line commands.
All other lines beginning with two spaces are considered command output.
Output lines ending with a space and the keyword
(re)are matched as Perl-compatible regular expressions.
Lines ending with a space and the keyword
(glob)are matched with a glob-like syntax. The only special characters supported are
?. Both characters can be escaped using
\, and the backslash can be escaped itself.
Output lines ending with either of the above keywords are always first matched literally with actual command output.
Lines ending with a space and the keyword
(no-eol)will match actual output that doesn't end in a newline.
Actual output lines containing unprintable characters are escaped and suffixed with a space and the keyword
(esc). Lines matching unprintable output must also contain the keyword.
Anything else is a comment.
- cram-0.6.tar.gz (24 KB, requires Python 2.4-2.7 or Python 3.1-3.2)
You can use pip to install Cram:
$ pip install cram
Or you can install Cram using GNU make:
$ wget https://bitheap.org/cram/cram-0.6.tar.gz $ tar zxvf cram-0.6.tar.gz $ cd cram-0.6 $ make install
Cram will print a dot for each passing test. If a test fails, a unified
context diff is
printed showing the test's expected output and the actual output.
Skipped tests (empty tests and tests that exit with return code
are marked with
s instead of a dot.
For example, if we run Cram on its own example tests:
.s.! --- /home/brodie/src/cram/examples/fail.t +++ /home/brodie/src/cram/examples/fail.t.err @@ -3,21 +3,22 @@ $ echo 1 1 $ echo 1 - 2 + 1 $ echo 1 1 Invalid regex: $ echo 1 - +++ (re) + 1 Offset regular expression: $ printf 'foo\nbar\nbaz\n\n1\nA\[email protected]\n' foo + bar baz \d (re) [A-Z] (re) - # + @ s. # Ran 6 tests, 2 skipped, 1 failed.
Cram will also write the test with its actual output to
When you're first writing a test, you might just write the commands and
run the test to see what happens. If you run Cram with
--interactive, you'll be prompted to merge the actual output back into
the test. This makes it easy to quickly prototype new tests.
You can specify a default set of options by creating a
[cram] verbose = True indent = 4
Is the same as invoking Cram with
To change what configuration file Cram loads, you can set the
environment variable. You can also specify command line options in the
CRAM environment variable.
Note that the following environment variables are reset before tests are run:
TMPare set to the test runner's
LANGUAGEare set to
TZis set to
COLUMNSis set to
GREP_OPTIONSare set to an empty string.
Cram also provides the following environment variables to tests:
CRAMTMP, set to the test runner's temporary directory.
TESTDIR, set to the directory containing the test file.
Also note that care should be taken with commands that close the test
stdin. For example, if you're trying to invoke
ssh in a
test, try adding the
-n option to prevent it from closing
Version 0.6 (Aug. 1, 2013)
Added the long option
Added support for specifying options in
.cramrc(configurable with the
--shelloption to change the shell tests are run with. Contributed by Kamil Kisiel.
Added Arch Linux package metadata (in
contrib/). Contributed by Andrey Vlasovskikh.
Fixed shell commands unintentionally inheriting Python's
SIGPIPEhandler (causing commands that close pipes to print
EPIPEunder PyPy when applying patches in
TESTFILEtest environment variable (set to the name of the current test).
Fixed GNU patch 2.7 compatibility by using relative paths instead of absolute paths. Contributed by Douglas Creager.
Fixed name clashes in temporary test directories (e.g., when running two tests with the same name in different folders).
Backwards compatibility: Fixed improper usage of the subprocess library under Python 3. This fixes Python 3.3 support, but breaks support for Python 3.1-3.2.3 due to a bug in Python. If you're using Python 3.0-3.2, you must upgrade to Python 3.2.4 or newer.
Version 0.5 (Jan. 8, 2011)
The test format has changed: Matching output not ending in a newline now requires the
(no-eol)keyword instead of ending the line in
Matching output containing unprintable characters now requires the
(esc)keyword. Real output containing unprintable characters will automatically receive
If an expected line matches its real output line exactly, special matching like
(glob)will be ignored.
Regular expressions ending in a trailing backslash are now considered invalid.
--indentoption for changing the default amount of indentation required to specify commands and output.
Added support for specifying command line options in the
--verboseoptions can now be used together.
When running Cram under Python 3, Unicode-specific line break characters will no longer be parsed as newlines.
Tests are no longer required to end in a trailing newline.
Version 0.4 (Sep. 28, 2010)
The test format has changed: Output lines containing regular expressions must now end in
(re)or they'll be matched literally. Lines ending with keywords are matched literally first, however.
Regular expressions are now matched from beginning to end. In other words
\d (re)is matched as
In addition to
(glob)has been added. It supports
?, and escaping both characters (and backslashes) using
Environment settings have changed: The
-Dflag has been removed,
$TESTDIRis now set to the directory containing the
$CRAMTMPis set to the test runner's temporary directory.
patch(1). Instead of
.tfiles during merges, diffs are applied using
patch(1). This prevents matching regular expressions and globs from getting clobbered.
.errfiles are now removed when tests pass.
Cram now exits with return code
1if any tests failed.
If a test exits with return code
80, it's considered a skipped a test. This is useful for intentionally disabling tests when they only work on certain platforms or in certain settings.
The number of tests, the number of skipped tests, and the number of failed tests are now printed after all tests are finished.
--quietto suppress diff output.
Added contrib/cram.vim syntax file for Vim. Contributed by Steve Losh.
Version 0.3 (Sep. 20, 2010)
Implemented resetting of common environment variables. This behavior can be disabled using the
Changed the test runner to first make its own overall random temporary directory, make
tmpinside of it and set
TMPDIR, etc. to its path, and run each test with a random temporary working directory inside of that.
--keep-tmpdir. Temporary directories are named by test filename (along with a random string).
--interactiveto merge actual output back to into tests interactively.
Added ability to match command output not ending in a newline by suffixing output in the test with
Version 0.2 (Sep. 19, 2010)
- Changed the test runner to run tests with a random temporary working directory.
Version 0.1 (Sep. 19, 2010)
- Initial release.
Download the official development repository using Git:
git clone https://github.com/brodie/cram.git
Test Cram using Cram:
Get a test coverage report using coverageppy:
Visit GitHub if you'd like to fork the project, watch for new changes, or report issues.