
Working with non-python tests
*****************************


A basic example for specifying tests in Yaml files
==================================================

Here is an example "conftest.py" (extracted from Ali Afshnars special
purpose pytest-yamlwsgi plugin).   This "conftest.py" will  collect
"test*.yml" files and will execute the yaml-formatted content as
custom tests:

   # content of conftest.py

   import pytest

   def pytest_collect_file(parent, path):
       if path.ext == ".yml" and path.basename.startswith("test"):
           return YamlFile(path, parent)

   class YamlFile(pytest.File):
       def collect(self):
           import yaml # we need a yaml parser, e.g. PyYAML
           raw = yaml.load(self.fspath.open())
           for name, spec in raw.items():
               yield YamlItem(name, self, spec)

   class YamlItem(pytest.Item):
       def __init__(self, name, parent, spec):
           super(YamlItem, self).__init__(name, parent)
           self.spec = spec

       def runtest(self):
           for name, value in self.spec.items():
               # some custom test execution (dumb example follows)
               if name != value:
                   raise YamlException(self, name, value)

       def repr_failure(self, excinfo):
           """ called when self.runtest() raises an exception. """
           if isinstance(excinfo.value, YamlException):
               return "\n".join([
                   "usecase execution failed",
                   "   spec failed: %r: %r" % excinfo.value.args[1:3],
                   "   no further details known at this point."
               ])

       def reportinfo(self):
           return self.fspath, 0, "usecase: %s" % self.name

   class YamlException(Exception):
       """ custom exception for error reporting. """

You can create a simple example file:

   # test_simple.yml
   ok:
       sub1: sub1

   hello:
       world: world
       some: other

and if you installed PyYAML or a compatible YAML-parser you can now
execute the test specification:

   nonpython $ py.test test_simple.yml
   =========================== test session starts ============================
   platform linux2 -- Python 2.7.3 -- pytest-2.3.5
   collected 2 items

   test_simple.yml .F

   ================================= FAILURES =================================
   ______________________________ usecase: hello ______________________________
   usecase execution failed
      spec failed: 'some': 'other'
      no further details known at this point.
   ==================== 1 failed, 1 passed in 0.05 seconds ====================

You get one dot for the passing "sub1: sub1" check and one failure.
Obviously in the above "conftest.py" you'll want to implement a more
interesting interpretation of the yaml-values.  You can easily write
your own domain specific testing language this way.

Note: "repr_failure(excinfo)" is called for representing test failures. If
  you create custom collection nodes you can return an error
  representation string of your choice.  It will be reported as a
  (red) string.

"reportinfo()" is used for representing the test location and is also
consulted when reporting in "verbose" mode:

   nonpython $ py.test -v
   =========================== test session starts ============================
   platform linux2 -- Python 2.7.3 -- pytest-2.3.5 -- /home/hpk/p/pytest/.tox/regen/bin/python
   collecting ... collected 2 items

   test_simple.yml:1: usecase: ok PASSED
   test_simple.yml:1: usecase: hello FAILED

   ================================= FAILURES =================================
   ______________________________ usecase: hello ______________________________
   usecase execution failed
      spec failed: 'some': 'other'
      no further details known at this point.
   ==================== 1 failed, 1 passed in 0.05 seconds ====================

While developing your custom test collection and execution it's also
interesting to just look at the collection tree:

   nonpython $ py.test --collectonly
   =========================== test session starts ============================
   platform linux2 -- Python 2.7.3 -- pytest-2.3.5
   collected 2 items
   <YamlFile 'test_simple.yml'>
     <YamlItem 'ok'>
     <YamlItem 'hello'>

   =============================  in 0.05 seconds =============================
