py library overview

Grig Gheorghiu

What is the py library?

• Collection of modules that aim to be as simple and "pythonic" as possible
• Mantra: "No API"!!!
• Avoid the F word (Framework)
• Intention is to address several issues with the standard library:
• testing (py.test)
• distributed programming (py.execnet)
• coroutines (py.magic.greenlet)
• producing XML/HTML (py.xml)
• logging (py.log)
• path management: local FS, subversion (py.path, py.magic.autopath)
• explicit name export control (py.initpkg)

py.test: general operation

• "No API" approach to running unit tests
• No need to inherit test classes from framework class
• test functions/methods start w. test_ and test classes with Test
• test modules start with test_ or end with _test
• Order of test execution is guaranteed within a module
• No alphanumerical sorting order to worry about
• Test collection process walks current directory and its subdirectories
• Collection is customizable per directory via configuration files
• Test process starts as soon as the first test item is collected
• Collection process is iterative and does not need to complete before the first test items are executed
• Test items (classes, methods, functions) can be selectively run via keywords

py.test: assertions and exception handling

• Normal assert statements can be used
• No need for special assertEqual syntax
• When assertions fail, py.test prints useful tracebacks:
• lines in the method containing the assertion, up to and including the failure
• actual and the expected values involved in the failed assertion
• only the relevant portions of tracebacks are included for easier debugging
• Assertions about exceptions via py.test.raises:
• py.test.raises(Exception, "func(*args, **kwargs)")
• py.test.raises(Exception, func, *args, **kwargs)
• Example:

alist = [5, 2, 3, 1, 4]
py.test.raises(NameError, "alist.sort(int_compare)")
py.test.raises(ValueError, alist.remove, 6)

py.test: test fixture management

• setup and teardown hooks for managing test fixture/state
• module-level state: setup_module/teardown_module
• invoked once per test module
• reusing database connections across multi-class test
• test initialization when only top-level test functions are used
• Class-level state: setup_class/teardown_class
• invoked once for each test class
• useful when multiple test clases are in same test module
• Method-level state: setup_method/teardown_method
• invoked once for each test method
• equivalent to setUp and tearDown in unittest
• All setup/teardown hooks are optional

py.test: maintaining state

def setup_module(module):
    module.TestState.classcount = 0

class TestState:

    def setup_class(cls):
        cls.classcount += 1

    def setup_method(self, method):
        self.methodcount = self.classcount + 1

    def test_state1(self):
        self.methodcount += 1
        assert self.classcount == 1
        assert self.methodcount == 3

    def test_state2(self):
        self.methodcount += 1
        assert self.classcount == 1
        assert self.methodcount == 3

py.test: command-line options

• verbose: -v
• do not capture stdout/stderr: -s
• exit on first failing test: -x
• do not reduce traceback output: --fulltrace
• do not do any 'magic' (e.g. interpreting assert statements): --nomagic
• show collected tests without running them: --collectonly
• selectively run tests by name: -k 'test_method_or_class_name'
• use tkinter front-end: --tkinter
• stay in loop and run test modules that get modified: --looponfailing
• run py.test with given Python executable: --exec=EXECUTABLE

py.test: customizing the test collection

• Test collection is customizable via conftest.py files
• Example: excluding directories from test collection:

import py
class ExcludeDirectory(py.test.collect.Directory):
    def recfilter(self, path):
        if path.basename == 'exclude_dir':
            return False
        return super(ExcludeDirectory, self).recfilter(path)

Directory = ExcludeDirectory

• See more comprehensive example for ReST syntax checking: py/documentation/conftest.py

Greenlets: implementing coroutines in Python

• py.magic.greenlet: C extension module, can be used with standard Python interpreter
• Micro-threads with no implicit scheduling, i.e. coroutines
• Allow you to exit a function by 'yielding' a value and then re-enter the function at exactly the point the execution left off
• generalization of generators
• generators can be easily implemented with greenlets
• Typical use case for greenlets/coroutines is turning asynchronous or event-based code into normal sequential control flow code

• for event in get_event(): do_something_with_event()

• Applications: GUI event loops, Web form processing, SAX-style XML parsing
• Coroutines will be implemented in the standard Python library via 'enhanced generators' (see PEP 342)

Greenlet example: XML parsing

from py.magic import greenlet
import xml.parsers.expat

def send(arg):
    greenlet.getcurrent().parent.switch(arg)

def start_element(name, attrs):
    send(('START', name, attrs))
def end_element(name):
    send(('END', name))
def char_data(data):
    data = data.strip()
    if data: send(('DATA', data))

def greenparse(xmldata):
    p = xml.parsers.expat.ParserCreate()
    p.StartElementHandler = start_element
    p.EndElementHandler = end_element
    p.CharacterDataHandler = char_data
    p.Parse(xmldata, 1)

def iterxml(xmldata):
    g = greenlet(greenparse)
    data = g.switch(xmldata)
    while data is not None:
        yield data
        data = g.switch()

if __name__ == "__main__":
    for data in iterxml("somexmldata"):
        # do something with data

py.xml: easy XML generation

• XML generation in 3 easy steps:
• Inherit from py.xml.Namespace
• Use method names for XML tags
• Use keyword arguments for XML attributes

class ns(py.xml.Namespace):
    "my custom xml namespace"

doc = ns.rsp(
    ns.event(name="Piggies July Meeting",
             start_date="2005-07-21", start_time="19:00:00"),
    ns.event(name="Piggies August Meeting",
             start_date="TBA", start_time="19:00:00"),
    stat="OK",
    version="1.0")
print doc.unicode(indent=2).encode('utf8')
# generated output:
#<rsp stat="OK" version="1.0">
#  <event name="Piggies July Meeting" start_date="2005-07-21" start_time="19:00:00"/>
#  <event name="Piggies August Meeting" start_date="TBA" start_time="19:00:00"/></rsp>

py.xml: easy HTML generation

• HTML generation in 3 easy steps:
• Import html namespace
• Use objects in html namespace for HTML tags
• Use keyword arguments for HTML attributes

from py.xml import html  # html namespace

paragraphs = "First Paragraph", "Second Paragraph"
doc = html.html(
   html.head(
        html.meta(name="Content-Type", value="text/html; charset=latin1")),
   html.body(
        [html.p(p) for p in paragraphs]))
print unicode(doc).encode('latin1')
# generated output (on one line):
#<html><head><meta name="Content-Type" value="text/html; charset=latin1"></meta>
#</head><body><p>First Paragraph</p><p>Second Paragraph</p></body></html>

py.log: keyword-based logging

• Goal: no print statements!
• Log producers: declare keywords; used directly by application code
• Calling a method on a log object adds method name to keywords
• Log consumers: associated with keywords; receive messages from producers; log to stdout, files, databases, syslog, email, etc.
• A consumer is simply a function that receives a message and processes it
• If no consumer is declared, messages get printed to stdout by default

# logmodule.py
log = py.log.Producer("logmodule") # keywords: ('logmodule',)
def runcmd(cmd):
    log.cmd(cmd) # keywords: ('logmodule', 'cmd')
def post2db(data):
    log.post2db(data) # keywords: ('logmodule', 'post2db')
if __name__ == "__main__":
    runcmd("Running command CMD1")
    post2db("PASS")
    #prints following to stdout (default):
    #[logmodule:cmd] Running command CMD1
    #[logmodule:post2db] PASS

py.log: configuring logging on the fly

• Consumers can be associated with keywords dynamically
• Modules that import log objects can redirect keywords to specific consumers:
• we want to print all messages from logmodule to stderr
• we don't want to print any db-related messages

import py
from logmodule import runcmd, post2db

py.log.setconsumer("logmodule", py.log.STDERR)
py.log.setconsumer("logmodule post2db", None)

runcmd("Running command CMD2")
post2db("FAIL")
#prints following to stderr:
#[logmodule:cmd] Running command CMD2

py.log: defining severity levels

log = py.log.Producer("")
log.debug = py.log.Producer("logfile debug")
log.info = py.log.Producer("console info")
log.error = py.log.Producer("console logfile error")

logfile = "/tmp/myapp.out"
py.log.setconsumer("logfile", py.log.Path(logfile))
py.log.setconsumer("console", py.log.STDOUT)

l = open(logfile, 'a', 1)
def console_logfile_logger(msg):
    print >>sys.stdout, str(msg)
    print >>l, str(msg)
py.log.setconsumer("console logfile", console_logfile_logger)

log.debug("DEBUG MESSAGE") # prints "[logfile:debug] DEBUG MESSAGE" to logfile
log.info("INFO MESSAGE") # prints "[console:info] INFO MESSAGE" to stdout
log.error("ERROR MESSAGE") # prints "[console:logfile:error] ERROR MESSAGE" to stdout and logfile
log.warn("WARNING MESSAGE") # prints "[warn] WARNING MESSAGE" to stdout (default consumer)
# to disable debug messages: py.log.setconsumer("logfile debug", None)