CALDAVTESTER
============
# INTRODUCTION
CalDAVTester is a Python app that will run a series of scripted tests
against a CalDAV server and verify the output, and optionally measure
the time taken to complete one or more repeated requests. The tests are
defined by XML files and ancillary HTTP request body files. A number of
different verification options are provided.
Many tests are included in this package.
CalDAVTester can be extended to run tests against any type of HTTP server
protocol by simply defining a new set of XML files.
# INSTALL
Get CalDAVTester from github, create a virtualenv to run it in and install
into the virtualenv:
git clone https://github.com/apple/ccs-caldavtester.git
cd ccs-caldevtester
virtualenv venv
source venv/bin/activate
pip install -r requirements.txt
# COMMAND LINE OPTIONS
CalDAVTester is run via the `testcaldav.py` script:
testcaldav.py
[-s filename]
[-x dirpath]
[--basedir dirpath]
[--ssl]
[--all]
[--random]
[--random-seed SEED]
[--stop]
[--print-details-onfail]
[--always-print-request]
[--always-print-response]
[--exclude filename]
[--observer OBSERVER]
file1 file2 ...
-s : filename specifies the file to use for server information
(default is 'serverinfo.xml').
-x : directory path for test scripts
(default is 'scripts/tests').
--basedir : directory path for serverinfo.xml, test/ and data/,
overrides -s and -x values
-p : filename specifies the file to use to populate the server with
data. Server data population only occurs when this option is
present.
-d : in conjunction with -p, if present specifies that the populated
data be removed after all tests have completed.
--ssl : run tests using SSL/https connections to the server.
--all : execute all tests found in the working directory. Each .xml
file in that directory is examined and those corresponding to the
caldavtest.dtd are executed.
--random : randomize the order in which the tests are run.
--random-seed SEED : a specific random seed to use.
--stop : stop running all tests after one test file fails.
--print-details-onfail : print HTTP request/response when a test fails.
--always-print-request : always print HTTP request.
--always-print-response : always print HTTP response.
--exclude FILE : when running with --all, exclude the file from the test run.
--observer OBSEREVER : specify one or more times to change which classes are
used to process log and trace messages during a test. The OBSERVER name must
be the name of a module in the observers package. The default observer is the
"log" observer. Available observers are:
"log" - produces an output similar to Python unit tests.
"trace" - produces an output similar to the original output format.
"loadfiles" - prints each test file as it is loaded.
"jsondump" - prints a JSON representation of the test results.
file1 file2 ...: a list of test files to execute tests from.
# QUICKSTART
Edit the serverinfo.xml file to run the test against your server setup.
Prior to running, make sure you are in the virtualenv:
source venv/bin/activate
Run `testcaldav.py --all` on the command line to run the tests. The app
will print its progress through the tests.
# EXECUTION PROCESS
1. Read in XML config.
2. Execute <start> requests.
3. For each <test-suite>, run each <test> the specified number of times,
executing each <request> in the test and verifying them.
4. Delete any resources from requests marked with 'end-delete'.
5. Execute <end> requests.
# XML SCRIPT FILES
## serverinfo.dtd
Defines the XML DTD for the server information XML file:
ELEMENT <host>
host name for server to test.
ELEMENT <nonsslport>
port to use to connect to server (non-SSL).
ELEMENT <sslport>
port to use to connect to server (SSL).
ELEMENT <authtype>
HTTP authentication method to use.
ELEMENT <certdir>
Base directory for TLS client certs.
ELEMENT <waitcount>
For requests that wait, defines how many iterations to wait for
[Default: 120].
ELEMENT <waitdelay>
For requests that wait, defines how long between iterations to
wait for in seconds [Default: 0.25].
ELEMENT <waitsuccess>
For requests with the wait-for-success options, defines how many
seconds to wait [Default: 10].
ELEMENT <features>
list of features for the server under test.
ELEMENT <feature>
specific feature supported by the server under test,
used to do conditional testing.
ELEMENT <substitutions>
used to encapsulate all variable substitutions.
ELEMENT <substitution>
a variable substitution - the repeat attribute can
be used to repeat the substitution a set number of
times whilst generating different substitutions.
ELEMENT <key>
the substitution key (usually '$xxx:').
ELEMENT <value>
the substitution value.
ELEMENT <repeat>
allow repeating substitutions for the specified count.
## caldavtest.dtd:
Defines the XML DTD for test script files:
ATTRIBUTE ignore-all
used on the top-level XML element to indicate whether this test
is run when the --all command line switch for testcaldav.py is
used. When set to 'no' the test is not run unless the file is
explicitly specified on the command line.
ELEMENT <description>
a description for this test script.
ELEMENT <require-feature>
set of features.
ELEMENT <feature>
feature that server must support for this entire test
script to run.
ELEMENT <exclude-feature>
set of features.
ELEMENT <feature>
feature that server must not support for this entire test
script to run.
ELEMENT <start>
defines a series of requests that are executed before testing
starts. This can be used to initialize a set of calendar
resources on which tests can be run.
ELEMENT <end>
defines a series of requests that are executed after testing is
complete. This can be used to clean-up the server after testing.
Note that there are special mechanisms in place to allow
resources created during testing to be automatically deleted
after testing, so there is no need to explicitly delete those
resources here.
ELEMENT <test-suite>
defines a group of tests to be run. The suite is given a name
and has an 'ignore' attribute that can be used to disable it.
ATTRIBUTE name
name/description of test-suite.
ATTRIBUTE ignore
if set to 'yes' then the entire test-suite will be skipped.
ATTRIBUTE only
if set to 'yes' then all other test-suites (except others with
the same attribute value set) will be skipped.
ELEMENT <require-feature>
set of features.
ELEMENT <feature>
feature that server must support for this test
suite to run.
ELEMENT <exclude-feature>
set of features.
ELEMENT <feature>
feature that server must not support for this test
suite to run.
ELEMENT <test>
defines a single test within a test suite. A test has a name,
description and one or more requests associated with it. There
is also an 'ignore' attribute to disable the test. Tests can be
executed multiple times by setting the 'count' attribute to a
value greater than 1. Timing information about the test can be
printed out by setting the 'stats' attribute to 'yes'.
ATTRIBUTE name
name of test.
ATTRIBUTE count
number of times to run the test. This allows tests to be
easily repeated.
ATTRIBUTE stats
if set to 'yes' then timing information for the test will be
printed.
ATTRIBUTE ignore
if set to 'yes' then the entire test will be skipped.
ELEMENT <require-feature>
set of features.
ELEMENT <feature>
feature that server must support for this test
to run.
ELEMENT <exclude-feature>
set of features.
ELEMENT <feature>
feature that server must not support for this test
to run.
ELEMENT <description>
detailed description of the test.
ELEMENT <pause>
halt tests and wait for user input. Useful for stopping tests to set a
break point or examine server state, and then continue on.
ELEMENT <request>
defines an HTTP request to send to the server. Attributes on the
element are:
ATTRIBUTE auth
if 'yes', HTTP Basic authentication is done in the request.
ATTRIBUTE user
if provided this value is used as the user id for HTTP Basic
authentication instead of the one in the serverinfo
file.
ATTRIBUTE pswd
if provided this value is used as the password for HTTP
Basic authentication instead of the one in the serverinfo
file.
ATTRIBUTE cert
if provided this value is used as the file name for a TLS
client certificate to be used with the request.
ATTRIBUTE end-delete
if set to 'yes', then the resource targeted by the request
is deleted after testing is complete, but before the
requests in the <end> element are run. This allows for quick
clean-up of resources created during testing.
ATTRIBUTE print-response
if set to 'yes' then the HTTP response (header and body) is
printed along with test results.
ATTRIBUTE wait-for-success
if set to 'yes' then the HTTP request will repeat over and over
for a set amount of time waiting for the verifiers to pass. If
time expires without success then the overall request fails. The
length of time is controlled by the <waittime> element in the
serverinfo file (defaults to 10 seconds).
ELEMENT <require-feature>
set of features.
ELEMENT <feature>
feature that server must support for this request
to run.
ELEMENT <exclude-feature>
set of features.
ELEMENT <feature>
feature that server must not support for this request
to run.
ELEMENT <method>
the HTTP method for this request. There are some 'special' methods that do some useful 'compound' operations:
1) DELETEALL - deletes all resources within the collections specified by the <ruri> elements.
2) DELAY - pause for the number of seconds specified by the <ruri> element.
3) GETNEW - get the data from the newest resource in the collection specified by the <ruri> element and put its URI
into the $ variable for later use in an <ruri> element.
4) WAITCOUNT N - wait until at least a certain number of resources "N" appear in a collection.
5) WAITDELETEALL - wait until at least a certain number of resources appear in a collection, then delete all child
resources in that collection.
6) GETCHANGED - the tool tracks the Etags on resources retrieved via GET. This special method will poll the specified
resource until the Etag returned in the response is different from the one found in the most recent
test.
6) GETOTHER - the tool finds the newest sibling resource to the one specified in the <ruri> element.
6) GETCONTAINS XXX - the tool finds the child resource whose content contains the supplied text "XXX".
ELEMENT <ruri>
the URI of the request. Multiple <ruri>'s are allowed with DELETEALL only.
The characters "**" may be used to cause a random uuid to be inserted where
those two characters appear. The characters "##" may be used to insert the
current test count iteration where those two characters occur.
ELEMENT <header>
can be used to specify additional headers in the request.
ELEMENT <name>
the header name.
ELEMENT <value>
the header value.
ELEMENT <data>
used to specify the source and nature of data used in the
request body, if there is one.
ATTRIBUTE substitutions
if set to 'yes' then '$xxx:' style variable substitutions
will be performed on the data before it is sent in the request.
ATTRIBUTE generate
if set to 'yes' then a basic calendar data "fuzzing" is done to
the source data to make it unique and up to date.
ELEMENT <content-type>
the MIME content type for the request body.
ELEMENT <filepath>
the relative path for the file containing the request body
data.
ELEMENT <generator>
a callback and set of arguments used to generate the data.
ELEMENT <callback>
the name of the generator method to execute.
ELEMENT <arg>
arguments sent to the generator method.
ELEMENT <name>
the name of the argument.
ELEMENT <value>
values for the argument.
ELEMENT <substitute>
a set of substitution variables to use on this data only.
ELEMENT <name>
the variable name.
ELEMENT <value>
the variable value.
ELEMENT <verify>
if present, used to specify a procedures for verifying that the
request executed as expected.
ELEMENT <require-feature>
set of features.
ELEMENT <feature>
feature that server must support for this verification
to be checked.
ELEMENT <exclude-feature>
set of features.
ELEMENT <feature>
feature that server must not support for this verification
to be checked.
ELEMENT <callback>
the name of the verification method to execute.
ELEMENT <arg>
arguments sent to the verification method.
ELEMENT <name>
the name of the argument.
ELEMENT <value>
values for the argument.
ELEMENT <graburi>
if present, this stores the value of the actual request URI
used in a named variable which can be used in subsequent requests.
Useful for capturing URIs when the GETNEW method is used.
ELEMENT <grabcount>
if present, this stores the number of child responses in a
{DAV:}multistatus response into the named variable which
can be used in subsequent requests. This is useful for
capturing the current count so that a change in the count
can be tested for later.
ELEMENT <grabheader>
if present, this stores the value of the specified header
returned in the response in a named variable which can be used
in subsequent requests.
ELEMENT <grabproperty>
if present, this stores the value of the specified property
returned in a PROPFIND response in a named variable which can
be used in subsequent requests.
ELEMENT <grabelement>
if present, this stores the text representation of an XML
element extracted from the response body in a named variable
which can be used in subsequent requests.
ELEMENT <grabjson>
if present, this stores the text representation of a JSON
object extracted from the response body in a named variable
which can be used in subsequent requests.
ELEMENT <grabcalproperty>
if present, this stores a calendar property value in a named
variable which can be used in subsequent request. The syntax for
<name> element is component/propname (e.g. "VEVENT/SUMMARY").
ELEMENT <grabcalparameter>
if present, this stores a calendar parameter value in a named
variable which can be used in subsequent request. The syntax for
<name> element is component/propname/paramname$propvalue where
the option $propvalue allows a specific property to be selected
(e.g. "VEVENT/DTSTART/TZID", or
"VEVENT/ATTENDEE/PARTSTAT$mailto:user01@example.com").
# VERIFICATION Methods
## acltems
Performs a check of multi-status response body and checks to see
whether the specified privileges are granted or denied on each
resource in the response for the current user (i.e. tests the
DAV:current-user-privilege-set).
Argument: 'granted'
A set of privileges that must be granted.
Argument: 'denied'
A set of privileges that must be denied denied.
Example:
<verify>
<callback>multistatusitems</callback>
<arg>
<name>granted</name>
<value>DAV:read</value>
</arg>
<arg>
<name>denied</name>
<value>DAV:write</value>
<value>DAV:write-acl</value>
</arg>
</verify>
## calandarDataMatch
Similar to data match but tries to "normalize" the calendar data so that e.g., different
ordering of properties is not significant.
Argument: 'filepath'
The file path to a file containing data to match the response body to.
Example:
<verify>
<callback>dataMatch</callback>
<arg>
<name>filepath</name>
<value>resources/put.ics</value>
</arg>
</verify>
## dataMatch
Performs a check of response body and matches it against the data in the specified file.
Argument: 'filepath'
The file path to a file containing data to match the response body to.
Example:
<verify>
<callback>dataMatch</callback>
<arg>
<name>filepath</name>
<value>resources/put.ics</value>
</arg>
</verify>
## dataString
Performs a check of response body tries to find occurrences of the specified strings or the
absence of specified strings.
Argument: 'equals'
One or more strings that must match exactly in the data (case-sensitive).
Argument: 'contains'
One or more strings that must be contained in the data (case-sensitive).
Argument: 'notcontains'
One or more strings that must not be contained in the data (case-sensitive).
Example:
<verify>
<callback>dataString</callback>
<arg>
<name>contains</name>
<value>BEGIN:VEVENT</value>
</arg>
<arg>
<name>notcontains</name>
<value>BEGIN:VTODO</value>
</arg>
</verify>
## freeBusy
Performs a check of the response body to verify it contains an
iCalendar VFREEBUSY object with the specified busy periods and
types.
Argument: 'busy'
A set of iCalendar PERIOD values for FBTYPE=BUSY periods
expected in the response.
Argument: 'tentative'
A set of iCalendar PERIOD values for FBTYPE=BUSY-TENTATIVE
periods expected in the response.
Argument: 'unavailable'
A set of iCalendar PERIOD values for FBTYPE=BUSY-UNAVAILABLE
periods expected in the response.
Argument: 'duration'
If present the period values being checked use duration rather then
end time.
Example:
<verify>
<callback>freeBusy</callback>
<arg>
<name>busy</name>
<value>20060107T010000Z/20060107T020000Z</value>
<value>20060107T150000Z/20060107T163000Z</value>
<value>20060108T150000Z/20060108T180000Z</value>
</arg>
<arg>
<name>unavailable</name>
<value>20060108T130000Z/20060108T150000Z</value>
</arg>
<arg>
<name>tentative</name>
<value>20060108T160000Z/20060108T170000Z</value>
<value>20060108T210000Z/20060108T213000Z</value>
</arg>
</verify>
## header
Performs a check of response header and value. This can be used to
test for the presence or absence of a header, or the presence of a
header with a specific value.
Argument: 'header'
This can be specified in one of three forms:
'headername' - will test for the presence of the response
header named 'header name'.
'headername$value' - will test for the presence of the
response header named 'headername' and also check that its
value matches 'value'.
'!headername' - will test for the absence of a header named
'headername' in the response header.
Example:
<verify>
<callback>header</callback>
<arg>
<name>header</name>
<value>Content-type$text/plain</value>
</arg>
</verify>
## jcalDataMatch
Like calendarDataMatch except that comparison is done using jCal data.
## jsonPointerMatch
Compares the response with a JSON pointer and returns TRUE if there
is a match, otherwise False.
The pointer is the absolute pointer from the root down. A JSON object's
string value can be checked by append "~$" and the string value to test
to the JSON pointer value. To test for a null value append "~~". A single
"." can be used as a reference-token in the JSON pointer to match against
any member or array item at that position in the document.
Argument: 'exists'
JSON pointer for a JSON item to check the presence of
in the response.
Argument: 'notexists'
JSON pointer for a JSON item to check the absence of
in the response.
Example:
<verify>
<callback>jsonPointerMatch</callback>
<arg>
<name>exists</name>
<value>/responses/response</value>
</arg>
<arg>
<name>notexists</name>
<value>/responses/response/name~$ABC</value>
</arg>
<arg>
<name>exists</name>
<value>/responses/./name~$XYZ</value>
</arg>
</verify>
## multistatusItems
Performs a check of multi-status response body and checks to see
what hrefs were returned and whether those had a good (2xx) or bad
(non-2xx) response code. The overall response status must be 207.
Argument: 'okhrefs'
A set of hrefs for which a 2xx response status is required.
Argument: 'badhrefs'
A set of hrefs for which a non-2xx response status is required.
Argument: 'prefix'
A prefix that is appended to all of the specified okhrefs and
badhrefs values.
Example:
<verify>
<callback>multistatusitems</callback>
<arg>
<name>okhrefs</name>
<value>/calendar/test/1.ics</value>
<value>/calendar/test/2.ics</value>
<value>/calendar/test/3.ics</value>
</arg>
<arg>
<name>badhrefs</name>
<value>/calendar/test/4.ics</value>
<value>/calendar/test/5.ics</value>
<value>/calendar/test/6.ics</value>
</arg>
</verify>
## postFreeBusy
Looks for specific FREEBUSY periods for a particular ATTENDEE.
Argument: 'attendee'
Calendar user address for attendee to match.
Argument: 'busy'
Period for FBTYPE=BUSY to match.
Argument: 'tentative'
Period for FBTYPE=BUSY-TENTATIVE to match.
Argument: 'unavailable'
Period for FBTYPE=BUSY-UNAVAILABLE to match.
Argument: 'events'
Number of VEVENTs (in extended free-busy request) to match.
Example:
<verify>
<callback>postFreeBusy</callback>
<arg>
<name>attendee</name>
<value>$cuaddr1:</value>
</arg>
<arg>
<name>busy</name>
<value>20060101T230000Z/20060102T000000Z</value>
</arg>
</verify>
## prepostcondition
Performs a check of response body and status code to verify that a
specific pre-/post-condition error was returned. The response status
code has to be one of 403 or 409.
Argument: 'error'
The expected XML element qualified-name to match.
Example:
<verify>
<callback>prepostcondition</callback>
<arg>
<name>error</name>
<value>DAV:too-many-matches</value>
</arg>
</verify>
## propfindItems
Performs a check of propfind multi-status response body and checks to see
whether the returned properties (and optionally their values) are good (2xx) or bad
(non-2xx) response code. The overall response status must be 207.
Argument: 'root-element'
Expected root element for the XML response. Normally this is DAV:multistatus
but, e.g., MKCOL ext uses a different root, but mostly looks like multistatus
otherwise.
Argument: 'okprops'
A set of properties for which a 2xx response status is required. Two forms can be used:
'propname' - will test for the presence of the property named
'propname'. The element data must be a qualified XML element
name.
'propname$value' - will test for the presence of the property
named 'propname' and check that its value matches the provided
'value'. The element data must be a qualified XML element name.
XML elements in the property value can be tested provided proper
XML escaping is used (see example).
'propname!value' - will test for the presence of the property
named 'propname' and check that its value does not match the provided
'value'. The element data must be a qualified XML element name.
XML elements in the property value can be tested provided proper
XML escaping is used (see example).
Argument: 'badhrefs'
A set of properties for which a non-2xx response status is
required. The same two forms as used for 'okprops' can be used
here.
Example:
<verify>
<callback>propfindItems</callback>
<arg>
<name>okprops</name>
<value>{DAV:}getetag</value>
<value>{DAV:}getcontenttype$text/plain</value>
<value>{X:}getstate$<X:ok/></value>
</arg>
<arg>
<name>badprops</name>
<value>{X:}nostate</value>
</arg>
</verify>
## propfindValues
Performs a regular expression match against property values. The overall
response status must be 207.
Argument: 'props'
A set of properties for which a 2xx response status is required. Two forms can be used:
'propname$value' - will test for property value match
'propname!value' - will test for property value non-match
Argument: 'ignore'
One or more href values for hrefs in the response which will be
ignored. e.g. when doing a PROPFIND Depth:1, you may want to
ignore the top-level resource when testing as only the
properties on the child resources may be of interest.
Example:
<verify>
<callback>propfindValues</callback>
<arg>
<name>props</name>
<value>{DAV:}getcontenttype$text/.*</value>
<value>{DAV:}getcontenttype!text/calendar</value>
</arg>
<arg>
<name>ignore</name>
<value>/calendars/test/</value>
</arg>
</verify>
## statusCode
Performs a simple test of the response status code and returns True
if the code matches, otherwise False.
Argument: 'status'
If the argument is not present, the any 2xx status code response
will result in True. The status code value can be specified as
'NNN' or 'Nxx' where 'N' is a digit and 'x' the letter x. In the
later case, the verifier will return True if the response status
code's 'major' digit matches the first digit.
Example:
<verify>
<callback>statusCode</callback>
<arg>
<name>status</name>
<value>2xx</value>
</arg>
</verify>
## xmlDataMatch
Compares the response with an XML data file and returns TRUE if there
is a match, otherwise False.
Argument: 'filepath'
The file path to a file containing data to match the response body to.
Argument: 'filter'
Any specified XML elements will have their content removed from the
response XML data before the comparison with the file data is done.
This can be used to ignore element values that change in each request,
e.g., a time stamp.
Example:
<verify>
<callback>xmlDataMatch</callback>
<arg>
<name>filepath</name>
<value>resource/test.xml</value>
</arg>
<arg>
<name>filter</name>
<value>{DAV:}getlastmodified</value>
</arg>
</verify>
## xmlElementMatch
Compares the response with an XML path and returns TRUE if there
is a match, otherwise False.
The path is the absolute xpath from the root element down. Attribute, attribute-value
and text contents tests of the matched element can be done using:
[@attr] - "attr" is present as an attribute
[@attr=value] - "attr" is present as an attribute with the value "value"
[=text] - node text is "text".
[!text] - node text is not "text".
[*text] - node text contains "text".
[$text] - node text does not contain "text".
[+text] - node text starts with "text".
[^tag] - node has child element "tag".
[^tag=text] - node has child element "tag" with text "text".
[|] - node is empty.
[||] - node is not empty.
[json] - node contains valid JSON data.
[icalendar] - node contains valid iCalendare data.
Each path segment can now have its own test and "../" can be used to move up to
the parent. This allows testing for an element matching specific content plus
its sibling matching other specific content. e.g., "/{D}A/{D}B[=b]/../{D}C[=c]
which checks for an element {D}A with two child elements {D}B and {D}C each
with a specific value.
Argument: 'parent'
ElementTree style path for an XML element to use as the root for any
subsequent "exists" or "notexists" tests. This is useful for targeting
a specific resource in a Depth:1 multistatus response.
Argument: 'exists'
ElementTree style path for an XML element to check the presence of
in the response.
Argument: 'notexists'
ElementTree style path for an XML element to check the absence of
in the response.
Example:
<verify>
<callback>xmlDataMatch</callback>
<arg>
<name>exists</name>
<value>{DAV:}response/{DAV:}href</value>
</arg>
<arg>
<name>notexists</name>
<value>{DAV:}response/{DAV:}getetag</value>
</arg>
</verify>