FunkLoad

FunkLoad is a functional and load web tester

Table of Contents

1 Introducing FunkLoad
1.1 What is FunkLoad ?
1.2 Where to find FunkLoad ?
1.3 Installation
1.4 Examples
1.5 Documentation
1.6 Credits
2 Test runner
2.1 Usage
2.2 Examples
2.3 Options
3 Benching
3.1 Principle
Page
Test
Cycle
Cycles
3.2 Bench runner
Usage
Examples
Options
3.3 Tips
4 Bench report
4.1 Usage
4.2 Examples
4.3 Options
5 Test Recorder
5.1 Recording a new FunkLoad test
5.2 The fl-record command
Usage
Examples
Options
6 Credential server
7 Monitor server
8 The FunkLoadTestCase
8.1 Browser API
get
post
exists
setBasicAuth
clearBasicAuth
setUserAgent
addHeader
clearHeaders
debugHeaders
8.2 XML RPC API
xmlrpc
8.3 Assertion helpers API
getLastUrl
getLastBaseUrl
listHref
getBody
The response object
8.4 Configuration file API
conf_get
conf_getInt
conf_getFloat
 conf_getList
8.5Logging
logd
logi
8.6 Lipsum API
Lipsum
getWord
getUniqWord
getSubject
getSentence
getParagraph
getMessage
getPhoneNumber
getAddress
8.7 Utils
xmlrpc_get_credential
xmlrpc_list_groups
xmlrpc_list_credentials
9 FunkLoadDocTest
10 Other Test Cases
10.1 The ZopeTestCase
zopeRestart
zopePackZodb
zopeFlushCache
zopeAddExternalMethod
10.2 CPSTestCase
cpsCreateSite
cpsLogin
cpsLogout
cpsCreateGroup
cpsVerifyGroup
cpsCreateUser
cpsVerifyUser
cpsSetLocalRole
cpsCreateSection
cpsCreateWorkspace
cpsCreateDocument
cpsCreateNewsItem
cpsChangeUiLanguage
cpsListDocumentHref
cpsSearchDocId
11 Bug reporting

1 Introducing FunkLoad
1.1 What is FunkLoad ?
FunkLoad is a functional and load web tester, written in Python, whose main use cases are:

Functional testing of web projects, and thus regression testing as well.

Performance testing: by loading the web application and monitoring your servers it helps you to
pinpoint bottlenecks, giving a detailed report of performance measurement.
Load testing tool to expose bugs that do not surface in cursory testing, like volume testing or longevity
testing.
Stress testing tool to overwhelm the web application resources and test the application recoverability.
Writing web agents by scripting any web repetitive task, like checking if a site is alive.

Main FunkLoad features are:

FunkLoad is free software distributed under the GNU GPL.

Functional test are pure Python scripts using the pyUnit framework like normal unit test. Python enable
complex scenarios to handle real world applications.

Truly emulates a web browser (single-threaded) using an enhanced Richard Jones' webunit:

basic authentication support

cookies support
 referrer support
fetching css, javascript and images
emulating a browser cache
file upload and multipart/form-data submission
post any kind of content type like application/xml
https support
https with ssl/tls by providing a private key and certificate (PEM formatted)
http_proxy support
Advanced test runner with many command-line options:

set the target server url

display the fetched page in real time in your browser
debug mode to display http headers
check performance of a single page (or set of pages) inside a test
green/red color mode
select or exclude tests cases using a regex
support normal pyUnit test
support doctest from a plain text file or embedded in python docstring
Turn a functional test into a load test: just by invoking the bench runner you can identify scalability and
performance problems.

Detailed bench reports in ReST or HTML (and PDF via ps2pdf) containing:

the bench configuration

tests, pages, requests stats and charts.
the 5 slowest requests.
monitoring one or many servers cpu usage, load average, memory/swap usage and network
traffic charts.
an http error summary list
View a report example.

Differential reports to compare 2 bench reports giving a quick overview of scalability and velocity
changes. View a diff report example.

Easy test customization using a configuration file or command line options.

Easy test creation using embeded TCPWatch as proxy recorder, so you can use your web browser and
produce a FunkLoad test automatically, including file upload or any ajax call.

Provides web assertion helpers to check expected results in responses.

Provides helpers to retrieve contents in responses page using DOM.

Provides a funkload.CPSTestCase to ease Zope and Nuxeo CPS testing.

Easy to install (EasyInstall) and use, see examples in the demo folder.

Works fine with CPS or any Zope application, CherryPy, Nuxeo EP or any JBoss Seam JSF MyFaces
applications ...

1.2 Where to find FunkLoad ?

Either:

Latest stable package using EasyInstall:

EasyInstall the latest snapshot

Bleeding edge svn sources:

See CHANGES file for information about distribution contents.

1.3 Installation
See the INSTALL file for requirements and installation.

1.4 Examples
See the demo folder for test scripts and the report example.

For package installed with easy_install you need to run fl-install-demo to extract the demo examples.

1.5 Documentation
This page is the main FunkLoad documentation, there are also:

CHANGES for information about distribution contents.

INSTALL for requirement and installation.
API documentation generated with epydoc.
SLIDES introducing FunkLoad.

1.6 Credits
Thanks to Frank Cohen's TestMaker framework, Richard Jones and all contributors.

2 Test runner
A FunkLoad test can be used like a standard unittest using a unittest.main() and a 'python MyFile.py'.

To ease testing FunkLoad come with an advanced test runner to override the static configuration file.

The loop-on-pages option enable to check response time of some specific pages inside a test without changing
the script, which make easy to tune a page in a complex context. Use the debug option to find the page
numbers.

Note that fl-run-test can be used to launch normal unittest.TestCase and (if you use python2.4) doctest in a
plain text file or embedded in a python docstring. The --debug option makes doctests verbose.

2.1 Usage

2.2 Examples
2.3 Options

3 Benching
The same FunkLaod test can be turned into a load test, just by invoking the bench runner fl-run-bench.

3.1 Principle
Here are some definitions used in bench mode:

CUs: Concurrent Users, which is the number of threads.

STPS: Average of Successful Tests Per Second during a cycle.
 SPPS: Average of Successfully Page Per Second during a cycle.
RPS: Average Request Per Second, successfully or not.
max[STPS|SPPS|RPS]: maximum of STPS|SPPS|RPS for a cycle.

Page
A page is an http get/post request with associated sub requests like redirects, images or links (css, js files).
This is what users see as a single page.

Test
A test is made with 3 methods: setUp/test_name/tearDown. During the test_name method each get/post
request is called a page.

Cycle
A cycle is a load of n concurrents test during a 'duration' period. Threads are launched every 'startupdelay'
seconds, each thread executes test in a loop.

Once all threads have been started we start to record stats.

Only tests that end during the 'duration' period are taken into account for the test stats (in the representation
below test like [---X are not take into account).

Only pages and requests that finish during the 'duration' are taken into account for the request and pages
statistic

Before a cycle a setUpCycle method is called, after a cycle a tearDownCycle method is called, you can use
these methods to test differents server configuration for each cycle.

Cycles
FunkLoad can execute many cycles with different number of CUs, this way you can find easily the maximum
number of users that your application can handle.

Running n cycles with the same CUs is a good way to see how the application handles a writing test over time.

Running n cycles with the same CUs with a reading test and a setUpCycle that change the application
configuration will help you to find the right tuning.
3.2 Bench runner
Usage

Examples

Options

3.3 Tips
Here are few remarks/advices to obtain workable metrics.
 Since it uses significant CPU resources, make sure that performance limits are not hit by FunkLoad
before your server's limit is reached. Check this by launching a bench from another host.

Having a cycle with one user gives a usefull reference.

A bench is composed of a benching test (or scenario) run many times. A good benching test should not
be too long so you have a higher testing rate (that is, more benching tests can come to their end).

The cycle duration for the benching test should be long enough. Around 5 times the duration of a single
benching test is a value that is usually a safe bet. You can obtain this duration of a single benching test
by running fl-run-test myfile.py MyTestCase.testSomething.

Rationale : Normally a cycle duration of a single benching test should be enough. But from the testing
platform side if there are more than one concurrent user, there are many threads to start and it takes
some time. And on from the tested platform side it is common that a benching test will last longer and
longer as the server is used by more and more users.

You should use many cycles with the same step interval to produce readable charts
(1:10:20:30:40:50:60 vs 1:10:100)

A benching test must have the same number of page and in the same order.

Use a Makefile to make reproductible bench.

There is no debug option while doing a bench (since this would be illegible with all the threads). So, if a
bench fails (that is using fl-run-bench), use fl-run-test -d to debug.

Using fl-record is very easy and very fast to create a scenario. But since it doesn't support HTTPS, the
good practise is to first record a scenario with fl-record on HTTP, and then change the url back to https
in your FunkLoad test configuration file.

Always use description in post/get/xmlrpc, this improves the readability of the report.

4 Bench report
To produce an HTML or ReST report you need to invoke the fl-build-report, you can easily produce PDF report
using Firefox 'Print To File' in PostScript then use the ps2pdf converter.

4.1 Usage

fl-build-report analyze a FunkLoad bench xml result file and output a report.

4.2 Examples

4.3 Options
5 Test Recorder
5.1 Recording a new FunkLoad test
Starting with FunkLoad 1.3.0 you can use fl-record to record your navigator activity, this requires the
TCPWatch python proxy see INSTALL for information on how to install TCPWatch.

1. Start the recorder:

This will output something like this:

2. Setup your browser proxy and play your scenario

in Firefox: Edit > Preferencies > General; Connection Settings set localhost:8090 as your
HTTP proxy

Play your scenario using your navigator

Hit Ctrl-C to stop recording:

3. Replay you scenario:

You should see all the steps on your navigator.

4. Implement the dynamic part and assertion

Code the dynamic part like getting new url of a created document
Add assertion using FunkLoad helpers
Use a credential server if you want to make a bench with different users or simply don't
want to hard code your login/password.

Note that fl-record works fine with multi-part encoded form and file upload but will failed to record https
session.

Note that fl-record will handle automaticly JSF Myfaces token, which enable to easily record and play any
JBoss Seam application.

5.2 The fl-record command

Usage

Examples

Options

6 Credential server
If you are writing a bench that requires to be logged with different users FunkLoad provides an xmlrpc
credential server to serve login/pwd between the different threads.

It requires 2 files (like unix /etc/passwd and /etc/group) the password file have the following format:

The group file format is:

Setup a configuration file like in the demo/cmf folder, then start the credential server:
More options:

See the funkload-demo/cmf example for a credential configuration file.

7 Monitor server
If you want to monitor a linux server health during the bench, you have to run a monitor xmlrpc server on the
target server, this require to install the FunkLoad package.

On the server side you need to install the FunkLoad tool then launch the server using a configuration file
(example in the demo/simple folder.):

On the bench host side setup your test configuration like this:

Then run the bench, the report will include server stats.

Note that you can monitor multiple hosts and that the monitor is linux specific.

8 The FunkLoadTestCase
FunkLoadTestCase extends the pyUnit unittest.TestCase with browser capabilities, configuration file helpers
and assertions helpers. FunkLoad provides also some tools to generate random inputs and communicate with
credential servers.

Here is an overview of the api, you can find more on

8.1 Browser API

get

This emulates a browser http GET link. It will fetch the url, submits appropriate cookies, follow redirection,
register new cookies, load css and javascript.

It also simulates a browser cache by not reloading a css, a javascript or an image twice.

Note that this is an emulation with some limitation:

It is single threaded (it loads images one after the other)

It does not interpret javascript
See trac tickets that starts with Browser: for other limitations

This method returns a webunit HTTPResponse.

Parameters:

url a valid url

params a dico of parameters that going to be append to the url like url?key1=value1&...
description is used on the bench report to describe the user action
ok_codes is a list of http expected code like [200:301] if the http response is not in the list get will
raise a test failure exception, if not provided assume that the default list is [200, 301, 302].

Note that if the url already contains encoded parameters you should not use the params parameter.

post

Same interface than the get() but it uses a http post method. You can upload a file by setting a params like
this:

You can even post any contents like this (available since 1.9.0):

exists

Return True if the http response code is 200, 301 or 302, and return False if http code is 404 or 503, other
codes will raise a test failure exception.

setBasicAuth

Next requests will use the http basic authentication.

clearBasicAuth

Remove basic auth credential set by setBasicAuth.

setUserAgent

New in 1.3.0. version.

Set a User-Agent http header for the next requests, the default browser behaviour is to use the agent defined in
the configuration file under [main] user_agent or to use the default FunkLoad/version string. Using this method
enable to change the user agent during a test case.

addHeader
New in 1.3.0. version.

Add an http header for the next requests.

clearHeaders

New in 1.3.0. version.

Remove all headers previously added by addHeader or setUserAgent, and remove the referer as well.

debugHeaders

New in 1.11.0. version.

Display headers sent to the server for the next get or post request.

8.2 XML RPC API

You can test or bench xmlrpc services using the following API.

xmlrpc

Call the method_name at url using xmlrpclib. You can use the setBasicAuth method before to handle the http basic
authentication. Note that due to xmlrpclib limitation you can not use an http proxy.

Parameters:

url the url of the xmlrpc server

method_name the name of the procedure to call
params a list of parameters to pass to the method
description is used on the bench report to describe the action

8.3 Assertion helpers API

FunkLoad uses the unittest assertion (assert_, assertEquals, fail, ...), but provides some methods to check the
http response. After fetching a page you can use the following methods.

getLastUrl

Return the last accessed page url taking care of redirects.

getLastBaseUrl

Return the <base /> href value of the last accessed page.

listHref
Return a list of href anchor url present in the last html response, filtering href using the url_pattern regex if
present and filtering the text content of the link with content_pattern if present.

getBody

Return the last response content.

The response object

The response returned by a get or post are webunit HTTPResponse object

getDOM return a SimpleDOM interface of the fetched html page, see the webunit SimpleDOM api instructions
for details.

8.4 Configuration file API

A FunkLoadTestCase class uses a configuration file to setup variable configuration, like the base server url to
be tested, the test description, credential access, logging files and other test specific parameters. The test
configuration file have the same name of the FunkLoadTestCase with a '.conf' extension. See documented
examples in the demo folder (fl-install-demo).

conf_get

Return an entry from the configuration file. Note that the entry may be overriden by a command line option.

Parameters:

section the section in the configuration file.

key the key.
default a default value.

conf_getInt
Return an integer.

conf_getFloat
Return a float.

conf_getList
Additional parameter:

separator the default separator is a colon ':'.

Return a list

8.5 Logging
A FunkLoadTestCase store its results in an xml file (like request and test result) and put other log information
into a text log and/or output to the console.

logd

Debug log message

logi

Information log message

8.6 Lipsum API

To generate dummy document contents you can use the funkload.Lipsum api, this is a very simple "Lorem
ipsum" generator.

You can see some examples by doing:

Lipsum

Parameters:

vocab a list of word, Lipsum provide 3 lists V_ASCII, V_DIAC, V_8859_15

chars the list of char used to build an identifier
sep some separators used in sentences like coma, question mark ...

getWord

Return a random word from the vocabulary.

getUniqWord

Generate a kind of uniq id.

getSubject

Return a subject of length word.

Parameters:

length the number of words in the subject

prefix a prefix to add at the beginning of a the subject
uniq add an uniq identifier in the subject
length_min/max the words length is a random between min and max
getSentence

Return a sentence with some separators and and a ending point.

getParagraph

Return a paragraph of length sentences.

getMessage

Return a message with length Paragraphs.

getPhoneNumber

Return a random phone number.

Parameters:

lang can be fr or en_US

format can be short, medium or long

getAddress

Return a random address.

8.7 Utils
To communicate with FunkLoad services like the credential server, there are some wrappers in the utils
module.

xmlrpc_get_credential

Return a tuple login, password of a user that belong to group if specified.

xmlrpc_list_groups
List groups name served by the credential server.

xmlrpc_list_credentials
List all login/password served by the credential server.

9 FunkLoadDocTest
Since FunkLoad 1.5 you can use funkload easily from a doctest:
FunkLoadDocTest exposes the same API than The FunkLoadTestCase.

10 Other Test Cases

10.1 The ZopeTestCase
This class extends the FunkLoadTestCase providing common Zope tasks.

zopeRestart

Stop and Start the Zope server.

Parameters:

zope_url the zope url.

admin_id and admin_pwd the zope admin credential.
time_out maximum time to wait until the zope server restart.

zopePackZodb

Pack a zodb database.

Parameters:

database the database to pack.

days removing previous revision that are older than days ago

zopeFlushCache

Remove all objects from all ZODB in-memory caches.

zopeAddExternalMethod

Add an External method an run it.

10.2 CPSTestCase
This class extends the ZopeTestCase providing common Nuxeo CPS tasks. You need to import the
CPSTestCase that works with your CPS for example CPS338TestCAse or CPS340TestCase.

cpsCreateSite
Build a new CPS site.

Parameters:

admin_id and admin_pwd the zope admin credential.

manager_id and manager_pwd the cps manager credential.
zope_url the Zope server url [*].
site_id the CPS site id.
[*] if the zope_url and site_id is not given we guess it using the server_url

cpsLogin

CPS log in.

cpsLogout
Logout the user logged in using cpsLogin.

cpsCreateGroup

Create a CPS group.

cpsVerifyGroup

Create a CPS group if not present.

cpsCreateUser

Create a CPS users.

cpsVerifyUser
Create a CPS users if not present.

cpsSetLocalRole

Grant role to name in url.

cpsCreateSection

cpsCreateWorkspace
cpsCreateDocument

Create a random document in the parent_url container.

cpsCreateNewsItem

Create a simple news in the parent_url container.

cpsChangeUiLanguage

Change the ui locale selection

cpsListDocumentHref

Return a clean list of document href that matches pattern in the previous page fetched.

cpsSearchDocId

Return the list of url that ends with doc_id, using catalog search.

11 Bug reporting
The list of open task and bugs are here: http://svn.nuxeo.org/trac/pub/report/12

If you want to report a bug or if you think that something is missing, send me an email bdelbosc _at_
nuxeo.com.

Generated on: 2010-07-27 20:11 UTC.