GeoServerDeveloperManual PDF

GeoServer Developer Manual
Release 2.3.0
GeoServer
March 19, 2013
Contents
1 Introduction 3
1.1 License . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Tools 7
2.1 Java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Maven . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3 Git . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3 Source Code 9
3.1 Git . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2 Committing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.3 Repository distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.4 Repository structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.5 Codebase structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.6 Git client conguration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.7 Development workow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.8 More useful reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
4 Quickstart 17
4.1 Check out source code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.2 Build with Maven . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.3 Generate Eclipse project les with Maven . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.4 Import modules into Eclipse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.5 Run GeoServer from Eclipse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.6 Access GeoServer front page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
5 Maven Guide 25
5.1 Installing Maven . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.2 Running Maven . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.3 Building . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.4 Skipping tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
5.5 Building ofine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
5.6 Building extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
5.7 Recover Build . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
i
5.8 Proles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
5.9 Eclipse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
5.10 Building the web module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5.11 Running the web module with Jetty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5.12 Installing the Oracle module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
6 Eclipse Guide 31
6.1 Importing modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
6.2 Running and debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
6.3 Eclipse preferences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
7 Findbugs Guide 39
7.1 Running the Findbugs Report Locally with Maven . . . . . . . . . . . . . . . . . . . . . . . . 39
7.2 Using the Findbugs Eclipse Plugin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
7.3 Fixing Bugs from Findbugs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
8 Programming Guide 43
8.1 OWS Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
8.2 REST Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
8.3 Web User Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
8.4 Wicket Development In GeoServer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
8.5 Extension Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
8.6 WPS Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
8.7 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
8.8 Versioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
8.9 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
8.10 App-Schema Online Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
9 Release Schedule 135
9.1 Release branches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
9.2 Release cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
9.3 Development phases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
9.4 Commit rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
10 Release Guide 139
10.1 Before you start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
10.2 Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
10.3 Versions and revisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
10.4 Release in JIRA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
10.5 Build the Release . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
10.6 Test the Artifacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
10.7 Publish the Release . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
10.8 Create the download page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
10.9 Post the Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
10.10 Announce the Release . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
11 Release Testing Checklist 147
11.1 Artifact size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
11.2 Demos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
11.3 Sample requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
11.4 Map preview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
11.5 KML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
11.6 GeoWebCache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
12 Cite Test Guide 153
ii
12.1 Check out CITE tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
12.2 Run WFS 1.0 tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
12.3 Run WFS 1.1 tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
12.4 Run WMS 1.1 tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
12.5 Run WCS 1.1 tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
12.6 Run WCS 1.0 tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
12.7 Teamengine Web Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
13 Translating GeoServer 161
13.1 Translating the UI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
13.2 Translating documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
14 Policies and Procedures 163
14.1 Committing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
14.2 Submitting Patches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
14.3 Code Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
14.4 Community Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
14.5 GeoServer Improvement Proposals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
14.6 Community Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
14.7 Project Steering Committee . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
iii
iv
GeoServer Developer Manual, Release 2.3.0
Welcome to the GeoServer Developer Manual. The manual is for those who want to help with the develop-
ment process, including source code, software releasing, and other administrative work.
Contents 1
2 Contents
CHAPTER 1
Introduction
Welcome to GeoServer development. The project makes use of a number of resources:
http://geoserver.org/display/GEOS/Welcome (conuence wiki)
https://github.com/geoserver/geoserver (github source code)
http://jira.codehaus.org/browse/GEOS (codehaus issue tracker)
GeoServer User Manual
GeoServer Developer Manual
Communication channels:
http://blog.geoserver.org/
geoserver-devel email list
geoserver-users email list
irc://irc.freenode.net/#geoserver
We have a number of build servers employed to assist with day to day activities:
http://hudson.opengeo.org/hudson/view/geoserver/ (main build server)
http://ofce.geo-solutions.it/jenkins/ (windows build server)
http://aaime.no-ip.org/jenkins/ (JDK 7)
http://aaime.no-ip.org/sonar/ (Sonar)
Notication email lists:
https://groups.google.com/forum/#!forum/geoserver-commits
https://groups.google.com/forum/#!forum/geoserver-extra-builds
Question and answer:
http://gis.stackexchange.com/questions/tagged/geoserver
http://stackoverow.com/questions/tagged/geoserver
3
1.1 License
GeoServer is free software and is licensed under the GNU General Public License.:
GeoServer, open geospatial information server
Copyright (C) 2001 - 2011 The Open Planning Project dba OpenPlans
http://openplans.org
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version (collectively, "GPL").
As an exception to the terms of the GPL, you may copy, modify,
propagate, and distribute a work formed by combining GeoServer with the
Eclipse Libraries, or a work derivative of such a combination, even if
such copying, modification, propagation, or distribution would otherwise
violate the terms of the GPL. Nothing in this exception exempts you from
complying with the GPL in all respects for all of the code used other
than the Eclipse Libraries. You may include this exception and its grant
of permissions when you distribute GeoServer. Inclusion of this notice
with such a distribution constitutes a grant of such permissions. If
you do not wish to grant these permissions, remove this paragraph from
your distribution. "GeoServer" means the GeoServer software licensed
under version 2 or any later version of the GPL, or a work based on such
software and licensed under the GPL. "Eclipse Libraries" means Eclipse
Modeling Framework Project and XML Schema Definition software
distributed by the Eclipse Foundation and licensed under the Eclipse
Public License Version 1.0 ("EPL"), or a work based on such software and
licensed under the EPL.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Suite 500, Boston, MA 02110-1335 USA
Additionally:
Several les supporting GZIP compressions (GZIPFilter, GZIPResponseStream, GZIPResponseWrap-
per are provided using:
/
*
*
Copyright 2003 Jayson Falkner (jayson@jspinsider.com)
*
This code is from "Servlets and JavaServer pages; the J2EE Web Tier",
*
http://www.jspbook.com. You may freely use the code both commercially
*
and non-commercially. If you like the code, please pick up a copy of
*
the book and help support the authors, development of more free code,
*
and the JSP/Servlet/J2EE community.
*
*
Modified by David Winslow <dwinslow@openplans.org>
*
/
SetCharacterEncodingFilter and RewindableInputStream makes use of code provided under Apache
License Version 2.0.
4 Chapter 1. Introduction
UCSReader is provided using Apache License Version 1.1.
1.1. License 5
6 Chapter 1. Introduction
CHAPTER 2
Tools
The following tools need to installed on the system before a GeoServer developer environment can be set
up.
2.1 Java
Developing with GeoServer requires a Java Development Kit (JDK) 6 or greater, available from Oracle.
Note: While it is possible to use a JDK other than the one provided by Oracle, it is recommended that the
Oracle JDK be used.
2.2 Maven
GeoServer uses a tool known as Maven to build. The current recommended version of Maven is 2.2.1 and
is available from Apache. While 2.2.1 is recommended any version greater than 2.0.8 should also work.
Maven tracks global settings in your home directory .m2/settings.xml. This le is used to control global
options such as proxy settings or listing repository mirrors to download from.
2.3 Git
GeoServer source code is stored and version in a git repository on github There are a variety of git clients
available for a number of different platforms. Visit http://git-scm.com/ for more details.
7
8 Chapter 2. Tools
CHAPTER 3
Source Code
The GeoServer source code is located on GitHub at https://github.com/geoserver/geoserver.
To clone the repository:
% git clone git://github.com/geoserver/geoserver.git geoserver
To list available branches in the repository:
% git branch
2.1.x
2.2.x
*
master
To switch to the stable branch:
% git checkout 2.2.x
3.1 Git
Those coming from a Subversion or CSV background will nd the Git learning curve is a steep one. Luckily
there is lots of great documentation around. Before continuing developers should take the time to educate
themselves about git. The following are good references:
The Git Book
A nice introduction
3.2 Committing
In order to commit to the repository the following steps must be taken:
1. Congure your git client for cross platform projects. See notes below.
2. Register for commit access as described here.
3. Fork the canonical GeoServer repository into your github account.
4. Clone the forked repository to create a local repository
9
5. Create a remote reference to the canonical repository using a non-read only URL
(git@github.com:geoserver/geoserver.git).
Note: The next section describes howthe git repositories are distributed for the project and howto manage
local repository remote references.
3.3 Repository distribution
Git is a distributed versioning system which means there is strictly no notion of a single central repository,
but many distributed ones. For GeoServer these are:
The canonical repository located on GitHub that serves as the ofcial authoritative copy of the source
code for project
Developers forked repositories on GitHub. These repositories generally contain everything in the
canonical repository, as well any feature or topic branches a developer is working on and wishes to
back up or share.
Developers local repositories on their own systems. This is where development work is actually
done.
Even though there are numerous copies of the repository they can all interoperate because they share a
common history. This is the magic of git!
In order to interoperate with other repositories hosted on GitHub, a local repository must contain remote
references to them. A local repository typically contains the following remote references:
A remote called origin that points to the developers forked GitHub repository.
A remote called upstream that points to the canonical GitHub repository.
Optionally, some remotes that point to other developers forked repositories on GitHub.
To set up a local repository in this manner:
1. Clone your fork of the canonical repository (where bob is replaced with your GitHub account
name):
% git clone git@github.com:bob/geoserver.git geoserver
% cd geoserver
2. Create the upstream remote pointing to the canonical repository:
% git remote add upstream git@github.com:geoserver/geoserver.git
Or if your account does not have push access to the canonical repository use the read-only url:
% git remote add upstream git://github.com/geoserver/geoserver.git
3. Optionally, create remotes pointing to other developers forks. These remotes are typically read-only:
% git remote add aaime git://github.com/aaime/geoserver.git
% git remote add jdeolive git://github.com/jdeolive/geoserver.git
3.4 Repository structure
A git repository contains a number of branches. These branches fall into three categories:
10 Chapter 3. Source Code
1. Primary branches that correspond to major versions of the software
2. Release branches that are used to manage releases of the primary branches
3. Feature or topic branches that developers do development on
3.4.1 Primary branches
Primary branches are present in all repositories and correspond to the main release streams of the project.
These branches consist of:
The master branch that is the current unstable development version of the project
The current stable branch that is the current stable development version of the project
The branches for previous stable versions
For example at present these branches are:
master - The 2.3.x release stream, where unstable development such as major new features take place
2.2.x - The 2.2.x release stream, where stable development such as bug xing and stable features take
place
2.1.x - The 2.1.x release stream, which is at end-of-life and has no active development
3.4.2 Release branches
Release branches are used to manage releases of stable branches. For each stable primary branch there is a
corresponding release branch. At present this includes:
rel_2.2.x - The stable release branch
rel_2.1.x - The previous stable release branch
Release branches are only used during a versioned release of the software. At any given time a release
branch corresponds to the exact state of the last release from that branch. During release these branches are
tagged.
Release branches are also present in all repositories.
3.4.3 Feature branches
Feature branches are what developers use for day-to-day development. This can include small-scale bug
xes or major new features. Feature branches serve as a staging area for work that allows a developer to
freely commit to them without affecting the primary branches. For this reason feature branches generally
only live in a developers local repository, and possibly their remote forked repository. Feature branches
are never pushed up into the canonical repository.
When a developer feels a particular feature is complete enough the feature branch is merged into a primary
branch, usually master. If the work is suitable for the current stable branch the changeset can be ported
back to the stable branch as well. This is explained in greater detail in the Development workow section.
3.5 Codebase structure
Each branch has the following structure:
3.5. Codebase structure 11
build/
doc/
src/
data/
build - release and continuous integration scripts
doc - sources for the user and developer guides
src - java sources for GeoServer itself
data - a variety of GeoServer data directories / congurations
3.6 Git client conguration
When a repository is shared across different platforms it is necessary to have a strategy in place for dealing
with le line endings. In general git is pretty good about dealing this without explicit conguration but to
be safe developers should set the core.autocrlf setting to input:
% git config --global core.autocrlf input
The value input essentially tells git to respect whatever line ending form is present in the git repository.
Note: It is also a good idea, especially for Windows users, to set the core.safecrlf option to true:
% git config --global core.safecrlf true
This will basically prevent commits that may potentially modify le line endings.
Some useful reading on this subject:
http://www.kernel.org/pub/software/scm/git/docs/git-cong.html
https://help.github.com/articles/dealing-with-line-endings
http://stackoverow.com/questions/170961/whats-the-best-crlf-handling-strategy-with-git
3.7 Development workow
This section contains examples of workows a developer will typically use on a daily basis. To follow
these examples it is crucial to understand the phases that a changeset goes though in the git workow. The
lifecycle of a single changeset is:
1. The change is made in a developers local repository.
2. The change is staged for commit.
3. The staged change is committed.
4. The committed changed is pushed up to a remote repository
There are many variations on this general workow. For instance, it is common to make many local commits
and then push them all up in batch to a remote repository. Also, for brevity multiple local commits may be
squashed into a single nal commit.
3.7.1 Updating from canonical
Generally developers always work on a recent version of the ofcial source code. The following example
shows how to pull down the latest changes for the master branch from the canonical repository:
% git checkout master
% git pull upstream master
Similarly for the stable branch:
% git pull upstream 2.2.x
3.7.2 Making local changes
As mentioned above, git has a two-phase workow in which changes are rst staged and then committed
locally. For example, to change, stage and commit a single le:
# do some work on file x
% git add x
% git commit -m "commit message" x
Again there are many variations but generally the staging process involves using git add to stage les
that have been added or modied, and git rm to stage les that have been deleted. git mv is used to
move les and stage the changes in one step.
At any time you can run git status to check what les have been changed in the working area and what
has been staged for commit. It also shows the current branch, which is useful when switching frequently
between branches.
3.7.3 Pushing changes to canonical
Once a developer has made some local commits they generally will want to push them up to a remote
repository. For the primary branches these commits should always be pushed up to the canonical reposi-
tory. If they are for some reason not suitable to be pushed to the canonical repository then the work should
not be done on a primary branch, but on a feature branch.
For example, to push a local bug x up to the canonical master branch:
# make a change
% git add/rm/mv ...
% git commit -m "making change x"
% git pull upstream master
% git push upstream master
The example shows the practice of rst pulling from canonical before pushing to it. Developers should
always do this. In fact, if there are commits in canonical that have not been pulled down, by default git will
not allow you to push the change until you have pulled those commits.
Note: A merge commit may occur when one branch is merged with another. A merge commit occurs
when two branches are merged and the merge is not a fast-forward merge. This happens when the target
branch has changed since the commits were created. Fast-forward merges are worth reading about.
An easy way to avoid merge commits is to do a rebase when pulling down changes:
3.7. Development workow 13
% git pull --rebase upstream master
The rebase makes local changes appear in git history after the changes that are pulled down. This allows the
following merge to be fast-forward. This is not a required practice since merge commits are fairly harmless,
but they should be avoided where possible since they clutter up the commit history and make the git log
harder to read.
3.7.4 Working with feature branches
As mentioned before, it is always a good idea to work on a feature branch rather than directly on a primary
branch. A classic problem every developer who has used a version control system has run into is when
they have worked on a feature locally and made a ton of changes, but then need to switch context to work
on some other feature or bug x. The developer tries to make the x in the midst of the other changes and
ends up committing a le that should not have been changed. Feature branches are the remedy for this
problem.
To create a new feature branch off the master branch:
% git checkout -b my_feature master
% # make some changes
% git add/rm, etc...
% git commit -m "first part of my_feature"
Rinse, wash, repeat. The nice about thing about using a feature branch is that it is easy to switch context to
work on something else. Just git checkout whatever other branch you need to work on, and then return
to the feature branch when ready.
Note: When a branch is checked out, all the les in the working area are modied to reect the current
state of the branch. When using development tools which cache the state of the project (such as Eclipse) it
may be necessary to refresh their state to match the le system. If the branch is very different it may even
be necessary to perform a rebuild so that build artifacts match the modied source code.
3.7.5 Merging feature branches
Once a developer is done with a feature branch it must be merged into one of the primary branches and
pushed up to the canonical repository. The way to do this is with the git merge command:
% git merge my_feature
Its as easy as that. After the feature branch has been merged into the primary branch push it up as described
before:
3.7.6 Porting changes between primary branches
Often a single change (such as a bug x) has to be committed to multiple branches. Unfortunately primary
branches cannot be merged with the git merge command. Instead we use git cherry-pick.
As an example consider making a change to master:
% # make the change
% git add/rm/etc...
% git commit -m "fixing bug GEOS-XYZ"
We want to backport the bug x to the stable branch as well. To do so we have to note the commit id of the
change we just made on master. The git log command will provide this. Lets assume the commit id is
123. Backporting to the stable branch then becomes:
% git cherry-pick 123
% git pull --rebase upstream 2.2.x
% git push upstream 2.2.x
3.7.7 Cleaning up feature branches
Consider the following situation. Adeveloper has been working on a feature branch and has gone back and
forth to and from it making commits here and there. The result is that the feature branch has accumulated
a number of commits on it. But all the commits are related, and what we want is really just one commit.
This is easy with git and you have two options:
1. Do an interactive rebase on the feature branch
2. Do a merge with squash
Interactive rebase
Rebasing allows us to rewrite the commits on a branch, deleting commits we dont want, or merging com-
mits that should really be done. You can read more about interactive rebasing here.
Warning: Much care should be taken with rebasing. You should never rebase commits that are public
(that is, commits that have been copied outside your local repository). Rebasing public commits changes
branch history and results in the inability to merge with other repositories.
The following example shows an interactive rebase on a feature branch:
% git checkout my_feature
% git log
The git log shows the current commit on the branch is commit 123. We make some changes and commit
the result:
% git commit "fixing bug x" # results in commit 456
We realize we forgot to stage a change before committing, so we add the le and commit:
% git commit -m "oops, forgot to commit that file" # results in commit 678
Then we notice a small mistake, so we x and commit again:
% git commit -m "darn, made a typo" # results in commit #910
3.7. Development workow 15
At this point we have three commits when what we really want is one. So we rebase, specifying the revision
immediately prior to the rst commit:
% git rebase -i 123
This invokes an editor that allows indicating which commits should be combined. Git then squashes the
commits into an equivalent single commit. After this we can merge the cleaned-up feature branch into
master as usual:
% git merge my_feature
Again, be sure to read up on this feature before attempting to use it. And again, never rebase a public
commit.
Merge with squash
The git merge command takes an option --squash that performs the merge against the working area
but does not commit the result to the target branch. This squashes all the commits from the feature branch
into a single changeset that is staged and ready to be committed:
% git merge --squash my_feature
% git commit -m "implemented feature x"
3.8 More useful reading
The content in this section is not intended to be a comprehensive introduction to git. There are many things
not covered that are invaluable to day-to-day work with git. Some more useful info:
10 useful git commands
Git stashing
GeoTools git primer
CHAPTER 4
Quickstart
A step by step guide describing how to quickly get up and running with a GeoServer development envi-
ronment. This guide assumes that all the necessary Tools are installed.
Note: This guide is designed to get developers up and running as quick as possible. For a more compre-
hensive guide see the Maven Guide and the Eclipse Guide.
Check out source code
Build with Maven
Generate Eclipse project les with Maven
Import modules into Eclipse
Run GeoServer from Eclipse
Access GeoServer front page
4.1 Check out source code
Check out the source code from the git repository.:
git clone git://github.com/geoserver/geoserver.git geoserver
To list the available branches.:
% git branch
2.1.x
2.2.x
*
master
Choose master for the latest development.:
Or chose a stable branch for versions less likely to change often:
17
In this example we will pretend that your source code is in a directory called geoserver, but a more
descriptive name is recommended.
4.2 Build with Maven
Change directory to the root of the source tree and execute the maven build command:
cd geoserver/src
mvn clean install
A successful build will result in something like the following output:
[INFO]
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] ------------------------------------------------------------------------
[INFO] GeoServer ............................................. SUCCESS [10.271s]
[INFO] GeoServer Maven Plugins ............................... SUCCESS [0.865s]
[INFO] Configuration Deployment PlugIn ....................... SUCCESS [3.820s]
[INFO] GeoServer Maven Archetypes ............................ SUCCESS [0.054s]
[INFO] GeoServer WFS Output Format Archetype ................. SUCCESS [0.390s]
[INFO] Core Platform Module .................................. SUCCESS [5.270s]
[INFO] Data Module ........................................... SUCCESS [4.521s]
[INFO] Open Web Service Module ............................... SUCCESS [2.730s]
[INFO] Main Module ........................................... SUCCESS [10.077s]
[INFO] Web Coverage Service Module ........................... SUCCESS [3.785s]
[INFO] Web Coverage Service 1.1.1 Module ..................... SUCCESS [5.254s]
[INFO] Validation Module ..................................... SUCCESS [1.131s]
[INFO] Web Feature Service Module ............................ SUCCESS [6.695s]
[INFO] Web Feature Service Module ............................ SUCCESS [1.197s]
[INFO] Web Map Service Module ................................ SUCCESS [8.519s]
[INFO] Geoserver REST Support Code ........................... SUCCESS [3.366s]
[INFO] GeoWebCache (GWC) Module .............................. SUCCESS [0.255s]
[INFO] Web Application Module ................................ SUCCESS [27.386s]
[INFO] Community Space ....................................... SUCCESS [0.312s]
[INFO] GeoServer Extensions .................................. SUCCESS [0.071s]
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
4.3 Generate Eclipse project les with Maven
Generate the eclipse .project and .classpath les:
mvn eclipse:eclipse
4.4 Import modules into Eclipse
1. Run the Eclipse IDE
2. Open the Eclipse Preferences
18 Chapter 4. Quickstart
3. Navigate to Java, Build Path, Classpath Variables and click New...
4. Create a classpath variable named M2_REPO and set the value to the location of the local Maven
repository, and click Ok
5. Click Ok to apply the new Eclipse preferences
6. Right-click in the Package Explorer and click Import...
4.4. Import modules into Eclipse 19
7. Select Existing Projects into Workspace and click Next
8. Navigate to the geoserver/src directory
9. Ensure all modules are selected and click Finish
4.4. Import modules into Eclipse 21
4.5 Run GeoServer from Eclipse
1. From the Package Explorer select the web-app module
2. Navigate to the org.geoserver.web package
3. Right-click the Start class and navigate to Run as, Java Application
4. After running the rst time you can return to the Run Configurations dialog to ne tune your
launch environment (including setting a GEOSERVER_DATA_DIRECTORY).
5. If you are working on GeoServer 1.7.x or earlier Start class is located in the web module in the
package org.vfny.geoserver.jetty.
4.6 Access GeoServer front page
After a few seconds, GeoServer should be accessible at: http://localhost:8080/geoserver
The default admin password is geoserver.
4.6. Access GeoServer front page 23
CHAPTER 5
Maven Guide
A reference for building GeoServer with Maven.
5.1 Installing Maven
See Tools.
5.2 Running Maven
Maven provides a wide range of commands used to do everything from compiling a module to generating
test coverage reports. Most maven commands can be run from the root the source tree, or from a particular
module.
Note: When attempting to run a maven command from the root of the source tree remember to
change directory from the root the checkout into the src directory.
When running a command from the root of the source tree, or from a directory that contains other modules
the command will be run for all modules. When running the command from a single module, it is run only
for that module.
5.3 Building
The most commonly maven command used with GeoServer is the install command:
mvn clean install
While the clean command is not necessary, it is recommented. Running this command does the following:
compiles source code
runs unit tests
installs artifacts into the local maven repository
25
5.4 Skipping tests
Often it is useful to skip unit tests when performing a build. Adding the ag -DskipTests to the build
command will only compile unit tests, but not run them:
mvn -DskipTests clean install
5.5 Building ofine
Maven automatically downloads dependencies declared by modules being built. In the case of SNAPSHOT
dependencies, Maven downloads updates each time it performs the rst build of the day.
GeoServer depends on SNAPSHOT versions of the GeoTools library. The automatic download can result
in lengthy build time while Maven downloads updated GeoTools modules. If GeoTools was built locally,
these downloads are unecessary.
Also, if GeoTools is being modied locally, then the local versions rather than SNAPSHOT versions of
modules should be used.
This can be remedied by running maven in ofine mode:
mvn -o clean install
In ofine mode Maven will not download external dependencies, and will not update SNAPSHOT depen-
dencies.
5.6 Building extensions
By default, extensions are not included in the build. They are added to the build explicitly via proles. For
example the following command adds the restconfig extension to the build:
mvn clean install -P restconfig
Multiple extensions can be enabled simultaneously:
mvn clean install -P restconfig,oracle
A special prole named allExtensions enables all extensions:
mvn clean install -P allExtensions
5.7 Recover Build
After xing a test failure; you can resume from a specic point in the build:
mvn install -rf extension/wps
Recover from a 301 Redirect
A long standing bug in Maven from 2.0.10 handling of 301 errors when a artifact has been moved.
The work around is to run Maven with the option:
26 Chapter 5. Maven Guide
mvn install -Dmaven.wagon.provider.http=httpclient
This is not a common issue.
5.8 Proles
Additional proles are dened in the pom.xml les providing optional build steps. Proles are directly
enabled with the -P ag, others are automatically activated based on platform used or a -D property being
dened.
To build the release module as part of your build:
-Drelease
To include remote tests:
-PremoteOwsTests
Proles are also used manage optional extensions community plugins:
-Pproxy
-Poracle
-Pupload
-Pwps
Additional proles are dened in the pom.xml les providing optional build steps. Proles are directly
enabled with the -P ag, others are automatically activated based on platform used or a -D property being
dened.
To build javadocs with UML graph:
-Duml
To build the release module as part of your build:
-Drelease
To include the legacy moduled:
-Plegacy
To include remote tests:
-PremoteOwsTests
Proles are also used manage several of the optional community plugins:
-Pupload
-Pwps
-Pproxy
5.9 Eclipse
The maven eclipse plugin is used to generate eclipse projects for a set of modules:
mvn eclipse:eclipse
5.8. Proles 27
After which the modules can be imported into an eclipse workspace.
A useful feature of the plugin is the ability to download associated source code for third party dependen-
cies. This is done with the downloadSources ag:
mvn -DdownloadSources eclipse:eclipse
Warning: The rst time you enable the downloadSources ag the build will take a long time as it will
attempt to download the sources for every single library GeoServer depends on.
5.10 Building the web module
When the web module is installed, it does so with a particular conguration built in. By default this is the
minimal conguration. However this can be customized to build in any conguration via the configId
and configDirectory ags. For example:
mvn clean install -DconfigId=release -DconfigDirectory=/home/jdeolive/geoserver_1.7.x/data
The above command builds the web module against the release conguration that is shipped
with GeoServer. The configId is the name of the conguration directory to include, and
the configDirectory is the parent directory of the conguration directory to include. The
configDirectory can either be specied as an absolute path like in the above example, or it can be
specied relative to the web module itself:
mvn clean install -DconfigId=release -DconfigDirectory=../../../data
The above command does the same as the rst, however references the congDirectory relative to the web
module. This path, ../../../data, can be used if the GeoServer checkout has the standard layout.
5.11 Running the web module with Jetty
The maven jetty plugin can be used to run modules which are web based in an embedded Jetty container:
cd geoserver_2.0.x/src/web/app
mvn jetty:run
Note: This command must be run from the web/app module, it will fail if run from elsewhere.
The above command will run GeoServer with the built in data directory. To specify a different data directory
the GEOSERVER_DATA_DIR ag is used:
mvn -DGEOSERVER_DATA_DIR=/path/to/datadir jetty:run
5.12 Installing the Oracle module
To congure GeoServer to include the Oracle datastore extension module, do the following:
Obtain the appropriate Oracle JDBC driver (possibly by downloading from Oracle). Install it in the Maven
repository using the command:
mvn install:install-file -Dfile=ojdbc14.jar -DgroupId=com.oracle -DartifactId=ojdbc14 -Dversion=10.2.0.3.0 -Dpackaging=jar -DgeneratePom=true
Congure the Eclipse project using:
mvn -o -P oracle,oracleDriver eclipse:eclipse
(The oracle prole includes the GeoServer Oracle module, while the oracleDriver prole includes the
proprietary Oracle JDBC driver. These are separate because there are situations where the Oracle JDBC JAR
should not be included with the build.)
Finally, in Eclipse import the oracle module project. Refresh the web-app project to congure the depen-
dency on the oracle project.
When GeoServer is run, Oracle should be provided as a Vector Data Source on the New Data source page
5.12. Installing the Oracle module 29
CHAPTER 6
Eclipse Guide
A reference for developing GeoServer with Eclipse.
Importing modules
Running and debugging
Setting the data directory
Changing the default port for Jetty
Conguring JNDI resources in Jetty
Starting Jetty with an open SSL port
Eclipse preferences
Code formatting
Code templates
Text editors
Compiler
6.1 Importing modules
See the Eclipse section of the Maven Guide.
6.2 Running and debugging
Run or debug the class org.geoserver.web.Start in the web-app module. The steps to do so are
detailed in the Quickstart.
6.2.1 Setting the data directory
If unset, GeoServer will default to the minimal directory inside of the web-app module for its data direc-
tory. To change this:
1. Open Debug Configurations... from the Eclipse menu
31
2. Select the Start conguration, select the Arguments panel and specify the
-DGEOSERVER_DATA_DIR parameter, setting it to the absolute path of the data directory
Note: If you have checked out with git svn you may wish to use a symbolic in web/app correctly reference
minimal
6.2.2 Changing the default port for Jetty
If unset, Jetty will default to port 8080. To change this:
1. Open the Arguments panel of the Start conguration as described in the above section
32 Chapter 6. Eclipse Guide
2. Specify the -Djetty.port parameter, setting it to the desired port
6.2.3 Conguring JNDI resources in Jetty
JNDI resources such as data sources can be congured by supplying a Jetty server conguration le named
in the system property jetty.config.file, specied as a VM argument in the Arguments panel of
the launch conguration for Start. The path to the conguration le is relative to the root of the web-app
module, in which the launch conguration runs. For example:
-Djetty.config.file=../../../../../settings/jetty.xml
The following Jetty server conguration le congures a JNDI data source jdbc/demo that is a connection
pool for an Oracle database:
<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN" "http://jetty.mortbay.org/configure.dtd">
<Configure class="org.mortbay.jetty.Server">
<New class="org.mortbay.jetty.plus.naming.Resource">
<Arg>jdbc/demo</Arg>
<Arg>
<New class="org.apache.commons.dbcp.BasicDataSource">
<Set name="driverClassName">oracle.jdbc.driver.OracleDriver</Set>
<Set name="url">jdbc:oracle:thin:@oracle.example.com:1521:demodb</Set>
<Set name="username">claudius</Set>
<Set name="password">s3cr3t</Set>
<Set name="maxActive">20</Set>
<Set name="maxIdle">10</Set>
<Set name="minIdle">0</Set>
<Set name="maxWait">10000</Set>
<Set name="minEvictableIdleTimeMillis">300000</Set>
<Set name="timeBetweenEvictionRunsMillis">300000</Set>
<Set name="numTestsPerEvictionRun">20</Set>
6.2. Running and debugging 33
<Set name="poolPreparedStatements">true</Set>
<Set name="maxOpenPreparedStatements">100</Set>
<Set name="testOnBorrow">true</Set>
<Set name="validationQuery">SELECT SYSDATE FROM DUAL</Set>
</New>
</Arg>
</New>
</Configure>
Jetty does not mandate a reference-ref in GeoServer WEB-INF/web.xml, so there is no need to modify
that le. No Jetty-specic information is required inside the GeoServer web-app module or data directory,
so JNDI resources can be tested under Jetty for later deployment under Tomcat. See also the tutorial Setting
up a JNDI connection pool with Tomcat in the GeoServer User Manual.
6.2.4 Starting Jetty with an open SSL port
The SSL port used 8843.
1. Open the Arguments panel of the Start conguration.
2. Specify the -Dssl.hostname parameter, setting it to the full qualied host name of the box running
Jetty.
On rst time startup, a key store is created in <home directory>/.geoserver/keystore.jks. The
password is changeit and the key store contains a self signed certicate for the host name passed in the
ssl.hostname parameter.
Test the SSL connection by opening a browser and entering https://ux-desktop03.mc-
home.local:8843/geoserver. The browser should complain about the self singed certicate which
does not hurt for test and development setups.
6.3 Eclipse preferences
6.3.1 Code formatting
1. Download http://svn.osgeo.org/geotools/trunk/build/eclipse/formatter.xml
2. Navigate to Java, Code Style, Formatter and click Import...
3. Select the formatter.xml le downloaded in step 1
4. Click Apply
6.3. Eclipse preferences 35
6.3.2 Code templates
1. Download http://svn.osgeo.org/geotools/trunk/build/eclipse/codetemplates.xml
2. Navigate to Java, Code Style, Code Templates and click Import...
3. Select the codetemplates.xml le downloaded in step 1
4. Click Apply
6.3.3 Text editors
1. Navigate to General, Editors, Text Editors
2. Check Insert spaces for tabs
3. Check Show print margin and set Print margin column to 100
4. Check Show line numbers
5. Check Show whitespace characters (optional)
Note: Showing whitespace characters can help insure that unecessary whitespace is not unintention-
aly comitted.
6.3. Eclipse preferences 37
6. Click Apply
6.3.4 Compiler
1. Navigate to Java, Compiler, Building
2. Expand Output folder and add .svn/ to the list of Filtered resources
3. Click Apply
CHAPTER 7
Findbugs Guide
Findbugs is a static analysis tool for Java bytecode. It operates by searching compiled Java code for common
programming errors such as dereferencing null pointers. In GeoServer we use Findbugs to augment the
unit test suite.
We publish the analysis results as part of GeoServers nightly builds. You can view them online at
http://gridlock.opengeo.org:8080/hudson/job/geoserver-trunk-ndbugs/ . Click the FindBugs Warnings
link in the left sidebar to view the bug report for the latest build.
See Also:
More information on Findbugs is available at http://ndbugs.sourceforge.net/ .
7.1 Running the Findbugs Report Locally with Maven
Findbugs is integrated with Maven, enabling automatic running of Findbugs checks locally.
7.1.1 Generating a report
The basic form for running a report is to use the findbugs:findbugs goal for Maven. This requires
bytecode to already be generated, so the full command is:
$ mvn compile findbugs:findbugs
This command respects proles for Community Modules. For example, you can generate a Findbugs report
for the monitoring community module using:
$ mvn compile findbugs:findbugs -Pmonitoring
7.1.2 Filters
Findbugs patterns vary in their reliability - sometimes a questionable construct is the right solution to a
problem. In order to avoid distractions from bug patterns that dont always indicate real problems, the
GeoServer project maintains an bug lter that has been vetted by the project. When using Maven, you can
enable this lter with:
39
$ mvn compile findbugs:findbugs -P findbugs \
-Dfindbugs.excludeFilterFile=/path/to/src/maven/findbugs/findbugs-excludeNone.xml
Note: Please note that its necessary to specify the full path to the exclude lter.
7.1.3 Browsing Bugs
The findbugs:findbugs goal generates a report of Findbugs violations in XML format, stored in
target/findbugsXml.xml in each project. For manual inspection, Findbugs provides a graphical
browser with descriptions of the potential problems found, and sorting/ltering/linking to sources for
each detected violation. To launch it, switch into a module rst and use the findbugs:gui Maven goal:
$ cd main/
$ mvn findbugs:gui
Warning: If you do not change directories Maven will traverse all modules and launch the ndbugs
browser for each in sequence!! If you do so, you can cancel with Control+C in the terminal window
where Maven is running.
Figure 7.1: The Findbugs bug browser running against the GeoServer Main module
See Also:
40 Chapter 7. Findbugs Guide
More information on the Findbugs Maven plugin is available at http://mojo.codehaus.org/ndbugs-
maven-plugin/ .
7.2 Using the Findbugs Eclipse Plugin
Findbugs integrates with Eclipse Guide as well. This allows you to run bug reports and browse their results
in the same tool you use to edit code.
For more information on installing and using the Findbugs plugin for Eclipse, see
http://ndbugs.sourceforge.net/manual/eclipse.html .
Figure 7.2: The Findbugs Eclipse perspective browsing the Geoserver Main module
7.3 Fixing Bugs from Findbugs
We are always looking to eliminate bugs from GeoServer! If you want to help out, one way is to remove
one of the Findbugs rules from the exclusion lter, then re-run the analysis locally. Once you x all the
bugs, submit a patch to GeoServer for review. A core developer will review the changes, and well be able
to make our lax Findbugs lter that much stricter.
7.2. Using the Findbugs Eclipse Plugin 41
42 Chapter 7. Findbugs Guide
CHAPTER 8
Programming Guide
8.1 OWS Services
This section provides an overview of how RESTful services work in GeoServer.
8.1.1 OWS Services Overview
TBA
8.1.2 Implementing a simple OWS service
This section explains How to Create a Simple GeoServer OWS service for Geoserver using the following
scenario. The service should supply a capabilities document which advertises a single operation called
sayHello. The result of a sayHello operation is the simple string Hello World.
Contents
Implementing a simple OWS service
Setup
Creating the Plug-in
Trying it Out
*
Bundling with Web Module
*
Running from Source
Setup
The rst step in creating our plug-in is setting up a maven project for it. The project will be called hello.
1. Create a new directory called hello anywhere on your le system.
2. Add a maven pom called pom.xml to the hello directory:
43
<?xml version="1.0" encoding="ISO-8859-1"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>

<parent>
<groupId>org.geoserver</groupId>
<artifactId>community</artifactId>
<version>2.2.0</version>
</parent>
<artifactId>hello</artifactId>
<packaging>jar</packaging>
<version>1.0</version>
<name>Hello World Service Module</name>

<dependencies>
<dependency>
<artifactId>main</artifactId>
</dependency>
</dependencies>
<repositories>
<repository>
<id>opengeo</id>
<name>opengeo</name>
<url>http://repo.opengeo.org</url>
</repository>
</repositories>
</project>
1. Create a java source directory, src/main/java under the hello directory:
hello/
+ pom.xml
+ src/
+ main/
+ java/
Creating the Plug-in
A plug-in is a collection of extensions realized as spring beans. In this example the extension point of
interest is a HelloWorld POJO (Plain Old Java Object).
1. Create a class called HelloWorld:
import java.io.IOException;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
44 Chapter 8. Programming Guide
public class HelloWorld {
public HelloWorld() {
// Do nothing
}
public void sayHello(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
response.getOutputStream().write( "Hello World".getBytes() );
}
}
The service is relatively simple. It provides a method sayHello(..) which takes a HttpServletRequest,
and a HttpServletResponse. The parameter list for this function is automatically discovered by the
org.geoserver.ows.Dispatcher.
1. Create an applicationContext.xml declaring the above class as a bean.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">
<beans>

<bean id="helloService" class="HelloWorld">
</bean>

<bean id="helloService-1.0.0" class="org.geoserver.platform.Service">

<constructor-arg index="0" value="hello"/>

<constructor-arg index="1" ref="helloService"/>

<constructor-arg index="2" value="1.0.0"/>

<constructor-arg index="3">
<list>
<value>sayHello</value>
</list>
</constructor-arg>
</bean>
</beans>
At this point the hello project should look like the following:
hello/
+ pom.xml
+ src/
+ main/
+ java/
+ HelloWorld.java
+ applicationContext.xml
8.1. OWS Services 45
Trying it Out
1. Install the hello module:
[hello]% mvn install
[hello]% mvn install
[INFO] Scanning for projects...
[INFO] ----------------------------------------------------------------------------
[INFO] Building Hello World Service Module
[INFO] task-segment: [install]
[INFO] ----------------------------------------------------------------------------
[INFO] [resources:resources]
[INFO] Using default encoding to copy filtered resources.
[INFO] [compiler:compile]
[INFO] Compiling 1 source file to /home/ak/geoserver/community/hello/target/classes
[INFO] [resources:testResources]
[INFO] Using default encoding to copy filtered resources.
[INFO] [compiler:testCompile]
[INFO] No sources to compile
[INFO] [surefire:test]
[INFO] No tests to run.
[INFO] [jar:jar]
[INFO] Building jar: /home/ak/geoserver/community/hello/target/hello-1.0.jar
[INFO] [jar:test-jar {execution: default}]
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: /home/ak/geoserver/community/hello/target/hello-1.0-tests.jar
[INFO] [install:install]
[INFO] Installing /home/ak/geoserver/community/hello/target/hello-1.0.jar to /home/ak/.m2/repository/org/geoserver/hello/1.0/hello-1.0.jar
[INFO] Installing /home/ak/geoserver/community/hello/target/hello-1.0-tests.jar to /home/ak/.m2/repository/org/geoserver/hello/1.0/hello-1.0-tests.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6 seconds
[INFO] Finished at: Fri Sep 21 14:52:31 EDT 2007
[INFO] Final Memory: 27M/178M
[INFO] -----------------------------------------------------------------------
1. Copy target/hello-1.0.jar into the WEB-INF/lib directory of your GeoServer install
2. Restart GeoServer
3. Visit:
http://<host>/geoserver/ows?request=sayHello&service=hello&version=1.0.0
request the method we dened in our service
service the name we passed to the Service descriptor in the applicationContext.xml
version the version we passed to the Service descriptor in the applicationContext.xml
Note: A common pitfall is to bundle an extension without the applicationContext.xml le. If you
receive the error message No service: ( hello ) this is potentially the case. To ensure the le is present
inspect the contents of the hello jar present in the target directory of the hello module.
Bundling with Web Module
An alternative to plugging into an existing installation is to build a complete GeoServer war that includes
the custom hello plugin. To acheive this a new dependency is declared from the core web/app module on
the new plugin project. This requires building GeoServer from sources.
1. Build GeoServer from sources as described here.
2. Install the hello module as above.
3. Edit web/app/pom.xml and add the following dependency:
<dependency>
</dependency>
4. Install the web/app module
[web/app] mvn install
A GeoServer war including the hello extension should now be present in the target directory.
Note: To verify the plugin was bundled properly unpack geoserver.war and inspect the contents of the
WEB-INF/lib directory and ensure the hello jar is present.
8.1. OWS Services 47
Running from Source
During development the most convenient way to work with the extension is to run it directly from sources.
1. Setup GeoServer in eclipse as described here.
2. Move the hello module into the GeoServer source tree under the community root module.
3. Edit the community/pom.xml and add a new prole:
<profile>
<id>hello</id>
<modules>
<module>hello</module>
</modules>
</profile>
4. If not already done, edit web/app/pom.xml and add the following dependency:
<dependency>
</dependency>
5. From the root of the GeoServer source tree run the following maven command:
6. In eclipse import the new hello module and refresh all modules.
7. In the web-app module run the Start.java main class to start GeoServer.
Note: Ensure that the web-app module in eclipse depends on the newly imported hello module. This
can be done by inspeceting the web-app module properties and ensuring the hello module is a projcet
dependency.
8.2 REST Services
This section provides an overview of how RESTful services work in GeoServer.
8.2.1 Overview
GeoServer uses a library known as Restlet for all REST related functionality. Restlet is a lightweight rest
framework written in Java that integrates nicely with existing servlet based applications.
REST dispatching
In GeoServer, all requests under the path /rest are considered a call to a restful service. Every call of this
nature is handled by a rest dispatcher. The job of the dispatcher is to route the request to the appropriate end
point. This end point is known as a restlet.
Restlets are loaded from the spring context, and therefore are pluggable.
Restlets
A restlet is the generic entity which handles calls routed by the dispatcher, and corresponds to the class
org.restlet.Restlet. One can extend this class directly to implement a service endpoint. Alternatively
one can extend a subclass for a specialized purpose. Namely a nder, which is described in the next section.
Finders and resources
Restful services are often implemented around the concept of resources. A nder is a special kind of restlet
whose job is to nd the correct resource for a particular request. The resource then serves as the nal end
point and handles the request. The appropriate classes from the restlet library are org.restlet.Finder
and org.restlet.resource.Resource.
Representations
Arepresentation, commonly referred to as a format, is the state of a particular state or encoding of a resource.
For instance, when a request for a particular resource comes in, a representation of that resource is returned
to the client.
8.2.2 Implementing a RESTful Service
This section describes how to implement a restful service in GeoServer, using a Hello World example.
The service will be extremely simple and will return the text Hello World from a GET request.
8.2. REST Services 49
Prerequisites
Before being able to proceed, GeoServer must be built on the local system. See the Source Code and Quickstart
sections for details.
Create a new module
1. Create a new module named hello_rest somewhere on the le system.
2. Add the following pom.xml to the root of the new module:
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd ">
<artifactId>hello_rest</artifactId>
<version>1.0-SNAPSHOT</version>
<name>hello_rest</name>
<dependencies>
<dependency>
<artifactId>rest</artifactId>
</dependency>
<dependency>
<classifier>tests</classifier>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.mockrunner</groupId>
<artifactId>mockrunner</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.5</source>
<target>1.5</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
1. Create the directory src/main/java under the root of the new module:
[hello_rest]% mkdir -p src/main/java
Create the resource class
1. The class org.geoserver.rest.AbstractResource is a convenient base class available
when creating new resources. Create a new class called HelloResource in the package
org.geoserver.hellorest, which extends from AbstractResource.
package org.geoserver.hellorest;
import java.util.List;
import org.geoserver.rest.AbstractResource;
import org.geoserver.rest.format.DataFormat;
import org.restlet.data.Request;
import org.restlet.data.Response;
public class HelloResource extends AbstractResource {
@Override
protected List<DataFormat> createSupportedFormats(Request request, Response response) {
return null;
}
}
2. The rst method to implement is createSupportedFormats(). The purpose of this method is to
create mapping from an extension, to a particular format. For now the goal will be to return the text
Hello World when a .txt extension is requested by the client.
import java.util.ArrayList;
import org.geoserver.rest.format.StringFormat;
...
@Override
protected List<DataFormat> createSupportedFormats(Request request, Response response) {
List<DataFormat> formats = new ArrayList();
formats.add(new StringFormat( MediaType.TEXT_PLAIN ));
return formats;
}
3. The next step is to override the handleGet() method. This method is called when a GET request is
made for the resource.
@Override
public void handleGet() {
//get the appropriate format
DataFormat format = getFormatGet();
//transform the string "Hello World" to the appropriate response
getResponse().setEntity(format.toRepresentation("Hello World"));
}
The above makes use of the getFormatGet() method, whose purpose is to determine the extension
being requested by the client, and look up the appropriate format for it. In this case when the client
requests the .txt extension, the StringFormat setup in the previous step will be found.
Create the application context
1. The next step is to create an application context that tells GeoServer about the resource created in the
previous section. Create the directory src/main/resources under the root of the hello_rest
module:
[hello_rest]% mkdir src/main/resources
2. Add the following applicationContext.xml le to the src/main/resources directory under
the root of the hello_rest module.
<beans>
<bean id="hello" class="org.geoserver.hellorest.HelloResource"/>
<bean id="helloMapping" class="org.geoserver.rest.RESTMapping">
<property name="routes">
<map>
<entry>
<key><value>/hello.{format}</value></key>
<value>hello</value>
</entry>
</map>
</property>
</bean>
</beans>
There are two things to note above. The rst is the hello bean which is an instance of the
HelloResource class created in the previous section. The second is the helloMapping bean, which
denes a template for the uri in which the resource will be accessed. The above mapping species
that the resource will be located at /rest/hello.{format} where format is the representation
being requested by the client. As implemented hello.txt is the only supported representation.
Test
1. Create the directory /src/test/java under the root of the hello_rest module:
[hello_rest]% mkdir -p src/test/java
2. Create a new test class called HelloResourceTest in the package org.geoserver.hellorest,
which extends from org.geoserver.test.GeoServerTestSupport:
import org.geoserver.test.GeoServerTestSupport;
public class HelloResourceTest extends GeoServerTestSupport {
public void test() throws Exception {
}
}
3. Add a statement which makes a GET request for /rest/hello.txt and asserts it is equal to the
string Hello World:
public void test() throws Exception {
assertEquals( "Hello World", getAsString("/rest/hello.txt"));
}
4. Build and test the hello_test module:
[hello_rest]% mvn install
8.2.3 Implementing a RESTful Service with Maps
The previous section showed how to implement a very simple restful service. This section will show how
to make use of some existing base classes in order to save time when supporting additional formats for
representing resources.
The class used is org.geoserver.rest.MapResource. The idea with MapResource is that a resource
is backed by a data structure contained by a java.util.Map. With this map, the MapResource class can
automatically create representations of the resource in either XML or JSON.
Prerequisites
This section builds off the example in the previous section Implementing a RESTful Service.
Create a new resource class
1. Create a new class called HelloMapResource in the package org.geoserver.hellorest, which
extends from MapResource:
import java.util.Map;
import org.geoserver.rest.MapResource;
public class HelloMapResource extends MapResource {
@Override
public Map getMap() throws Exception {
return null;
}
}
2. The rst method to implement is getMap(). The purpose of this method is to create a map based
data structure which represents the resource. For now a simple map will be created with a single key
message, containing the Hello World string:
@Override
public Map getMap() throws Exception {
HashMap map = new HashMap();
map.put( "message", "Hello World");
return map;
}
Update the application context
1. The next step is to update an application context and tell GeoServer about the new resource created in
the previous section. Update the applicationContext.xml le so that it looks like the following:
<beans>
<bean id="helloMap" class="org.geoserver.hellorest.HelloMapResource"/>
<map>
<entry>

<value>helloMap</value>
</entry>
</map>
</property>
</bean>
</beans>
There are two things to note above. The rst is the addition of the helloMap bean. The second is a
change to the helloMapping bean, which now maps to the helloMap bean, rather than the hello
bean.
Test
1. Create a new test class called HelloMapResourceTest in the package
org.geoserver.hellorest, which extends fromorg.geoserver.test.GeoServerTestSupport:
public class HelloMapResourceTest extends GeoServerTestSupport {
}
2. Add a test named testGetAsXML() which makes a GET request for /rest/hello.xml:
...
import org.w3c.dom.Document;
import org.w3c.dom.Node;
...
public void testGetAsXML() throws Exception {
//make the request, parsing the result as a dom
Document dom = getAsDOM( "/rest/hello.xml" );
//print out the result
print(dom);
//make assertions
Node message = getFirstElementByTagName( dom, "message");
assertNotNull(message);
assertEquals( "Hello World", message.getFirstChild().getNodeValue() );
}
3. Add a second test named testGetAsJSON() which makes a GET request for /rest/hello.json:
...
import net.sf.json.JSON;
import net.sf.json.JSONObject;
...
public void testGetAsJSON() throws Exception {
//make the request, parsing the result into a json object
JSON json = getAsJSON( "/rest/hello.json");
print(json);
//make assertions
assertTrue( json instanceof JSONObject );
assertEquals( "Hello World", ((JSONObject)json).get( "message" ) );
}
[hello_rest]% mvn clean install -Dtest=HelloMapResourceTest
8.2.4 Implementing a RESTful Service with Reection
The previous section showed how to save some effort when implementing a RESTful service by taking
advantage of the MapResource base class. This section will show how to make use of a different base
class, but for a similar purpose.
The class used is org.geoserver.rest.ReflectiveResource. The idea with
ReflectiveResource is that a resource is backed by an arbitrary object. The ReflectiveResource
class uses reection to automatically create representations of the resource as XML or JSON.
Prerequisites
This section builds off the example in the previous section Implementing a RESTful Service with Maps.
Create a new java bean
1. The use of ReflectiveResource requires we have an underlying object to to work with. Create a
class named Hello in the package org.geoserver.hellorest:
public class Hello {
String message;
public Hello( String message ) {
this.message = message;
}
public String getMessage() {
return message;
}
}
Create a new resource class
1. Create a new class called HelloReflectiveResource in the package
org.geoserver.hellorest, which extends from ReflectiveResource:
import org.geoserver.rest.ReflectiveResource;
public class HelloReflectiveResource extends ReflectiveResource {
@Override
protected Object handleObjectGet() throws Exception {
return null;
}
}
2. The rst method to implement is handleObjectGet(). The purpose of this method is to return the
underlying object representing the resource. In this case, an instance of the Hello class created in the
previous step.
@Override
protected Object handleObjectGet() throws Exception {
return new Hello( "Hello World" );
}
Update the application context
1. The next step is to update an application context and tell GeoServer about the new resource created in
the previous section. Update the applicationContext.xml le so that it looks like the following:
<beans>
<bean id="helloMap" class="org.geoserver.hellorest.HelloMapResource"/>
<bean id="helloReflective" class="org.geoserver.hellorest.HelloReflectiveResource"/>
<map>
<entry>


<value>helloReflective</value>
</entry>
</map>
</property>
</bean>
</beans>
There are two things to note above. The rst is the addition of the helloReflective bean. The
second is a change to the helloMapping bean, which now maps to the helloReflective bean.
Test
1. Create a new test class called HelloReflectiveResourceTest
in the package org.geoserver.hellorest, which extends from
org.geoserver.test.GeoServerTestSupport:
public class HelloReflectiveResourceTest extends GeoServerTestSupport {
}
2. Add a test named testGetAsXML() which makes a GET request for /rest/hello.xml:
...
import org.w3c.dom.Document;
import org.w3c.dom.Node;
...
public void testGetAsXML() throws Exception {
//make the request, parsing the result as a dom
Document dom = getAsDOM( "/rest/hello.xml" );
print(dom);
//make assertions
Node message = getFirstElementByTagName( dom, "message");
assertNotNull(message);
assertEquals( "Hello World", message.getFirstChild().getNodeValue() );
}
3. Add a second test named testGetAsJSON() which makes a GET request for /rest/hello.json:
...
import net.sf.json.JSON;
import net.sf.json.JSONObject;
...
public void testGetAsJSON() throws Exception {
//make the request, parsing the result into a json object
JSON json = getAsJSON( "/rest/hello.json");
print(json);
//make assertions
assertTrue( json instanceof JSONObject );
JSONObject hello = ((JSONObject) json).getJSONObject( "org.geoserver.hellorest.Hello" );
assertEquals( "Hello World", hello.get( "message" ) );
}
[hello_rest]% mvn clean install -Dtest=HelloMapReflectiveResourceTest
8.3 Web User Interface
8.3.1 Overview
GeoServer uses a web application framework known as Wicket for its user interface. Wicket differs from
most Java web frameworks in that it is component based rather than JSP template based. This makes
Wicket a more natural web framework for many Java programmers who are more familiar with Swing
programming than web programming.
Plug-ins
Because of its component based nature Wicket components can be loaded from the classpath. Which means
that web applications can be built in a modular fashion, rather than in a monolithic fashion.
GeoServer takes this concept one step further to provide a pluggable user interface, in which Wicket com-
ponents can be plugged in via Spring and the regular GeoServer plug-in mechanism.
Each component that is plugged in is described by a component descriptor. A component descriptor is an
instance of the org.geoserver.web.ComponentInfo class:
public abstract class ComponentInfo<C extends Component> implements Serializable {
/
**
*
the id of the component
*
/
String id;
/
**
*
the title of the component
*
/
String title;
/
**
*
The description of the component
*
/
String description;
/
**
*
the class of the component
*
/
Class<C> componentClass;
}
A ComponentInfo instance contains meta information about the component being plugged in such as its
title and description, as well as the class which implements the component.
Each subclass of ComponentInfo represents a specic extension point. For instance the class
org.geoserver.web.MenuPageInfo represents the extension point for main pages, ie pages that are
linked to from the main menu of the application.
8.3.2 Implementing a Wicket UI Extension
This section describes howto implement an extension to the GeoServer Wicket user interface. The extension
will be extremely simple and will be a basic page that is linked from the main menu and displays the
message Hello World.
Prerequisites
Before being able to proceed, GeoServer must be built on the local system. See the Source Code and Quickstart
sections for details.
Create a new module
1. Create a new module named hello_web somewhere on the le system
2. Add the following pom.xml to the root of the new module:
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd ">
<parent>
<artifactId>web</artifactId>
</parent>
<artifactId>hello_web</artifactId>
<name>hello_web</name>
<dependencies>
<dependency>
<groupId>org.geoserver.web</groupId>
<artifactId>web-core</artifactId>
8.3. Web User Interface 59
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<configuration>
</configuration>
</plugin>
</plugins>
</build>
</project>
[hello_web]% mkdir -p src/main/java
Create the page class
1. The class org.geoserver.web.GeoServerBasePage is the base class for all pages in GeoServer.
Create a new class called HelloPage in the package org.geoserver.helloweb, which extends
from GeoServerBasePage:
package org.geoserver.helloweb;
import org.geoserver.web.GeoServerBasePage;
public class HelloPage extends GeoServerBasePage {
}
2. The rst task is to implement the constructor. In Wicket a page or component builds itself in its
constructor. This page is basic and will simply create a label which has the value Hello World!:
import org.apache.wicket.markup.html.basic.Label;
...
public HelloPage() {
add( new Label( "hellolabel", "Hello World!") );
}
In the above code, an instance of Label is created. The rst argument to its constructor is the compo-
nent id. In Wicket every component must have an id. In the next section this id will be used to bind
the component to its HTML presentation. The second argument to the Label constructor is the value
of the world, in this case the string Hello World!
Create the page presentation
1. With the page completed, the next step is to create the HTML presentation for the page. To do this
create a le named HelloPage.html in the same directory as the HelloPagejava class:
<html>
<body>
<wicket:extend>
<div wicket:id="hellolabel"></div>
</wicket:extend>
</body>
</html>
There are few things to note about the HTML. The rst is the use of the <wicket:extend> element.
This tells wicket that HelloPage is an extension of another page, in this case GeoServerBasePage,
and it should inherit presentation from that page.
The second thing to note is the attribute wicket:id on the <div> element. This is what binds the
<div> tag to the Label component created in the previous section. The value of wicket:id must
match the id given to the component, in this case hellolabel.
Create the i18n le
With Wicket (and any web application framework), any string that appears in the web application should
be interationalized. In GeoServer, this is performed by creating an internationalization (i18n) le named
GeoServerApplication.properties.
1. Create the directory src/main/resources under the root of the hello_web module:
[hello_web]% mkdir -p src/main/resources
2. Create the (i18n) le GeoServerApplication.properties in the src/main/resources direc-
tory:
HelloPage.page.title=Hello
HelloPage.page.description=A page to say hello
HelloPage.title=Hello Page Title
HelloPage.description=This is the description of the page
The rst two keys in the above i18n le declare the title of the page and the description of the page.
This will be the title of the link to the page and the tooltip for the page link. The next two keys are the
title and description that are displayed on the page itself.
Create the application context
1. The nal step is to create an application context which tells GeoServer about the page created in
the previous section. Add the following applicationContext.xml le to the src/main/java
directory, under the root of the hello_web module:
<beans>
<bean id="helloPage" class="org.geoserver.web.MenuPageInfo">
<property name="id" value="helloPage"/>
<property name="titleKey" value="HelloPage.page.title"/>
<property name="descriptionKey" value="HelloPage.page.description"/>
<property name="componentClass" value="org.geoserver.helloweb.HelloPage"/>
</bean>
</beans>
8.3. Web User Interface 61
The above bean declaration declares an instance of the MenuPageInfo class which is a descriptor for
pages linked from the main page of the GeoServer web application. The property titleKey is the
title of the page and it receives the value of the title i18n key created in the previous section. Similar
for the the descriptionKey property.
Test the extension
At this point, the hello_web module should look like the following:
hello_web/
pom.xml
src/main/resources
GeoServerApplication.properties
src/main/java
applicationContext.xml
org/geoserver/helloweb/
HelloPage.java
HelloPage.html
1. Build the hello_web module:
[hello_web]% mvn install
2. Copy the hello_web-1.0-SNAPSHOT.jar le from the hello_web/target directory into the
WEB-inf/lib directory of a GeoServer installation:
[hello_web]% cp target/hello-1.0-SNAPSHOT.jar /home/bob/geoserver-2.0/webapps/geoserver/WEB-INF/lib
3. Start or restart GeoServer
4. Navigate to http://localhost:8080/geoserver/web
Upon success a link titled Hello should appear in the menu on the left side of the main GeoServer page.
Following the link brings up the HelloPage
8.4 Wicket Development In GeoServer
This page explains the steps to follow in creating a page for the Wicket-based conguration interface in
GeoServer 2. For more information on Wicket, check out the project site at http://wicket.apache.org/.
8.4.1 Adding a Page
In Wicket, each page has one corresponding Java class. To add a page, you need to create a new class that
extends org.geoserver.web.GeoServerBasePage. You will also want to create a link somewhere that
brings the user to your page. (In general, Wicket pages do not have reliable URLs, so you must explicitly
create a link in an existing page and let Wicket generate the proper URL.) In the case where your class
does not require arguments to its constructor, you can insert a link using Spring. Creating links in Spring
requires that your page link text be internationalizable. Well discuss internationalization in more depth
later. The simplest possible Wicket extension for GeoServer involves 3 les. There is a Java class denition
(this would be in src/main/java/org/geoserver/web/example/MyPage.java).
package org.geoserver.web.example;
8.4. Wicket Development In GeoServer 63
public class MyPage extends GeoServerBasePage {
// We will fill in the rest later, for now the page can just be blank
}
There would also need to be a Spring application context document
(src/main/java/applicationContext.xml):
<bean class="org.geoserver.web.MenuPageInfo" id="myPageLink">

<property name="id" value="mypage"/>

<property name="titleKey" value="org.geoserver.web.example.MyPage.page.title"/>

<property name="descriptionKey" value="org.geoserver.web.example.MyPage.page.description"/>

<property name="componentClass" value="org.geoserver.web.example.MyPage"/>

<property name="icon" value="imgs/some-image.png"/>

<property name="category" ref="someCategory"/>

<property name="order" value="100"/>
</bean>
The third necessary le is the default dictionary for internationalized strings, at
src/main/resources/GeoServerApplication.properties:
org.geoserver.web.example.MyPage.page.title=My Example Page
org.geoserver.web.example.MyPage.page.description=An example page for developers trying to extend the GeoServer UI.
If you create a jar with these three les and add it to the GeoServer classpath, you should see the new link
in the left-hand menu.
8.4.2 Adding to a Page
At this point youve added a page to the UI, but its not very interesting. In Wicket, pages provide their
content with an HTML le stored with the same name as the Java code except for the extension. There
are a few details about these les that differ from standard HTML; for one thing, they must be valid XML
for Wickets parser to work. In addition, Wicket uses a few special elements to specify where the Java
code should hook into the HTML. The following are used quite extensively in the GeoServer administrative
console.
Wicket Element Purpose
<foo
wicket:id="bar"></foo>
A wicket:id attribute tells Wicket the name to be used for an
element when attaching Wicket Components
<wicket:child/> Requires no contents, but species that classes extending this
page can insert content at this position.
<wicket:extend></wicket:extend> Species that the enclosed content will be inserted into a parent
page at the point indicated by <wicket:child/>
<wicket:panel></wicket:panel>Similar to wicket:extend, but used in creating custom
components rather than extending pages.
<wicket:head></wicket:head> Indicates a section (like a CSS or JavaScript include) that should
be added to the header of pages that include this markup (can be
used for pages or panels).
<wicket:link></wicket:link> Encloses sections in which Wicket will rewrite links to pages, CSS
les, and other resources that it manages. (This lets you refer to
resources using paths relative to the Java source and not the
rendered HTML.)
<wicket:message
key="i18nKey"> Default
Text </wicket:message>
Tells Wicket to look up a string in the internationalization
database and replace the provided text if one is found.
Wicket provides quite a few components, of which several can be seen in the Wicket Component Reference.
In general, Wicket components require a Model object which handles the getting, setting, and conversion
to/fromString of the value associated with a component. For the purposes of this example, we will focus on
one of the simplest, the Label, which simply replaces the contents of the element it is bound to with a value
provided at runtime. Continuing the example from above, we can pass a String to the Labels constructor
and it is transparently converted to a Model:
package org.geoserver.web.example;
import org.apache.wicket.markup.html.basic.Label;
public class MyPage extends GeoServerBasePage{
public MyPage(){
add(new Label("label", "Hello World"));
}
}
The corresponding HTML source would live at src/main/java/org/geoserver/web/example/MyPage.html:
<html>
<head></head>
<body>
<wicket:extend>
Greetings, GeoServer User! My message for you is <span wicket:id="label"> thanks for using GeoServer </span>.
</wicket:extend>
</body>
</html>
Of course, there are much more complicated (and useful) things we can do with Wicket, but this example
demonstrates the most common usage; just adding some behavior to an HTML element.
8.4.3 Adding a Link Outside the Navigation Menu
Of course, we cant have everything in the sidebar menu; for one thing, it denes only a static set of links
while GeoServer is bound to contain lots of resources that vary from conguration to conguration. For
8.4. Wicket Development In GeoServer 65
another, some pages need to have arguments to their constructors. If you want to add a customlink to some
page, you can use a Wicket Link component and customize the onClick behavior to call the appropriate
constructor. (You can use setResponsePage in other methods that handle user input as well, such as on
form submits. Check the Wicket documentation for more information.) An example:
//...
import org.apache.wicket.markup.html.link.Link;
//...
add(new Link("link"){
public void onClick(){
setResponsePage(new MyPage());
}
});
The corresponding HTML would look like:
Follow this lovely <a href="#" wicket:id="link">link</a>.
8.4.4 Making it Internationalizable
In the GeoServer UI, we use a customized resource lookup utility within Wicket to allow any module to
provide resource strings. All you need to do is include your I18N (Internationalization) values in a Java
Properties le named GeoServerApplication.properties in the resources directory of your src
directory (ie, project/src/main/resources/GeoServerApplication.properties).
The <wicket:message> element makes it quite easy to make text internationalizable, but in the event
that you need to insert a value into a sentence at a position that changes dependent on the language, youll
need to use something more complicated.
In Wicket, I18N value strings can dene parameters which provide the ability to place dynamic values into
internationalized strings.
See Also:
http://wicket.apache.org/docs/wicket-1.3.2/wicket/apidocs/org/apache/wicket/model/StringResourceModel.html
for details.
8.4.5 Adding Resources
Often in HTML, you will need to include assets such as CSS les, JavaScript libraries, or images to include
in your page. Wicket allows you to specify URLs to these relative to your Java source le, using relative
paths enclosed in <wicket:link> tags. Wicket will rewrite these links at runtime to use the correct path.
However, such resources are not inherited from parent classes, so if you need to include a resource in
multiple packages you will need to extract the functionality that uses it to a new class that can be shared
between the two. See the XMLEditor component in the core module of GeoServers UI for an example of
a component that does this.
8.4.6 UI Design Guidelines
A brief listing of UI design guidelines for Wicket pages in GeoServer follows.
Forms In forms, group each eld as an item in an unordered list with a label and a form eld.
For radio buttons and checkboxes, the label should come after the eld; for all others the
label should precede the eld. For example:
<ul>
<li>
<label for="foo"><wicket:message key="foo"> Foo </wicket:message></label>
<input wicket:id="foo" type="text"></input>
</li>
</ul>
Avoid requiring special knowledge from the user. For example, where a list of values is re-
quired, provide a widget that allows manipulating the list one element at a time rather
than expecting a comma-separated list of values.
Custom Components We recommend creating a reusable Wicket component for any complex
values that might need to be edited by users, such as a bounding box or a list of free strings.
By extracting this into a component, it is much simpler to provide consistent, rich editing
for users.
8.5 Extension Points
8.6 WPS Services
This section provides an overview of how WPS services work in GeoServer.
8.6.1 WPS design guide
This guide serves as an introduction to the WPS module. As such, it does not contain:
a primer to the WPS protocol, that can be found in the WPS specication (the module implements the
WPS 1.0 specication).
it does not repeat again what can be already found in the classes javadocs
it does not explain how to implement a OWS service using the GeoServer OWS framework, that is
left to its dedicated guide.
In short, it provides a global vision of how the module ts togheter, leaving the details to other information
sources.
General architecture
Note: We really need to publish the Javadocs somewhere so that this document can link to them
The module is based on the usual GeoServer OWS framework application:
a set of KVP parsers and KVP readers to parse the HTTP GET requests, found in the
org.geoserver.wps.kvp package
a set of XML parsers to parse the HTTP POST requests, found int the org.geoserver.wps.xml
and org.geoserver.wps.xml.v1_0_0
a service object interface and implementations responding to the various WPS methods, in particu-
lar org.geoserver.wps.DefaultWebProcessingService, which in turn delegates most of the
work to the GetCapabilities, DescribeProcess and ExecuteProcess classes
8.5. Extension Points 67
a set of output transformers taking the results generated by DefaultWebProcessingService
and turning them into the appropriate response (usually, XML). You can nd some of those in the
org.geoserver.wps.response package, whilst some others are generic ones that have been
parametrized and declared in the Spring context (see the applicationContext.xml le).
The module uses extensively the following GeoTools modules:
net.opengis.wps which contains EMF models of the various elements and types described in the
WPS schemas. Those objects are usually what ows between the KVP parsers, XML decoders, the
service implementation, and the output transformers
gt-xsd-wps and gt-xsd, used for all XML encoding and decoding needs
gt-process that provides the concept of a process, with the ability to self describe its inputs and
outputs, and of course execute and produce results
The processes
The module relies on gt-process SPI based plugin mechanism to lookup and use the processes available
in the classpath. Implementing a new process boils down to:
creating a ProcessFactory implementation
creating one or more Process implementations
registering the ProcessFactory in SPI by adding the factory class name in the
META-INF/services/org.geotools.process.ProcessFactory le
The WPS module shows an example of the above by bridging the Sextante API to the GeoTools process
one, see the org.geoserver.wps.sextante package. This also means its possible to rely on libraries
of existing processes provided they are wrapped into a GeoTools process API container.
An alternative way of implementing a custom WPS process, based on Java Annotations, is described in the
Implementing a WPS Process section.
Bridging between objects and I/O formats
The WPS specication is very generic. Any process can take as input pretty much anything, and return
anything. It basically means WPS is a complex, XML based RPC protocol.
Now, this means WPS can trade vector data, raster data, plain strings and numbers, spreadsheets or word
processor and whatever else the imagination can lead one to. Also, given a single type of data, say a plain
geometry, there are many useful ways to represent it: it could be GML2, or GML3, or WKT, WKB, or a one
row shapele. Different clients will nd some formats easier than others to use, meaning the WPS should
try to offer as many option as possible for both input and output.
The classes stored in the org.geoserver.wps.ppio serve exactly this purpose: turning a represen-
tation format into an in memory object and vice versa. A new subclass of ProcessParameterIO
(PPIO) is needed each time a new format for a known parameter type is desired, or when a pro-
cess requires a new kind of parameter, and it then needs to be registered in the Spring contex so that
ProcessParameterIO.find(Parameter, ApplicationContext) can nd it.
Both the XML reader and the XML encoders do use the PPIO dynamically: the WPS document structure is
made so that parameters are actually xs:Any, so bot
The code providing the description of the various processes also scans the available
ProcessParameterIO implementations so that each parameter can be matched with all formats in
which it can be represented.
Filtering processes
By default GeoServer will publish every process found in SPI or registered in the Spring context.
The org.geoserver.wps.process.ProcessFilter interface can be implemented to exert some con-
trol over how the processes are getting published. The interface looks as follow:
public interface ProcessFilter {
ProcessFactory filterFactory(ProcessFactory pf);
}
An implementation of ProcessFilter can decide to return null to the filterFactory call in order to have
all the processes inside such factory be hidden from the user, or to wrap the factory so that some of its
functionality is changed. By wrapping a factory the following could be achieved:
Selectively hide some process
Change the process metadata, such as its title and description, and eventually add more translations
of the process metadata
Hide some of the process inputs and outputs, eventually defaulting them to a constant value
Exert control over the process inputs, eventually refusing to run the process under certain circum-
stances
For the common case of mere process selection a base class is provided,
org.geoserver.wps.process.ProcessSelector, where the subclasses only have to double
check if a certain process, specied by Name is allowed to be exposed or not.
The GoeServer code base sports by default two implementation of a ProcessFilter:
org.geoserver.wps.UnsupportedParameterTypeProcessFilter, which hides all the pro-
cesses having an input or an output that the available ProcessParameterIO classes cannot handle
org.geoserver.wps.DisabledProcessSelector, which hides all the processes that the ad-
ministrator disabled in the WPS Admin page in the administration console
Once the ProcessFilter is coded it can be activated by declaring it in the Spring application context, for
example the ProcessSelector subclass that controls which processes can be exposed based on the WPS
admin panel conguration is registered in applicationContext.xml as follows:

<bean id="unsupportedParameterTypeProcessFilter" class="org.geoserver.wps.UnsupportedParameterTypeProcessFilter"/>
<bean id="configuredProcessesFilter" class="org.geoserver.wps.DisabledProcessesSelector"/>
Implementation level
At the moment the WPS is pretty much bare bones protocol wise, it implements only the required behaviour
leaving off pretty much everything else. In particulat: - GetCapabilities and DescribeProcess are
supported in both GET and POST form, but Execute is implemented only as a POST request - there is no
raster data I/O support - there is no asynch support, no process monitoring, no output storage abilities. -
there is no integration whatsoever with the WMS to visualize the results of an analysis (this will require
output storage and per session catalog extensions) - the vector processes are not using any kind of disk
buffering, meaning everything is kept just in memory (wont scale to bigger data amounts) - there is no set
of demo requests nor a GUI to build a request. That is considered fundamental to reduce the time spent
trying to gure out how to build a proper request so it will be tackled sooner rather than later.
8.6. WPS Services 69
The transmute package
The org.geoserver.wps.transmute package is an earlier attempt at doing what PPIO is doing. It is
attempting to also provide a custom schema for each type of input/output, using subsetted schemas that
do only contain one type (e.g., GML Point) but that has to reference the full schema denition anyways.
Note: This package is a leftover, should be completely removed and replaced with PPIO usage instead. At
the moment only the DescribeProcess code is using it.
8.6.2 Implementing a WPS Process
This section describes how to implement a WPS process for use in GeoServer. It demonstrates the develop-
ment artifacts and build steps necessary to create a WPS process, deploy it in GeoServer, and test it.
The example process used is a simple Hello World process which accepts a single input parameter and
returns a single text output.
Prerequisites
Before starting, GeoServer must be built on the local system. See the Source Code and Quickstart sections for
details. GeoServer must be built with WPS support as described in the Maven Guide section. Specically,
make sure GeoServer is built using the -Pwps prole.
Alternatively, the custom WPS plug-in can be deployed into an existing GeoServer instance (which must
have the WPS extension installed).
Create the process module
To create a new WPS process plug-in module the rst step is to create a Maven project. For this example
the project will be called hello_wps.
1. Create a new directory named hello_wps somewhere on the le system.
2. Add the following pom.xml to the root of the new module in the hello_wps directory:
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd ">
<artifactId>hello_wps</artifactId>
<name>hello_wps</name>
<dependencies>
<dependency>
<groupId>org.geotools</groupId>
<artifactId>gt-process</artifactId>
<version>8-SNAPSHOT</version>
</dependency>
<dependency>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<configuration>
</configuration>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<id>opengeo</id>
<name>opengeo</name>
<url>http://repo.opengeo.org</url>
</repository>
</repositories>
</project>
[hello_wps]% mkdir -p src/main/java
The project should now have the following structure:
hello_wps/
+ pom.xml
+ src/
+ main/
+ java/
Create the process class
1. Create the package that will contain the custom WPS process.
Package naming plays an important role in creating a WPS process. The rightmost part of the
package name is the namespace for the WPS process. For this example, create a package named
org.geoserver.wps.gs inside the src/main/java directory structure. The namespace for the new
WPS process is gs:
[hello_wps]% mkdir -p src/main/java/org/geoserver/wps/gs
2. Create the Java class that implements the custom WPS process.
Create a Java class called HelloWPS.java inside the created package:
package org.geoserver.wps.gs;
import org.geotools.process.factory.DescribeParameter;
import org.geotools.process.factory.DescribeProcess;
import org.geotools.process.factory.DescribeResult;
import org.geotools.process.gs.GSProcess;
@DescribeProcess(title="helloWPS", description="Hello WPS Sample")
public class HelloWPS implements GSProcess {
@DescribeResult(name="result", description="output result")
public String execute(@DescribeParameter(name="name", description="name to return") String name) {
return "Hello, " + name;
}
}
Register the process in GeoServer
GeoServer uses the Spring Framework to manage instantiation of components. This mechanism is used
to register the process with GeoServer when it starts, which will make it discoverable via the WPS service
interface.
1. Create a directory src/main/resources under the root of the new module:
[hello_wps]% mkdir -p src/main/resources
The project should now have the following directory structure:
hello_wps/
+ pom.xml
+ src/
+ main/
+ java/
+ resources/
2. Create an applicationContext.xml in the src/main/resources directory with the following
contents:
<beans>
<bean id="helloWPS" class="org.geoserver.wps.gs.HelloWPS"/>
</beans>
Build and Deploy
To build the custom process, run the following command from the root of the project:
mvn clean install
This cleans the build area, compiles the code, and creates a JAR le in the target directory. The JAR le
name is determined by the name and version given to the project in the pom.xml le. (for this example it
is hello_wps-2.2-SNAPSHOT.jar).
To deploy the process module, copy this JAR le into the /WEB-INF/lib directory of GeoServer and then
restart the instance.
Note: For alternative deployment options (i.e. running from source), see the Trying it out section inside
Implementing a simple OWS service
Test
You can verify that the new process was deployed successfully by using the WPS Request Builder. The
WPS Request Builder is a utility that allows invoking WPS processes through the GeoServer UI. Access this
utility by navigating to the WPS Request Builder in the Demos section of the GeoServer Web Admin Interface.
In the WPS Request Builder select the process called gs:helloWPS from the Choose process dropdown.
The request builder displays an interface which allows calling the process, based on the parameters and
outputs described in the capabilities of the process (which are dened by the process class annotations).
The following image shows the WPS Request Builder running the gs:helloWPS process. Enter the desired
parameter and click on Execute process to run it. A window with the expected result should appear.
8.6.3 Implementing a Rendering Transformation
Rendering Transformations are a special kind of WPS process which run within the GeoServer WMS ren-
dering pipeline to transform data in order to provide more effective visualization. This section describes
how to implement a rendering transformation process in Java.
Rendering transformations are very general, and can transform both the content and the format of input
data. Content transformation typically involves complex geospatial processing which requires access to the
entire dataset (in contrast to geometry transformations, which operate on a single spatial feature at a time).
Format transformation converts from vector to raster or vice-versa in order to produce an output format
appropriate to the desired visualization (for instance, a raster for displaying continuous surfaces, or vector
data for displaying discrete objects).
For more information about the function and use of rendering transformations within GeoServer, refer to
the Rendering Transformations section of the GeoServer User Guide.
Lifecycle of a Rendering Transformation
To implement a rendering transformation it is useful to understand their lifecycle and operation within
GeoServer. A rendering transformation is invoked in an SLD by providing a <Transformation> element
inside a <FeatureTypeStyle>. This element species the name of the transformation process and the
names and values of the process parameters. As an example, the following is a portion of an SLD which
uses the gs:Heatmap transformation:
1 <FeatureTypeStyle>
2 <Transformation>
3 <ogc:Function name="gs:Heatmap">
Figure 8.1: WPS Request Builder, showing gs:HelloWPS process parameters
4 <ogc:Function name="parameter">
5 <ogc:Literal>data</ogc:Literal>
6 </ogc:Function>
8 <ogc:Literal>weightAttr</ogc:Literal>
9 <ogc:Literal>pop2000</ogc:Literal>
10 </ogc:Function>
12 <ogc:Literal>radiusPixels</ogc:Literal>
13 <ogc:Function name="env">
14 <ogc:Literal>radius</ogc:Literal>
15 <ogc:Literal>100</ogc:Literal>
16 </ogc:Function>
17 </ogc:Function>
19 <ogc:Literal>pixelsPerCell</ogc:Literal>
20 <ogc:Literal>10</ogc:Literal>
21 </ogc:Function>
23 <ogc:Literal>outputBBOX</ogc:Literal>
25 <ogc:Literal>wms_bbox</ogc:Literal>
26 </ogc:Function>
27 </ogc:Function>
29 <ogc:Literal>outputWidth</ogc:Literal>
31 <ogc:Literal>wms_width</ogc:Literal>
32 </ogc:Function>
33 </ogc:Function>
35 <ogc:Literal>outputHeight</ogc:Literal>
37 <ogc:Literal>wms_height</ogc:Literal>
38 </ogc:Function>
39 </ogc:Function>
40 </ogc:Function>
41 </Transformation>
42 ...
During WMS requests which uses an SLD specifying a transformation, the arguments passed to the trans-
formation process are assembled from the parameters and values specied in the SLD. Some argument
values may be determined dynamically from SLD variables (as shown above in lines 22-40).
Before the transformation process is executed, it is given an opportunity to rewrite the query GeoServer
makes to the source datastore, via the optional invertQuery or invertGridGeometry methods. This
allows a transformation to enlarge the query extent, since some kinds of transformations may need to
include data which lies outside the original query window.
The query is then performed against the source datastore, and the transformation process is executed
against the resulting dataset. The transformation returns a computed output dataset of the same or dif-
ferent format. If the output has a different coordinate system (CRS) than the requested map it is reprojected
automatically. Finally, the output dataset is passed on through the rendering pipeline to be styled by the
symbolizers dened in the SLD <FeatureTypeStyle>.
Transformation process class
Like other WPS processes, rendering transformations are implememented as Java classes. A pro-
cess class implements the GSProcess marker interface, and is registered with GeoServer via an
applicationContext.xml le. For further information about the basic steps for creating, building and
deploying a GeoServer WPS process in Java refer to the Implementing a WPS Process section.
WPS processes must provide metadata about themselves and their parameters. The easiest way to do this
is to use the GeoTools annotation-based Process API, which uses Java annotations to specify metadata. For
example, the code below shows the process metadata specied for the gs:Heatmap rendering transforma-
tion:
@DescribeProcess(title = "Heatmap",
description = "Computes a heatmap surface over a set of irregular data points as a GridCoverage.")
public class HeatmapProcess implements GSProcess {
GeoServer instantiates a single instance of each rendering transformation class. This means that rendering
transformation classes must be stateless, since they may be called concurrently to service different requests.
This is ensured by avoiding declaring any instance variables within the class. For complex transformations
it may be desirable to implement an auxiliary class to allow the use of instance variables.
execute method
Like all process classes, a rendering transformation class must declare an execute method, which is called
by GeoServer to perform the transformation. The signature of the execute method species the types of
the input parameters and the process result.
The declaration of the execute method for the Heatmap transformation is:
@DescribeResult(name = "result", description = "The heat map surface as a raster")
public GridCoverage2D execute(
// tranformation input data
@DescribeParameter(name = "data", description = "Features containing the data points")
SimpleFeatureCollection obsFeatures,
// process parameters
@DescribeParameter(name = "radiusPixels",
description = "Radius to use for the kernel, in pixels")
Integer argRadiusPixels,
@DescribeParameter(name = "weightAttr",
description = "Featuretype attribute containing the point weight value",
min = 0, max = 1)
String valueAttr,
@DescribeParameter(name = "pixelsPerCell",
description = "Number of pixels per grid cell (default = 1)",
min = 0, max = 1)
Integer argPixelsPerCell,
// output map parameters
@DescribeParameter(name = "outputBBOX",
description = "Georeferenced bounding box of the output")
ReferencedEnvelope argOutputEnv,
@DescribeParameter(name = "outputWidth", description = "Width of the output raster")
Integer argOutputWidth,
@DescribeParameter(name = "outputHeight", description = "Height of the output raster")
Integer argOutputHeight,
) throws ProcessException {
...
Input parameters
The supported process input parameters are dened as parameters to the execute method. The metadata
for them is supplied via @DescribeParameter annotations.
To accept the input data to be transformed, the process must dene one input parameter of type
SimpleFeatureCollection or GridCoverage2D. In the invoking SLDonly the name of this parameter
is specied, since GeoServer provides the dataset to be transformed as the parameter value.
Any number of other parameters can be dened. Parameters can be mandatory or optional (optional pa-
rameters have a value of null if not present). Lists of values can be accepted by dening an array-valued
parameter.
Some transformations require information about the request map extent and coordinate system, and request
image width and height. Situations where these are required include:
the transformation operation depends on the request resolution
the transformation computes a raster result in the request coordinate system to ensure optimal visual
quality
These values can be obtained from SLD predened variables and passed in via parameters of types
ReferencedEnvelope and Integer. (See the Variable Substitution in SLD section in the User Guide for
details of all predened variables available.)
In the case of the Heatmap transformation, the request resolution is used to determine the ground size of the
radiusPixels parameter, and the output raster is computed in the request coordinate systemto avoid un-
desired reprojection. To support this the transformation denes the required outputBBOX, outputWidth
and outputHeight parameters. These are supplied by predened SLD variables as shown in lines 22-40
of the SLD snippet above.
Transformation output
The output of a transformation is a new dataset of type SimpleFeatureCollection or
GridCoverage2D. This is specied as the return type of the execute method. Name and description
metadata is provided by the @DescribeResult annotation on the execute method.
If the output dataset is not in the coordinate system requested for map output, GeoServer reprojects it
automatically. As noted in the previous section, there may be situations where it is desirable to avoid this.
In this case the transformation must ensure that the output has the appropriate CRS.
Query rewriting
If required, the rendering transformation has the ability to alter the query made against the source dataset.
This allows expanding the extent of the data to be read, which is necessary for some kinds of transfor-
mations (in particular, ones whose result is determined by computing over a spatial window around the
input). This also allows controlling query optimizations (for instance, ensuring that geometry decimation
does not prevent point features from being read).
Query rewriting is performed by providing one of the methods invertQuery or invertGridGeometry.
These methods have the general signature of:
X invertX( [inputParam,]
*
Query targetQuery, GridGeometry targetGridGeometry)
The targetQuery parameter is the query constructed from the original request.
The targetGridGeometry parameter is the georeferenced extent of the requested output map. It is not
used in the dataset query, but may be needed for use in conjunction with the transformation parameters
to determine how to rewrite the query. For instance, if a parameter is specied in output units, the output
extent information is required to transform the value into units applicable in the input CRS.
In addition, these methods can accept any number of the input parameters dened for the execute
method. If dened these parameters must be annotated with @DescribeParameter in the same way
as in the execute method.
invertQuery method
This method is called when the rendering tranformation applies to vector data (the data input is of type
SimpleFeatureCollection).
The method returns a new Query value, which contains any required alterations of extent or query opti-
mizations. This is used to query the source dataset.
The Heatmap process implements the invertQuery method in order to enlarge the query extent by the
ground size corresponding to the radiusPixels parameter. To allow converting the pixel size into a
ground distance the input parameters providing the output map extents are also required. The signature of
the implemented method is:
public Query invertQuery(
@DescribeParameter(name = "radiusPixels",
description = "Radius to use for the kernel", min = 0, max = 1)
Integer argRadiusPixels,
// output image parameters
@DescribeParameter(name = "outputBBOX",
description = "Georeferenced bounding box of the output")
ReferencedEnvelope argOutputEnv,
@DescribeParameter(name = "outputWidth",
description = "Width of the output raster")
Integer argOutputWidth,
@DescribeParameter(name = "outputHeight",
description = "Height of the output raster")
Integer argOutputHeight,
Query targetQuery, GridGeometry targetGridGeometry
) throws ProcessException {
...
invertGridGeometry method
This method is called when the rendering tranformation applies to raster data (the data input is of type
GridCoverage2D).
The method returns a new GridGeometry value, which is used as the query extent against the source
raster dataset.
Summary
In summary, the key features of a rendering transformation process class are:
There must be an input parameter which is a FeatureCollection or a GridCoverage2D
It may be useful to have input parameters which provide the request map extent and image dimen-
sions
There must be a single result of type FeatureCollection or GridCoverage2D
The optional invertQuery or invertGridGeometry methods may be supplied to rewrite the ini-
tial data query
The transformation process class must be stateless
8.7 Testing
This section provides an overview of how testing in GeoServer works and a brief guide for developers to
help with the process of writing tests.
8.7.1 Libraries
GeoServer utilizes a number of commonly used testing libraries.
JUnit
The well known JUnit framekwork is the primary test library used in GeoServer. The current version used
is Junit 4.x. While it is possible to continue to write JUnit 3.x style tests with JUnit 4 it is required that
developers write in the JUnit4 style with annotations.
Current version: 4.10
XMLUnit
The XMLUnit library provides a convenient way to make test assertions about the structure of XML docu-
ments. Since many components and services in GeoServer speak XML it is a very useful library.
MockRunner
The MockRunner framework provides a set of classes that implement the various interfaces of the J2EE and
Java Servlet apis. It is typically used to create HttpServletRequest , HttpServletResponse, etc...
objects for testing servlet based components.
Current version: 0.3.6
EasyMock
The EasyMock library is a mocking framework that is used to simulate various objects without actually
creating a real version of them. This is an extermely useful tool when developing unit tests for a componenet
a, that requires component b when component b may not be so easy to create from scatch.
8.7. Testing 79
8.7.2 Testing Categories and Terminology
Software testing falls into many different categories and the GeoServer code base is no exception. In the
GeoServer code base one may nd different types of tests.
Unit
Tests that excercise a particular method/class/component in complete isolation. In GeoServer these are
tests that typically dont extend from any base class and look like what one would typically expect a unit
test to look like.
Integration/Mock
Tests that excercise a component by that must integrate with another component to operate. In GeoServer
these are tests that somehow mock up the dependencies for the component under test either by creating it
directly or via a mocking library.
System
Tests that excercise a component or set of components by testing it in a fully running system. In GeoServer
these are tests that create a fully functional GeoServer system, including a data directory/conguration and
a spring context.
8.7.3 Writing System Tests
Systemtests are the most common type of test case in GeoServer, primarily because they are the easiest tests
to write. However they come with a cost of performance. The GeoServer system test framework provides
a fully functional GeoServer system. Creating this system is an expensive operation so a full system test
shold be useful only when its really necessary. Developers are encouraged to consider a straight unit or
mock tests before resorting to a full system test.
In GeoServer system tests extend from the org.geoserver.test.GeoServerSystemTest class. The
general lifecycle of a system test goes through the following states:
1. System initialization
2. System creation
3. Test execution
4. System destruction
Phases 1 and 2 are referred to as the setup phase. It is during this phase that two main operations are
performed. The rst is the creation of the GeoServer data direcetory on disk. The second is the creation of
the spring application context.
Single vs Repeated Setup
By default, for performance reasons, the setup phase is executed only once for a system test. This can
however be congured by annotating the test class with a special annotation named TestSetup. For
example, to specify that the setup should be executed many times, for each test method of the class:
@TestSetup(run=TestSetupFrequency.REPEAT)
public class MyTestCase extends GeoServerSystemTestSupport {
...
}
This however should be used only as a last resort since as mentioned before a repeated setup makes the
test execute very slowly. An alternative to a repeated setup is to have the test case revert any changes
that it makes during its execution, so that every test method can execute in a consistent state. The
GeoServerSystemTestSupport contains a number of convenience methods for doing this. Consider
the following test:
@Before
public void revertChanges() {
//roll back any changes made
revertLayer("foo");
}
@Test
public void testThatChangesLayerFoo() {
//change layer foo in some way
}
}
The test makes some changes to a particular layer but uses a before hook to revert any such changes. In
general this is the recommended pattern for system tests that must are not read-only and must modify
conguration or data to execute.
Method Level SetUp
A third method of controlling test setup frequency is available at the test case level. Annotating a test
method with the RunTestSetup annotation will cause the test setup to be run before the test method is
executed. For example:
@Before
public void revertChanges() {
//roll back any changes made
revertLayer("foo");
}
@Test
public void test1() {
}
@Test
}
@Test
@RunTestSetup
}
8.7. Testing 81
@Test
}
}
In the above method the test setup will be run twice. Once before the entire test class is run, and again
before the test3 method is executed.
Setup/Teardown Hooks
There are a number of ways to hook into test lifecycle to provide setup and tear down functionality.
JUnit @Before, @After, @BeforeClass, @AfterClass
As with a any JUnit test various annotations are available to preformtasks at various points of test life cycle.
However with a GeoServer system test one must be weary of the task having a dependency on the system
state. For this reason the GeoServerSystemTestSupport class provides its own callbacks.
setUpTestData
This callback method is invoked before the system has been created. It is meant to provide the test with a
way to congure what conguration gets created in the Geoserver data directory for the test. By default
the test setup will create a standard set of vector layers. This method is where that should be changed, for
instnace to indicate that the test requires that raster layers be created as well. For example:
public class MySystemTest extends GeoServerSystemTestBase {
protected void setUpTestData(SystemTestData testData) {
// do the default by calling super
super.setUpTestData(testData);
// add raster layers
testData.setUpDefaultRasterLayers();
}
}
Depending on whether the test uses a single or repeated setup this method will be called once or many
times.
onSetUp
This callback method is invoked after the system has been created. It is meant for standard post system
initialization tasks. Like for instance changing some service conguration, adding new layers, etc...
Depending on whether the test uses a single or repeated setup this method will be called once or many
times. For this reason this method can not be used to simply initialize elds of the test class. For instance,
consider the following:
Catalog catalog;
@Override
protected void onTestSetup(SystemTestData testData) throws Exception {
// add a layer named foo to the catalog
Catalog catalog = getCatalog();
catalog.addLayer(new Layer("foo"));
// initialize the catalog field
this.catalog = catalog;
}
@Test
catalog.getLayerByName("foo");
}
@Test
catalog.getLayerByName("foo");
}
}
Since this is a one time setup, the onSetUp method is only executed once, before the test1 method. When the
test2 method is executed it is actually a newinstance of the test class, but the onTestSetup is not re-executed.
The proper way to this initalization would be:
Catalog catalog;
@Override
protected void onTestSetup(SystemTestData testData) throws Exception {
// add a layer named foo to the catalog
catalog.addLayer(new Layer("foo"));
// initialize the catalog field
this.catalog = catalog;
}
@Before
public void initCatalog() {
this.catalog = getCatalog();
}
}
8.7.4 Writing Mock Tests
Mock tests, also referred to as integration tests, are a good way to test a component that has depdendencies
on other components. Often the case is that it is not simple to create the dependent component.
A mock test is just a regular unit test that uses functions from the EasyMock library to create mock objects.
There is however a base class named GeoServerMockTestSupport that is designed to provide a recre-
ated set of mock objects. These pre created mock objects are designed to mimic the objects as they would
be found in an actual running system. For example:
public class MyMockTest extends GeoServerMockTestSupport {
8.7. Testing 83
@Test
public void testFoo() {
//get the mock catalog
//create the object we actually want to test
Foo foo = new Foo(catalog);
}
}
Like system tests mock tests utilize a one-time setup with the same setUpTestData and onSetUp callbacks.
The benet of mock tests over systemtests is the setup cost. Mock tests essentially have no setup cost which
means they can execute very quickly helping to keep overall bulid times down.
EasyMock Class Extension
By default EasyMock can only mock up interfaces. To mock up classes requires the EasyMock classexten-
sion jar and also the cglib library. These can be declared in a maven pom like so:
<dependency>
<groupId>org.easymock</groupId>
<artifactId>easymockclassextension</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>cglib</groupId>
<artifactId>cglib-nodep</artifactId>
<scope>test</scope>
</dependency>
The change is mostly transparent, however rather than importing org.easymock.EasyMock one must
import org.easymock.classextension.EasyMock.
8.7.5 Maven Dependencies
All of the GeoServer base test classes live in the main module. However since they live in the test packages
a special dependency must be set up in the pom of the module depending on main. This looks like:
<dependency>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
Furthermore, in maven test scope dependencies are not transistive in the same way that regular dependen-
cies are. Therefore some additional dependencies must also be declared:
<dependency>
<scope>test</scope>
</dependency>
<dependency>
<groupId>xmlunit</groupId>
<artifactId>xmlunit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.easymock</groupId>
<artifactId>easymock</artifactId>
<scope>test</scope>
</dependency>
8.7.6 Online Tests
Often a test requires some external resource such as a database or a server to operate. Such tests should
never assume that resource will be available and should skip test execution, rather than fail, when the test
is not avaialble.
JUnit4 provides a handy way to do this with the org.junit.Asssume class. Methods of the class are
called from a @Before hook or from a test method. For example consider the common case of connecting
to a database:
public class MyTest {
Connection connect() {
//create a connection to the database
try {
Conection cx = ...
return cx;
}
catch(Exception e) {
LOGGER.log(Level.WARNING, "Connection failed", e);
return null;
}
}
@Before
public void testConnection() {
Connection cx = connect();
org.junit.Assume.assumeNotNull(cx);
cx.close();
}
@Test
// test something
}
}
In the above example the assumeNotNull method will throw back an exception telling JUnit to simply
forgo execution of the test.
8.8 Versioning
Versioning refers to the ability of storing the history of the changes occurred to a feature, along with the
user tha performed the change and an optional comment, performing point in time extractions of the values
and eventually rolling back the changes to a previous state.
8.8. Versioning 85
GeoServer uses the GeoTools versioning PostGIS data store to provide two different kinds of versioning
services:
versioned WFS
GeoServer Synchronization Service, a distributed version control system managing synchronization
between a number of nodes, called unites, and a central orchestrating node called central
8.8.1 Versioned WFS
Versioned WFS is an extension of standard WFS that uses a versioned store and provides extra requests to
deal with history, point in time extraction and rollbacks. The following documents provide insight on its
design, current implementation status and usage:
WFS versioning requirement specication
First off, we dont have a full requirements list, but a general set of wishes from the following pages:
http://docs.codehaus.org/display/GEOS/GeoCollaborator
http://lists.eogeo.org/pipermail/opensdi/2005-April/000071.html
http://lists.eogeo.org/pipermail/opensdi/2005-November/000277.html
From these pages we can gather the following requirements:
1. Wiki for geospatial data and for us the two dening features of a wiki are the attribution and the
ability to diff and roll back.
2. The CVS of geospatial data
3. Easier to use than commercial softwares (less maintenance I guess?)
4. Implemented as a set of plug-insinto the WFS transaction calls, among others, such as validation (so
that bad data is not accepted) and change notication (mails, rss, and so on)
5. Some kind of delayed commit, Another possible plug-in could be peer review holding area, where
some one must explicitly approve a change instead of checking after and rolling it back;
6. Patches, itd also be nice if there were an easy way to make a geospatial patch, based on specic
contraints such as version numbers, area, user that provided them.
7. Checkouts, as in a classic version control system: caching and consuming of WFS. If you have a
version table, that has timestamps as to the last updates, then the client really only needs to access
the WFS once. And indeed in some cases you could even send out clients with the WFS layers pre-
cached. When a client examines the layer again, they dont need to download the full WFS, they can
just check the version table if there are updates since the last time they checked. If there are, then they
can just download those directly, put them in the cacheand also The version WFS would become
the version control, and one could do diffs of ones locally modied les against the server. This also
goes back to low and no bandwidth situations - les could be updated asynchronously, an update can
come on a cd/dvd/rewire drive from the main ofce.
Extra requirements that do not appear to be cited in the above documents, but seem to be important to me
as well:
Object of versioning: Id say, given the simple feature nature of our implementation, that we want to
version multiple layers at the same time, since changes on one layer may raise incompatibilities on
other layers. For example, what if you move a road so that it overlaps with a building? Moreover,
standard WFS-T calls are designed to work against multiple feature types in each call.
Branches: requirement number 5 could be turned into a request to handling branches, which is some-
thing that ArcSDE does, even if in SDE these are just called versions.
Tags, since we want to be able and extract maps in a veried state.
Versioned tree space management: after a while every heavily used versioning systemshows the need
for cleanup, to reduce the load on the server, speed up searches and general functioning.
Speed: we need to decide which operations should be speedy, which should be costly. Out of the
box, Id say we want the last revision of each branch to be quick to check out, almost as if it wasnt
versioning around, and have to pay an extra cost for intermediate revisions and diffs.
The above should take into consideration space too. Hum, speed, space, ease of implementation, pick any
two? Other interesting questions are:
Do we want to be able and version schema changes as well? What happens the day we need to
add/remove/change an attribute in the feature?
Do we want to keep data structures untouched, and allow people to keep on accessing them read only as
if versioning was not around, or do we make it explicit that this is a different kind of beast and must
be accessed accordingly?
Comments
Cameron Shorter: To be effective, this functionality will require User Authentication and Authoriza-
tion. This needs to be mixed with business rules. Group A is allowed to enter new features, Group B
is allowed to edit existing features, Group C is allowed to set the Reviewed attribute to TRUE.
Brent Owens: Branches? Branches are great for software because updating software can take a while
and they are a handy way to stay in sync and merge back into trunk. Data on the other hand, it often
goes through QAbefore it gets thrown into the central repository. Imjust not seeing the need to make
a branch of a dataset. A copy of a dataset? Yes, a branch, no.
Brent Owens: Long transactions: Im updating an area of a dataset, I have it locked so no one else can
edit it while I am, but my update takes 2 days to process. Can we handle long transactions? Do we
want to tackle it for this?
Brent Owens: As Cameron stated, we need to allow for business rules to plug in. There is no way we
can tackle this for all cases, but I think we can cover a few common/simple ones.
Spatial data versioning experiences
The following list of article contain real world experiences about spatial data and versioning. If you know
more, please contribute.
Using ArcSDE Versions in the Real World <http://gis.esri.com/library/userconf/proc01/professional/papers/pap232/p232.htm>
Comments
Brent Owens: Having the history queryable is a huge bonus. Often I had to go back and nd out what
features were changed during a certain time frame. Or I had to use a snapshot of the dataset from a
certain time. The versioning was built into our system so this was easy to do, and very useful; we just
kept copies of each feature with a ag stating if it is the current working version.
Brent Owens: Deep histories (storing all history from every version) was also a great way to see how
the data was changing over the years and for generating some statistics on how it was changing: this
many roads became paved this year.
8.8. Versioning 87
WFS-T extensions
That is, how do we perform versioning versioning operations, and eventually provide enough information
to build a client using some form of checkout, with the WFS Transaction and GetFeature calls? Are these
enough?
GetFeature call must be extended so that we can specify which branch/tag we want to checkout, eventually
which revision, if we go for the svn revision model, or which date. GetFeature response should include at
least revision and tag/branch information.
Transaction calls should be exdended to handle branch/tag and revision number so that we can check the
modication is going over an up to date feature (do we need that? guess so, unless we go for a reserved
checkout model leveraging lock calls?). Transaction responses should be extended to provide conict infor-
mations (and eventually new revision numbers if we go the svn model?).
The same goes for lock calls, we need to specify what we do want to lock (at least the branch/tag, not sure
we may want to lock a specic revision).
Most of the above requirements, beside the extended answers, come as natural if branch and revision are
included in the feature type as extra attributes. We need those in order to build a checkout, but also to
check for conicts during transactional updates. As Jody suggested, the ranges of revision numbers could
be provided thru a DescribeFeatureType call, using a choice construct.
An alternative could be to have a separate WFS server (different getCapabilities addresses) for each branch
of the server. This implies a vision of versioning that spans the entire server. The main drawback I see
is that this way the versioning handling seems to be locked to a specic datastore, or to an extension that
can perform versioning independent of the datastore. Moreover, it apparently implies that all feature types
server by the server are versioned.
Versioning at the datastore level allows to have a single server with multiple and separate versioned sets of
feature type, along with non versioned types as well. Versioning at the WFS level seems to deny it.
Plus, depending on the implementation, it may be hard to add versioning on top of a data store that does
not support it on its own, the implications would be that:
the original data set has to stay unchanged, so it cannot be served as is beyond GetFeatures. It also
seems to me more common to have people ask for the last version or a specic tag.
if versioned operations go on for enough time, most of the data volume is in version tables, so I guess
it may be just sensible to start by putting the shapele or whatever the format is in the versioned
datastore itself.
Scope and scheduling
Versioning will be implemented in two distinct phases:
Basic versioning support: attribution, versions, rollback, possibly checkout support. No branches, no
tags. Just get on par with OpenStreetMap functionality.
Enterprise versioning support: add branches and tags, merges and eventually checkout support
should it have been excluded from the rst phase due to time constraints.
Versioning WFS - Versioning approaches classication
Its quite clear that the requirements for a Versioning WFS sum up to building a full edged version control
system for features, although we may want to nd ways to implement it step by step. So, it may be helpful
to compare different approaches at tackling some of the implementation aspects of a version control system.
Difference handling
Given two revisions of the same feature, how do you store the difference between them?
Copy: we have two integral copies of the features
Diff: only the differences are stored. There may be multiple ways to do so: * keeping a map of
xpath/old value for every changed attribute * encoding the two revisions in GML, and do a text diff
between them * keeping a map of xpath -> diff, where the difference of each attribute is encoded as a
text difference (less tags around, and less context...), or as a binary difference (serialize the attribute,
and then diff the byte arrays?)
Adaptive, try diff, revert to copy if diff does not provide any space benet.
Copy has the advantage of speed, getting a certain revision is just a matter of selection, whilst patch may use
less space, especially if the feature is big, but requires both the selection of patches and their application to
the integral data copy we have somewhere (how the integral data copy is managed is discussed in another
part of this document).
For a comparison of different diff computation algorithms, see: An Empirical Study of Delta Algorithms.
Versioning direction and integral copies
Do we keep backwards or forward changes?
Backward changes: the last revision is an integral copy, going backwards means applying one or
more reverse patches (this is the svn, cvs approach);
Forward changes: we do start from an integral copy (eventually an empty data set), and register
normal patches to store the differences.
Were do you have the states that differences refer to in order to build another revision of data?
last revision of each trunk/branch/tag
in the root of the versioning tree
Backward changes ensures fast access to the last revision, forward changes allows to handle multiple
branches of data in a more natural way, because revisions can be thought as a tree starting with a sin-
gle root (so a branch is just recorded as a set of forward changes from the common point where it branched
off).
Backwards changes need last revision integral copies, whilst forward changes do need at least an integral
copy in the root of the versioning tree.
Forward changes calls for fast to compute differences in order to get decent speed on the common case.
ArcSDE uses forward changes coupled with difference by copy approach and integral copy in the root. Svn
and csv use backward changes coupled with diffs and integral copy as the last revision.
Branches and tags management
Are these special beasts, or does the data structure seamlessly handle those too?
no special management in SVN and ArcSDE, the same data structure rules them all;
as an extra structure on top of the base with CVS (attic and le infos).
In the ArcSDE case, forward changes provide a natural t to handle whatever trunk copy you may think
of, branches and tags are really the same. The same goes for SVN.
8.8. Versioning 89
If would be nice to have the same conceptual ease.
Space management
How do we manage space and speed up a bogged down versioning server?
by removing unneeded tags, branches;
by removing part of the history;
by removing intermediate revisions between two valuable ones (which means removing the interme-
diate changes in copy approach, or merging diffs in the diff approach).
Versioning implementations
Before starting to work some existing versioing implementations have been analyzed:
Subversion
Subversion is a well known version control system.
The model visible to the user is well known, notable features in the context of feature versioning are:
Revisions numbers are related to the whole tree.
Trunk, tag, and branches are simply conventions, all copies are treated equals through a smart copy
mechanism.
The checkout allows for ofine operation;
Difference computation is efcient and can handle both text and binary les (based on the VDelta
algorithm).
Client side view of the world Its interesting to note how commands do behave on the client side, and
how they do parallel with the WFS.
There are a few similiarities in all commands:
If no revision number is provided, the last revision is assumed.
Almost all commands support authentication provided thru username/password, and these are
stored in a special folder so that authentication is required just once (and its not in the checkout,
but in the user home).
commands can work both on the checkout or directly against the server.
Lets have a look at some interesting commands that we may want to replicate in a versioned WFS:
svn cat provides a way to list the content of an URL at the last revision, or at a specied revision. It can
work only against a list of les, but its relatively similar to GetFeature, and provides no information
to build a checkout, just the plain content.
svn info lists informations about a le or directory, specically, revision the le is at, author, date and revision of the last change occurred before the current revision (you can specify a target revision in svn info, so the last change may be different depending on the chosen target revision). This is interesting:
a le is considered to be at the last revision, not at the revision it was last changed. This
is consistent with the global revision number, which is really associated with the full tree,
branches included.
each le has a kind of internal versioning that knows the le has been changed 20 times, but
this is not shown to the user, only the global revision number shows up.
svn log provides a history of changes for a specic URL and revision, citing for each change revision,
author, date and commit log, eventually it may return a list of all les changed during the commit.
The output format can be plain text (human readable) or XML (machine processable), and eventually
a limit to the number of changes reported can be specied.
svn diff returns the difference between two revisions of a given path, or even between two revisions
at different paths. The difference is encoded as a standard unix diff.
svn checkout checks out the content from the repository. It builds a quite specic checkout format,
containing informations about the le revisions, full server URL for each le (which allows to indenty
the branch the le comes from) and a base copy that allows to perform status and difng opera-
tions without being connected to the server. The latter is most important because it allows for far
more efcient and pleasand usage when the versioning server is not in your local network, or when
connection is not available at all.
svn commit commits local changes to the server. Its a command that requires a local checkout, so
its quite unlike the WFS Transaction that does not need it, but at the same time similar, since it sends
diffs to the server which are akin to updates.
svn revert reverts local changes, so again works against a local checkout. Theres no such a thing as a
server revert, going back adds to the revision history, unlike SDE or Oracle where a revert wipes out
the reverted changes. In order to revert a committed change in svn, you have to perform a reverse
merge, that is, svn merge -r m:n where m> n, and then commit the change, or performa direct reverse
merge against the server, svn merge -r m:n svn://svn.myserver.org/trunk/path....
svn merge applies changesets from one url to another. The changeset to extract is specied by means
of an URL and two revision numbers, and its applied to a specied path. The version working against
two remote URL is something compatible with WFS on a conceptual level, much like a combined
GetFeature and Transaction operation.
svn import, allows to check in a new subtree of les in the server. This is interesting from a WFS
perspective if we think of it as a way to create a new versioned feature type, since Transaction already
allows for insertions.
Versioning storage implementations The way subversion manages versioning on the server side is prob-
ably less known.
Subversion may use two different backends, the FSFS backend, and the Berkeley DB backend. Whats most
surprising is that not only the two backends are using different storage supports, but they dont behave
in the same way either, versioning wise: one works computing forward changes, whilst the other does
backward ones.
The informations here are a summary of the subversion design document and a mail discussion I had with
subversion developers.
FSFS backend The FSFS backend is a forward change type, that is, each revision of a le stores the delta
required to go from a prevision revision to the current one.
To avoid obvious efciency problem when the revision history of a le becomes long, a skip list approach
is used in order to reduce the number of forward deltas that need to be applied, reducing the number of
deltas to be applied from O( n ) to O(log n).
Some revisions do not store the delta from a base revision that allows us to perform jumps as big as 2n.
Moreover, each time a le on a branch is modied for the rst time a full copy of the le for that branch is
generated, and it becomes the new base full copy for the le in the branch.
8.8. Versioning 91
In the following discussion I provide some more detail. When I say le version I do mean the actual n-th
version of the le, I do not refer a global version number.
I dont have the exact algorithm, but a way could be to compare each revision against the last power of
2 lower, so if youre committing then 18thversion of a le, you delta against the 16th, which contains the
delta against the 8th, and then against the 4th, and nally the 2nd and the 1st (which is a full copy).
File version 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Delta computed against version 1 2 2 4 4 4 4 8 8 8 8 8 8 8 8
The rst drawback of this approach is that you have to sum all the differences occurred between the last
power of 2 and the current revision, so the patch may become (very) big. One way to avoid that would be
to insert full copies at selected points in the sequence so that the biggest patch does not store more than 64
or 128 changes.
The second drawback of this approach is that you have to know how many versions of a le are there in
order to retrieve the diffs required to rebuild the n-th version of a le. This is a little harder to turn into a
database query, but it could be solved by having a reference table that store the powers of two, and have a
subquery select all power of two lower than the current le version. The BDB backend
The BDB backed stores backwards changes, so the last revision on trunk is always a full copy, whilst on a
branch the revision is a full copy if the le has been modied on the branch, otherwise its just a pointer to
the trunk revision the branch originate from.
As with the FSFS backend, the BDB backed uses skip lists in order to rebuild faster old revisions, but since
the reference copy is a moving target, also the delta need to be recomputed at each commit.
When you commit a new le, the 4th, 8th, 16th, ... level ancestors are re-deltied against the last revision.
So, in the following example, if you request the 3rd version of a le youll have to apply deltas contained in
3, 19, 23 (in reverse order) to the full copy in 24.
File version 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Delta against version 17 18 19 20 21 22 23 24 17 18 19 20 21 22 23 24 21 22 23 24 22 23 24
The positive side of this approach is that the last revision is usually quick to build, on the downside the
insert process is complex as it requires recomputing deltas, and gathering a revision other than the last one
of trunk goes through the same complex process as FSFS, because not all les on a branch may have been
modied (and thus be full copies).
Oracle Worskspace Manager
Workspace manager is an Oracle module that allows for version control like features while retaining the
ability to work against the database as if the versioning systemwas not there. This system is very powerful,
but shows some notable differences from normal version control systems.
Workspace manager is a forward change with full copies system, just like ArcSDE, thought the implemen-
tation is quite different, and whilst not have a concept of revision, it still provides branches, tags and change
dates. It has no attribution, so its not possible to know who did perform a specic change.
The main versioning concepts in Workspace manager are workspaces (branches) and savepoints (tags).
Each workspace can have zero or more child workspaces, and zero or more parent workspaces (multiparent
workspaces have been introduced in Oracle 10g).
Workspace aware client applications do select which workspace they do work in, and then operate against
the database as usual, with the notable difference that each DML statement does in fact create new rows in
the versioned table, so that fully history is recorded.
Merges can occur two ways:
from parent to child workspace: these are called refreshes, and can be manual, or set as automatic
(continually refreshed workspaces), so that each change on the parent workspace is directly reected
on child workspaces (not sure how conicts are handled should they arise, the guide does not tell).
From child to parent: this is called merge. A merge can also close the child workspace (its an option).
Once work is done on a workspace the latter can be deleted, this implies the deletion of all information
about that workspace too.
Save points are a way to establish tags on a workspace. They are used for a few purposes:
to mark an interesting state of the workspace that can be consulted aftewards. It is possible to enter
a save point afterwards and have a read only view of the database.
to mark a position and eventually revert to its state. A rollback to a save point will remove all changes
that have been performed after the save point, without any possibility to undo the rollback (so it
removes the intermediate changes from the database, it does not create new changes that set the state
back to the savepoint like a svn like rollback);
to mark the start of a new child workspace (much like in CVS branching best practices).
An extra set of lock operations is provided, besides the usual lock primitives, that allow inter workspace
locking, allowing a row to be locked in both child and parent workspace, limiting the likeliness of conicts.
The versioning implementation is in fact quite complex, and its not explained in any detail in Oracle doc-
umentation.
When a table is marked as versioned, a few things occur:
the table gets renamed to <table>_lt, and a few new colums appear: version, createTime, retireTime,
nextVer, delStatus, ltLock.
A set of views is built to gather signicant information about the versioned table.
First and foremost, a new view named <table> is created, so that the user has the impression that the
original table is still there, and a set of trigger is created so that each insert/update/delete operation on
top of the view really alters the versioned table. The view in turn selects only live rows associated to the
current workspace, so that changing workspace or savepoint changes the rows returned by the view. A
slew of other views is built to enabled the creation of diffs, to query about parent/child conicts or long
lived locks.
In the WMSYS schema a lot of support tables is available that identies association between version num-
bers and workspace and save point names, as well as parent child relations, support for global lock and the
like.
Versioning WFS - ArcSDE
ArcSDE is a forward change feature version system, capable of handling multiple branches, and based on
a set of database tables. The standard parlance in ArcSDE is a little different than the one used in SVN:
version: branch/trunch/tag, that is, the equivalent of an svn copy. Trunk is called the default
version;
state: revision
post a version B to version A: performa merge of changes occurred in B onto A, where B was branched
off earlier. This usually also closes version B.
In ArcSDE states are maintained as sets of adds and deletes to the initial state of data, and each add or
update is a full copy, not a diff.
The following diagram provides a picture of entities involved into versioning.
8.8. Versioning 93
Each version is identied by a name and owned by some user. The access level may be private (only the
owner can read and write), protected (only the owner can write) or public. Each version has a parent, and
it can be posted back to that parent once done (I havent seen examples of posting to another version, dont
know if its possible).
Each version refers a state, which is in turn a set of modications on the feature set, and has a parent state,
thus building a tree of states. Each state hold modications for a number of features, and the list of feature
tables modied in a certain state is held in the ModiedTables associated records.
Both version and state do refer a lineage, which the documentation does not comment much about. Any-
ways, since an efcient way to look up the ancestors of a given state or version is needed, lineage probably
stores an ancestors path or a similar way to encode a hierarchy. For example, version 6 lineage may be
something like 0.1.2.6. For a discussion of ways to encode a hierarchy in a sql database refer to the DBAzine
article Nested sets and materialized paths.
For each feature table FeatureX in the database, we do have a AddFeatureX table whose structure is
the same as the feature table table, along with a reference to the state the record is associated to, and a
DelFeatureX, which holds just a reference to the deleted object, along with the last state where the object
was alive, and the state where the object has been deleted. For each added feature, a record appears in
the AddFeatureX table, for each deleted feature a record appears in the DelFeatureX table, and for each
updated feature a record appears in both tables.
The record in the AddFeatureX tables always represents the new full state of a feature, so no delta compu-
tation is performed: this allows a faster retrieval of current state at the expense of extra space consumption.
On the bright side, the features at a certain state can be retrieved with a single SQL query. Assuming the
state ancestry is maintained with the materialized path method, the query would look like:
select id, att1, ... attn
from FeatureX
where id not in (
select id
from DelFeatureX, States
where DelFeatureX.stateId = States.id
and States.lineage in (0, 0.1, 0.1.2, 0.1.2.6)
)
union
select id, att1, ...., attn
from AddFeatureX, States
where AddFeatureX.state_id =
(
select max(state_id)
from AddFeatureX af2
where af2.id = AddFeatureX.id
)
and AddFeatureX.state_id = States.id
and AddFeatureX.id not in (
select id
from DelFeatureX, States
where DelFeatureX.stateId = States.id
8.8. Versioning 95
and DelFeatureX.stateId <> AddFeatureX.stateId
)
that is, all original features that have not been updated of deleted during the states history, union the last
record of all features that have been added or updated, and not been deleted during the states history.
Here we suppose we know the lineage of the state and that we wont incur in sql length problems... this
is a little nave of course, an alternative would be to keep full ancetry information in a separate table that
can be quied, allowing from something like state_id in (select parent from state_hierarchy where child =
current_state_id)). Of course such a table is bound to become very big.
OpenStreetMap
From the OpenStreetMap home page:
OpenStreetMap is a project aimed squarely at creating and providing free geographic data such as
street maps to anyone who wants them;
Anyone can edit the current map using one of our map editors, or upload and share your GPS track-
logs. Before you can upload, or edit, you will need to create an account.
OpenStreetMap is one of a kind in many ways, from the feature concept, to geometry handling, versioning
approach, and API. Lets analyze the most interesting parts of it.
Feature concept Features in OpenStreetMap are tagged geometries, where tagging is completely open
ended, a map from keys to values. So, we have node, segment, ways and areas, where each record can have
a different set of attributes.
This is very different from a WFS approach, where data is typed and has a foreseen number of attributes.
The pro is that any kind of information can be stored, the cons is that there is no type checking and you
cannot handle data making assumptions on data structure (so no SLD or lters, for example).
Geometry concept Geometries are not simple features, they are topologies:
Node is the basic concept, its a point. A node cab be shared by various segments (as start or end
point).
Segment is a straight line between two nodes used to build ways and areas. Segments can be shared
between various ways and areas;
Way is a set of segments.
Area is a set of segments as well. Apparently holes are not supported, not sure about multipolygons.
As said above, each can be tagged with extra information.
Versioning All data in OSM is apparently stored in a mysql database, whose schema is provided here:
http://svn.openstreetmap.org/sql/mysql-schema.sql. The schema is peculiar, there are no foreign keys or
spatial indexes that I can see.
First thing you can note is that theres a split between a set of current_xxx tables and xxx tables: the former
represent the current state of the database, the last revision of it, whilst xxx tables do represent the history.
This is evidently done to increase performance, since the most accessed data is the live one anyways.
Each table representing a feature (nodes, segments, areas, ways) has a user attribution, a time stamp, and a
visibility ag.
A version attribute is found in tables related to areas and ways, but not in segments and nodes (current
tables do not have a version number). It seems the system relies more on timestamps than version numbers
to do its work?
Tag handling is mixed, for nodes and segments its a single text eld, whilst for area and ways the thing is
handled with a normalized one to many approach.
API API is based on a REST approach. The API is simple, a full description can be found here:
http://wiki.openstreetmap.org/index.php/REST.
A few considerations here:
all calls are authenticated using basic authentication, in order to provide attribution;
theres no transaction concept, in order to add a new way you have perform three different requests,
to separately add nodes, segments and areas. This seems to be quite fragile data consistency wise.
there is no commit message support, neither dirty area support, so its hard to tell what was done in
each commit... this is consistent with the lack of transaction thought.
theres no explicit rollback operation, thought data structures are in place to support it?
even if version number are handled internally, they are not shown in the xml format.
Versioning implementation
These pages describe the current implementation of the versioning WFS.
Early implementation proposal
Chris already provided his own vision about an implementation on the OpenSDI mailing list in April 2005.
Its a nice starting point, and also works in a way different than ArcSDE, so its an interesting read.
What Im envisioning at the moment is a table in a database that records every single update/insert action:
It holds the diff, ie the changes between what the feature was and what the new one is.
It holds the username of the person who modied, and their comments on the change (like a cvs
commit comment, why they made the change, what their source was, ect.).
It holds the featureid of the feature changed (which is a foreign key to the tiger-an db).
And it holds a location, or perhaps a bounding box, of the change. I suppose this could just follow
the foreign key, but I think I may prefer the actual location. The nice part about this last
thing is that it allows you to get the list of diffs on a zoomed in part of the map, that are the ones
actually relevant for the part of the map you are looking at. Im thinking something similar to
subversion, where there is just a global revision number, for all changes, and those changes then affect
certain features.
So if you were super zoomed in, and hit the edit button, then the history column for the page would
just be the numbers of the revisions that affected that area, instead of the normal wiki way where all
the
numbers are present. As for implementation, to retrieve the history column for any map youre
zoomed in on, you use WFS.
8.8. Versioning 97
You would expose the diff table as WFS, and since it has locations of all the diffs, and feature ids
(which in this case are the global revision numbers). To get the history for a given map, you would
just do a WFS query on the bounding box (and indeed this makes it trivial to do things like give me
all commits in this bounding box made by cholmes, and the like, as its just a WFS query). And
exposing the diff history as WFS would make it easier to implement in different clients, and indeed
to maybe even use another WFS. We would just need to come up with a spatial wiki history gml
application schema, and specify how to ll the elds on each commit (or perhaps make a special wiki
transaction xml request? Though if the diff history was wfs-t then it could just be a second wfs insert
transaction (Ah! Way too meta!)).
Interesting points in my opinion:
single version table for full data
commit bbox
versioning a la svn (thought it seems to me hard to implement with decent performance)
serving the diff table with wfs again
Datastore design
The solution is a mix between Oracle, ArcSDe, svn and the Chris proposal. From a classication point of
view, its a forward only versioning with no diffs, and no branches, using progressive revision numbers
like svn.
Here is the diagram:
The Data table represents a single versioned feature type, has all the attributes of the normal feature type,
and a few extra columns:
rowId: the primary key if the versioning was not there, backed by a sequence.
revisionCreated: the revision where the row was introduced (as a result of an insert or of a row
update).
revisionExpired: the revision where the row was marked as old (as a result of a deletion or an
update). For rows that are not expired, its assumed to be MAXLONG (because, as youll see, this
eases querying).
The Data table primary key is (rowId, revisionCreated). RevisionExpired should be indexed as well.
The ChangeSet represents the change in a set of rows contained in versioned feature type as a result of a
Transaction or rollback operation. The columns are:
revision: backed by a sequence, its the global revision number, backed by a sequence (primary key
of this table).
author: is the id/name/whatever of the person/system that performed the change.
date: the full timestamp (milliseconds included) of the change.
bbox: the area that the changes involved, expressed in ESPG:4326 (since different feature types may
be expressed in different CRS).
message: the commit message for the change.
Each changeset is associated to a list of involved tables, in order to ease nding the changed records for
each changeset.
The VersionedTable table is a list containing the currently versioned feature types:
versionedTableId: backed by a sequence, its the primary key;
name: the versioned table name
Common operations implementations in terms of SQL queries Its interesting to see how this design
allows for relatively simple queries for the most common operations:
Gather a specic revision, with lter:
select
*
from data
where <filter encoded in sql>
and revisionCreated <= [revision]
and revisionExpired > [revision]
Last version, with lter:
select max(revision)
from changes
and then the same as gathering a specific revision
State at a certain point in time:
select max(revision)
from changes
where date < [specifiedDate]
and then the same as getting a specific revision.
Rollback to a specic revision, but keeping history: First a set of intersting queries in order to under-
sand whats going on: All modied rows between revision and current, and matching a certain lter
at some point during the evolution of that branch:
select distinct(rowId) from data
where revisionCreated > [revision]
and [filter]
Original state of all rows existing at the specied revision (and matching the lter):
select data1.
*
from data data1
and data1.revisionCreated <= [revision]
and data1.revisionExpired > [revision]
and data1.rowId in (
select data2.rowId
from data data2
where data2.revisionCreated > [revision]
and [filter])
Current state of all rows created and not deleted after the specied revision:
select
*
from data
and revisionExpired = [maxLong]
and rowId not in (
select rowId
where data1.revisionExpired > [revision]
and [filter])
So basically reverting means running the following queries:
8.8. Versioning 99
-- update old revisions, equates to a deletion
-- for those that were not there
update data
set revisionExpired = [newRevision]
from data
and revisionExpired = [maxLong]
and rowid in (
select rowId
from data data2
where data2.expired > [revision]
and [filter]
)
-- inserts the old values for all data that
-- has been updated or deleted between the two revisions
insert into data
select d.
*
(besides rev numbers), [newRevision], [maxLong]
from data d
where d.revisionCreated >= [revision]
and d.revisionExpired > [revision]
and d.rowId in (
select data2.rowId
from data data2
where data2.revisionExpired > [revision]
and [filter]
)
Difng between revision n and revision m Last value of everything changed between n and m (m
> n), and satisfying a lter in one of the past states (if row is not changed, it wont have old states
between n and m):
select
*
from data d1
where revisionCreated <= m
and revisionExpired >= m
and rowId in (
select rowid
from data d2
where d2.rowid = d1.rowid
and revisionCreated < d1.revisionCreated
and revisionExpired > n
and [filter])
order by rowId
Value at n of everything changed between n and m (eventually deleted):
select
*
from data d1
where revisionCreated <= n
and revisionExpired > n
and rowId in (
select rowid
from data d2
where d2.rowid = d1.rowid
and revisionCreated > d1.revisionCreated
and [filter])
order by rowId
Then the two sets must be scanned in parallel like in a merge sort and diffs must be generated (the
diff format is still a matter of discussion). This unfortunately works ne only for single column keys,
for multicolumns its not as obvious, especially if the primary key is allowed to change like in the
multicolumn d mappers. Hmmm... this must be forbidden in fact for identity chain to work (rowid
is what keeps togheter the rows history...).
Moreover, the discussion assumes lter can be encoded in sql, and it may not be the case... this
complicates matters a lot since the lter is to be applied in a subquery that does not return values.
Getting change history for a table, with eventual lter on area or user:
select date, author, revision, message
from ChangeSets
where revision in (
select revision
from TablesChanged
where versionedTableId = [tableId])
and bbox && [searchArea]
and user = [specifiedUser]
Performance tests One of the main concerns given the data structures we are setting up is scalability, that
is, we do expect a performance hit due to versioning, and wonder how severe it is compared to revision
numbers in the database and the actual modications performed by each release.
Agood implementation should degrade no worse than O, where n is the total number of versioned records.
Well, to my surprire, it seems the above table setup, with proper indexes, is less than linear, but almost
constant .
To asses performance Ive setup a little data ller and then a query benchmark. The data ller:
sets up a spatial table in Postgis with a linestring geometry, a text attribute, a numeric id and revision
columns (testData table);
sets up a versioned spatial table with the same structure as the previous, but without the revision
columns, as a reference of the performance that can be obtained without versioning around;
inserts a certain amount of data in testData as the rst revision, lling the lat/lon space with a regular
grid (each feature occupies a cell). Geometries are random, but guaranteed to t in their cell.
starts versioning data modiying for each revision a certain number of features, and marking as
expired the current version of it;
ll the reference data table with a snapshot of the last revision;
does a vacuum analyze to make sure optimizer knows about data distribution.
The query benchmark instead performs a few queries against reference and versioned data:
an extraction of the full data set from reference, and then last revision from versioned, and a few
snapshots as specif versions;
an extraction of a bbox (big enough to be timed), and the same against the last revision and specic versions.
The above is run twice to make sure the are no caching effects around, and in fact, the second
run does not seem to hit the disk at all, but runs against the le system cache.
Tests have been performed on an Intel Core 2 Duo 2.13Ghz, 2GB RAM, and two 7200 rpm disks in RAID 0,
Windows XP professional, Postgres 8.1.3 and Postgis 1.1.4 congured as out of the box, no extra tweaking
on Postgres memory settings.
Here are the results with 100.000 reference features, and 4000 revisions modifying each 30 records (thus,
120.000 more records in the database), 220.000 records total:
8.8. Versioning 101
Reference data:
Running: select
*
from testDataRef
Elapsed: 1.157 s, returned records:100000
Running: select
*
from testData where expired = 9223372036854775807
Running: select
*
from testData where revision <= 0 and expired > 0
Running: select
*
Running: select
*
Running: select
*
from testDataRef where geom && GeometryFromText(POLYGON((0 0, 80 0, 80 80, 0 80, 0 0)), 4326)
Running: select
*
from testData where expired = 9223372036854775807 and geom && GeometryFromText(POLYGON((0 0, 80 0, 80 80, 0 80, 0 0)), 4326)
Running: select
*
from testData where revision <= 3999 and expired > 3999 and geom && GeometryFromText(POLYGON((0 0, 80 0, 80 80, 0 80, 0 0)), 4326)
And here are the results with 100.000 reference features, and 10000 revisions modifying each 30 records
(thus, 300.000 more records in the database), 400.000 records total:
Running: select
*
from testDataRef
Running: select
*
from testData where expired = 9223372036854775807
Running: select
*
Running: select
*
Running: select
*
Running: select
*
from testDataRef where geom && GeometryFromText(POLYGON((0 0, 80 0, 80 80, 0 80, 0 0)), 4326)
Running: select
*
from testData where expired = 9223372036854775807 and geom && GeometryFromText(POLYGON((0 0, 80 0, 80 80, 0 80, 0 0)), 4326)
Running: select
*
from testData where revision <= 9999 and expired > 9999 and geom && GeometryFromText(POLYGON((0 0, 80 0, 80 80, 0 80, 0 0)), 4326)
As you can see, despite the second run has to deal with twice the number of records in the versioned table,
timings are the same.
You may wonder where is the magic. Well, the magic is good indexes and the Posgtres 8.0 onwards newly
aquired ability to inspect multiple indexes during a single query, and do bitmap merging before accessing
the actual data (this is used by the spatial queries). This is important, results wont be
The table creation queries are here:
create table testData (id bigint, txt varchar(256), revision bigint, expired bigint not null, primary key (revision, id))
select AddGeometryColumn(testdata, geom, 4326, LINESTRING, 2)
create index testDataGeomIdx on testData using gist (geom gist_geometry_ops)
create index testDataRevIdx on testData (expired, id)
create table testDataRef (id bigint, txt varchar(256), primary key (id))
select AddGeometryColumn(testdataref, geom, 4326, LINESTRING, 2)
create index testDataRefGeomIdx on testDataRef using gist (geom gist_geometry_ops)
Observe the primary key order, which allows queries needing only revision to use the primary key as an
index, and the other index, that allows the same for expired (id is there again to help queries that have to
expire a certain record).
If you want to reproduce the test on your PC, the source of the benchmark is attached.
Versioning WFS - GT2 Datastore extensions
Versioning requires some changes on the datastore interface.
Schema handling Assuming that a vesioned datastore is able to handle both versioned and non ver-
sioned feature types and that it can make them change state, the following additions to FeatureType
getSchema(String) and String[] getTypeNames() in order to chech wheter a type is versioned, and in or-
der to transition it between the versioned and non versioned states:
boolean isVersioned(String typename) throws IOException;
void setVersioned(String typename) throws IOException;
setVersioned should be optional, throwing UnsupportedOperationException on datastore that do not pro-
vide it, and an IOException in the case where it does not make sense to version a feature type, such as the
ChangeSet one, which is used to provide logging (see Versioning WFS - Database design).
Data gathering Querying the datastore should take versioning into account.
The Query interface already provides a String getVersion which is WFS standard compliant. If no version
is provided, the latest version will be assumed.
Well use it, thought it would be nice to expand its semantics to take care of branches and time based
extractions:
1024: extract revision 1024
projectX:1024: extract revision 1024 from branch projectX
20061203122500: time based extraction using ISO like format (December 12 2006, 12:25:00)
projectX:20061203122500: as above, with branch specication.
Data writing Here things get messier. Write side of the world does not have any notion of versioning,
since Query is basically not used at all in the writing interfaces (nor DataStore, nor FeatureStore). Moreover,
here both version and user id shall be provided in order to perform the operation against a versioned
datastore.
The best guess I have at the moment is to leverage the Transaction inteface, and in particular the Trans-
action properties, thus making revision were working against and user id two well known transaction
properties the versioned datastore should use. If the two above are not available the versioned datastore
8.8. Versioning 103
can either fall back on some default user provided during datastore instantiation and last revision, or throw
an exception.
WFS Extensions
Versioned features support requires a protocol that knows about the extra features such as history and
snapshot gathering, as well as conicts and rollbacks. Here we discuss an extension to the WFS proto-
col, although a simpler REST implementation would be appealing as well and deserves further inspection
(JSON seems to be a popular choice for RESTful implementations). Were starting with WFS-T because we
already have a solid base supporting it, so we an focus on what is new, instead of writing an entirely new
API. But if there is consensus around a REST equivalent of WFS(-T) becomes we would be very interested
in supporting it and extending it for versioning.
Authentication It is advisable that calls to WFS use some kind of authentication in order to attribute
each change to the actual user that performed it. Well, in fact even a plain WFS-T should be subjected to
authentication, in order to avoid data vandalism.
In any case, the authentication process should be out of band with respect to the WFS protocol calls, for
example using HTTP basic/digest authentication. Geocollaborator discussion includes some info about
authentication libraries and schemes we could use.
It would be interesting to hear from client developers what authorization mechanisms they would be able
to support.
WFS protocol extensions The WFS protocol must be extended in order to support versioning, that is
revision awarness, logs, differences.
Yet, WFS clients unaware of versioned WFS should be able to work against treating it like a plain WFS, just
like Oracle does when an unaware client works against a versioned table:
a plain GetFeature call must return the latest revision;
Transaction, in absence of branches, works just like usual from the user point of view (but of course
creates new versioned records instead of altering existing ones);
Lock, in absence of branches, works the same as usual, too, so its intended to work against the
last revision (probably some changes are needed to handle actual record changes during the lock
duration).
Besides that, real versioning requires both changes in the existing calls, and a new set of calls as well.
Note: Warning This poses a problem, because WFS request/response elements are dened in a XML
schema. Extensions will be performed using XML schema standard mechanisms, that is, extension and
type substitution. Were possible and sensible, WFS 1.0 extension points provided for vendor parameters
will be used.
Versioning WFS will be realized on top of the Geoserver 1.6.x WFS module, which is designed priarily for
WFS 1.1, but supports 1.0 as well. The following schema elements do extend WFS 1.1 schema, but well
support WFS 1.0 as well. Extensions do not depend on specic WFS 1.1 features, thus will be applied the
same way to both WFS protocols. GetFeature
We obviosly need to be able and retrieve features at a specic revision. In WFS an hook is already available
to support versioned request, in particular in the Query element of a GetFeature request:
<xsd:complexType name="QueryType">
...
<xsd:attribute name="featureVersion" type="xsd:string" use="optional">
<xsd:annotation>
<xsd:documentation>
For systems that implement versioning, the featureVersion
attribute is used to specify which version of a particular
feature instance is to be retrieved. A value of ALL means
that all versions should be retrieved. An integer value
i, means that the ith version should be retrieve if it
exists or the most recent version otherwise.
</xsd:documentation>
</xsd:annotation>
</xsd:attribute>
...
This would make us standard compliant with WFS and limit ltering capabilities to equality, thought we
have to implement ALL as well. The documentation explicitly says either ALL or an integer must be used,
but well be more lenient and support other formats as well:
numeric version
timestamp (in some ISO standard, locale independent format)
branch:version
branch:timestamp
ALL
FIRST (rst state committed in a branch)
LATEST (head of a branch). This does not pose validation issues, since version attribute is a generic
string. LATEST should become the default value for Query featureVersion attribute, that is, if not
specied, the server acts as if the latest version is required.
Returned feature collections should provide a version number, so that clients can build a checkout and
just ask for differences in subsequent requests. This can be done by extending FeatureCollection into a
VersionedFeatureCollection, being part of the same substitution group as FeatureCollection, which reports
the last version for each feature type (given that some feature types may not be versioned, and others may
come from different versioning datastores):
<VersionedFeatureCollection ... >

...
<FeatureTypeVersions>
<FeatureTypeVersion typeName="ft1" featureVersion="1156"/>
<FeatureTypeVersion typeName="ft2" featureVersion="1256"/>

</FeatureTypeVersions>
</VersionedFeatureCollection>
Providing version numbers is an optimization that can be added later, since basic versioning functionality
is there even without checkouts.
Here we should add an extension of GetFeature that is able to return features modied by a specic user
GetLog Setting up an equivalent of svn log, given our current infrastructure, could be done by simply
exposing the ChangeSet table as a read only feature type, since it has a bbox attribute which is a geometry.
8.8. Versioning 105
There are a few drawbacks though:
its not possible to get a log for a specic feature type (since feature types modied in a tranaction are
stored in an associated transaction)
the output format is not designed for readability
its not possible to use all the formats for the version (time based, for example)
So, a new call is required, that mimics GetFeature, but allows to specify in the Query a starting and ending
feature version. This can be done by extending Query into a DifferenceQuery that supports a fromFeature-
Version attribute, and creating the log operation accordingly (by copy and paste from GetFeature, since
complex type restrictions are not widely supported by parsers).
<xsd:complexType name="GetLogType">
<xsd:complexContent>
<xsd:extension base="wfs:BaseRequestType">
<xsd:sequence>
<xsd:element ref="wfsv:DifferenceQuery" maxOccurs="unbounded"/>
</xsd:sequence>
<xsd:attribute name="resultType"
type="wfs:ResultTypeType" use="optional"
default="results">
</xsd:attribute>
<xsd:attribute name="outputFormat"
type="xsd:string" use="optional"
default="text/xml; subtype=gml/3.1.1">
<xsd:attribute name="maxFeatures"
type="xsd:positiveInteger" use="optional">
</xsd:extension>
</xsd:complexContent>
</xsd:complexType>
<xsd:complexType name="DifferenceQueryType">
<xsd:sequence>
<xsd:element ref="ogc:Filter" minOccurs="0" maxOccurs="1"/>
</xsd:sequence>
<xsd:attribute name="typeName" type="xsd:QName" use="required"/>
<xsd:attribute name="fromFeatureVersion" type="xsd:string" default="FIRST"/>
<xsd:attribute name="toFeatureVersion" type="xsd:string" use="optional" default="LAST"/>
</xsd:complexType>
GetLog has basically the same schema as GetFeatures. Having an output format choice is important to
have a variety of clients and display use it (see next paragraph). Having a maxFeatures attributes allows
to limit the number of log entries returned, and closely mimics svn log limit xxx. resultType allows to
cheaply count how many log entries are there (not sure this is necessary, may well be removed).
Query has been replaced by DifferenceQuery, a versioned companion with two differences compared to the
standard Query:
Does not have a property list, since its meant for extracting diffs and logs from a feature type, not
extracting actual features.
Has starting and ending version.
The default GetFeature output format is GML, which is good for WFS clients to map, but hard for a human
being to read. I guess a client may want to show the output in html or pure text, because in this respect the
call would be more similar to a WMS GetFeatureInfo call. So, GetLog will provide multiple representations
just like GetFeature, and support both GML and a human readable format.
GetDiff An equivalent of svn diff could be interesting because it would allow to:
Pinpoint what changed between two versions on the attribute level.
Perform rollbacks, just gather a reverse diff from the server and use it to build a transaction call.
A GetDiff call would really look like a GetFeature call, but using DifferenceQuery instead of a plain Query:
<xsd:complexType name="GetDiffType">
<xsd:extension base="wfs:BaseRequestType">
<xsd:sequence>
<xsd:element ref="wfsv:DifferenceQuery" maxOccurs="unbounded"/>
</xsd:sequence>
<xsd:attribute name="user" type="xsd:string" use="optional"/>
<xsd:attribute name="outputFormat"
type="xsd:string" use="optional"
default="application/xml; subtype=wfsv-transaction/1.1.0">
</xsd:attribute>
</xsd:extension>
</xsd:complexType>
The default result would be a Transaction call that can be applied on the server to merge the diff on another
branch (when branches will be supported) or to performa rollback. The standard (unversioned) transaction
call is especially well suited, because it allows for specication of what needs to be created, deleted and
udpated in order to update data to a specic revision, so its suggested to be the default output format. To
simplify things for clients, only d lters will be added in the resulting Transaction call (to avoid having
light clients to implement a full OGC Filter handling).
Rollback This operation is not really required if GetDiff is implemented, since GetDiff + Transaction could
be used, but:
The difference maybe be very big, creating issues on unreliable or slow networks, since the diff must
travel the network twice (rst as a GetDiff result, then as a Transaction call).
As the SQL call sampler shows, a direct server side rollback can be a lot more efcient.
Since both of these can be thought as optimizations, this could be thought as an optional feature that only
advanced Versioning WFS implementations do support.
A rollback call should identify which changes should be rolled back, so in principle it requires the revision
at which we want to roll back, and a commit message. This can be implemented as a new element type in
transaction (RollbackElementType), and is further discussed in the Transaction paragraph.
Merge The same considerations made for rollback can be made for a cross branch merge. Since the rst
implementation does not consider branches, merge wont be discussed further.
Transaction Transaction modications for versioning need to consider that a server will probably serve
both versioned and non versioned feature types, possibly have multiple versioned datastores, and that
clients using the plain WFS protocol should be allowed to participate and work against a versioned WFS
too, so modications need to be minor, and optional. In particular, there is a need for:
a place where commit message can be specied (optional).
returning the new revision information.
handling rollbacks and merges
handling conicts, at least against clients that do know about versioning.
8.8. Versioning 107
The commit message can be stored into the Transaction handle attribute. Whilst this bends a little its in-
tended usage, it also provide a reference message for clients that are unaware of versioning, but that do set
some handle message for the sake of error reporting.
New revision information in the response can be stored among the Action elements of a response, since
they are designed to carry generic messages too. It would be something like:
<wfs:TransactionResponse ...>
<wfs:TransactionSummary>
...
</wfs:TransactionSummary>
<wfs:TransactionResults>
<wfs:Action code="revision" locator="The handle provided in the Transaction request">
<wfs:Message>15213</wfs:Message>
</wfs:Action>
</wfs:TransactionResults>
<wfs:InsertResults>
...
</wfs:InsertResults>
</wfs:TransactionResponse>
Rollback and merges can be handled with new elements that leverage the vendor extension mechanisms
for Transaction elements. The new RollbackElementType would be very similar to the DeleteElementType,
with a typename and a lter, and would require just the specication of the rollbackTo revision as an extra
attribute.
<xsd:element name="Rollback" type="wfsv:RollbackType" substitutionGroup="wfs:Native">
</xsd:element>
<xsd:complexType name="RollbackType">
<xsd:extension base="wfs:NativeType">
<xsd:sequence>
<xsd:element ref="wfsv:DifferenceQuery" minOccurs="1" maxOccurs="1"/>
</xsd:sequence>
<xsd:attribute name="handle" type="xsd:string" use="optional"/>
<xsd:attribute name="user" type="xsd:string" use="optional"/>
</xsd:extension>
</xsd:complexType>
The lter allows to select which features need to be rolled back. This allows for rolling back changes in a
specic area, or with other specic characteristics. The user attribute allows for gathering only the changes
performed by a specic user, so it acts like a lter, but its separate since the user id is not among the feature
type attributes.
Finally, lets handle conicts. Version control system usually do not allow to commit a change if the server
state changed in the meantime, and thats a very basic security measure to avoid losing changes and pre-
vent conicts. But in our case, we do want to support versioned WFS unaware clients, so bypassing that
mechanism is easy: we have to accept calls that do not specify a reference revision, allowing the overwrite
of changes performed between GetFeature and Transaction (unless the client did set a Lock). A cong-
uration parameter should also allow administrators to put unaware clients out of the game, since these
unchecked calls are dangerous for data consistency. Aware clients can leverage extra checks by setting a
featureRevision on their update/delete elements, and the server should throw an exception if xxx is not the
last revision for the features hit by the update/delete lters. This means the approach is to allow clients to
leverage extra check, but without enforcing them. The extended transaction elements would be:
<xsd:complexType name="VersionedUpdateElementType" >
<xsd:extension base="wfs:UpdateElementType">
<xsd:attribute name="featureVersion" type="xsd:string" use="required">
</xsd:attribute>
</xsd:extension>
</xsd:complexType>
<xsd:complexType name="VersionedDeleteElementType" >
<xsd:extension base="wfs:DeleteElementType">
<xsd:attribute name="featureVersion" type="xsd:string" use="required">
</xsd:attribute>
</xsd:extension>
</xsd:complexType>
Reference XSD The above snippets have been gathered in an extension XSD le that can be analyzed
along with OGC ones.
Versioning WFS - Sample calls
The sample versioning conguration contains a set of sample calls that can be performed against the ver-
sioning datastore to try out its functionality. Here are the calls with some narrative explaining them.
Assess initial situation All of the calls hit the topp:archsites feature type. This call extract two of the three
feature were going to modify with the transaction.
<wfs:GetFeature service="WFSV" version="1.0.0"
xmlns:topp="http://www.openplans.org/topp"
xmlns:wfs="http://www.opengis.net/wfs"
xmlns:ogc="http://www.opengis.net/ogc"
xsi:schemaLocation="http://www.opengis.net/wfs
http://schemas.opengis.net/wfs/1.0.0/WFS-basic.xsd">
<wfs:Query typeName="topp:archsites">
<ogc:Filter>
<ogc:FeatureId fid="archsites.1"/>
</ogc:Filter>
</wfs:Query>
</wfs:GetFeature>
This is a standard WFS GetFeature, but you may have noticed that the service hit is WFSV. Also, if you
run the request youll notice archsites.25 is not there, because were going to create it with the next request.
Transaction This is a standard WFS transaction, versioning is totally transparent (that is, this transaction
call is exactly the same you would performon a standard WFS, the difference is that the backend will record
the change instead of just applying it to the data).
<wfs:Transaction service="WFSV" version="1.0.0"
xmlns:ogc="http://www.opengis.net/ogc" xmlns:wfs="http://www.opengis.net/wfs"
xmlns:gml="http://www.opengis.net/gml"
8.8. Versioning 109
http://schemas.opengis.net/wfs/1.0.0/WFS-transaction.xsd
http://www.openplans.org/topp
http://localhost:8080/geoserver/wfsv?request=DescribeFeatureType&version=1.0.0&typeName=topp:archsites"
handle="Updating Signature rock label">
<wfs:Insert>
<topp:archsites>
<topp:cat>2</topp:cat>
<topp:str1>Alien crash site</topp:str1>
<topp:the_geom>
<gml:Point srsName="http://www.opengis.net/gml/srs/epsg.xml#26713">
<gml:coordinates decimal="." cs="," ts=" ">604000,4930000</gml:coordinates>
</gml:Point>
</topp:the_geom>
</topp:archsites>
</wfs:Insert>
<wfs:Update typeName="topp:archsites">
<wfs:Property>
<wfs:Name>str1</wfs:Name>
<wfs:Value>Signature Rock, updated</wfs:Value>
</wfs:Property>
<ogc:Filter>
<ogc:FeatureId fid="archsites.1" />
</ogc:Filter>
</wfs:Update>
<wfs:Delete typeName="topp:archsites">
<ogc:Filter>
</ogc:Filter>
</wfs:Delete>
</wfs:Transaction>
The result is:
<wfs:WFS_TransactionResponse version="1.0.0" xmlns:wfs="http://www.opengis.net/wfs"
xmlns:ogc="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.opengis.net/wfs http://localhost:8080/geoserver/schemas/wfs/1.0.0/WFS-transaction.xsd">
<wfs:InsertResult>
</wfs:InsertResult>
<wfs:TransactionResult handle="Updating Signature rock label">
<wfs:Status>
<wfs:SUCCESS />
</wfs:Status>
</wfs:TransactionResult>
</wfs:WFS_TransactionResponse>
Grabbing the versioning log The following call retrieves the change log for the topp:archsites feature
type:
<wfsv:GetLog service="WFSV" version="1.0.0"
outputFormat="HTML"
xmlns:wfsv="http://www.opengis.net/wfsv"
xsi:schemaLocation="http://www.opengis.net/wfsv
http://localhost:8080/geoserver/schemas/wfs/1.0.0/WFS-versioning.xsd">
<wfsv:DifferenceQuery typeName="topp:archsites" fromFeatureVersion="1" toFeatureVersion="13"/>
</wfsv:GetLog>
resulting in a single entry (the entries from 1 to 6 report the operations needed to version enable the tables).
Note how the transaction handle was used as the commit message, and also the fact the user is unknown,
because in this demo the http basic authentication was not used.
Revision Author Date Message
7 anonymous 10/10/07 9.47 Updating Signature rock label
Alternatively, the GetLog result can be encoded in GML. GetFeature with version support
Lets do a before and after comparison. This rst GetFeature tries to retrieve the value of the features at
revision 1.
outputFormat="GML2"
<wfs:Query typeName="topp:archsites" featureVersion="1">
<ogc:Filter>
</ogc:Filter>
</wfs:Query>
</wfs:GetFeature>
This one retrieves the results at revision 1, that is, before the transaction occurred. We can see features
archsites.1 and archsites.2, whilst archsites.26 is not there (it has been created at revision 7).
<wfs:FeatureCollection xmlns="http://www.opengis.net/wfs" xmlns:wfs="http://www.opengis.net/wfs"
xmlns:topp="http://www.openplans.org/topp" xmlns:gml="http://www.opengis.net/gml"
xsi:schemaLocation="http://www.openplans.org/topp http://localhost:8080/geoserver/wfs?service=WFS&version=1.0.0&request=DescribeFeatureType&typeName=topp:archsites http://www.opengis.net/wfs http://localhost:8080/geoserver/schemas/wfs/1.0.0/WFS-basic.xsd">
<gml:boundedBy>
<gml:Box srsName="http://www.opengis.net/gml/srs/epsg.xml#26713">
<gml:coordinates xmlns:gml="http://www.opengis.net/gml" decimal="." cs="," ts=" ">
591950,4914730 593493,4923000
</gml:coordinates>
</gml:Box>
</gml:boundedBy>
<gml:featureMember>
<topp:archsites fid="archsites.1">
<topp:str1>Signature Rock</topp:str1>
<topp:the_geom>
593493,4914730
</gml:coordinates>
</gml:Point>
</topp:the_geom>
8.8. Versioning 111
</topp:archsites>
</gml:featureMember>
<gml:featureMember>
<topp:str1>No Name</topp:str1>
<topp:the_geom>
591950,4923000
</gml:coordinates>
</gml:Point>
</topp:the_geom>
</topp:archsites>
</wfs:FeatureCollection>
Querying for revision 7 offers a different result:
outputFormat="GML2"
<wfs:Query typeName="topp:archsites">
<ogc:Filter>
</ogc:Filter>
</wfs:Query>
</wfs:GetFeature>
<wfs:FeatureCollection xmlns="http://www.opengis.net/wfs" xmlns:wfs="http://www.opengis.net/wfs"
xmlns:topp="http://www.openplans.org/topp" xmlns:gml="http://www.opengis.net/gml"
xsi:schemaLocation="http://www.openplans.org/topp http://localhost:8080/geoserver/wfs?service=WFS&version=1.0.0&request=DescribeFeatureType&typeName=topp:archsites http://www.opengis.net/wfs http://localhost:8080/geoserver/schemas/wfs/1.0.0/WFS-basic.xsd">
<gml:boundedBy>
<gml:Box srsName="http://www.opengis.net/gml/srs/epsg.xml#26713">
593493,4914730 604000,4930000
</gml:coordinates>
</gml:Box>
</gml:boundedBy>
<gml:featureMember>
<topp:str1>Alien crash site</topp:str1>
<topp:the_geom>
604000,4930000
</gml:coordinates>
</gml:Point>
</topp:the_geom>
</topp:archsites>
<gml:featureMember>
<topp:str1>Signature Rock, updated</topp:str1>
<topp:the_geom>
593493,4914730
</gml:coordinates>
</gml:Point>
</topp:the_geom>
</topp:archsites>
</wfs:FeatureCollection>
Here we can see archsites.2 has been removed (deleted during the transaction) and archsites.26 appears.
GetDiff Diff returns the difference between two revisions, eventually its possible to specify a lter to
gather the diff concerning a specic feature set. Also notice that the output format is HTML, but if you
dont specify it, youll get the log encoded as a WFS Transaction (the transaction that applied to the initial
revision bring you to the specied destination revision).
<wfsv:GetDiff service="WFSV" version="1.0.0"
outputFormat="HTML"
<wfsv:DifferenceQuery typeName="topp:archsites" fromFeatureVersion="1"/>
</wfsv:GetDiff>
The output, in human readable HTML format, is:
Feature type archsites, diff from version 1 to version CURRENT
Feature archsites.26, inserted, feature content:
*
cat: 2
*
str1: Alien crash site
*
the_geom: POINT (604000 4930000)
Feature archsites.2, deleted, old feature content:
*
cat: 2
*
str1: No Name
*
the_geom: POINT (591950 4923000)
Feature archsites.1, updated, modified attributes:
+-----------+------------------+------------------------+
| Attribute | Value at 1 | Value at CURRENT |
+===========+==================+========================+
| str1 | Signature Rock |Signature Rock, updated |
+-----------+------------------+------------------------+
8.8. Versioning 113
If no output format is specied, the GetDiff will return a WFS transaction, that is, the actions needed to
turn the features from version 1 to version CURRENT. The GetDiff can also work backwards, that is, one
could specify toFeatureVersion < fromFeatureVersion, this would return a reverse diff, which seen as a
transaction, is the set of command needed to rollback the changes. Yet, its advisable to use the specic
Rollback element for actually performing rollbacks, because it guarantees deleted features are restored with
the original feature id (perfoming the transaction returned by a backwards diff would create a new feature
id instead).
Rollback Versioning WFS introduces a new transaction element, Rollback, that can be used to undo
changes performed between two revisions, eventually limiting the rollback to features touched by a specic
user, or satisfying a specic lter.
handle="Rolling back previous changes">
<wfsv:Rollback safeToIgnore="false" vendorId="TOPP" typeName="archsites" toFeatureVersion="1"/>
</wfs:Transaction>
and the result would be:
<wfs:WFS_TransactionResponse version="1.0.0" xmlns:wfs="http://www.opengis.net/wfs"
xmlns:ogc="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.opengis.net/wfs http://localhost:8080/geoserver/schemas/wfs/1.0.0/WFS-transaction.xsd">
<wfs:InsertResult>
<ogc:FeatureId fid="none" />
</wfs:InsertResult>
<wfs:TransactionResult>
<wfs:Status>
<wfs:SUCCESS />
</wfs:Status>
</wfs:TransactionResult>
</wfs:WFS_TransactionResponse>
Versioning aware clients and transactions Versioning aware clients should try to handle updates keeping
the possibily of conicts in mind. A conict occurrs if the same feature is modied by two users in the same
time frame. Suppose the rst one managed to commit its changes. The second one should try to avoid
overwriting the rst user changes, and instead be notied that it does not have the latest revision. In that
case, the client should rst update (by getting a diff), handle the eventual conict locally, and then try to
commit again.
The following example shows a transaction that should fail, because the client declares of being at a revision
number thats no more the latest revision:
handle="Trying an update with wrong version">
<wfsv:VersionedUpdate typeName="topp:archsites" featureVersion="1">
<wfs:Property>
<wfs:Name>str1</wfs:Name>
<wfs:Value>You wont see me updated</wfs:Value>
</wfs:Property>
<ogc:Filter>
</ogc:Filter>
</wfsv:VersionedUpdate>
</wfs:Transaction>
Phase two implementation proposal
Branching support could be added to the previous design by using an approach similar to Oracle and
ArcSDE, that is, tagging each path in the versioning tree with a number, and creating a new one each
time a tag or a branch is created. This number is called componentId here.
The BranchTag table holds both tag and branch names (branch is used to discriminate branches and tags)
and ComponentLineage holds a set of all fathers of a specic path component so that the whole path that
let to a specic branch can be rebuilt.
Of course all the above queries can be modied to take into account branches adding extra subqueries, and
thus adding a performance overhead. On the bright side, this schema allows for cheap branching (since it
does not have to create extra row copies).
Taking into consideration branches in the API The DataStore API provides reading hooks for versioning
using Query.getVersion() which is a string and can be freely intepreted. So, if we wanted to encode version
10 of branch gizmo we would use gizmo:10.
Writing does not use Query at all. In the no branch case, all the relevant informations such as user and
commit message have been encoded as Transaction properties. Version is no need in this picture, because
we cannot write again a specic revision, but only against latest. When branch comes into the picture, we
see that branch information need to be encoded as a transaction property, leaving to a bit of inconsistency,
because in the reading case it makes more sense to have it in the Query (it matches the WFS level) whilst
8.8. Versioning 115
in the writing case we need to encode it in the transaction. Moreover, phase one did not leave any place to
store the branch information, unless we devise some fancy way to provide that one along with the handle
(not really good looking). This leaves us out to encode branch information in a new attribute, something
we tried to avoid with commit message included as the handle.
8.8.2 GeoServer Synchronization Service
GeoServer Synchronization Service is a distributed version control system managing synchronization be-
tween a number of nodes, called unites, and a central orchestrating node called central. The following
documents provide insight on its design, current implementation status and usage:
Introduction
The GSS (GeoServer Synchronization Service) module has been developed to support a geographic dis-
tributed versioning system in a proof of concept for a larger data distribution and editing project. As such
it proves feasibility of the distributed version control system, but does not provide any friendly tooling and
is setup with strong limitations in mind:
each node data is edited directly via a customized uDig that can handle and resolve conicts. No
extensions to the existing WFS or versioned WFS module has been created to handle conicts, they
are to be solved directly on the PostGIS database (to which the customized uDig connects directly)
the setup requires a central node that orchestrates the synchronization, there is no peer to peer support
there is no support for databases other than PostGIS, and even that case is limited to tables that have
no referential integrity
General system structure
The GSS module goal is to keep in sync a distributed network of GeoServer instances, each using the same
layers and each storing the layers in a PostGIS versioning data store.
Each node in the network is using a versioning PostGIS datastore, it has its own history and revision num-
bering, however the changes occurred on one node can propagate to all the other nodes (see Layers for the
propagation rules).
The network is composed of three types of nodes:
unit: its a generic node where data modications take place
central: its the central node that performs all the synchronization based on a predetermined schedule.
No one can directly edit on central, every change gets pulled in from units instead.
administration unit: a special unit that can edit administration tables that will drive central
The GSS module will be a single one, it will be congured to play the necessary role on each node.
Central polls all of the units in a round robin fashion, grabs changes fromthemand pushes the changes that
it got from the other units to the remote unit. This means central has knowledge of all the units available
in the network. It also means that, due to synchronization lag, a unit own database contents wont be the
same as the central one, it will miss some modications made by other units and will have some local
modications that have not made into the central database yet.
This may result in conicting changes. See Conicts for details about the nature and handling of a conict.
Each unit will be equipped with a customized uDig that will connect directly to PostGIS using a versioning
PostGIS data store. This client will understand the contents of the administration tables of the node (in
particular, the conicts one) and enforce special editing. This means, in particular, that the GeoServer
running in each unit will have no other services running other than GSS itself.
Layers
The system handles three types of layers:
Published layers: published layers can be edited only by the administration unit, every other unit
receives changes from central but cannot perform local ones. Its a publish-subscribe model from
central to the non administration units. The publish only nature of the layer will be enforced by the
uDig client, that wont allow the layer to be edited.
Backup layers: backup layers can be edited by every unit, but are setup so that each unit has its own
records that no other unit can modify, and vice versa (by using a unique identier that labels the
record as such). In this case we have a publish-subscribe model from units to central, where central
acts as a overview and backup mechanism.
Synchronized layers: these layers can be edited by all remote units at the same time and thus changes
to features will be traded in both directions.
Examples of published layers:
base layers prepared by the administration unit
Examples of backup layers:
conict tables
Examples of synchronized layers:
All shared layers that are supposed to be edited by the units and that have competency overlap be-
tween the units
Conicts
A conict arises when a change on a feature goes both ways during a synchronization, that is, when central
is trying to push a change on a feature that has been also modied in the unit since the last successful
synchronization. In that case the change coming from central is applied and the unit local one is registered
in a special conict table that the client will use to show the user the differences in changes and allow the
user to act on it.
In case of a conict the user can either:
accept the change coming from central thus ignoring the local one
force the local change to overwrite the one coming from central
eventually merge the two, accepting parts of each.
This results in a new modication to the feature that will either ow back to central during the next syn-
chronization or cause a new conict, in case another modication is coming down from central.
The uDig client will interpret the contents of the local CONFLICTS table and wont allow direct editing of
a feature marked as in conict until the conict is resolved. The client will provide the user with tools to
identify and x the conicts interactively.
8.8. Versioning 117
Connectivity map
The connectivity map shows the geographic area associated to each unit with a color coding expressing the
freshness of the last successful synchronization. It will be generated starting from housekeeping informa-
tion that Central keeps up to date each time a synchronization occurs.
Note: The current implementation does not provide a connectivity map, though it is not hard to add one
GeoServer conguration
Each unit GeoServer will be locked down so that no actual service, besides GSS itself, will be available. The
GSS communications will be using a predetermined user-name and password with no integration with the
eventual user table.
Known limitations
This design assumes there will not be any layer creation to be propagated and that there will be no layer
structure changes either. If those needs arise the tables will have to be created off line on the entire network
before starting to use them, and database structural changes will have to be performed manually with the
synchronization protocol put on hold for the entire operation.
The GeoTools data store model upon all this work is based is simply not referential integrity aware, cannot
deal with linked tables. As a result no attempt will be made to handle data with referential integrity con-
straints (the store can be modied to support referential integrity, but that will require a complete overhaul
of the storage structures and code accessing them).
Table structures will be modied when they are version enabled (in particular, primary key will change)
so existing foreign keys will loose their meaning and/or cause the system to misbehave (it is possible to
change the structure of the versioning tables so that the original table is left untouched, that will require a
complete overhaul of the storage structures and code accessing them).
Core synchronization protocol
Protocol overview
Central will contact each unit in a periodic fashion and synchronize all of the shared tables, each one ac-
cording to its nature. The interval of synchronization will be different for each unit to take into account the
different connectivity features of each installation (speed, delay, reliability). For each unit and each layer
the protocol will use two calls from central to the unit:
1. (Central Unit) GetCentralRevision(Layer). Central asks the Unit about the last Central revision
that the Unit got differences to. Unit responds with the revision number.
2. (Central Unit) PostDiff(Layer, from, to, transaction). Central posts to Unit a synchronization
document, the set of differences occurred on Layer between the last known revision number known
to Unit and the current one. Unit compares that with the local modications occurred since the last
PostDiff call, nds out conicts, applies only the non conicting changes, stores the conicts in the
conict table, marks down in a history table the new revision number, the list of ids of features under
conict, and the new central revision number
3. (Central Unit) GetDiff(Layer, from). Central asks for the differences occurred since the last known
Unit revision number. Unit builds the difference skipping along the way all the commits incoming
from Central as well as all the changes resulted in conicts and returns the cleaned up difference as a
synchronization document to Central. Central applies the differences and marks down the new Unit
revision in a metadata table.
In case of connectivity failure the unit will be rescheduled for synchronization at a later time. All the three
types of tables can use the protocol. For some types of tables the differences in one of the two calls will be
consistently empty.
Protocol restart
The network between Central and the Unit cannot be trusted to be reliable, thus the protocol need to take
into account the need of partial synchronization restarts.
The rst call, GetCentralRevision, is read only on both sides, so repeating it wont cause any issue.
The second call, PostDiff, is read only on the Central side and write only on the Unit side. Failure scenarios:
If the connection drops whilst the synchronization document is being sent Unit wont perform any
local change, thus the protocol can be restarted without problems.
If Unit manages to commit the changes but the connection drops before it can return the OK, the proto-
col can be safely restarted from the beginning, it will simply result in multiple updates accumulating
on the Unit side
The third call, GetDiff, is read only on the Unit side and write only on the Central side. Failure scenarios:
If the connection drops while the request is being sent the protocol can be restarted from the begin-
ning, or be restarted from GetDiff itself, without issues
If the connection drops while the synchronization document is being returned the server side wont
commit the changes and, again, the protocol can be restarted either from GetDiff or from the begin-
ning.
Synchronization document
A synchronization document is an XML document stating the layer being synchronized, the revisions at
which the difference starts and ends, and the difference itself, expressed as a WFS 1.0 transaction summa-
rizing all the changes occurred:
<gs:Synchronization typeName=parcels fromRevision=10 toRevision=12>
<wfs:Transaction>
<wfs:Insert>
...
</wfs:Insert>
....
</wfs:Transaction>
<gs:Synchronization>
Computing the synchronization document requires computing all the changes accumulated since the last
successful synchronization, but it requires skipping the changes that occurred in it, since those are changes
coming from the other side, that the other side is already aware of.
On the server side this is easy, it is sufcient to record the last revision at which the changes coming from
a Unit in a particular layer occurred, and then skip them when generating the differences for the Unit. By
virtue of the protocol structure, there will be at most one to skip (since the changes coming from the Unit
are at the end of the synchronization).
8.8. Versioning 119
On the client side its harder, as there is no way to guarantee there will just be one PostDiff in the queue.
If a PostDiff succeeds on the Unit but the message fails to go back to Central (timeout, network failure) the
latter will issue another PostDiff some time after that, which could contain more changes. So on the Unit
side more synchronizations and local changes can intermingle.
This is the reason why Unit records all successful PostDiff calls, and it will use that record to create a
synchronization document that contains only the changes occurred locally up to the last successful PostDiff.
The latter requires an explanation: why only up to the last successful synchronization? Because the uDig
client, operating in parallel, might have lined up more changes between the last PostDiff and GetDiff,
meaning some of them might introduce conicts. Those will be taken into consideration only after being
compared with a new PostDiff from the server.
Network communications
All of the communications between Central and the various Unit will happen following an OGC style:
HTTP protocol using GET and POST requests, HTTP basic authentication, and exchange of XML documents
between the two sides.
Protocol granularity and concurrency
The synchronization protocol is layer based, meaning the number of communications required to fully
synchronize a unit is three times the number of layers to be synchronized, each time paying a network
delay.
The choice of layer granularity is taken in order to avoid exchanging and parsing big XML documents,
as well as avoiding dealing with big amounts of data during the conict computation (both are memory
bound at the moment).
In order to work properly the protocol cannot update in parallel the same layer from two different units
(thus the communication must be driven by Central as opposed as from the units, the latter would require
locking). Its however possible to synchronize two layers in parallel, either from the same unit or from two
different units, as the updates on those would be independent of each other.
The rst implementation will use a simple linear scan, but a more sophisticated approach can be taken
in the future to reduce the time needed to perform a synchronization and better use the available Central
network capacity.
Primary key generation
Primary key generation is happening all over the distributed network, features generated in the various
nodes will eventually be merged in Central and distribute to the rest of the units.
This means the system has to make sure no two primary key will ever conict. This section discusses three
possible algorithms for conict avoidance
GUID based
Most distributed version control systems (Git, Mercurial) use Globally Unique Identiers (GUID) to mark
items that have to be uniquely identied across the distributed network.
A GUID is a 128bit number generated by an algorithm providing uniqueness guarantees across a dis-
tributed network. Often, but not always, the algorithm uses the Ethernet card MAC address and a very
precise notion of the current time to generate such identier.
The positive side of this approach is that it does not require any conguration nor synchronization protocol
modications, the downside is that the generated number is long and hard to read.
Explicit discriminator
An explicit discriminator approach uses a locally unique number (such as the one generated by a local
sequence) plus a separate identier for the Unit. Assuming unitId is a number the following expression
will generate unique ids:
unitId
*
1000000000000 + localSequenceValue
This expression generates 128bit numbers just like a GUID, allows for up to one million different units, and
makes it possible to know which unit generated a certain feature by just looking at the identier. The main
downside is that it requires conguration of a unique unitId, if by mistake two Units are congured with
the same id the synchronization will consistently fail.
Central replacement
In this approach all units will generate locally unique ids using sequences. When the synchronization
happens the GSS in central will look at the newly inserted features and generate a new unique identier
based on one or more local sequences, and then send back to the Unit a replacement id for each feature.
The advantage of this approach is lack of conguration and the usage of standard 64 bit sequences.
There are two signicant downsides to this approach:
It makes the synchronization protocol more complex, and makes it harder to cleanly restart in face of
network communication issues. An extra call is required, as well as the storage of the id replacement
table in order to restart the protocol shall this extra exchange fail to take place due to a network issue.
If the synchronization happens as a user is still editing the features the change of id will break the
user edits requiring a restart of the operation with consequent time lost and user frustration.
Chosen implementation
The systemwill be initially implemented along the GUIDapproach, which guarantees key uniqueness with
the least effort and the best guarantees of proper operation.
Database tables
Central management tables
Central needs to maintain a list of all the units available around in order to schedule and perform the
updates.
8.8. Versioning 121
Table 8.1: UNITS
Colum Type Description
UnitId Long A unique code that identies the unit
Name Var-
char(256)
The name of the management unit. Used in reports and in the connectivity
map.
Address Var-
char(1024)
Full URL of the GSS service hosted on the Unit (as
http://host:port/path/to/gss, usually http://host:8080/geoserver/gss)
User Var-
char(256)
The user name used to log into the units GSS to perform synchronization (null
if no basic authentication is required)
Pass-
word
Var-
char(256)
The password used to log into the units GSS to perform synchronization.
TimeS-
tart
Time Beginning of the contact window
TimeEnd Time End of the contact window
Interval Integer Synchronisation interval (how many minutes to wait between one
synchronization cycle and the next)
AreaOfIn-
terest
Polygon The geographic area managed by the unit (used for the connectivity map)
Errors Boolean Flag raised when the last synchronization failed
Central also need to maintain a list of tables that need to be synchronized with each unit.
Table 8.2: UNIT_TABLES
Column Type Description
UnitId Long A unique code that identies the unit
TableId Long The identier of the table to be synchronized
LastSynchronization Time Time of the last successful synchronization
LastFailure Time Time of the last failed synchronisation
LastCentralRevision Long Central revision number at the last successful synchronization
LastUnitRevision Long Unit revision at the last successful synchronization
Finally each table needs to be described:
Table 8.3: SYNCH_TABLES
TableId Long Identier
Name Varchar(256) The table name
Type Char(1) The type of table: P (published), B (backup), 2 (two way synchronized)
Since the system design does not take into consideration the creation of new tables there wont be a need to
synchronize the tables with the clients.
Unit management tables
Each unit also needs to keep track of the layers to be synchronized:
TableId Long Identier
Name Varchar(256) The table name
Type Char(1) The type of table: P (published), B (backup), 2 (two way synchronized)
The following table records the synchronization history and allows to generate a clean GetDiff response to
Central:
Table-
Name
Var-
char(256)
The table to be synchronized
LocalRe-
vision
Long The local revision number at which the Central PostDiff has been committed
(might be null if the Central difference document was empty)
Central-
Revision
Long The central revision number included in the PostDiff operation
Table 8.6: SYNCH_CONFLICTS
TableName Var-
char(256)
The table containing the conicting feature
FeatureId Long The identier of the conicting feature
LocalRevi-
sion
Long The local revision before the central incoming change generated the
conict
Resolved Boolean Conict resolution ag
Difference Text The difference that would bring the current central feature to the one
edited locally
Known limitations
The current implementation is a proof of concept with these major objectives:
prove that synchronization can be attained
make it solid against network failures
make it implementable in a short time
There are other important objectives that have been sacriced in order to attain the above objectives.
Someone needs to create the tables to be synchronized manually on both ends, and then register them in
the proper metadata tables. Would be nicer if Central could post a feature type description to the units
and have the tables be created instead (obvious issues: mismatch between raw SQL types and what can be
expressed in a XML schema).
The service has no support for modication of the table structure.
No support for foreign keys.
The service assumes the feature type schemas on both sides are the same, there are no checks.
At the moment there is some redundancy in the manual work of setting up the synchronized tables. Central
could learn which tables the unit wants to synch up via a GetCapabilities of sort for example, and/or could
push a list of tables to be synchronized
The Central part has no service protocol, though it might be interesting to have a way to interact with it:
get a list of units and layers (capabilities)
informations about latest synchronization per unit or per layer
run manually the synchronization per unit, per layer (as opposed to waiting the service to do it auto-
matically)
8.8. Versioning 123
The service obviously lack a GUI to setup layers to be synchronized on the unit side and units, layers and
synchronization details (frequency, time windows) on the central side. At the moment everything has to
be setup manually in the database. The system requires usage of UUID primary keys in all synchronized
tables. There is the issue of generating them for data loading purposes. PostgreSQL has an installable
module that allows for the generation of such keys1, otherwise the table can be setup in PostGIS using
manual SQL and then it can be populated using a script leveraging versioning PostGIS datastore to insert
the data (even before version enabling the table).
8.9 Security
This section provides an overview of the GeoServer security subsystem api.
8.9.1 Security Manager
The GeoServerSecurityManager class is the main facade for the security subsystem. It plays a simi-
lar role to that of the GeoServer and Catalog interfaces for the conguration and catalog subsystems
respectively. Some of the duties of this class include:
Access to security services such as user group and role services, authentication providers, etc...
Manage the lifecycle of security services
CRUD operations for security service congurations
Access to various singleton classes such as ResourceAccessManager, KeyStoreProvider,
AuthenticationCache, GeoServerSecurityFilterChain, etc..
Implement the spring security AuthentictionProviderManager interface providing the list of active
Authentication Provider instances.
8.9.2 Security Services
GeoServerSecurityService is the base interface for all security services, including:
User/group services
Role services
Authentication lters
Authentication providers
Master password providers
The interface provides some common methods for all security services, including:
void initializeFromConfig(SecurityNamedServiceConfig config);
Every type of security service corresponds to a conguration object for that service. The conguration
object is a simple java bean that contains the service conguration. These objects are persisted via xstream
in the GeoServer data directory. See Security Plugin Provider for more details.
Security services are created via methods provided by the securty manager. For instance to create a new
xml user group service:
GeoServerSecurityManager mgr = ...;
XMLSecurityServiceConfig config = new XMLSecurityServiceConfig();
config.setName("foo");
config.setFilename("users.xml");
mgr.saveUserGroupService(config);
An instance of a security service is obtained by looking it up by the name matching its conguration object:
GeoServerUserGroupService ugService = mgr.loadUserGroupService("foo");
The cong object for the service can be looked up in a similar way:
SecurityUserGroupServiceConfig config = mgr.loadUserGroupServiceConfig("foo");
And modied accordingly:
config.setPasswordEncoderName("digest");
mgr.saveUserGroupServiceConfig(config);
UserGroup Service/Store
The GeoServerUserGroupService and GeoServerUserGroupStore interfaces provides a database
for users and groups. A GeoServerUserGroupService may be read only in that it acts solely as a source
without the ability to create new or edit existing users and groups.
The interfaces provide access to users and groups:
GeoServerUserGroupService ugService = mgr.loadUserGroupService("default");
ugService.getUsers();
GeoServerUser user = ugService.getUserByUsername("admin");
ugService.getGroups();
GeoServerUserGroup group = ugService.getGroupByGroupname("users");
The service interface advertises whether it is read only:
if (ugService.canCreateStore()) {
GeoServerUserGroupStore store = ugService.createStore();
store.addUser(new GeoServerUser("bob"));
store.store();
}
The GeoServerUserGroupService implements the spring security UserDetailsService interface in
order to integrate with existing facilities such as remember me services which require a user details service
for loading user information at runtime.
Role Service/Store
GeoServerRoleService and GeoServerRoleStore provide a database for roles and role associations
for users and groups. Like user group services a GeoServerRoleService may be read only:
GeoServerRoleService roleService = mgr.loadRoleService("default");
roleService.getRoles();
roleService.getRolesForUser("admin");
roleService.getRolesForGroup("users");
8.9. Security 125
if (roleService.canCreateStore()) {
GeoServerRoleStore store = roleService.createStore();
GeoServerRole role = new GeoServerRole("ROLE_FOO");
store.addRole(role);
store.associateRoleToGroup(role, "users");
store.store();
}
Authentication Provider
GeoServerAuthenticationProvider is an extension of the spring security
AuthenticationProvider interface and is responsible for performing authentication of user cre-
dentials.
The class extends the AuthenticationProvider contract and provides methods for authentication that
provide access to the current request to make it easier for providers that require request information to
perform authentication:
@Override
public final Authentication authenticate(Authentication authentication)
throws AuthenticationException {
return authenticate(authentication, request());
}
/
**
*
Same function as {@link #authenticate(Authentication)} but is provided with
*
the current request object.
*
/
public abstract Authentication authenticate(Authentication authentication,
HttpServletRequest request) throws AuthenticationException;
The list of active authentication providers is maintained by the GeoServerSecurityManager which extends
the spring security AuthenticationProviderManager interface.
Authentication Filter
GeoServerSecurityFilter is the base class for servlet lters that play a part in the authentication pro-
cess. Such lters can play two roles. The rst is to gather authentication credentials to passed off to a
provider for actual authentication. An example would be a lter for doing HTTP basic auth.
The second role is to perform pre-authentication in which involves doing authentication by recognizing
authentication that has already taken place outside of GeoServer. An example would be when using
a security proxy such as Siteminder or a J2ee authentication which involves delegating to the servlet
container for doing authentication.
Security lters are maintained in the lter chain which maintains the mapping of the lters to be applied to a
specic type of request. For example the lters applied to a web ui request are different than those applied
to an OGC or REST request.
Password Policy
PasswordPolicy is the interface for validating user passwords,applying constraints such as password
length, character mix, etc...
MasterPasswordProvider
Security service that provides a method for obtaining the GeoServer master password. The master pass-
word serves two purposes.
1. Is the password for the GeoServer root account
2. Protects the GeoServer keystore that is used to store encryption keys
8.9.3 Security Plugin Provider
The GeoServerSecurityProvider is the actual extension point that allows for the plugging in of in-
stances of the services discussed above. The single interface covers all the security services.
For each type of security service the provider has two methods to implement. For example with a user
group service:
public Class<? extends GeoServerUserGroupService> getUserGroupServiceClass() {
return null;
}
public GeoServerUserGroupService createUserGroupService(SecurityNamedServiceConfig config)
throws IOException {
return null;
}
The rst method reports on the specic class of user group service it implements.
This is how a specic security provider is chosen from a specic conguration object.
SecurityNamedServiceConfig.getClassName() is used to locate the provider.
8.9. Security 127
The second method creates an instance of the security service from a specied conguration object.
Providers are registered via spring, for example:
<bean id="ldapSecurityProvider" class="org.geoserver.security.ldap.LDAPSecurityProvider">
<constructor-arg ref="geoServerSecurityManager"/>
</bean>
8.9.4 Security Conguration
Service Conguration
As mentioned above each type of security service corresponds to a conguration class. The
SecurityNamedServiceConfig is the base class for all such conguration classes and maintains three
properties that all classes inherit. The rst is name for the conguration:
/
**
*
The name of the service.
*
/
String getName();
This name is used to reference both the conguration directly, or to the corresponding service implementa-
tion. For example consider a user group service named foo:
GeoServerUserGroupService service = mgr.loadUserGroupService("foo");
SecurityUserGroupServiceConfig config = mgr.loadUserGroupServiceConfig("foo");
The second property is the fully qualied class name of the service implementation that the cong object
corresponds to:
/
**
*
Name of class for implementation of the service.
*
/
String getClassName();
For instance consider creating an XML user group service:
XMLUserGroupServiceConfig config = new XMLUserGroupServiceConfig();
config.setClassName(XMLUserGroupService.class.getName());
The third property is an internal identier, similar to how catalog and conguration objects have an id:
/
**
*
Internal id of the config object.
*
<p>
*
This method should be used by client code.
*
</p>
*
/
String getId();
The main purpose of this id is to detect if the security service cong has been persisted or not.
Service conguration objects are persisted in the geoserver data directory under the security directory.
Under security are sub directories for each service type:
security/
auth/
filter/
masterpw/
pwpolicy/
role/
usergroup/
Under each directory are additional subdirectories for each named service of that type. For example, out of
the box GeoServer security is congured with the following:
Single user/group service named default
Single role service named default
Single authentication provider named default
This would correspond to the following directory structure:
security/
auth/
default/
role/
default/
usergroup/
default/
Lets say an additional authentication provider named ldap was added. The tree would look like:
security/
auth/
default/
ldap/
.
.
.
Inside each named conguration directory is a le named config.xml that contains the direct xstream se-
rialization of the conguration object. For example the default user group service conguration is persisted
in the le security/usergroup/default/config.xml and looks like:
<userGroupService>
<id>7aacccc3:13660a38ccb:-7ffd</id>
<name>default</name>
<className>org.geoserver.security.xml.XMLUserGroupService</className>
<fileName>users.xml</fileName>
<checkInterval>10000</checkInterval>
<validating>true</validating>
<passwordEncoderName>pbePasswordEncoder</passwordEncoderName>
<passwordPolicyName>default</passwordPolicyName>
</userGroupService>
Global Conguration
Aside from conguration objects for the various security services is the SecurityManagerConfig class
which provides the same function but for global security settings. It contains a number of conguration
properties such as the active role service, the list of authentication providers and lters making up the
active Authentication Chain, and conguration for remember me services.
Interacting with the global conguration is much like interacting with a security service conguration:
SecurityManagerConfig config = mgr.getSecurityConfig();
config.setEncryptingUrlParams(false);
8.9. Security 129
config.getAuthProviderNames().add("ldap");
config.saveSecurityConfig(config);
8.9.5 Authentication Chain
The GeoServerSecurityFilterChain class is a data structure that maintains mappings from request
type to a list of named security lters. This class is persisted with the rest of the global security conguration
as available as a property of the SecurityManagerConfig class:
GeoServerSecurityFilterChain filterChain = mgr.getSecurityConfig().getFilterChain();
The lterChain is essentially a map whose keys are strings corresponding to ant request patterns. The
values are lists of _auth_lter names.
GeoServerSecurityFilterChainProxy is an extension of the spring security FilterChainProxy
and is responsible for creating the actual lter chain from the GeoServerSecurityFilterChain cong-
uration object.
8.10 App-Schema Online Tests
The ofine tests in app-schema-test suite use properties les as data source. In reality, properties les are
only used as testing means, whereas in production, users would use databases as data source. Users would
often encounter problems/bugs that cannot be recreated using properties les, which raises the need to run
with a test database. Moreover, Niels joining support to increase performance can only be tested online,
and we need to ensure that it works with the current features/bug xes covered in the tests.
8.10.1 Prerequisites
This requires installation of Oracle driver in Maven repository:
mvn install:install-file -Dfile=ojdbc14.jar -DgroupId=com.oracle -DartifactId=ojdbc14 -Dversion=10.2.0.3.0 -Dpackaging=jar
You would also need to have test databases for both Oracle and Postgis. Then follow these steps:
Create oracle.properties and postgis.properties in {user directory}/.geoserver directory.
Populate each properties le with database details, e.g.:
password=onlinetestuser
passwd=onlinetestuser
user=onlinetestuser
port=5432
url=jdbc\:postgresql\://localhost:5432/onlinetest
host=localhost
database=onlinetest
driver=org.postgresql.Driver
dbtype=postgisng
8.10.2 Running tests from Maven
Without specifying any prole, the default Maven conguration for app-schema-test is to run ofine tests
only.
To run online tests, enable the prole:
-Papp-schema-online-test
This prole enables the data reference set tests and ofine tests to run online. Data reference set tests are
online tests based on data and use cases from GeoScience Victoria. Each is explicit for a database type
(Oracle and Postgis) and has a copy to run with joining enabled.
The ofine tests are congured to run online with joining through separate modules for each database:
app-schema-oracle-test and app-schema-postgis-test. These modules are placeholders for pom.xml les
containing database specic parameters. This makes it easy to identify when a test fails with a particular
database when running from Maven/buildbot.
Memory requirements
The online tests require more memory than usual, so specifying the usual -Dtest.maxHeapSize=256m is not
enough. Specify Dtest.maxHeapSize=1024m instead.
When the build is successful, you would see this in the Reactor Summary:
[INFO] Application Schema Integration Online Test with Oracle Database SUCCESS [5:52.980s]
[INFO] Application Schema Integration Online Test with Postgis Database SUCCESS [1:42.428s]
8.10.3 Running tests from JUnit
There is no need to import the online test modules as they are empty and you cannot run the tests through
them in Eclipse.
To run ofine tests (in app-schema-test/src/test/java/org/geoserver/test) with a test database, enable
joining and specify the database. Add these parameters in VM Arguments for postgis:
-Dapp-schema.joining=true -DtestDatabase=postgis -Xmx256m
Similarly, to test with oracle:
-Dapp-schema.joining=true -DtestDatabase=oracle -Xmx256m
Additionally for Oracle, you also need to add ojdbc14.jar in the test Classpath.
Note: Please note that you should only run the tests in org.geoserver.test package with the above param-
eters, since the data reference tests in org.geoserver.test.onlineTest package contain non-joining tests which
would fail.
You do not need to specify these VM Arguments for running data reference tests (in app-schema-
test/src/test/java/org/geoserver/test/onlineTest). However, you would still need to specify the Oracle
JDBC driver in the Classpath for Oracle specic tests. Data reference tests package also requires 768m
memory to run from JUnit.
8.10. App-Schema Online Tests 131
8.10.4 Adding new tests
When adding new tests to app-schema-test suite (except for onlineTest package for data reference tests),
please note the following:
Test ofine only
If your test is a special case and does not need to be tested online, exclude them in both app-schema-oracle-
test and app-schema-postgis-test pom.xml and ignore the points beyond this. Otherwise, read on.
idExpression
If your test database does not use primary keys, ensure idExpression is specied for the top level element
in your mapping le.
Multi-valued properties ordering
When testing multi-valued properties, the order of the values could vary depending on the data source
type. To be safe, compare your values as a list, instead of evaluating individual xpath node against a single
value for such properties. E.g.:
List<String> names = new ArrayList<String>();
names.add("New Group");
names.add("-Xy");
String name = evaluate("//gsml:MappedFeature[@gml:id=" + id
+ "]/gsml:specification/gsml:GeologicUnit/gml:name[1]", doc);
assertTrue(names.contains(name));
names.remove(name);
name = evaluate("//gsml:MappedFeature[@gml:id=" + id
+ "]/gsml:specification/gsml:GeologicUnit/gml:name[2]", doc);
assertTrue(names.contains(name));
names.remove(name);
assertTrue(names.isEmpty());
This is because of the difference in the handling of queries with joining. Joining uses order by when query-
ing tables. When the tests run ofine, property data store returns data from properties le unordered.
When joining is enabled:
If the multi-valued properties are not feature chained, the order is unpredictable.
If the multi-valued properties are feature chained, they are ordered by the foreign key used in feature
chaining.
Column names in upper case
Ensure column names in mapping les are in upper case, even if they are in lower case in the properties
le. This is to avoid failures with Oracle database, due to OracleDialect not wrapping names with escape
characters. To work around this, the script for online tests creates the columns in upper case, therefore
should be referred by with upper case.
Functions in feature chaining
If using feature chaining, avoid using functions in sourceExpression for linking attributes, i.e. attribute
used in both OCQL and linkField. This is because functions used in feature chaining are not supported
with joining support.
WMS tests
If you are testing Application Schema WMS support behaviour, it is highly recommended to also perform
the optional perceptual diff tests, which are included in both online as well as ofine unit tests. Perceptual
diff tests for app-schema WMS support will only be performed if:
-Dorg.geotools.image.test.enabled=true
and Perceptual Diff is installed on the computer from which the tests are executed.
3D tests
There are a number of tests that try out 3D features in App-schema. To run these as online tests against a
postgis or oracle database, a number of prerequisites must be met.
For PostGIS:
You must use postgis 2 to support 3D.
In your postgis, if it hasnt been done yet, this command must be executed:
http://spatialreference.org/ref/epsg/4979/postgis/ to support srid 4979 (wgs84 with 3d)
For Oracle:
You must use Oracle 11g Release 2, preferably the latest version that can be downloaded for best 3D
support
Oracle does NOT support WKT parsing of 3d geometries, so some extra DBA work is needed to set
this up. Otherwise the online tests, which rely on WKT to enter data in the database, will fail.
You need the following package SC4O (Spatial Companion for Oracle), created Simon Greener:
download at http://www.spatialdbadvisor.com/les/SC4O.zip. It has an installation script for linux
and windows that must be run from the server that runs oracle. The package will provide JTS func-
tionality that can be called from PL/SQL.
If the online test user is different from the user used for installation of the pack-
age, the online test user must be given permission to use the package. You must
also execute as an admin user the following command for the online test user: CALL
DBMS_JAVA.GRANT_PERMISSION(onlinetestuser,java.lang.RuntimePermission,getClassLoader,);
Afterwards, you have to tell the online testing system to use the JTS method for wkt parsing rather
than the regular oracle method SDO_GEOMETRY. You do this with the systemproperty -Dwktparser.
The method you need is SC4O.ST_GeomFromEWKT but you need to specify the user where the pack-
age was installed. For example, I installed the package using the System user. Then I gave onlinetes-
tuser permission to execute it. I run the test with -Dwktparser=System.SC4O.ST_GeomFromEWKT
8.10. App-Schema Online Tests 133
CHAPTER 9
Release Schedule
Starting with version 2.2 GeoServer releases follow a time boxed model in which releases occur at regular
predictable frequencies rather than at ad-hoc dates. In a time boxed the software is released at predictable
frequencies with whatever xes, improvements, and feature are available at the time of release. This differs
from the previous feature based model in which releases occur when a certain number of key features have
accumulated.
To compensate for the inherent unpredictability of the release contents the model includes strict rules about
what type of development is appropriate during specic periods of a branches life cycle. These rules include
a suitably long hardening period for the unstable branch to mature and eventually become stable.
9.1 Release branches
At any given time GeoServer is undergoing two to three main branches of development.
1. The stable branch in which only bug xing, and smaller scale feature development occurs on
2. The unstable (master) branch in which more experimental and larger scale feature development occurs
3. The maintenance branch which was the previously stable branch that is nearing end of life and sees
only the most stable development, typically mostly bug xes.
9.2 Release cycle
On the stable branches release follow a monthly cycle in a which a new minor release occurs once a month.
On the unstable branch releases follow a 6 month cycle in which a new major release occurs once every 6
months. The following diagram illustrates the cycle:
Things to note:
135
monthly releases on the stable branch
a four month open development period followed by two months hardening period on the unstable
branch
beta releases are supposed to be released out of the unstable series on a monthly basis across the
switch between open development and hardening, followed by the rst release candidate
release candidates are pushed out every two weeks until we reach a stable code base, which will be
released as the new major stable release
the rst release candidate marks the branch off of a new trunk, on which open development starts
again
The time of the rst release candidate marks the origin of a new stable branch in which the unstable branch
becomes the stable branch, and the stable branches becomes a maintenance branch.
Every month, on the same day, a new release is issued from the stable branch using whatever revision of
GeoServer/Geotools passed the last CITE tests. The release is meant to improve upon the previous release
in both functionality and stability, so unless the project steering committee determines reasons to block the
release it will happen regardless of what bug reports are in Jira. Pending resourcing, they can be xed in
the next release that comes out one month later.
At any given point in time there are two branches under the development, the stable branch and the mas-
ter/unstable branch. Once every six months when the creation of a new stable branch occurs a third active
maintenance branch is created. This branch is kept up to date with the stable stable for a period of one-
month after which the nal release on that branch is created. That said there is nothing against a developer
continuing to maintain the branch or creating release from it, it is just not expected.
9.3 Development phases
The type of development that can occur on a branch is dictated by where the branch is in terms of its life
cycle.
9.3.1 Stable branch
The type of acceptable development on the stable branch does not change much over its lifetime. It is meant
for bug xes and new features that do not affect the GeoServer API or signicantly affect the stability. A
PSC vote (with eventual proposal) can be called in case a signicant new feature or change needs to be back
ported to the stable branch overriding the above rules.
If, for any reason, a release is delayed the next release will be rescheduled 30 days after the last release (that
is, delaying the whole train of remaining releases).
9.3.2 Unstable branch
The type of development on the master/unstable branch changes over its lifetime fromfour months of open
development to two months of stable/hardening development.
9.3.3 Open development phase
The open development phase starts when the new stable release is branched off, and ends when hardening
starts, four months after the new stable release is made.
136 Chapter 9. Release Schedule
During this phase developers are free to commit stability undermining changes (even signicant ones).
Those changes still need to be voted as GSIP anyways to ensure, as usual, resourcing checks, API consis-
tency and community review.
After three months from the release of the stable series a rst beta will be released, one month after that the
second beta will be released and the branch will switch into hardening mode.
9.3.4 Hardening phase
The hardening phase starts when the second beta is released and continues through all release candidate
(RC) releases. The rst RC is released one month after the second beta, and then bi-weekly releases will be
issued until no major issues will be reported by the user base, at which point the last RC will be turned into
the new stable release.
During hardening only bug-xes and new non core plugins can be developed
9.4 Commit rules
The following are guidelines as to what types of commits can occur in any particular phase. While the PSC
reserves the right to vote and override the committing guidelines the following rules reect the current
policies.
The hardening phase, and by extension the stable and open phases, can receive any of the following types
of commits:
bug xes
documentation improvements
new plugins contributed as community or extension modules (with a cautionary note that during
hardening the attention should be concentrated as much as possible on getting the new release stable)
In addition the stable phase can receive any of the following types of commits:
new minor core functionality (e.g. new output format)
new API that we can commit to for a long period of time (provided its not a change to existing API
unless the PSC votes otherwise). GeoServer is not a library, so dening API can be hard, but any class
that can be used by pluggable extension points should be changed with care, especially so in a stable
series
In addition to the above the stable phase can receive the following types of changes provided there are no
reservations or concerns about them from the PSC. Such changes may be subject to voting:
promotion of extensions to core
core changes that are unlikely to affect the stability of the upcoming release (if the PSC is ok better
land them right after a release to get as a large window for testing as possible)
back port of larger changes that have proven to be working well on trunk for an extended period of
time
During the open development phase all types of commits are fair game but of course large changes are still
subject to proposals and reviews.
9.4. Commit rules 137
138 Chapter 9. Release Schedule
CHAPTER 10
Release Guide
This guide details the process of performing a GeoServer release.
10.1 Before you start
10.1.1 GeoTools release
For any non-beta release (including release candidates) a GeoServer release requires a corresponding
GeoTools release. Therefore before you start you should coordinate a GeoTools release. Either perform-
ing the release yourself or asking on the GeoTools devel list for a volunteer to perform the release.
TODO: point to geotools release docs
10.1.2 Notify developer list
It is good practice to notify the GeoServer developer list of the intention to make the release a few days in
advance, even though the release date has been agreed upon before hand.
10.2 Prerequisites
The following are necessary to perform a GeoServer release:
1. Commit access to the GeoServer Git repository
2. Build access to Hudson
3. Edit access to the GeoServer Blog
4. Administration rights to GeoServer JIRA
5. Release/le management privileges in SourceForge
139
10.3 Versions and revisions
When performing a release we dont require a code freeze in which no developers can commit to the
repository. Instead we release from a revision that is known to pass all tests, including unit/integration
tests as well as CITE tests.
To obtain the GeoServer and Geotools revisions that have passed the CITE test, navigate to the latest Hud-
son run of the CITE test and view its console output and select to view its full log. For example:
http://hudson.opengeo.org/hudson/view/cite/job/cite-wfs-1.1/813/consoleFull
Perform a search on the log for Git revision and you should obtain the following.:
version = 2.2-SNAPSHOT
Git revision = 4ea8d3fdcdbb130892a03f27ab086068b95a3b01
Git branch = 4ea8d3fdcdbb130892a03f27ab086068b95a3b01
build date = 03-Aug-2012 03:39
geotools version = 8-SNAPSHOT
geotools revision = 73e8d0746a4527e46a46e5e8bc778ca92ca89130
Since we dont make any release from master, ensure you select the right CITE test that passed to obtain the
right revision.
Since most GeoServer releases require an ofcial GeoTools release the GeoTools revision is usually not
needed. But if performing a beta release it is allowed to release directly from a specic GeoTools revision.
10.4 Release in JIRA
Run the geoserver-release-jira job in Hudson. The job takes the following parameters:
VERSION
The version to release, same as in the previous section. This version must match a version in
JIRA.
NEXT_VERSION
The next version in the series. All unresolved issues currently ls against VERSION will be
transitioned to this version.
JIRA_USER
A JIRA user name that has release privileges. This user will be used to perform the release in
JIRA, via the SOAP api.
JIRA_PASSWD
The password for the JIRA_USER.
This job will perform the tasks in JIRA to release VERSION. Navigate to JIRA and verify that the version
has actually been released.
10.5 Build the Release
Run the geoserver-release job in Hudson. The job takes the following parameters:
BRANCH
140 Chapter 10. Release Guide
The branch to release from, 2.2.x, 2.1.x, etc... This must be a stable branch. Releases are not
performed from master.
REV
The Git revision number to release from. eg, 24ae10fe662c..... If left blank the latest revision
(ie HEAD) on the BRANCH being released is used.
VERSION
The version/name of the release to build, 2.1.4, 2.2, etc...
GT_VERSION
The GeoTools version to include in the release. This may be specied as a version number such
as 8.0 or 2.7.5. Alternatively the version may be specied as a Git branch/revision pair in
the form <branch>@<revision>. For example master@36ba65jg53...... Finally this value
may be left blank in which the version currently declared in the geoserver pom will be used
(usually a SNAPSHOT). Again if performing a non-beta release this version must be a version
number corresponding to an ofcial GeoTools release.
GWC_VERSION
The GeoWebCache version to include in the release. This may be specied as a version num-
ber such as 1.3-RC3. Alternatively the version may be specied as a Git revision of the form
<branch>@<revision> such as master@1b3243jb.... Finally this value may be left blank
in which the version currently declared in the geoserver pom will be used (usually a SNAP-
SHOT).Git Again if performing a non-beta release this version must be a version number corre-
sponding to an ofcial GeoTools release.
GIT_USER
The Git username to use for the release.
GIT_EMAIL
The Git email to use for the release.
This job will checkout the specied branch/revision and build the GeoServer release artifacts against the
GeoTools/GeoWebCache versions specied. When successfully complete all release artifacts will be up-
loaded to the following location:
http://gridlock.opengeo.org/geoserver/release/<RELEASE>
Additionally when the job completes it res off two jobs for building the Windows and OSX installers.
These jobs run on different hudson instances. When those jobs complete the .exe and .dmg artifacts will
be uploaded to the location referenced above.
10.6 Test the Artifacts
Download and try out some of the artifacts from the above location and do a quick smoke test that there
are no issues. Engage other developers to help test on the developer list.
10.7 Publish the Release
Run the geoserver-release-publish in Hudson. The job takes the following parameters:
VERSION
10.6. Test the Artifacts 141
The version being released. The same value s specied for VERSION when running the
geoserver-release job.
BRANCH
The branch being released from. The same value specied for BRANCH when running the
geoserver-release job.
This job will rsync all the artifacts located at:
http://gridlock.opengeo.org/geoserver/release/<RELEASE>
to the SourceForge FRS server. Navigate to Sourceforge and verify that the artifacts have been uploaded
properly. Set the necessary ags on the .exe, .dmg and .bin artifacts so that they show up as the appro-
priate default for users downloading on the Windows, OSX, and Linux platforms.
10.8 Create the download page
Get the JIRA version for this release:
Go to JIRA
Select Change log
Open the release notes for the version being released
The version will be in the url, e.g. http://jira.codehaus.org/secure/ReleaseNote.jspa?projectId=10311&version=18700
-> 18700
Create the new download page:
Go to GeoServer web site and make sure you are logged in.
Select the New page link in the menu
Click select a page template to start from
Choose the Download template
Fill in the version, release date (e.g., May 17th, 2012) and the jira version
Set the page title to the version being released (e.g. GeoServer 2.2-RC3)
Save and check all the links are working
If you are releasing a stable version, edit the Stable version page and make it include the newly created
release page. If you are instead releasing a beta/RC, edit the Latest version page and make it include the
newly created release page.
10.9 Post the Documentation
Note: For now, this requires a user account on the OpenGeo server wedge.opengeo.org.
Note: This content will likely move to GitHub in the near future.
1. Open a connection to wedge.opengeo.org.
2. Create the following new directories:
/var/www/docs.geoserver.org/htdocs/a.b.c
/var/www/docs.geoserver.org/htdocs/a.b.c/developer
/var/www/docs.geoserver.org/htdocs/a.b.c/user
where a.b.c is the full release number.
3. Download the HTML documentation archive from the GeoServer download page, and extract the
contents of both user manuals to the appropriate directory:
cd /var/www/docs.geoserver.org/htdocs/a.b.c/
sudo wget http://downloads.sourceforge.net/geoserver/geoserver-a.b.c-htmldoc.zip
sudo unzip geoserver-a.b.c-htmldoc.zip
sudo rm geoserver-a.b.c-htmldoc.zip
4. Open the le /var/www/docs.geoserver.org/htdocs/index.html in a text editor.
5. Add a new entry in the table for the most recent release:
<tr>
<td><strong><a href="http://geoserver.org/display/GEOS/GeoServer a.b.c">a.b.c</a></strong></td>
<td><a href="a.b.c/user/">User Manual</a></td>
<td><a href="a.b.c/developer/">Developer Manual</a></td>
</tr>
6. Save and close this le.
10.10 Announce the Release
10.10.1 Mailing lists
Note: This announcement should be made for all releases, including betas and release candidates.
Send an email to both the developers list and users list announcing the release. The message should be
relatively short. The following is an example:
Subject: GeoServer 1.7.1 Released
The GeoServer team is happy to announce the release of GeoServer 1.7.1.
The release is available for download from:
http://geoserver.org/display/GEOS/GeoServer+1.7.1
This release comes with some exciting new features. The new and
noteworthy include:
*
KML Super Overlay and Regionating Support
*
KML Extrude Support
*
KML Reflector Improvements
*
Mac OS X Installer
*
Dutch Translation
*
Improved Style for Web Admin Interface
*
New SQL Server DataStore Extension
*
Improved Oracle DataStore Extension
*
Default Templates per Namespace
10.10. Announce the Release 143
Along with many other improvements and bug fixes. The entire change log
for the 1.7.1 series is available in the issue tracker:
http://jira.codehaus.org/browse/GEOS/fixforversion/14502
A very special thanks to all those who contributed bug fixes, new
features, bug reports, and testing to this release.
--
The GeoServer Team
10.10.2 SourceForge
1. Log in to SourceForge.
2. Edit the release, and scroll down to the bottom of the page.
3. Check the Im sure check box, and click the Send Notice button.
4. Repeat for the extension release.
10.10.3 GeoServer Blog
Note: This step requires an account on http://blog.geoserver.org
1. Log into the GeoServer Blog.
2. Create a new post. The post should be more colorful than the average announcement. It is meant
to market and show off any and all new features. Examples of previous posts:
http://blog.geoserver.org/2008/12/09/geoserver-171-released/
http://blog.geoserver.org/2008/10/27/geoserver-170-released/
3. Do not publish the post. Instead present it to the GeoServer outreach team for review, and they will
publish it.
Note: GeoServer wiki has to be updated as well.
1. Go to http://geoserver.org/display/GEOS/Download :
Click on Add Page on the right.
Copy the wiki source from an existing page and update the reference link to your released arti-
fact.
Edit the page location to correctly reect where the page should reside.
2. Finally, update the wiki link on http://geoserver.org/display/GEOS/Latest to your new page.
10.10.4 Update GeoServer homepage
1. Navigate to http://geoserver.org.
2. Click Log In in the top right corner.
3. Enter your Conuence user name and password and click Log In.
4. Navigate back to http://geoserver.org.
5. At the bottom of the page, click Edit.
6. Change the links at the top of the page to include the new GeoServer download page and the blog
post. For example:
h6. December 32, 2086: [GeoServer 3.1.1] released\! [See whats new...|http://blog.geoserver.org/2086/12/32/geoserver-3.1.1-released/]
7. When nished, click Save.
8. Verify that the links work on the homepage.
10.10.5 SlashGeo
Note: This announcement should be made only for ofcial releases. Not betas and release candidates.
Note: This step requires an account on http://slashgeo.org
1. Go to http://slashgeo.org, and log in, creating an account if necessary.
2. Click the Submit Story link on the left hand side of the page. Examples of previous stories:
http://technology.slashgeo.org/technology/08/12/09/1745249.shtml
http://industry.slashgeo.org/article.pl?sid=08/10/27/137216
10.10.6 FreeGIS
Note: This announcement should be made only for ofcial releases. Not betas and release candidates.
Send an email to bjoern dot broscheit at uni-osnabrueck dot de. Example:
10.10. Announce the Release 145
Subject: GeoServer update for freegis
GeoServer 1.7.1 has been released with some exciting new features. The big
push for this release has been improved KML support. The new and noteworthy
include:
*
KML Super Overlay and Regionating Support
*
KML Extrude Support
*
KML Reflector Improvements
*
Mac OS X Installer
*
Dutch Translation
*
Improved Style for Web Admin Interface
*
New SQL Server DataStore Extension
*
Improved Oracle DataStore Extension
*
Default Templates per Namespace
Along with many other improvements and bug fixes. The entire change log for
the 1.7.1 series is available in the issue tracker:
http://jira.codehaus.org/browse/GEOS/fixforversion/14502
10.10.7 FreshMeat
Note: This announcement should be made only for ofcial rel-eases. Not betas and release candidates.
Note: This step requires an account on http://freshmeat.net/
1. Go to http://freshmeat.net/ and log in.
2. Search for geoserver and click the resulting link.
3. Click the add release link at the top of the page.
4. Choose the Default branch
5. Enter the version and choose the appropriate Release focus.
Note: The release focus is usually 4,5,6, or 7. Choose which ever is appropriate.
6. Enter a succinct description (less than 600 characters) of the Changes.
7. Update the links to the following elds:
Zip
OS X package
Changelog
8. Click the Step 3 button.
9. Click the Finish button.
CHAPTER 11
Release Testing Checklist
A checklist of things to manually test for every release.
11.1 Artifact size
The binary release of GeoServer should be somehere around 45 - 46 megabytes.
11.2 Demos
Note: These are no longer available in GeoServer 2.0, well probably reinstate them later
To do the demo page, http://localhost:8080/geoserver/demo.do, and test all of the demos. This includes:
WFS-T demo
GeoRSS demo with Google Maps, Virtual Earth, and Yahoo Maps
WMS Overlay demo
WMS Example
11.3 Sample requests
Go to the sample request page, http://localhost:8080/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.web.demo.DemoRequestsPage,
and execute every sample request, ensuring the correct response for each request.
11.4 Map preview
1. Go to the map previewpage, http://atlas.openplans.org:8081/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.web.demo.MapPreviewPage
147
2. Click the OpenLayers link next to nurc:ArcSample
3. Go back to the map previewand click the GeoRSS itemin the drop down choice next to topp:states
4. Go back to the map preview and click the OpenLayers link next to topp:states.
5. Enable the options toolbar and specify the CQL lter:
STATE_ABBR EQ TX
148 Chapter 11. Release Testing Checklist
11.5 KML
1. Go back to the map preview and click the KML link next to topp:states
2. Open the result in Google Earth
3. Zoom out as far as possible and notice the smaller states (on the east coast) disappear.
4. Close Google Earth
Warning: If you do not shut down Google Earth it will cache information and throw off the next
steps.
5. Go to the feature type editor page for the topp:states feature type
11.5. KML 149
6. Change the KML Regionating Attribute to SAMP_POP and change the KML Regionating
Strategy to external-sorting:
.. image:: states_kml_config.png
7. Submit and Apply changes
8. Go back to the map preview page and again click the KML link next to topp:states, opening the
result in Google Earth
9. Zoom out as far as possible and notice the smaller population states (green) disappear
10. Go back to the map preview page and click the KML link next to nurc:Img_Sample, opening the
result in Google Earth
11. Zoom in and notice tiles load
12. Follow the link http://localhost:8080/geoserver/wms/kml?layers=topp:states&mode=refresh ,
opening the result in Google Earth
13. Notice the KML reload every time the camera is stopped
14. Edit the description template for the states layer as follows:
This is the state of ${STATE_NAME.value}.
<img src="http://www.netstate.com/states/symb/flags/images/${STATE_ABBR.value?lower_case}_fi.gif"/>
<br>
For more information visit <a href="http://en.wikipedia.org/wiki/${STATE_NAME.value}">Wikipedia</a>
15. Refresh the KML by moving the camera and click on a placemark
16. Append the parameter kmscore=0 to the above link and open the result in Google Earth
17. Notice the rasterized version of the KML
18. Follow the link http://localhost:8080/geoserver/wms/kml?layers=topp:states&mode=download ,
saving the result to disk.
19. Examine the le on disk and notice a raw dump of all placemarks for the layer.
11.5. KML 151
11.6 GeoWebCache
1. Go the geowebcache demo page, http://localhost:8080/geoserver/gwc/demo
2. Click the EPSG:4326" link for topp:states
3. Zoom in and notice the tiles load.
4. Repeat steps 2 to 3 for EPSG:900913
CHAPTER 12
Cite Test Guide
A step by step guide to the GeoServer Compliance Interoperability Test Engine (CITE).
Contents
Cite Test Guide
Check out CITE tools
Run WFS 1.0 tests
Run WFS 1.1 tests
Run WMS 1.1 tests
Run WCS 1.1 tests
Run WCS 1.0 tests
Teamengine Web Application
12.1 Check out CITE tools
The CITE tools are available at https://github.com/jdeolive/geoserver-cite-tools. The README le con-
tains the most update documentation of how to checkout and build the tools. The quick version is:
git clone git://github.com/jdeolive/geoserver-cite-tools.git
cd geoserver-cite-tools
git submodule update --init
mvn clean install
12.2 Run WFS 1.0 tests
Note: Running WFS 1.0 tests require PostGIS to be installed on the system.
1. Create a PostGIS user named cite:
createuser cite
2. Create a PostGIS databased named cite, owned by the cite user:
153
createdb -T template_postgis -U cite cite
3. Change directory to the citewfs-1.0 data directory and execute the script cite_data.sql:
psql -U cite cite < cite_data.sql
4. Start GeoServer with the citewfs-1.0 data directory. Example:
cd <root of geoserver install>
export GEOSERVER_DATA_DIR=<root of geoserver sources>/data/citewfs-1.0
./bin/startup.sh
5. Change directory back to the cite tools and run the tests:
ant wfs-1.0
With the following parameters:
(a) Capabilities URL
http://localhost:8080/geoserver/wfs?request=getcapabilities&service=wfs&version=1.0.0
(b) All tests included
12.3 Run WFS 1.1 tests
Note: Running WFS 1.1 tests require PostGIS to be installed on the system.
1. Create a PostGIS user named cite:
154 Chapter 12. Cite Test Guide
createuser cite
2. Create a PostGIS databased named cite, owned by the cite user:
createdb -T template_postgis -U cite cite
3. Change directory to the citewfs-1.1 data directory and execute the script dataset-sf0.sql:
psql -U cite cite < dataset-sf0.sql
4. Start GeoServer with the citewfs-1.1 data directory. Example:
cd <root of geoserver install>
export GEOSERVER_DATA_DIR=<root of geoserver sources>/data/citewfs-1.1
./bin/startup.sh
ant wfs-1.1
http://localhost:8080/geoserver/wfs?service=wfs&request=getcapabilities&version=1.1.0
(b) Supported Conformance Classes:
Ensure WFS-Transaction is checked
Ensure WFS-Xlink is unchecked
(c) GML Simple Features: SF-0
12.3. Run WFS 1.1 tests 155
12.4 Run WMS 1.1 tests
1. Start GeoServer with the citewms-1.1 data directory.
ant wms-1.1
http://localhost:8080/geoserver/wms?service=wms&request=getcapabilities&version=1.1.1
(b) UpdateSequence Values:
Ensure Automatic is selected
2 for value that is lexically higher
0 for value that is lexically lower
(c) Certification Profile : QUERYABLE
(d) Optional Tests:
Ensure Recommendation Support is checked
Ensure GML FeatureInfo is checked
Ensure Fees and Access Constraints is checked
For BoundingBox Constraints ensure Either is selected
(e) Click OK
12.5 Run WCS 1.1 tests
1. Start GeoServer with the citewcs-1.1 data directory.
ant wcs-1.1
(a) Capabilities URL:
http://localhost:8080/geoserver/wcs?service=wcs&request=getcapabilities&version=1.1.1
Click Next
12.5. Run WCS 1.1 tests 157
3. Accept the default values and click Submit
12.6 Run WCS 1.0 tests
Warning: The WCS specication does not allow a cite compliant WCS 1.0 and 1.1 version to co-exist.
To successfully run the WCS 1.0 cite tests the wms1_1-<VERSION>.jar must be removed from the
geoserver WEB-INF/lib directory.
1. Remove the wcs1_1-<VERSION>.jar from WEB-INF/lib directory.
2. Start GeoServer with the citewcs-1.0 data directory.
ant wcs-1.0
(a) Capabilities URL:
http://localhost:8080/geoserver/wcs?service=wcs&request=getcapabilities&version=1.0.0
(b) MIME Header Setup: image/tiff
(c) Update Sequence Values:
2 for value that is lexically higher
0 for value that is lexically lower
(d) Grid Resolutions:
0.1 for RESX
0.1 for RESY
(e) Options:
Ensure Verify that the server supports XML encoding is checked
Ensure Verify that the server supports range set axis is checked
(f) Schemas:
Ensure that original schemas is selected
(g) Click OK
12.6. Run WCS 1.0 tests 159
12.7 Teamengine Web Application
The Teamengine web application is useful for analyzing results of a test run. To run the web application
execute:
ant webapp
From the cite tools checkout. Once started the web app will be available at:
http://localhost:9090/teamengine
To run on a different port pass the -Dengine.port system property to ant command.
CHAPTER 13
Translating GeoServer
We would like GeoServer available in as many languages as possible, so we want your help to add local-
izations / translations, specically the GeoServer UI and documentation.
13.1 Translating the UI
The GeoServer UI stores text strings inside properties les. The default (English) les are named
GeoServerApplication.properties and are located in the following directories:
/src/web/core/src/main/resources/
/src/web/demo/src/main/resources/
/src/web/gwc/src/main/resources/
/src/web/security/src/main/resources/
/src/web/wcs/src/main/resources/
/src/web/wfs/src/main/resources/
/src/web/wms/src/main/resources/
To translate the GeoServer UI to another language, copy and rename each of these les to be
GeoServerApplication_[LANG].properties where [LANG] is the language code as dened in RFC
3066 For example, the language code for German is de and for Brazilian Portuguese is pt-BR.
Once created, each line in the les represents a string that will need to be translated. When nished, you
will need to commit these les or submit a JIRA issue with attached patch. See the section on Source Code
for more information on how to commit.
13.1.1 Editing in Eclipse
If you are using Eclipse, you can install the Eclipse ResourceBundle Editor. Once installed, you can edit the
src/main/resources/GeoServerApplication.properties les in all web-
*
projects (web-core,
web-demo, etc.) with the ResourceBundle editor.
13.2 Translating documentation
The GeoServer User Manual contains a wealth of information fromthe novice to the experienced GeoServer
user. It is written using the Sphinx Documentation Generator. The stable branch version of the User Manual
161
exists as the following URL:
http://docs.geoserver.org/stable/en/user/
Built from the following source les:
/doc/en/user/
To create a User Manual in a different language, rst create a directory called /doc/[LANG]/, where
[LANG] is the language code as dened in RFC 3066. The you can copy the contents of /doc/en/user/
to /doc/[LANG]/user and edit accordingly, or generate a new Sphinx project in /doc/[LANG]/user.
(See the Sphinx Quickstart <http://sphinx.pocoo.org/tutorial.html> for more information about creating a new
project.)
The GeoServer Sphinx theme exists at /doc/en/user/themes, so that can be copied (and modied if
desired) to /doc/[LANG]/user/themes.
When nished, you will need to commit the content (if you have commit rights) or submit a JIRA issue
with attached patch. See the section on Source Code for more information on how to commit. Setting up
the documentation to be hosted on docs.geoserver.org will require a project administrator; please send an
email to the mailing list for more details.
13.2.1 Tips
See the GeoServer Documentation Manual for more information about writing documentation.
The Developer Manual exists at /doc/en/developer. The same procedures for editing the User
Manual apply to the Developer Manual.
162 Chapter 13. Translating GeoServer
CHAPTER 14
Policies and Procedures
14.1 Committing
14.1.1 Getting commit access
There are two stages of commit access:
1. community module or extension commit access
2. core commit access
The rst stage of access allows a developer to commit only to the community module or extension for which
they are the maintainer. This stage of access can be obtained quite easily.
The second allows a developer to make commits to the core modules of geoserver. Being granted this
stage of access takes time, and is obtained only after the developer has gained the trust of the other core
committers.
Community commit access
The process of getting community commit access is as follows:
1. Email the developer list
This rst step is all about communication. In order to grant commit access the other developers on
the project must rst know what the intention is. Therefore any developer looking for commit access
must rst describe what they want to commit (usually a community module), and what it does.
2. Sign up for a GitHub account
GeoServer source code is hosted on Github and youll need an account in order to access it. You can
sign-up here.
3. Notify the developer list
After a developer has signed up on Github they must notify the developer list. A project despot
will then add them to the group of GeoServer committers and grant write access to the canonical
repository.
163
4. Fork the canonical GeoServer repository
All committers maintain a fork of the GeoServer repository that they work from. Fork the canonical
repository into your own account.
5. Congure your local setup
Follow this guide in the developer manual.
Core commit access
The process of obtaining core commit access is far less mechanical than the one to obtain community commit
access. It is based soley on trust. To obtain core commit access a developer must rst obtain the trust of the
other core commiters.
The way this is typically done is through continuous code review of patches. After a developer has submit-
ted enough patches that have been met with a postitive response, and those patches require little modica-
tions, the developer will be granted core commit access.
There is no magic number of patches that make the criteria, it is based mostly on the nature of the patches,
how in depth the they are, etc... Basically it boils down to the developer being able to show that they
understand the code base well enough to not seriously break anything.
14.1.2 Commit Guidelines
There is not much in the way of strict commit policies when it comes to committing in GeoServer. But over
time some rules and conventions have emerged:
1. Add copyright headers for new les
When adding new source les to the repository remember to add the standard copyright header:
/
*
Copyright (c) 2001 - 2013 OpenPlans - www.openplans.org. All rights reserved.
*
This code is licensed under the GPL 2.0 license, available at the root
*
application directory.
*
/
2. Do not commit large amounts of binary data
In general do not commit any binary data to the repository. There are cases where it is appropriate
like some data for a test case, but in these cases the les should be kept as small as possible.
3. Do not commit jars or libs, use Maven instead
In general never commit a depending library directly into the repository, this is what we use Maven
for. If you have a jar that is not present in any maven repositories, ask on the developer list to get it
uploaded to one of the project maven repositories.
4. Ensure code is properly formatted
Ensure that the IDE or editor used to edit source les is setup with proper formatting rules. This
means spaces instead of tabs, 100 character line break, etc...
If using Eclipse ensure you have congured it with the template and formatter used for GeoTools.
164 Chapter 14. Policies and Procedures
14.2 Submitting Patches
As with most open source project GeoServer is very happy to accept patches from the community. Patches
from the community have been a source of some of the best bug xes and improvements in GeoServer and
is a great way to give back to the project.
This document is a guide designed to help users through the process of successfully submitting a patch.
14.2.1 Source code
Before one can generate a patch source code is needed. See the developer quickstart for information about
obtaining a copy of the GeoServer sources.
In general developers will only apply patches that apply cleanly against the latest versions of the stable and
master branches. Therefore before you generate a patch it is important that you update your checkout to
the latest for the branch you are working on.
14.2.2 Generating a patch
There are a number of valid ways to generate a patch.
GitHub pull requests
The GeoServer git repository is hosted on github, which provides a very nice way to manage patches in the
way of pull requests. To issue a pull request requires that you fork the GeoServer git repo into your own
account.
Assuming that origin points to your github repo the the patch workow then becomes:
1. Make the change.
git checkout -b my_bugfix master
git add .
git commit-m "fixed bug xyz"
2. Push the change up to your github repository.
git push origin my_bugfix
3. Visit your github repo page and issue the pull request.
At this point the core developers will be notied of the pull request and reviewit at the earliest convenience.
Git diff
All git clients provide an easy way to generate a patch. Assuming you are using the command line tools a
simple workow for generating a patch is as follows.
1. Make the change as above.
2. Generate the patch.
git diff master > my_bugfix.patch
3. Open a JIRA ticket and attach the patch le to the ticket.
At this point the core developers will be notied of the ticket t and review it at the earliest convenience.
14.2. Submitting Patches 165
Unix diff
If you are not working from git, perhaps working from a source archive directly, it is always possible to
manually generate a patch with unix diff tools.
1. Back up the source tree.
cp geoserver geoserver.orig
2. Make the change.
3. Generate the diff.
diff -ru geoserver.orig geoserver > my_bugfix.patch
4. Open a JIRA ticket and attach the patch le to the ticket.
At this point the core developers will be notied of the ticket t and review it at the earliest convenience.
14.2.3 Patch guidelines
The following guidelines are meant to ensure that changes submitted via patch be as easy as possible to
review. The easier a patch is to review the easier it is to apply.
Ensure your IDE/editor is properly congured
Ensure that your development environment is properly congured for GeoServer development. Acommon
issue with patches from newcomers is that their text editor is congured to use tabs rather than spaces.
See the the Eclipse Guide for general information about formatting and ide setup.
Include only relevant changes
Ensure the patch only contains changes relevant to the issue you are trying to x. A common mistake is
to include whitespace and formatting changes along with the relevant changes. These changes, while they
may seem harmless, make the patch much harder to read.
Fix one thing at a time
Do not batch up multiple unrelated changes into a single patch. If you want to x multiple issues work on
them separately and submit separate patches for them.
Be patient
The core developers review community patches in spare time. Be cognizant of this and realize that just as
you are contributing your own free time to the project, so is the developer who is reviewing and applying
your patch.
14.3 Code Review
Reviewing code is an invaluble process that is used to ensure that the highest quality code possible gets
comitted into GeoServer at all times. There is no hard policy relating to code review such that every line
of code needs to be reviewed but the unofcial practice which has been adopted over time by GeoServer
developers is summed up as follows:
if the change is to a core module and the developer does not have core commit access it requires
review
if the change is non-trivial and to a core module, it should be reviewed
if the change is local to a module the developer maintains it usually does not require a review
The above guidelines are not rules. In general code review is always welcome and if there is a question as
to if a change should be reviewed or not, always heir on the side of getting it reviewed.
14.4 Community Process
This document describes the process that the GeoServer community uses to handle contributions and ad-
ditions to the project. This process is used to manage new features and additions to the project, as well as
for short and long term project planning.
The GeoServer community process adheres to the following guidelines to ensure that it retains open devel-
opment practices:
Transparency Project decisions are made in an open and transparent way. When a decision is made it
is clear as to why it was made.
Balance Every member of the developer community has an equal voice. Project decisions are made by
the community as a whole, and not subject to the whims of any one single developer or organizations.
14.4.1 Road map
The GeoServer road map is the center of all project planning. It communicates to the community what
is being worked on and when it is planned to be released. More specically the road map is a timeline
of planned releases. Each release has a date attached to it and contains a list of features and critical bug-
xes/improvements that will be implemented for that release.
The road map is structured into thee parts: short term, medium term, and long term. The short term road
map is made up of features scheduled for the next two releases. The medium term road map is made up
of features scheduled for the remainder the current stable development branch. The long term road map
consists of features scheduled against the next/unstable development branch.
Short term road map
The short term road map is the most important part of the overall road map as it pertains specically to
what is going to be released in the near future. This road map is updated on a weekly basis.
Items on the short term road map are described in JIRA as new features, improvements, or high priority
bug xes.
14.3. Code Review 167
Medium term road map
The medium term road map is a higher level view of what is to come in the current unstable development
branch (trunk). It is much less ner grained than the short term road map. Items on the short term road
map may or may not show up in JIRA. Medium term road map items usually take the form of a GSIP in
progress, or one being discussed, or just a community module being developed.
Long term road map
The long term road map is the highest level view of the road map and describes what is to come in future
development branches. Items on the long term road map are more or less just ideas or future plans. Long
term road map items are usually described on a RnD page until they become a reality.
Maintaining the short term road map
Being the most important part of the road map, the process for maintaining and updating the short term
road map is well dened. Road map updates occur weekly and take place on the developer list as follows:
1. At the beginning of the week an email is sent to the developer list with the subject line #roadmap
update <DATE>. This email is more or less an quick overview of the current state of the road map
and contains:
The next scheduled release date
A list of the unresolved road map items
2. Developers (or any community members) may reply to the initial email and provide progress updates
on road map items assigned to them. Developers may also propose changes and updates to the road
map. Examples of such updates include proposing:
new features to be added to the road map, see Adding features
features which need to be moved back to a future release due to lack of resourcing
features which need to be moved back to a future release due to other issues
a change to the scheduled release date
3. Developers and community members discuss the proposed changes on the list. Those updates which
are agreed upon are factored into the road map and it is updated.
Adding features
During the road map update developers propose new features and improvements to be added to the road
map. The following are the prerequisites for proposing a new feature:
1. The feature has a sponsor. This means either a developer willing to carry out the work or a customer
who is paying for it
2. The feature has has gone through the GSIP process if necessary. Whether a feature requires a GSIP is
decided by the community when the feature is proposed
After a feature has met the above prerequisites it is assigned to a release on the road map. The determining
factor for what release a feature should be assigned to is based on the estimate of the time to implement
the feature, and the current Release cycle for the branch the feature is being implemented on. If the time to
implement the feature does not allow it to be assigned to the next release it is scheduled accordingly into a
future release.
Release cycle
GeoServer follows a regular release cycle. Usually this cycle is a release every month. However once a
development branch has become stable the release cycle drops off to every few months. Similarly on an
unstable development branch the release cycle can be every few months, until the branch becomes stable
enough for monthly releases.
14.5 GeoServer Improvement Proposals
GeoServer Improvements Proposals (GSIP) are the formal mechanism used to manage any sort of major
change to GeoServer. Examples of changes which are managed by the GSIP process include:
major features
code re-architecture
community process improvements
intellectual property
14.5.1 How a GSIP works
The typical life cycle of a GSIP is as follows:
1. Developer has an intent to perform a major change
2. Developer communicates with the community about the change
3. Developer goes off and implements the change
4. Developer writes a GSIP and presents it to the community for feedback
5. The PSC votes on the GSIP
6. Developer commits changes upon receiving a positive vote
14.5.2 Voting on a GSIP
One of the duties of the GeoServer Project Steering Committee is to vote on GSIPs. The voting process
works as follows:
Each PSC member gets a single vote, which can be one of +1, -1, 0
Any PSC member that votes negatively against a proposal must provide a reasonable explanation as
to why
Any PSC member that votes negatively against a proposal has a limited time to provide constructive
feedback as to how the vote can be turned
The GSIP author must incorporate any reasonable feedback into the proposal
Any negative vote is reviewed to determine if criteria has been met to turn it to a positive vote
The proposal is considered successful after a majority of positive votes is a achieved and all feedback
from any negative votes has been addressed
14.5. GeoServer Improvement Proposals 169
14.5.3 Implementing a GSIP
GSIPs are written up on the wiki. To make a GSIP:
1. Navigate to the proposals under discussion page
2. Log in to the wiki and click the Add Page link
3. Use the Proposal template to create the new page
4. Name the new page GSIP ## - <TITLE> where:
(a) ## is the number of the GSIP
(b) <TITLE> is the title of the GSIP
5. Fill in the information in the page template, and click Save when complete.
14.5.4 GSIP FAQ
Q. When do I need to make a GSIP?
A: Generally you will make a GSIP when you want something major done in GeoServer and need feedback
from others. GSIPs are needed for things that will have an impact on other community members, and thus
should be talked about.
GSIPs are not intended to be a way to get feedback on experimental work. There are alternate mechanisms
for this such as creating an R&D page on the wiki, svn branches and spikes, etc... Often once the idea is
formalized through experimentation it can be turned in to an ofcial GSIP.
Q. Who can comment on a GSIP?
A: Anyone and everyone can comment on a GSIP including regular users, and it is encouraged. Comments
an take place on the email lists, or as comments on the wiki page for the GSIP itself. Feedback from the user
community is denitely desired so that the developers do not just making decisions in a vacuum. If you
speak up your voice will be heard.
Q. Who can vote on a GSIP?
A: Only PSC members can ofcially vote on a GSIP. But more often than not this vote is based on general
feedback from the user community.
Q: What happens if I propose a GSIP but I dont have the time to implement it?
A: If a proposal is made but is not completed then it will be taken from a release schedule and live as a
deferred proposal. A deferred proposal can be picked up by the original developer or another developer,
or it can live in limbo until it ofcially gets rejected.
Q: If I am a PSC member and I make a GSIP, do I still vote on it?
A: Yes.
14.6 Community Modules
This document describes the GeoServer community module process. It is a guide that describes how the
GeoServer project takes in contributions from the community.
In GeoServer a module can fall into one of three classes:
core, those modules which GeoServer requires to function and are distributed with the main
GeoServer distribution
extension, plug-ins available as separate artifacts from the main distribution
community, experimental or unstable modules which are not part of the release process
Every module added to GeoServer has its origin as a community module. If the module becomes stable
enough it will eventually become part of the main GeoServer distribution either as a core module, or as an
extension.
14.6.1 Creating a community module
Requirements
The single requirement for adding a community module is the approval of one Project Steering Committee
member.
Process
The following outlines the steps to be taken in order to add a new community module.
1. Get approval
The rst step is to get approval to add the community module. This involves rst explaining to the
rest of the GeoServer community the purpose and function of the extension to be added. The two best
ways to do this are:
(a) send an email to the developers list, or
(b) participate in a weekly IRC meeting
After explaining the intention, the approval of at least one Project Steering Committee member is
needed before proceeding. Getting approval is easy as long as it is explained that the extension will
be useful to other users or developers.
2. Get version control access
The next step is to create the community module in the subversion repository. To do this it is necessary
to be granted commit status. The process for signing up for version control access is dened in the
Committing section.
3. Add a new module
Once commit access is obtained the module can be added. All community modules live under the
directory community, directly under the root of the source tree. The community modules on trunk
can be found here.
For example, from the root of the GeoServer source tree:
14.6. Community Modules 171
[geoserver]% cd community
[geoserver/community]% svn mkdir myCommunityModule
[geoserver/community]% svn commit -m "adding my community module" myCommunityModule
4. Add a Maven POM
Every module in the build requires a maven pom le, pom.xml. Use the following as a template:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<parent>
<artifactId>geoserver</artifactId>
<version>1.7.0-SNAPSHOT</version> 
</parent>
<artifactId>myCommunityModule</artifactId>
<name>My Community Module</name>
<dependencies>

</dependencies>
</project>
Add the le to the root of the new community module, myCommunityModule/pom.xml
5. Add a build prole
The nal step involves adding the new module to the maven build, and in particular adding a build
prole for it. To do this:
(a) Edit community/pom.xml and add the following inside of the <profiles> element:
<profiles>
...
<profile>
<id>myComunityModule</id>
<modules>
<module>myCommunityModule</module>
</modules>
</profile>
</profiles>
(b) Edit web/app/pom.xml and add the following inside of the <profiles> element:
<profiles>
...
<profile>
<id>myCommunityModule</id>
<dependencies>
<dependency>
<artifactId>myCommuityModule</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>
.. warning::
If the community module depends on any other community modules,
they too should be included in the profile definition.
.. warning::
Ensure that the name of the profile matches the name of the
community module
14.6.2 Promoting a community module
Once a community modules becomes stable, it may be promoted to a core or extension module. Which
depends on the nature of the community module. If the module is plug-in based (ie. it provides functional-
ity that some users may want, but others may not) then it should become an extension. Otherwise it should
become a core module.
Requirements
The following properties must hold true in order to promote a community module:
1. The module has at least a handful of users
In order to avoid cluttering the main code base, only those community modules which are of interest
to at least 3 users (this may include the maintainer) are promoted.
2. The module has a designated and active maintainer
Every core and extension module requires a module maintainer. The job of the maintainer is to x
bugs and address issues which arise with the module. If a community module is promoted and the
maintainer drops off, the module is in danger of being demoted back to community status.
3. The module is considered stable by the majority of the PSC
A module will only be promoted if it is deemed stable by the majority of the PSC. Those PSC
members deeming it unstable must provide a reasonable justication for the assertion.
4. The module maintains 40% test coverage
A minimum of 40% test coverage must be maintained by the module in order to be promoted. Of
course higher coverage is encouraged. The more test coverage a community module the more credi-
bility it gets.
5. The module has no IP violations
The module must not contain any code with a license or copyright that violates the GPL.
6. The module has a page in the user manual
Each module needs a page in the user manual documenting its function and usage. Tutorials and
walk-throughs are encouraged.
7. The maintainer has signed the GeoServer Contributor Agreement
The Open Planning Project (TOPP) retains all copyright on code released as part of GeoServer. Since
core and extension modules are released along with the rest of GeoServer, the maintainer of said
modules must agree to assign copyright of code to TOPP.
Process
1. Submit a GeoServer Improvement Proposal
To promote a community module the contributor must create a GeoServer Improvement Proposals
(GSIP). The proposal must then go through the regular feedback and voting process.
2. Move the module
Once the proposal is accepted, the next step is to move the module out of the community space. Where
the module ends up depends on wether it is being promoted to a core module, or an extension.
Core modules
Core modules live under the root of the source tree:
[geoserver]% svn move community/myCommunityModule .
[geoserver]% svn commit -m "promoting my community module to a core module" myCommunityModule community/
Extensions
Extension modules live under the extension directory, under the root of the source tree:
[geoserver]% svn move community/myCommunityModule extension
[geoserver]% svn commit -m "promoting my community module to an extension" extension community
3. Update the build
Once the module has been moved, the maven build must be updated.
Core modules
(a) Edit community/pom.xml and remove the prole for the community module
(b) Edit pom.xml under the root of the source tree and add a module enty:
<modules>
...
<module>myCommunityModule</module>
</modules>
(a) Edit web/app/pom.xml and move the dependency on the community module into the main
dependencies section of the pom. Then remove the prole
Extensions
(a) Copy the prole for the community module from community/pom.xml to
extension/pom.xml
(b) Remove the prole from community/pom.xml
4. Update the release process
The next step is to include the new module in the release process.
Core modules
(a) Edit release/src.xml and add an <include> for the module:
...
<moduleSets>
<moduleSet>
...
<include>org.geoserver:myCommunityModule</include>
</moduleSet>
</moduleSets>
...
Extensions
(a) Create a new directory under release/extensions which matches the name of the extension
(b) Add the following to the new directory:
i. A license called <module>-LICENSE.txt which contains the license for the extension
ii. A readme called <module>-README.txt which contains instructions on how to install the
extension
Warning: Dont skip this step.
iii. Any static les that are required by the extension (example would be a proprietary driver
not available for download via maven)
(c) Create a release descriptor called ext-<module>.xml under the release directory which follows
the following structure (where %module% is the name of the module):
<assembly>
<id>%module%</id>
<formats>
<format>zip</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<fileSets>
<fileSet>
<directory>release/extensions/%module%</directory>
<outputDirectory></outputDirectory>
<includes>
<include>
*
</include>
</includes>
</fileSet>
<fileSet>
<directory>release/target/dependency</directory>
<includes>
<include>%module%-
*
.jar</include>
</includes>
</fileSet>
<fileSet>
<directory>release/extensions</directory>
<includes>
<include>LICENSE.txt</include>
</includes>
</fileSet>
</fileSets>
</assembly>
*
Add additional include elements in the second fileSet for
the jar dependencies of the module
*
Add additional include elements in the third fileSet for
the static file dependencies of the module
(d) Add a dependency from release/pom.xml to the extension module:
<dependencies>
...
<dependency>
<groupId>org.geoserver.extension</groupId>
<artifactId>%module%</artifactId>
<version>%version%</version>
</dependency>
...
</dependencies>
(e) Add an entry for the release descriptor to the root pom.xml of the source tree (ie. one step up
from the release directory):

<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptors>
<descriptor>release/src.xml</descriptor>
<descriptor>release/war.xml</descriptor>
<descriptor>release/javadoc.xml</descriptor>
<descriptor>release/bin.xml</descriptor>
<descriptor>release/doc.xml</descriptor>
...
<descriptor>release/ext-%module%.xml</descriptor>
</descriptors>
</configuration>
</plugin>
(a) Update the documentation
Add a page to the user manual for the new module.
Todo
Finish this by linking somwhere...
(b) Download the contributor agreement
The nal step in the process is to download and ll out the contributor agreement form. Follow
the instructions on the form to submit it.
14.6.3 Demoting a community module
For one reason or another a module is neglected and becomes unmaintained. When this happens the
GeoServer PSC essentially becomes the maintainer and may decide to do one of two things:
1. Assume maintainership
In this case someone (may be more than one person) on the PSC agrees to take on maintainership
duties responsibilities for the module, such as bug xing
2. Demote the module
If no one steps up to maintain the module it may be demoted back to community status. If and when
a module is demoted depends on the circumstances. If the module is relatively quiet in that it just
works and not many bug reports arise from it, it may be left alone and not demoted.
Requirements
The following properties must hold true in order to demote a module back to community status:
1. The module has no designated maintainer
The module maintainer has stepped down or is unreachable and has not been active for a number of
weeks.
2. The module is problematic
The module contains one or more issues with blocker status, or contains a handful of issues with
high priority.
Process
The following outlines the steps to demote a module to community status:
1. Call for a maintainer
Before demoting the module rst try to nd a new maintainer for it. Send an email to both the
developer and user list advertising the module is in danger of getting pushed back to community
status. Wait a few days to see if anyone steps up to take on maintainership.
2. Move the module and update the build
If no one steps up to take on the maintainer role, reverse the steps described here, taken to promote
the module. In summary:
(a) Move the module back to the community directory
(b) Disable any of the modules release artifacts
(c) Move the prole for the module from extension/pom.xml to community/pom.xml in the
case of an extension module
14.6.4 Stepping down from module maintainership
Often a module maintainer does not have the time or resources to continue to maintain a contribution. This
is understood and is a fact of life in the open source software world. However, to relieve the burden on the
project and PSC, the following steps taken by any maintainer stepping down are highly appreciated.
1. Give notice
The more time you can give to the project in lieu of your departure the better. Send an email to the
developers list as soon as you know you will be dropping off
2. Find a new maintainer
While often not possible, any attempt to nd a new maintainer for the module is greatly appreciated.
14.7 Project Steering Committee
Welcome to the GeoServer organizational system, as with any open source project we start with people.
14.7. Project Steering Committee 177
14.7.1 Summary
This document describes the role and responsibilities of the to be created Project Steering Committee, as
well as the process under which it operates. Much of the denition and inspiration for the GeoServer PSC
is taken from the MapServer Technical Steering Committee and the Plone foundation.
The committee is made up of individuals based on merit irrespective of organization ties.
14.7.2 Structure
The PSC is made up of individuals who are meant to represent the various communities which have a stake
in GeoServer. An odd number is chosen to facilitate the voting process and help prevent ties. However,
even with an odd number, the voting system may still allow for a tie in some cases. For this reason the PSC
has an appointed Chair, whose sole responsibility is to break ties among the PSC.
Turnover is allowed and expected to accommodate people only able to become active on the project in
intervals. A PSC member may step down at any time.
14.7.3 Process
The primary role of the PSC is to make decisions relating to project management. The following decision
making process is used. It is based on the Proposal-Vote system.
Issues that require a decision are based on GeoServer Improvement Proposals (GSIPs). For more on
making a proposal see GeoServer Improvement Proposals
Proposals may be made by any interested party (PSC, Non-PSC, committer,user,etc...)
Proposals should be addressed within one week of being submitted, votes take place at the subse-
quent IRC meeting.
Each PSC member may vote one of the following in support of the proposal:
+1 : For
-1 : Against
+0: Mildly for, but mostly indifferent
-0: Mildly against, but mostly indifferent
A -1 vote must be accompanied with a detailed reason of being against.
A vote is successful if there is a majority of positive votes.
A vote is unanimous, if there are either:
1. No -1 votes against it, or
2. No +1 votes for it.
In the event of an successful non unanimous vote, the following steps are taken:
Each member who votes -1 _may_ supply an alternative with which the original author can use
to rework the proposal in order to satisfy that PSC member.
If at least one -1 voting PSC member supplies some alternative criteria, the original author must
rework the proposal and resubmit, and the voting process starts again from scratch.
If no -1 voters are able to supply alternative criteria, the proposal is accepted.
In the event of an unsuccessful vote, the author may rework and submit. A proposal may not be
resubmitted after being rejected three times.
Note that a majority of positive votes does not need to be a majority of the full PSC, just a majority
of the active PSC - dened by those present at an IRC meeting or responding within 4 days on
email. PSC members need not sound in on every single motion, but are expected to be active -
the section on stepping down details that if a PSC is not active for 2 months they will
When not to use the GSIP
A GSIP is only needed for:
an action that has a major effect on others in the GeoServer community.
If an action will break backwards compatibility, or change core code, a GSIP is recommended.
A GSIP is NOT needed for:
an improvement that can go in a community module; or
a bug x that doesnt rework anything substantially
For minor decisions where feedback might be desired, the course of action to take is to consult the develop-
ment list or raise it in an irc meeting. The GeoServer Project recognizes that it is run those who are actually
doing the work, and thus we want to avoid high overhead for getting things done.
Note: Snap Decisions
For all decisions that are not ofcial GSIP proposals, those present (those in the IRCmeeting or who bother
to respond to an email within 4 days) are given the power to vote and decide an issue. The same voting
procedures are used, but any vote that meets a -1 from any party present (even a new user), should go to a
GSIP.
14.7.4 Responsibilities
Responsibilities of PSC members fall into the following categories:
1. Operations
2. Planning
Operations
Day to day project management. Duties include:
Note: Archive
Weekly IRC Meeting Attendance
PSC members are expected to attend one of the weekly IRC meetings. Of course this is not
always possible due to various reasons. If known in advance that a member cannot attend
a meeting the member should email the developer list. No reason need to be given for not
attending the meeting. Meeting absences are subject to the following policies.
If a member misses three consecutive meetings without prior notice, they are eligible to be voted off of the PSC.
If a member continuously misses IRC meetings (with or without notice) they may be asked to step down to make way
for a potentially more active member.
Illness and vacation are acceptable exceptions to the above policies. However extended absence from the project may
lead to the member being temporarily dismissed from the PSC.
Mailing List Participation
PSC members are expected to be active on both user and developer email lists, subject to open-source
mailing list etiquette of course.
It is a requirement that all PSC members maintain good public visibility with respect to activity and management of
the project. This cannot happen without a good frequency of email on the mailing lists.
Planning
Long term project management. Duties include:
Guiding Major Development Efforts
PSC members are expected to help guide the major development efforts of the project. This may include deciding which
development efforts should receive priority when different efforts are in conict.
The PSC has the right to veto any proposed development efforts.
A major development effort which is intended to become part of the core of GeoServer can be proposed by any interested
part, PSC, or non PSC. However, the effort must be approved by the PSC before it can begin.
Project Policies
The PSC is responsible for dening project policies and practiced. Examples include:
Development Practices
Code Reviews
Intellectual Property
Documentation Requirements
Commit Access
Testing Requirements
Branch Culture
Release Procedures
Frequency
Version numbering
Stable vs R&D
14.7.5 Current PSC
Alessio Fabiani
Andrea Aime
Ben Caradoc-Davies
Chris Holmes (Chair)
Christian Mueller
Gabriel Roldan
Jody Garnett
Jukka Rahkonen
Justin Deoliveira
Phil Scadden
Simone Giannecchini
14.7.6 PSC Voting procedure
Bootstrapping
First a chair is chosen by the current group of active committers. The Chair is then removed from the
nominee list.
Everyone on the email lists gets 5 votes for PSC,. Once the list is accepted by those nominated, a volunteer
will privately gather the votes posting the results. The 7 nominees receiving the most 5 votes will be selected
as the PSC.
Future PSC members
A new PSC member can be nominated at any time. Voting for a new PSC is done by current active PSC
members. There is no hard limit to the number of PSC members, but we want a relatively active PSC. PSC
nominations are generally given in recognition to very signicant contributions to the project. Membership
is open to non-technical people, for example if someone is to make huge advances to the documentations
or marketing of GeoServer, for example.
Since we demand a fairly active PSCwe expect turnover may be high compared to other projects, so initially
we will aim to keep it around 7 PSC members. But given sufcient reason we will expand that.
Nominated PSC members must recieve a majority of +1 votes from the PSC, and no -1s.
14.7.7 Stepping Down
If you nd you cannot make meetings for a month or two, by all means step aside. Thank you so much
for your time, if you want to groom a successor and then nominate them that is cool, but the nomination
process still applies.
If we do not hear from you for two months we will assume you lost, send out a search party and nominate
your replacement.
That is to say, status on PSC is lost if not active at all in a two month period of time. Of course you can come
back on to the PSC if you become active again, but a new nomination procedure will be needed.
14.7.8 Dissolution of PSC
If there are no suitable replacements, the PSC can decide to go down in number. If the number of active PSC
members drops below 5, however, then TOPP reserves the right to take back active control of the project or
to pass it on to an organization that will effectively steer it. We sincerely hope that will never happen, but
want a policy in place so that we avoid a zombie project, forcing someone else to eventually fork GeoServer
or some such.

GeoServerDeveloperManual PDF

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

GeoServerDeveloperManual PDF

Încărcat de

Drepturi de autor:

Formate disponibile

GeoServer Developer Manual

S-ar putea să vă placă și