Evaluation of Open Source (OS)

Content Management Systems (CMS):

Alfresco, Drupal, and Joomla!

18 September 2007
Evaluation of OS CMS: Alfresco, Drupal, Joomla! Absolute North Communications

Executive Summary
Businesses are using content management systems (CMS) to perform web administrative
functions, manage assets, provide personalization and localization features, and much
more. However, selecting an open source CMS is difficult because there are many
options. This report uses trend data from independent research organizations to form
evaluation criteria that can be used to further analyze CMS software. Evaluation criteria
were then used to evaluate three different open source products: Alfresco, Joomla!, and
Drupal to determine which products were suitable for different web requirements.

Results indicate that Alfresco is suitable for websites that require advanced functionality
such as detailed content analytics or document and knowledge management features.
Drupal is suitable for sites that require intermediate level functionality, such as the ability
to grant access of one portion of the site to a subset of users, or the ability to maintain
multiple sites. Joomla! is suitable for basic sites that require some intermediate level
functionality, such as blogs and forums.

2
Evaluation of OS CMS: Alfresco, Drupal, Joomla! Absolute North Communications

Table of Contents
Executive Summary ............................................................................................................ 2
Table of Contents................................................................................................................ 3
Introduction......................................................................................................................... 4
Definitions and Terms......................................................................................................... 5
Methodology ................................................................................................................... 5
Literature Review and Industry Research....................................................................... 6
Evaluation Criteria .......................................................................................................... 7
Candidate Selection ........................................................................................................ 8
Evaluation and Results.................................................................................................... 9
Conclusion ........................................................................................................................ 12
Appendix........................................................................................................................... 13
End notes........................................................................................................................... 16

3
Evaluation of OS CMS: Alfresco, Drupal, Joomla! Absolute North Communications

Introduction
Once upon a time, when businesses were starting to establish their web presences and
visitors had limited bandwidth, many websites functioned as brochures. Users visited
these websites, which consisted primarily of static HTML pages, to learn more about
products and services. Interaction with the company meant sending an email to the sales
representative or finding the toll-free telephone number to call technical support. Today,
as visitors grow more web savvy, and as access to high-bandwidth connections become
more prevalent, visitor expectations have changed. Visitors now expect training videos,
to be able to connect with other visitors via forums, to be able to find white papers that
answer specific questions, and much more. Additionally, as more people from different
countries gain internet access, businesses must find ways to globalize their web content.
Multinational companies often communicate with geographically and linguistically
diverse employees via internal intranets.

To help meet these needs, businesses are using Content Management Systems (CMS) in
conjunction with other products. CMS are used to perform web administrative functions,
manage assets, provide personalization and localization features, and much more. There
are many subcategories of CMS. A Web Content Management (WCM) system has
additional features specifically tailored to manage web site content. A Digital Asset
Management (DAM) system has additional features to support the ingesting, cataloguing,
storing, retrieving, and annotating of digital material. An Enterprise Content
Management (ECM) system is usually comprised of a WCM and DAM, and provides
additional knowledge management and document management features as well.

However, the dizzying array of options can be daunting for even the most seasoned
information technology (IT) administrator. CMS Matrix, an open source organization
that tracks and rates content management systems, lists over 800 different open source
CMS packages.i And, if we expand the list to include commercial packages, the list
grows longer.

If an administrator requires a CMS for a mid to large company, and there are funds for a
commercial solution, the selection process becomes easier. There are many respected
independent IT research firms, such as Gartner Incorporated, Forrester Research, and
International Data Corporation (IDC) that regularly determine which applications are
best-of-breed. The administrator can use this material, coupled with a thorough
understanding of business requirements to quickly identify viable candidates.

But, evaluating open source CMS packages is a bit more difficult. The few firms that
concentrate on open source applications may or may not include trend analysis. Those
that do often produce reports that are expensive for many. For example, the 2007 CMS
Watch Web Content Management System report ranges from $975 to $2975, depending
on licensing options.ii Although Gartner, Forrester, and IDC offer reports within the
same price ranges, if commercial vendors are positively reviewed, they are more likely to
purchase the rights to publish those reports on the web. Although the intent of these

4
Evaluation of OS CMS: Alfresco, Drupal, Joomla! Absolute North Communications

companies is to use these reports to attract potential customers, researchers interested in

open source can exploit this free source of information to obtain trend analysis data that
is standard in these reports.

This document uses that approach to evaluate three open source CMS packages: Joomla!,
Drupal, and Alfresco. These three packages are widely considered best-of-breed in the
open source community. Additionally, because this document focuses on web site
management, the CMS packages selected are those that have been categorized as WCM
software or have a significant WCM component. These packages have then been
evaluated to determine which type of website it best serves.

Definitions and Terms

Although the CMS and its subcategories have been briefly discussed, it is helpful to gain
a fuller understanding of each.
Content Management System (CMS). A content management system is
software that is used to support the creating, updating, publishing, translating,
archiving, and retiring of digital information. Standard features, such as tracking
the changes made to digital information are often included. CMS is a generic
term that can be applied to many different types of software. Subcategories of
CMS include web content management and digital asset management software.
Digital Asset Management (DAM). Digital asset management software has
additional features to support the ingesting, cataloguing, storing, retrieving, and
annotating of digital material. Traditionally used to support media organizations,
DAM software is being increasingly used to store corporate information, such as
webcasts, podcasts, or audio material.
Web Content Management (WCM). Web content management software has
additional features specifically tailored to manage web site content. Examples
include the ability to publish content on a web server or the ability to provide web
content in different languages (commonly part of a set of features related to
localization).
Enterprise Content Management (ECM). According to Association for
Information and Image Management (AIIM), ECM software are the technologies
used to capture, manage, store, preserve, and deliver content and documents
related to organizational processes.iii ECMs often contain include WCM, DAM,
and knowledge management features.

Methodology
A traditional software evaluation process involves approximately seven steps:
Requirements Gathering, Literature and Industry Research Review, Candidate Selection,
Request for Proposal, Vendor Demonstration, Formal Evaluation, and Software
Selection.

However, since the goal of this process is not to select software, but rather to determine
which software can be used to support which website goals, modifications to this process
have been made.

5
Evaluation of OS CMS: Alfresco, Drupal, Joomla! Absolute North Communications

1. Literature and Industry Research Review

2. Development of Open Source Evaluation Criteria
3. CMS Search and Candidate Selection
4. Formal Evaluation
5. Software Mapping and Categorization

First, research literature was reviewed to identify trends and commercially-used

evaluation criteria. Reports from independent research organizations were reviewed and
evaluation models identified. Next, review and trend data was distilled to generate
evaluation criteria suitable for open source software.

Next, data from independent open source research organizations were evaluated to
determine which CMS products were considered to be best-of-breed. Additionally
industry awards, awards were reviewed to find candidates. At the conclusion of this, three
candidates were identified for inclusion in this report.

Once candidates were identified, the actual software was evaluated using the open source
evaluation criteria, CMS Matrix data, and actual working copies of the software.
Documentation, as well as comments from users, were evaluated against the raw data.
As part of the evaluation, results were compiled in a spreadsheet and software was
categorized according to website level.

Literature Review and Industry Research

In 2006, Gartner published WCM and DAM: The Next Generation, which outlined key
issues and trends in web development that have affected the development of web content
management and digital asset management systems. Gartner found that WCM will
continue to offer core functionality, such as workflow management, library services, and
access to templates.iv However, because of changing user expectations, businesses are
beginning to create websites that are more targeted and focused on the customer.
Therefore, high-end CMS packages are offering analytics to track user behavior, a means
for feedback, and stronger integration with portal software. Previously advanced
features, such as blogs and forums are now standard. The Forrester Wave: Web Content
Management for External Sites, Q3 2007 echoes Gartners findings. v In addition to
addressing the need to deliver targeted information, the report also highlights the need for
consistent branding. Internal employees who dont have programming skills want to be
able to perform administrative functions such as create personalization rules or
administer multiple sites. Both reports indicate that presenting content in different
languages, reusing content, and having the ability to use metadata to tag content are vital
for high-functioning web sites.

The CMS market has changed significantly to address these trends. In particular, the
enterprise content management market has been volatile. As ECM vendors have realized
the importance of the web content management, many have rushed to acquire companies
with strong track records in this area. Consider IBMs acquisition of FileNet, SDLs
acquisition of Tridion, or Oracles recent acquisition of Stellent, which is one of the most

6
Evaluation of OS CMS: Alfresco, Drupal, Joomla! Absolute North Communications

feature-rich WCM packages on the market. Major ECM vendors have recognized that an
ECM suite is not complete without strong WCM. In response, WCM products not
associated with an ECM are quickly adding more features to stay competitive. Because
commercial ECMs can be quite expensive (ranging from $100K to over $1M, depending
on options purchased), standalone WCMs are a viable option, particularly for small to
mid-sized businesses.

Evaluation Criteria
The Forresters WCM Model, which maps different web goals to WCM features, served
as a starting point for the development of this reports evaluation criteria.vi The
evaluation criteria used in this report incorporates industry trends and basic website
goals. These criteria have been specifically formulated for open source applications. For
example, for commercial applications, the corporate commitment to a product is partially
determined by whether a vendor has been profitable and the number of employees
dedicated to the product line. For open source applications, one way to measure this is to
determine the size of the user community and the frequency of releases. So, an open
source project that is supported by two people and that has had no releases for the past
two years is the equivalent of a company that has been posting record losses and is facing
bankruptcy.

Table 1: Web Evaluation Criteria

Website Goal WCM Evaluation Criteria
Basic: Establishing
Web Presence
Intermediate: Adding
Internal and External
Functionality and
Standardization
Is it low cost?
Is technical support and developer support available (e.g.
documentation, forums, mail lists)?
Is work ongoing (updates, patches)? Has there been a
release, patch or update within the past year?
Can users publish a static page?
Are social networking plug-ins available (e.g. blogs,
forums, chat rooms)?
Can users check in and check out digital assets to a
centralized library? Is it version controlled?
Can people who dont know HTML enter and publish
content?
Is there multi-site management? Can you share content
among sites?
Can administration of the site be delegated?
Are search logs available? Basic statistical reports (e.g.
impressions, keywords)?
Is this product a portal or can it be integrated with one?
Is this product compliant with major W3C
recommendations/standards (e.g. 508C), accessibility
standards, and JSR 168 (for portals)?
Are templates available?
Can users customize the site?

7
Evaluation of OS CMS: Alfresco, Drupal, Joomla! Absolute North Communications

Table 1: Web Evaluation Criteria

Website Goal WCM Evaluation Criteria
Is localization supported?
Advanced: Targeted Can users customize workflow?
Operations and Are content analytical tools available?
Services Can users publish information in a variety of formats
(multi-channel publishing), such as Word, PDF, or XML?
Is site and archive retention available?
Are there metadata tags available for content?

Candidate Selection
The first step in candidate selection was to find a systematic means of sifting through the
800+ open source CMS packages. CMS Matrix provides a website for users in the
community to download and review CMS packages. Users review software using nine
categories: system requirements, security, support, ease of use, performance,
management, interoperability, flexibility, built-in applications, and commerce. Because
of the systematic nature of the review, this seemed to be a good starting point.

Only products that had, at minimum, 150 user reviews (indicating a large user base or at
least interest in the product) and that scored a six or higher in six of the nine categories
were considered as potential candidates. From these, the top five for each category was
selected.

Additionally industry awards, such as InfoWorlds Best of Open Source Software

(BOSSIES) and the Packt Open Source Content Management System awards were
reviewed to find potential candidates.vii viii The following tests were then applied to each
system potential candidate:

1. Software must provide basic WCM capability including the ability to publish
pages, connect to or function as a portal, and perform web administration, and
maintain user accounts.
2. Software must have an active user base, as is evidenced by the presence of forums
and/or mailing lists as well as user groups and/or conferences. Activity must have
been detected in these forums or lists within the past 14 days.
3. Software must have demonstrated evidence of product development within the
past year. This includes major or minor releases, patches, or testing.
4. Software demonstrates that there are plans for future upgrades or releases.

At the end of this process, three candidates emerged as clear contenders:

Joomla! is a content management system that allows users to build websites of

varying complexity using a combination of user entered data, custom code, and
extensions. Although the product is fairly young (the first release was in 2005), it
is built on the Mambo codebase, which was first developed in 2000. Joomla!
meets all basic criteria and was announced the winner of the Packt Publishing
Open Source Content Management System Award in 2006.ix

8
Evaluation of OS CMS: Alfresco, Drupal, Joomla! Absolute North Communications

Drupal is a CMS that provides a means for users to quickly build web sites.
There are many modules available right out of the box as well as extensions.
Drupal was originally built as a bulletin board system and became open source in
2001. Drupal won the CNet Webware 100 and was second runner-up (by one
vote) in the Packt Publishing Open Source Content Management System Award
in 2006.x
Alfresco is an Enterprise Content Management system that offers web and
document content management functionality as well as records, knowledge, and
image management features. Only the web content management features were
evaluated. Alfresco was founded in 2005 by the former Chief Operating Officer
of Business Objects, a major business intelligence software company. Since its
inception, Alfresco has won numerous awards, including the InfoWorld BOSSIE
and KMWorlds Trendsetting project of 2006.xi

Evaluation and Results

Overall, the software evaluations revealed that each product could be successfully used
for different types of sites. The following table presents a summary of the results. Refer
to Table 1: Web Evaluation Criteria for more information on the Web Site Goal/Criteria.
A detailed list of results appears in the appendix of this document.

Table 2: Evaluation Summary

Web Site Joomla! Drupal Alfresco
Goal/Criteria
Basic X (Good, but X X (But may be too
standards complex. This is an
compliance might issue because
be an issue.) technical support is
not free.)
Intermediate X X
Advanced X

Joomla! Joomla! handily fulfills the requirements needed by basic websites. In addition
to providing the standard features, Joomla! has a large and active development and user
community, user groups, and conferences. There are also many plugins to add
functionality. Web forms make updating content easy for non-HTML users.

9
Evaluation of OS CMS: Alfresco, Drupal, Joomla! Absolute North Communications

Figure 1: Joomla! Control Panel

However, site administration is substandard. Permissions are preset, not granular, so

administrators precisely control user access to different functions. Another drawback is
that Joomla! does not offer multi-site management, which would be needed to handle
more advanced business needs. There are issues with compliance to industry standards as
well. However, a commercial plugin can be purchased to supply this feature. Search
analytic tools are limited at best, but the software does support the use of Google
Analytics. There is also no support for secure socket layers, which is a basic requirement
for secure web sites. Note that Joomla! 1.5 will be released soon and may address some
of these issues.

Pros: Strong developer/user community, many templates, many plug-ins

Cons: Permissions not granular, can not administer multiple sites without paying
for a plugin, no SSL
Recommendation: Good option for Basic Web Sites

Drupal: Drupal is very simple to use and easy to administer. User permissions are
granular, so administrators can easily control the features users have access to. In
addition to meeting basic web requirements, it provides many advanced features that
make it a good option for intermediate-level web sites.

10
Evaluation of OS CMS: Alfresco, Drupal, Joomla! Absolute North Communications

Figure 2: Drupal Content Management Page

However, there are some simple functions, such as including an image on a page that can
be cumbersome initially. Drupal has a large development and user community. It allows
those without HTML knowledge to edit sites and has rudimentary change management
features. Like Joomla!, Drupal supports Google Analytics.

Pros: Strong developer/user community, multi-site management

Cons: Functionality for simple functions is sometimes cumbersome
Recommendation: Good option for both Basic and Intermediate Level Sites

Alfresco: Alfresco is the most robust of the WCM evaluated in this report. Alfresco
offers all of the functionality needed for basic websites (e.g. page publishing, web forms
entry, multi-site management, delegated administration) as well as for advanced sites.
Alfresco is compliant with many W3C and industry standards (such as 508 and JSR 168),
which means easier integration with other products. It also includes a virtual server that
provides previews of site changes. Users can publish to multiple channels such as PDF
or Word. The workflow can be customized to the business and is email based. In
addition to electronic image capture and document management capabilities, the
application supports java server faces, a huge advantage for those using Struts to develop
web applications. Content analytics and archived format conversion are some of the
features that make this product suitable for advanced websites.

11
Evaluation of OS CMS: Alfresco, Drupal, Joomla! Absolute North Communications

Figure 3: Alfresco Dashboard

Alfresco is not without drawbacks however. One major drawback is that although
Alfresco is free, technical support is not. Although user groups are forming, because
Alfresco is fairly new, there are not as many free, alternative sources of support.

Pros: Feature-rich, advanced capabilities, industry standard compliance

Cons: Technical Support
Recommendation: Good option for intermediate and advanced web sites

Conclusion
Today, businesses are using content management systems (CMS) to help them deliver
targeted information for visitors interested in their products and services. There are many
software packages available, however selecting a CMS is sometimes difficult because
trend and evaluation data does not always coexist in the same reports. Using a process
that incorporates trend analysis conducted by independent researchers and feature
analysis conducted by open source organizations can often facilitate decision making.

The results of this report indicate Alfresco, Joomla!, and Drupal have various strengths
and weaknesses. Alfresco provides advanced functionality, but may require more
expertise to administer since user support is somewhat lacking. Joomla! provides good
basic functionality and is very easy to use, but is missing some key features. Drupal is a
good solution for basic and intermediate websites and can be easily administered
(although some features are a bit cumbersome). Ultimately, the best path for business
owners is to use trend data, coupled with their own requirements to choose a CMS that
will support their web strategy, both today and in the future.

12
Evaluation of OS CMS: Alfresco, Drupal, Joomla! Absolute North Communications

Appendix
The following is the detailed evaluation criteria for each CMS candidate.

Drupal Evaluation

Version: 5.2
Website: www.drupal.org
Evaluator: Chantel Brathwaite
Evaluation Date: 8-Sep-07
Criteria Score Comments
Cost (Software + Tech
Support) 5
Developer Support 5 Huge community. Lots of support. Legacy
Hard to do some very basic things. But
Static Page Publishing 4 overall very good.
Social Networking Tools Large variety of plugins and core
(blogs, forums, etc.) 5 components.
Library Functions (Check No version control, but can interface with
In / Check Out) 3 systems to provide this.
Very intuitive although simple actions can be
Web forms content entry 4 complex.
Standards compliance
(508, JSR 168, etc.) 5
Multisite management 2 A little kludgy - must be done using scripts
Delegated administration 5 Fully customizable
Content sharing between
sites 3
Localization 3 Based on user preferences
Personalization
(customize look and feel) 5
Content analytics 3 Extremely limited, supports Google analytics
Segmentation and content and language - based on user
personalization 3 preferences
Multichannel (email,
newsletters, websites) 0
Archive and retention 0
Workflow Management 0
Metadata Support 0

13
Evaluation of OS CMS: Alfresco, Drupal, Joomla!
Joomla! Evaluation
Version: 1.0.7
Website: www.joomla.org
Evaluator: C. Brathwaite
Evaluation Date: 6-Sep-07
Criteria
Cost (Software + Tech Support)
Developer Support
Static Page Publishing
Social Networking Tools (blogs,
forums, etc.)
Library Functions (Check In /
Check Out)
Web forms content entry
Standards compliance (508, JSR
168, etc.)
Web forms content entry
Multisite management
Delegated administration
Content sharing between sites
Localization
Personalization (customize look
and feel)
Content analytics
Multichannel (email, newsletters,
websites)
Archive and retention
Workflow Management
Metadata Support
Absolute North Communications
Score Comments
5
Many user groups, forums, can interact with
5 the team. Very enthusiastic community.
5
Large variety of plugins and core
5 components.
No version control, but can interface with
2 systems to provide this.
5 Very intuitive
Fair. The team is addressing this but this
3 could hinder the adoption of this tool
5 Simple!
1 Only for pay (http://www.elearningforce.biz/)
Somewhat inflexible. Can't assign user to
more than one role. Author, editor,
publisher, manager, administrator,
3 superuser
0
4
5
has a little web analytics in core package,
but nothing in-depth. There are extensions.
Will need a different tool or extension to do
more analysis. Google analytics is
available for download. See what users are
searching, trends, report summaries.
2 Landing page optimization.
0
0
0
0
14
Evaluation of OS CMS: Alfresco, Drupal, Joomla! Absolute North Communications

Alfresco Evaluation
Version: 2.1
Website: www.alfresco.com
Evaluator: Chantel Brathwaite
Evaluation Date: 10-Sep-07
Criteria
Cost (Software + Tech Support)
Developer Support
Static Page Publishing
Social Networking Tools (blogs,
forums, etc.)
Library Functions (Check In /
Check Out)
Web forms content entry
Standards compliance (508, JSR
168, etc.)
Web forms content entry
Multisite management
Score Comments
Software is free but technical support is not.
2 Can be quite costly.
Not a huge user community yet, the forum
at Alfresco seems to be monitored but
providing free support would probably work
3 against their bottom line.
5
5
5 Since Alfresco is an ECM, it excels at this
5 Very intuitive
5 Great, one of the strengths of the tool
5
Can do this and have virtual server
5 previews

Delegated administration 5
Content sharing between sites 5
Localization 5
Personalization (customize look
and feel) 5

Content analytics 5 Robust package

Multichannel (email, newsletters,
websites) 5
Archive and retention 5
Workflow Management 5
Metadata Support 5

15
Evaluation of OS CMS: Alfresco, Drupal, Joomla! Absolute North Communications

End notes
i
CMSMatrix Home Page. http://www.cmsmatrix.org/matrix September 18, 2007.
ii
CMSWatch Pricing page. http://cmswatchstore.com/index.html. September 18, 2007.
iii
AIIM Home Page. http://www.aiim.org/about-ecm.asp. September 18, 2007.
iv
WCM and DAM: The Next Generation Gartner.
v
The Forrester Wave: Web Content Management For External Sites, Q3 2007 September 2007, page 2.
vi
The Forrester Wave: Web Content Management For External Sites, Q3 2007 September 2007, page 4.
vii
Packt Pub. http://www.packtpub.com/award
viii
InfoWorld. http://www.infoworld.com/article/07/09/10/37FE-boss-intro_1.html.
ix
Packt Publishing, http://www.packtpub.com/article/open-source-content-management-system-award-
winner-announced
x
CNet. http://drupal.org/node/152770
xi
Alfresco. http://www.alfresco.com/about/awards/

16