Cost-Benefit Analysis of SysML Modelling for the

Atomic Clock Ensemble in Space (ACES) Simulator

Julien Maurandy Eberhard Gill
Julien.maurandy@gmail.com E.K.A.Gill@tudelft.nl
TU Delft, Space Systems TU Delft, Space Systems Engineering
Engineering Kluyverweg 1
EADS Astrium GmbH 2629 HS Delft, The Netherlands

Achim Helm Roland Stalford

EADS Astrium GmbH EADS Astrium GmbH
Location Friedrichshafen Location Friedrichshafen
Claude-Dornier Straße Claude-Dornier Straße
88090 Immenstaad, Germany 88090 Immenstaad, Germany
Copyright © 2012 by Julien Maurandy, Ebehard Gill, Achim Helm and Roland Stalford. Published and used by INCOSE with permission.

Abstract. Systems Engineering has successfully supported the space industry since its
inception with methodologies and techniques to handle complex projects. However, the
conventional design approach, ‘Document-Based Systems Engineering’ (DBSE), is more and
more reaching its limits. This research evaluates the benefits and the cost associated with the
paradigm of ‘Model Based Systems Engineering’ (MBSE) instead of DBSE by applying the
Systems Modelling Language (SysML) in the frame of the Atomic Clock Ensemble in Space
(ACES) project. ACES is indeed deemed to be a suitable project for this study as it includes both
a very complex ground segment as well as a challenging space segment on board the
International Space Station.
Following an introduction to SysML and the ACES mission, we first develop the metrics on
which the cost-benefit analysis is based on. Then, we explain the methodology and models used
to perform the analysis which targets to characterise DBSE versus MBSE based on the ACES
ground segment development. Then, we use the Analytical Hierarchical Process to determine
weighted criteria where attention is also given to the transitional process between DBSE and
MBSE.
After a critical reflection upon the analysis methodology and its results, we focus on lessons-
learned from the use of SysML for the implementation of MBSE in space projects. We have
identified five key areas of lessons-learned for using MBSE with SysML itself as well as main
deficiencies for Enterprise Architect, the tool used to implement SysML. We conclude with
seven suggested improvements which are considered valuable to help improving the
performance and acceptance of MBSE for the development of (space) projects in the future.
Keywords: Model-Based Systems Engineering, SysML, Systems Engineering, Modelling,
Space Engineering, Cost-Benefit analysis, Atomic Clocks Ensemble in Space (ACES),
International Space Station (ISS)
 1 Introduction
As projects in the industry are getting bigger and more complex, they involve an increasing
number of stakeholders, not only on the end user side, but also on the rest of the life cycle of the
product, including design, development, manufacturing and end of life. Their requirements also
grow in number and complexity. Systems Engineering is the discipline that allows dealing with
this problem based on its processes, techniques and methodologies.
Though, those processes address very relevant questions, they are always ‘document based’,
meaning the information is stored and exchanged in the form of written documents. Model-
Based Systems Engineering (MBSE) provides a novel approach to Systems Engineering and is
presented as a candidate to solve many problems of nowadays Systems Engineering. SysML
appears in this context as a modelling language, and therefore enables MBSE. If MBSE with
SysML holds its promises, it may one day become a standard in Systems Engineering and project
management.

1.1 Research objectives and paper outline

Through this research, the cost and the benefits of Model Based Systems Engineering with
SysML are evaluated using an actual project as a support for the assessment. The objectives of
this research were:
• Identify the benefits of Model Based Systems Engineering with SysML, in the context of
ACES Ground segment's Mission Simulator. Aspects to be covered in that respect are
communicability improvements, completeness, and consistency.
• Trade these benefits against their cost. The trade-off shall address relevant weighed
parameters chosen in cooperation with EADS Astrium.
The paper is outlined as follow: it starts by an introduction to the research environment,
including MBSE, SysML, the project (ACES), and the different pieces of software used in this
research work. In part two and three the analysis methodology is presented and justified. Follows
then a selection of sample representative problems encountered during the modelling is
presented. Part five presents the results of the study. Part six through nine respectively expose a
critic view of the analysis performed, the lessons learned, suggestion to improve SysML, and the
conclusion.

1.2 Introduction to SysML

SysML is a language which allows describing systems. The language is defined in a
document called the SysML Specifications (1). Using this language to describe a system, one
builds a model of this system, and uses the Model-Based Systems Engineering (MBSE)
approach. A model is "a simplified description of a system or a process" according to (2). In
Model-Based Systems Engineering, the model will gather all the information related to the
project. It is the set of all elements, their properties and links between them. Note that it is
important to distinguish the tool (a piece of software) which allows making a model using
SysML, and actually modelling the system, from SysML itself. An analogy would be
distinguishing MS Word from English.
 SysML intends to support design activities by providing a convenient way of capturing and
handling design information. SysML intends to allow engineers to work more efficiently by
providing a single and intuitive “slot” for each piece of information: in that way, engineers
should always be able to know very quickly whether the information is available or not, and in
the event it is, what the value or the design decision that has been made is, and why it is such
value or design decision. Another difficulty SysML undertakes is the communication between
engineers, as well as between the company, its clients and its subcontractors. The customary
approach is to prepare a document, which may differ depending on the targeted entity. SysML
proposes constructs that allow gathering the relevant information in a specific region of the
model. Using those constructs, modellers can prepare a “view” of the model. Views will then be
automatically updated by the SysML modelling software after each modification of the
information contained in the model. Tools also often propose document generation capabilities,
which allows having always up-to-date documents as they are based on the model on which the
team works. The last difficulty undertaken has deeper origins. In traditional Systems
Engineering, complex systems are decomposed into a hierarchy of smaller blocks until the
smallest blocks are manageable by a team of engineers. This methodology has been now applied
for several decades, but according to (3), some systems are not decomposable in such a way
anymore, as of the many interactions between each block. As systems are getting increasingly
complex, the number of such indecomposable systems will keep increasing with time. Or in
other words, the risks of missing an crucial interaction between two different blocks will keep
increasing. The authors logically deduce that Systems Engineering has to adapt in order to
consider the system more as a whole and avoid missing any interactions, as many of nowadays
major project failures usually arise from unforeseen interactions. According to them, MBSE
could be the answer to this problem, as it could provide a holistic view of the system. The
purposes of MBSE with SysML can therefore be summarized in four points:
• Provide a support for high level design information storage
• Allow easy access to information and easy modification: increased consistency
• Enable enhanced communication between the different stakeholders of the project:
increased communicability
• Provide a holistic view of the System, which helps detecting unexpected interactions
within the system: decreased risks.
Through those objectives, SysML aims to improve development time, and reduce
development costs and risks. For more information about SysML, readers are encouraged to refer
to the relevant literature. Beside the SysML Specifications (1), several books provide an easy
access to SysML such as (4) and (5).

1.3 Related work

As SysML emerged only in 2006, scientific researches concerning SysML are therefore
limited. Most papers tend to assess pros and cons of Model-Based Engineering and Development
(which is not necessarily Systems Engineering). Fieber et al. (6) studied the usability of model-
driven software development in industrial projects. They concluded that model-driven does not
yet “deliver its promises”. A similar study is the one of Sascha Kirstan et al. (7). They could
identify several advantages, without actually proving they were reached. They also underline that
no cost/benefits analysis study has been done to this date concerning Model-based development.
 An assessment of SysML itself (8) was also done early in its development (2005) by Erik
Herzog and Asmus Pandikow. The authors explain they had at first a lot of "easy picks" against
SysML, but that they came to change their mind with updated versions. So much that they even
changed their initial goal and decided to contribute to the specifications. The paper gives in
particular a list of observations concerning SysML, and concludes that though there is still a long
way to go, SysML remains an impressive achievement which is worthy of attention. Nicolas
Belloir et al. published more recently a paper (9), identifying, among other things, the difficulty
to differentiate software and hardware past a certain level. Bone and Cloutier present in their
paper (10) the results of their survey, which give a good picture of Model-Based Systems
Engineering. Karban et al. (11) also give an interesting evaluation. They identify some traps in
which modellers can fall, and introduce “best practice, guidelines and lessons learnt” as way to
avoid those traps, “enhance productivity, and facilitate communication across the disciplines of
the project”. They also underline the necessity of providing the team with adequate tool training
and modelling mentoring.
In another category, some papers investigate how to work with SysML, and how to increase
its potential in particular domains. Yves Vanderperren and Wim Dehaene (12) investigated
model-driven development in the frame of “System-on-a-chip” (SoC). Karban et al. (13)
explored how to use SysML construct to re-use existing modelling elements developed in
previous projects.
Finally, it is worth mentioning that in parallel to this research, another one is carried out by
Dorus de Lange, to assess how well Model Based Systems Engineering could fit with concurrent
design at the ESTEC (14).

1.4 The ACES Mission

This analysis was performed using ACES (Atomic Clock Ensemble in Space) as a support
for the modelling activities. ACES is an initiative of the European Space Agency. It is a
scientific mission for fundamental physics (theory of relativity) based on a high accuracy
comparison of orbiting atomic clocks with ground based clocks. The ACES payload will be one
of the Columbus External Payloads accommodated on the nadir pointing External Payloads
Facility. The mission shall last at least 1.5 years, and aims to last 3 years.
ACES has 3 main objectives:
1. Demonstrate the high performance of a new generation of atomic clocks (1 s accuracy in
300 million years)
2. Demonstrate capability to compare ground clocks at high accuracies
3. Perform fundamental physics experiments with large improvements in measurement’s
precision.
The payload is composed of two atomic clocks. The output of each clock is compared in the
FCDP (Frequency Comparison and Distribution Package) through a complex servo‐loop system
in order to generate a signal combining the advantages of both clocks.
 ACES on the External
Payload Facility

Figure 1: ACES on the ISS (credits NASA and ESA)

To allow the comparison of space and ground time, a complex space-to-ground MicroWave
Link (MWL) system has been developed. The link's hardware is itself composed of two parts:
the Flight Segment (FS) and the Ground Terminals (MWL GT or GT). GT are placed next to
their associated atomic clock in order to minimize the time transfer inaccuracies a longer path
would introduce. In addition to the MWL, the ELT (European Laser Timing) subsystem has been
added to ACES. Used together with Satellite Laser Ranging SLR stations (SLR stations), it shall
provide an optical-band-based time transfer capability, while also allowing to perform radio
occultation experiments.
The Ground Segment is therefore composed of the following actors:
• Columbus Command Centre: it is the Telemetry and Telecommand interface for ACES. It
also provides all the orbitography information necessary for the experiments.
• ACES User Support and Operation Centre (USOC): it is ACES operation centre. That
means it gathers telemetry, processes it, makes it available for further processing to
scientists, schedules experiments, control the Ground Segment and sends commands to
the Flight Segment.
• MWL GT network: it is the primary on-ground system for time transfer experiments. It
generates telemetry and sends it to the USOC.
• ELT Data Centre: it controls and transmits commands from the USOC to the SLR
stations, and gathers the telemetry they generate in order to send it to the USOC.
• External Users: They include scientists, payload and SHM developers, as well as ESA's
Investigation Working Group. Each of them has a specific role to play in the mission, and
they are provided with workstations designed for their specific needs.
The ACES Simulator emerges in this context, in order to provide the USOC with the
capability of simulating the "presence" of each ground facility which is not directly located in its
premises. Such capability will be used to support verification and validation testing, as well as
troubleshooting analysis and operator training during ACES operational life. The modelling
activities therefore focused on the ACES Ground Segment, and in particular on the USOC and its
external facilities.
 1.5 Presentation of the software used
Enterprise Architect for SysML modelling
Enterprise Architect (EA) is developed by Sparx Systems and specializes in the building of
models and graphs using specific graphical languages. It is mainly developed as a UML tool, but
also integrates a SysML modelling capability. The version used during research work was
mainly the version 8, build 855. Shortly before the present paper was written, update to version
9, build 908 became possible and has been performed.
EC Pro for supporting the evaluations of the Benefits
EC Pro, or Expert Choice Pro, is a piece of software based on the Analytical Hierarchy
Process (AHP) developed by Saaty (15). The software basically takes the input parameters of the
AHP and gives its results. It is also provided with interesting additional tools to simplify the data
input procedure, and its post-processing. The version used during the research activities was the
version 9.5.

2 Analysis methodology
As defined in the objectives (section 1.1), the trade-off will cover two aspects:
• Identify the benefits of MBSE with SysML over Document Based Systems Engineering
(DBSE)
• Trade these benefits against its cost.
It has been decided to follow an approach suggested by (16), according to which, benefits
and costs should be assessed and traded-off separately. The generated output is expected to
provide a sounder basis for decision making. Once the costs and the benefits are evaluated, the
solutions can be compared in terms of their benefit to cost ratio. For each solution i, there is the
final score Fi which can be defined by
𝑆𝑆𝑖𝑖
𝐹𝐹𝑖𝑖 =
𝑅𝑅𝑖𝑖
where Si is the benefits score and Ri is the cost ratio. These three quantifies are unitless.

2.1 The benefits analysis

The methodology used to define how to assess the benefits of each solution was the
following: it was first decided to determine exactly what is aimed to when one writes a report, or
a document. It was found that, no matter what document is being written, the purpose is always
at least the following:
A document is written in order to give information a concrete support, so that this
information can be reread, either by the writer (to remind him-/her-self of all the ramifications of
the idea) or by another stakeholder (to transmit the idea to others). It therefore aims to store
information on one hand, and communicate it on the other.
This gives an idea of what is aimed at through DBSE, and therefore, through MBSE. The
question is thus: How to assess how well DBSE or MBSE allow reaching those objectives?
The Metrics
One approach is to define several metrics. According to (2), the definition of a metric is:
noun. (technical) a system or standard of measurement.
In order to be complete, the metric should address three questions:
• What characteristic is being measured? (e.g. the diameter)
• What is the object of the measurement? (e.g. a pipe)
• How is the characteristic measured? (e.g. using a slide gauge)
The characteristics to be measured shall be defined in accordance to the above-defined
objectives:
1. Information storage:
a. Completeness: is all the information there?
b. Consistency: are we sure that nowhere in the documentation, two completely
opposing statements are made?
c. Extendability: how easy is it to add information?
2. Communication:
a. Readability of design information: is it easy to transfer the information from its
concrete support into someone’s mind?
b. Capability of providing clear layering of the design (necessary for
subcontractors): are there clear separations between the different design levels?
MBSE and DBSE are then assessed based on measurements performed on their products: the
best solution being the one providing the best products. The set of objects of the measurement is
composed of the generated documentation using the two information-storage solutions dealt with
in the frame of this paper:
• MBSE with SysML: the generated SysML model
• DBSE: the generated report(s).
Characteristics and objects of measurement can then be combined to form a characteristic-
object couple (e.g. completeness of the SysML model or consistency of the report).
The last aspect of the metric to be undertaken is now “How to measure those
characteristics?” and “How to create a meaningful and clear basis for decision making out of
those characteristics and objects?” On one hand, a value has to be given to each couple
characteristic-object, and on the other hand a procedure to handle all those numbers and to
interpret them has to be issued.
The Analytical Hierarchy Process (AHP) (15) has been chosen to provide the answer to those
problems. This process will allow deriving both weighing factors for the characteristics and
scores for the objects, using pair-wise comparisons. The result of each comparison will be
justified based on an argumentation. The AHP will then take the results of those pair-wise
comparisons as an input to create weighing factors for the characteristics, and the scores for the
solutions. The total score of a solution i will then be the sum of its scores pondered by the weight
of their associated characteristic j

𝑆𝑆𝑖𝑖 = � 𝑤𝑤𝑗𝑗 𝑆𝑆𝑖𝑖𝑖𝑖

 where Si is the benefits score for the solution i, wj is the weight of the characteristic j, and Sij
is the score the evaluated solution i obtained in this specific characteristic j.
The context: requirement capturing and architecture design
The analysis has been done in two contexts: requirements definition phase and design phase.
The reason behind this separation is that MBSE with SysML has been used quite extensively as a
requirements handling tool within Astrium. Nevertheless, it is expected that MBSE is not well-
suited for requirements handling only.

2.2 The Cost analysis

The Cost Ratio
In this paper, two categories of costs have been identified: costs which are paid only once
(the transition from MBSE to DBSE for instance), and costs which have to be paid for every
piece of design information (e.g. hourly rate of an engineer). It is assumed here, that, if both the
quantity of design information, and the cost of their development and capturing, could be
quantified and measured, then they would be bound by an affine law. If CMBSE is the total
estimated cost of the project in the MBSE approach, and Qi the quantity of information, then:
𝐶𝐶𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀 = 𝑎𝑎𝑄𝑄𝑖𝑖 + 𝑏𝑏
where b corresponds to an initial cost, and a to the cost to be provided to treat one unit of
information. CMBSE is associated with the overall cost of the project: it shall be given in a
monetary unit, in our case the €. The quantity of information Qi is more problematic. One could
think of the number of bytes necessary to store it. This value however strongly depends on the
compression method which is used on the data. The information to be stored is, for instance,
“this part is divided in two subsystems, one is A and does this, the other is B and does that”.
Whether this information is stored as a sentence in a text file, or in a picture or as SysML
diagram is not important: to the System Engineer, it is always the same information. Another
solution would be using the estimated cost of the project in the DBSE paradigm. It may sound
like a dog chasing its tails since costs are being used to evaluate costs, but it is actually not the
case: the estimated development costs of the system using DBSE are used to deduce an estimate
of the development costs of the system using MBSE. Such an approach has the advantage that
projects costs models are already available and are all based on a DBSE approach. Thus,
assuming the quantity of design information is proportional to the estimated cost in the DBSE
paradigm, we may write:
𝐶𝐶𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀 = 𝑎𝑎′𝐶𝐶𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷 + 𝑏𝑏
where a’ is unitless and all other quantities are e.g. in €. If the input is the estimated
development costs of the system using DBSE, then, on one hand, when the costs of the DBSE
method are being evaluated, a’ obviously equals one, and b equals zero. On the other hand, when
the costs of the MBSE method are evaluated, a’ is expected to be lower than one (if MBSE holds
its promises), and b has a certain value to be determined. The cost can then be normalised, in
order to get a unitless number:
𝐶𝐶𝑖𝑖
𝑅𝑅𝑖𝑖 =
∑ 𝐶𝐶𝑖𝑖
Analysis of the initial cost
In order to support the costs evaluation, further analysis was performed on this topic. Two
questions were addressed:
• What does b depend on?
• What does a’ need to be for the costs of the MBSE solution to be lower than the costs of
the DBSE solution?
Parameters influencing the value of b
In order to determine what parameters could have an influence on b, one needs to know what
costs it encompasses. Based on the experience obtained during the modelling activities, it was
determined the following items are required to complete the transition from DBSE to MBSE:
• A fully trained and qualified personnel
• An infrastructure that supports MBSE with SysML (software licences, servers …)
• Modelling activities prepared (methodology defined, components libraries and
documentation created, etc.)
• If the transition is made after the kick-off of a project, the existing design information
needs to be "translated", and integrated into the model.
Already, one parameter influencing b clearly stands out: the size of the team working on the
project. It will, indeed, directly influence the training costs, and the cost of the infrastructure
through the number of licences. One can quickly see that b is linked to the number of team
members by an affine relationship: there are, on one hand, costs which linearly depend on the
number of members (training cost and cost of the licences), and on the other hand, costs which
do not (or weakly) depend on the number of members (project translation, methodology
definition, some infrastructure costs, documentation creation, etc.). One may therefore write
𝑏𝑏 = 𝑐𝑐𝑛𝑛𝑚𝑚 + 𝑑𝑑
where c is the cost of the transition per team member [€/person], nm is the number of team
members [person]d is the component which does not depend on the number of team members
[€].
The value of c can be further decomposed in

𝑐𝑐 = � 𝑝𝑝𝑖𝑖 𝑈𝑈𝑖𝑖
𝑖𝑖

where i is the index of the considered cost, pi is the proportion of the team members to
contribute to the cost (unitless, e.g. proportion of team members following the seminar), and Ui
is the unit cost of the item (€/person, e.g. cost of one seminar).
A target value for a’
One may now wonder what should the target value for a’ be. Depending on the total cost of
the project and the initial costs, what is the value of a’ that should be reached to ensure no money
was lost? If money is not lost then 𝐶𝐶𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀 ≤ 𝐶𝐶𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷 . Using the mathematical framework
previously defined :
𝐶𝐶𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀 𝑏𝑏
= 𝑎𝑎′ + ≤1
𝐶𝐶𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷 𝐶𝐶𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷
and once reworked, one obtains:
𝑏𝑏
𝑎𝑎′ ≤ 1 −
𝐶𝐶𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷
Note that, as the equation shows, if the transitioning costs are about 5% of the total estimated
cost in the DBSE approach, then the company target a total amount of work in the MBSE
approach of 100% - 5% = 95% of the one in the DBSE approach, in order to reach the
profitability limit.

3 Evaluation of the weighing factors

The weighing factors were issued on the base on an argumentation using inputs from the
industry. The results are presented in the matrix shown in Table 1. The level of importance is
converted into a number following the rule:
• equally important = 1
• equally important to moderately more important = 2
• moderately more important = 3
• moderately to strongly = 4
• strongly = 5
• strongly to very strongly = 6
A negative number indicates that the characteristic on top of the column is the most
important, whereas, a positive number indicates it is the characteristic on the left of the row that
is more important.
Table 1: Summary of choices made
Complet. Consis. Exten. Read. Layer.
Complet. 1 -3 3 -2 -5
Consis. 3 1 5 2 -2
Exten. -3 -5 1 -4 -6
Read. 2 -2 4 1 -3
Layer. 5 2 6 3 1
As those pair-wise comparisons are redundant starting from three items (A twice as
important as B, and B twice as important as C, implies A four times as important as C), they
allow for consistency check. The AHP thus defines a so-called inconsistency ratio. A value of 0
means perfect consistency. According to EC Pro, this matrix has an inconsistency ratio of 0.02,
which is very good and much lower than the 0.1 rule of the thumb given by (15). Using the AHP
process, weights reflecting the argumentations we held were issued (see Table 2).
 Table 2: Weights of each characteristic
Weight
Completeness 0.097
Consistency 0.261
Extendability 0.048
Readability 0.161
Layering 0.433

3.1 The Costs

The purpose of this paper is not to put an accurate price tag on MBSE with SysML, but
rather having a very rough estimate of the cost and lay down the basis for further research.
Several parameters have been defined in section 2.2, and were given a mathematical framework.
There are two categories of costs which have been identified: the system development costs,
which depend on the size of the project, and the once-in-a-lifetime costs associated to
transitioning to MBSE.

3.2 DBSE to MBSE: Transitioning Costs

In order to have a complete transition, several items are required
• A fully trained and qualified personnel
• An infrastructure that supports SysML (workstations, software, servers …)
• Modelling activities prepared (methodology defined, components libraries and
documentation created, etc.)
• If the transition is made after the kick-off of a project, the translation of the existing
design information, and integration into the model.
As previously discussed, some of those costs will depend on the size of the team assigned to
the project (training and infrastructure). So as to deal with those team-size dependent costs, the
framework previously defined is used.
Table 3: Evaluation of the costs of transitioning from DBSE to MBSE with SysML for
ACES project
Cost name Portion of staff Cost per person Scaled cost
Training / Seminar 0,1 3 500,00 € 350,00 €
Training / 1-month assimilation 1 6 000,00 € 6 000,00 €
Software Licenses 0,4 800,00 € 320,00 €
Low efficiency 1 10 800,00 € 10 800,00 €
Number of team-members 10 Total per person 17 470,00 €
Modelling Preparation 15 000,00 €
Total 189 700,00 €
 Anyone using this work for other purposes should of-course input the size of their team to
adapt the trade-off. In our case, the ACES Team was composed of 10 persons. All costs have
been investigated, and their evaluation has been justified. Table 3 shows a summary of these
estimations.
Evaluation of the development cost
These costs turned out to be particularly difficult to evaluate. In this study, it was decided
simply assume a value. Further research is needed to shed some light on the actual value of this
coefficient, and our cost model in general. It was assumed here that through experience,
improvements of the language and of the tools, an a’ of at least about 0.95 will be reached. This
number remains an assumption and, therefore, the result of the trade-off shall be handled with
extreme care, and modulated by the possible evolutions of this value.
Note that assuming such a value implies assuming that the profitability limit is reached.
Indeed, b in our case represents about 1% of the total estimated cost, which sets the profitability
limit at a’ ≤ 0.99.
The costs ratios
Knowing that the budget of ACES is about 35 Million € (17), this allows computing the cost
coefficients. The results are RMBSE = 0.489 and RDBSE = 0.511. Looking closer, one can quickly
realize these coefficients highly depend on the ratio of the total project cost to the number of
team members: the higher the ratio, the more the investment pays off for values of a' lower than
one. This particular aspect outlines the importance of a good evaluation of a’, or even of the
whole development cost model.

4 Sample problems encountered during modelling

Back in 2005, Herzog and Pandikow already shared their impression that SysML leaves too
much freedom to the modellers as compared to UML (8). Indeed, it seems that SysML, like
natural languages, does not always define a unique way to represent or express a given piece of
information, and the determination of which construct to use is left to the modellers. To that are
also added, problems encountered with the modelling tools as they do not always provide
accurate representations – for instance, and among others, the line is sometimes solid instead of
dashed, the stereotype is incorrect, or there is no representation for such item. This section
exposes the most important problems that were encountered during the modelling activities.

4.1 Characterisation of the data transfers

Astrium, as prime contractor for ACES, deals with many requirements, and interfaces. High
level interfaces on ACES Ground Segment basically consist of data-flows. The model needed
therefore to capture them and characterize them. It was necessary to store in the model and for
each data-flow, at least three properties:
• The information conveyed
• The protocol used
• The network on which the information is exchanged
SysML deals very well and easily with the information conveyed it-self. Two methods which
allow showing the items exchanged by two blocks are defined. However, giving more details
about the link is not as easy. SysML allows any kind of element (element, not link) to possess
Tagged Values. A tagged value is a variable which can be defined for specific stereotypes or
elements. Tagged values have (at least in Enterprise Architect) the drawback of not being
displayed unless explicitly asked. They also have to be defined on each element on which they
should appear, instead being associated to a stereotype. Nevertheless, even though it represents a
lot of work, using tagged values remains the only solution found for this problem. Because an
element inherits the tagged values of its typing element, it was decided the Flow Specifications
specifying the link would carry the tagged values.

4.2 Positioning the simulator in the Model Repository

The Simulator it-self is often referred to as a whole within the ACES team. This whole is
composed of two components, the Payload Simulator and the Mission Simulator. The latter is
then divided into the External Facilities Simulator and the OPS Support LAN Simulator. Each of
those components interfaces with different networks having different levels of Security: any
communication between them is therefore forbidden. In the end, they are three completely
independent facilities. The problem then stems from the fact that facilities within the ACES
project are "traditionally" classified depending on the network to which they interface, meaning
that all three Simulator facilities should belong to different facility groups. The questions were
then: is it possible for a structural subset to belong to several groups at the same time? If not, in
which group should then the simulator be classified?
Those questions have a direct impact on the product tree itself, making it difficult but yet
crucial to answer. SysML does allow, in diagrams, a block to compose several other blocks at the
same time. In that sense, SysML itself allows a structural subset to belong to several groups, but
the tree organisation of the model’s repository does not. Just like folders on a computer, a given
file can only be in one folder at a given time (copying the file creates two distinct identical files).
The only question left is now, in which "folder" should the simulator be?
It was decided the simulator would have its own package, gathering all three facilities. This
decision is supported by the fact that some elements, such as requirements, are shared by all
facilities, separating the facilities would therefore have been complex. This example shows that
the question of the model repository's organisation on large projects is not easy to answer.

4.3 Ground Segment Architecture and Data-flows

A very interesting problem which has also been encountered was the determination of “how
to gather data-flows”. Let us imagine, there are two files leaving from one facility to another.
Should those two files be gathered into one unique flow, or should two distinct flows be created?
On which criteria should this kind of decision be made? Creating a new flow is synonym of
creating a new flow port. The question could therefore be seen from a ‘port’ point of view. When
asked, someone giving the first answer coming to their mind would say: “just create ports which
actually exist in your design, and do not separate them or gather them, it would only make the
model confusing”. That sounds sensible, however which port should be modelled? Hardware
Ethernet port? Software SFTP port 21? The problem is then choosing which mapping between
the actual system and its model should be made. Indeed, there is no best mapping. In some cases
one may prefer only showing the hardware ports on high-level level diagrams, where
representing the data-flows in more details would make the diagrams very difficult to read, but
one may prefer showing the data-flows using one connector per file in a detailed design phase,
on a diagram showing only two facilities. Ideally, both should be simultaneously possible,
meaning that a mapping between high-level hardware ports, and low-level software port, or even
lower level file channels could be done. This is however not possible at the moment: SysML
does not allow breaking down a port into lower levels. There is one port, and this one port will
be used for all the levels of the design. This discussion reminds of the one in (9), where the
author underlines the difficulty of differentiating hardware and software components and the
associated confusion: obviously several appropriate constructs or semantic elements are missing
in SysML.

5 Results of the trade-off

After having gathered much modelling experience, the arguments were held, and the results
of the trade-off could be issued.

5.1 Architecture and behaviour context

In the context of Architecture and Behaviour capturing, through the arguments, MBSE was
found to be "moderatly better" than DBSE in the frame of completness (MBSE 0.667 vs. DBSE
0.333). Concerning consistency, extendability, and readability, MBSE was estimated to be
"strongly better" than DBSE (MBSE 0.833 vs. DBSE 0.167). Finally, regarding layering,
because MBSE with SysML succeeded so well to represent the system as a whole, it was found
difficult realize a separation between the different design layers. MBSE was therefore, in this
case, estimated to be "moderatly to strongly worse" than DBSE (MBSE 0.25 vs. DBSE 0.75).
The benefits scores obtained by each solution are therefore: SMBSE = 0.564 and SDBSE = 0.436.
Using the cost ratios (RMBSE = 0.489 and RDBSE = 0.511), it is then possible to derive the
solutions’ final scores pondered by their cost: FMBSE = 1.15 and FDBSE = 0.85. Our trade-off
therefore indicates it would have been an interesting and potentially promising idea to carry out
ACES architectural and behavioural development using MBSE. One must however keep in mind
that many assumptions were made to reach this result (in particular concerning the cost
assessment), and that some of them may be reviewed and supported by further research.

5.2 Requirements handling context

In the context of requirements capturing, and concerning completeness, extendability, and
layring, MBSE was found to be "equally good" as DBSE (MBSE 0.5 vs. DBSE 0.5). In the case
of consistency, MBSE was estimated to be "moderately to strongly better" than DBSE (MBSE
0.75 vs. DBSE 0.25). Finally, regarding readability, MBSE was found to be "very strongly
worse" than DBSE (MBSE 0.1 vs. DBSE 0.9), because of, at least in our case, a very awkward
diagram presentation, and a poorly functional table representation. This last point reflects our
belief that MBSE with SysML is not adapted to requirements handling only, and that
requirements diagram should only be used as interface between system model and requirements
handling methods or software.
The benefits scores obtained by each solution are therefore: SMBSE = 0.501 and SDBSE = 0.499.
Using the cost ratios (RMBSE = 0.489 and RDBSE = 0.511), it is then possible to derive the
solutions’ final scores pondered by their cost: FMBSE ≈ 1.02 and FDBSE ≈ 0.98.
 6 Reflection over the analysis and possible improvements
The evaluation of MBSE with SysML in general is based on the authors’ understanding of
SysML’s semantics. This understanding may also include potential misunderstanding, misuse or
misinterpretation of its different constructs. Should this have happened, it would show potential
improvement areas for SysML and/or its tool. Also, this analysis is provided as the first iteration
of a whole process. As such, it is not perfect and several axes for improvement have been
identified:
1. The first problem of this analysis is the number of considered solutions. There are
traditionally at least three solutions considered in cost/benefit analyses. This critic is also
supported by the impossibility to perform a consistency check on the different
evaluations given to each solution in each characteristic using the AHP. Indeed, as soon
as the number of solution reaches three, the pair-wise comparison becomes redundant. A
third solution which could be included could be Model Based Systems engineering using
only UML for instance.
2. Another axis to be investigated is the improvement of the result of all the comparisons
made in this research work. All comparisons have been done based on an argumentation
held by the authors. Though we tried to be as impartial as possible, this process still
leaves some room for subjective and wishful thinking. This aspect could be mitigated by
involving many more persons in the trade-off.
3. The costs evaluation has to be improved. It remains very approximate, and is based on
many assumptions. The model is only linear which was not actually verified, and is
probably not true. The value of a’ has been simply assumed. The evaluation of several
costs was also not done as they require a lot of attention and research. The only foreseen
method that could yield interesting results would be investing more time and research in
this topic.
4. The last axis of improvement identified concerns the results and the analysis performed
for the requirements capturing and handling. According to our results, there is almost
no difference between DBSE and MBSE with SysML (0.98 against 1.02) in the context
of requirements handling. However, those results do not reflect the overall feeling we had
during the modelling activities. It was felt that MBSE with SysML is not meant to be
used for requirements handling only. It is therefore expected this conclusion will change
if the characteristics used for the comparison are adapted to requirements handling.
There are, of course, other aspects in this analysis which could be improved. Those are
however believed to be the most important.

7 MBSE: Lessons learned

This section intended to provide a few tips for Systems Engineers desiring to step into Model
Based Systems Engineering.

7.1 SysML
Five lessons have been learned through this research:
1. Try to always define, beforehand, the lowest level the model will reach in each
package, or diagram category! This shall be done in order to avoid continuing the
 design after the point to which the model should be handed in to the subcontractor, or to a
specialized engineer. The purpose of the model has to defined and clear at all time.
2. Use separate high level packages for architecture, behaviour and behaviour
allocations! We suggest to completely separate architecture and behaviour. In frame of
this research, everything was gathered into one single project package. This resulted in
the necessity of using a hierarchy system of master package for each element, containing
composing block / behaviour packages, which makes project browser (or model
repository) very hard to navigate into. It also raises some problems concerning the level
at which such or such behaviour should be described. This is however only a suggestion:
more research has to be done on this point in order to determine what solution is the most
interesting.
3. Always take extreme care when showing diagrams to persons not introduced to
SysML! Systems Engineers are nowadays used to MS Visio diagrams in which an exact
and precise meaning is not associated with each type of arrows and blocks, and in which
the purpose of the diagram is supporting a paragraph, or a document. This leads to
misunderstandings, confusion and misinterpretations of the SysML diagrams. When a
modeller presents a diagram to other engineers, he or she needs to make sure, before even
explaining the diagram itself, that the others understood: the purpose of the diagram (e.g.
showing architecture and not functions), what each type of element represents (e.g.
physical block), and the meaning of each link (e.g. composition).
4. Do not use SysML for requirements handling only! since it was not meant to be used
as such, it becomes a bit awkward and unhandy as compared to other tools, which are
better suited to this kind of use.
5. Always Remember: though it provides a certain guidance in the design, it does not
substitute to experience in Systems Engineering!

7.2 Software tools: Enterprise Architect

It is important to remember, especially during the learning process, that SysML tools are, in
most (if not all) cases, UML tools which have been adapted for SysML. This results
sometimes in controls, options and functionalities which are neither adapted nor intended for
Systems Engineering, and makes the use of the tool quite disturbing and confusing. For instance,
it is always possible, in Enterprise Architect, to specify in which language a block is
programmed (as the concept of block derives from UML classes), regardless of what the block
represents, may it be a piece or software, a computer, a satellite or even a screw!

8 SysML: Suggestions for improvement

This section presents a set of ideas which are, when implemented, are expected to allow
significant improvements in the modelling processes and capabilities. Those improvements could
be reached by:
1. removing the conjugation of flow specifications. It is indeed believed that conjugation
of flow specifications may lead to misunderstandings and makes the model more difficult
to read. Instead of having item flows and flow specifications, it is suggested to provide
only one unique construct to specify what is flowing through a port. This construct would
be like the item flow, directly associated with the link itself, and not with the ports. Flow
 specifications could be kept, but they would be then automatically generated using the
information contained in the link. The advantages of this improvements would be:
a. flow specifications are generated automatically, so no need for conjugation
b. as the information is contained in the link, a description of the content of the link
can be shown above the connector, even for complex flows
c. the whole modelling process would be simpler as there is only one unique way to
specify what is going through a given connector
2. allowing to organize ports into a hierarchy. This could be a solution to the already-
discussed problem of the difficult and awkward differentiation between hardware and
software. It would also be an interesting solution for engineers who would like to
describe the systems at several levels: when one shows the whole ground segment, one
does not wish to see all the software ports and all the channels of each facility, as the
diagram would otherwise be unreadable. However, the deeper the model goes in the
design, the more interesting it gets to see those low-level ports. Nevertheless, there is, to
this date, no way to show that a port is actually a smaller part of a bigger port, like for
instance the HTTP port 80 is a "part" of the Ethernet port on the computer.
3. allowing organizing flow specifications into a hierarchy. When one deals with
complex data flows, it becomes interesting to define small "data packages", which can be
reused in other contexts.
4. providing SysML specifications with a dictionary from MBSE vocabulary. It could,
for instance, give a strict definition to "children elements". UML says children elements
are the specialized part of a generalisation relationship, but there are referred to in the
Practical Guide (4) as elements contained by another element in the model repository.
5. completely separating SysML specifications from UML. SysML specifications are, to
this date, only describing the part of SysML which has been added to UML. It is
impossible for someone not familiar with UML to learn SysML from its specifications,
and is very difficult for someone familiar with SysML to find the information he or she is
looking for.
6. providing a tabular display of information other types of diagrams than
requirements. Sometimes block definition diagram are used only to "define" block, and
no links are shown on those diagrams. Another example could be, in activity diagrams,
when a switch-like decision structure is used, and it has many cases, it becomes laborious
to use a diagram. In such cases, it would be interesting to give the modeller the possibility
to use tables.
7. allowing to have packages within a block to sort and organize embedded elements.
Note that this problem may be solved directly by the software, by organizing the
embedded elements directly in a folder-like hierarchy.

9 Conclusion
Model-based Systems Engineering in general and SysML in particular have raised attention
over the past years. Though MBSE is a young approach of Systems Engineering, it could impose
itself in the industry in the years to come. SysML is a graphical language which enables MBSE.
This paper analyzed both and evaluated their maturity, through their benefits and their costs in
the framework of a space project.
 The following key advantages have been identified:
• Project documentation is nearly perfectly consistent
• Relevant information is easily found
• Well structured information and clear separation between structure and behaviour
• Model and diagrams point out missing design information
• Opens the way to automated Systems Engineering, through automatic document
generation, assisted design verification and system behaviour simulation
Nevertheless, the following drawbacks have been identified:
• Steep learning curve: much effort is required to learn and use SysML
• It requires a significant knowledge before its advantages start showing
• Diagrams are very easily misinterpreted if SysML is not learned beforehand
• Tools are not completely ready yet
• Some things are difficult to model, where a simple sentence in the final report would have
been enough
To support decision makers on whether they should use MBSE with SysML or not, a trade-
off methodology has been developed. The advantages are very well reflected in the results of this
trade-off as it shows MBSE with SysML seems to be a more interesting option than DBSE.
These results, though, have to be modulated as the trade-off itself needs to be refined: more
accurate and potentially different results are expected if the costs estimation is improved by
further research (in particular, the evaluation of a’), and if the given evaluations in each of the
characteristic are supported by a poll, or at least by the opinions of several modellers.
Through this work, it was shown that Model-Based Systems Engineering with SysML
exhibits undeniable advantages over Document-Based Systems Engineering. These advantages
do, nevertheless, not undoubtedly outweigh its cost and other drawbacks. Tools are also not fully
mature, and SysML, in general, has not reached its final potential yet. Industry will, most
certainly, expect more from MBSE with SysML before it can generally be accepted as a
standard.

References
1. Object Management Group. OMG Systems Modeling Language (OMG SysML™) version
1.2. 2010. Standard Specification URL: http://www.omg.org/spec/SysML/1.2/.
2. WordReference.com. Concise Oxford English Dictionary. WordReference.com. [Online]
University Press, 2011. http://www.wordreference.com/definition/.
3. Warwick, Graham and Norris, Guy. Design for Success. Aviation Week and Space
Technology. 2010, Nomvember 1/8, 2010.
4. Friedenthal, Sanford, Moore, Alan and Steiner, Rick. A Practical Guide to SysML. s.l. :
Morgan Kaufmann Publishers, 2009. ISBN-13: 978-0123786074.
5. Weilkiens, Tim. Systems Engineering with SysML/UML: Modeling, Analysis, Design. s.l. :
Morgan Kaufmann, 2008. ISBN-13: 978-0123742742.
6. Fieber, Florian, Regnat, Nikolaus and Rumpe, Bernhard. Assessing usability of model
driven development in industrial projects. Aachen : RWTH Aachen University, 2009.
7. Kirstan, Sascha and Zimmermann, Jens. Evaluating costs and benefits of model-based
development of embedded software systems in the car industry – Results of a qualitative
Case Study. Munich : s.n., 2010.
8. Herzog, Erik and Pandikow, Asmus. SysML - An Assessment. s.l. : INCOSE, 2005.
9. Belloir, Nicolas, et al. Utilisation de SysML pour la modélisation des réseaux de. s.l. :
Université de Pau et des pays de l’Adour, 2008.
10. Bone, Mary et Cloutier, Robert. The Current State of Model Based Systems Engineering:
Results from the OMG™ SysML Request for Information 2009. Hoboken : Stevens
Institute of Technology, 2009.
11. Karban, R., et al. Exploring Model Based Engineering for Large Telescopes - Getting
started with descriptive models. Garching bei München : European Southern
Observatory, 2008.
12. Vanderperren, Yves and Dehaene, Wim. UML 2 and SysML: an Approach to Deal with
Complexity in SoC/NoC Design. Leuwen : Katholieke Universiteit Leuven, 2005.
13. Karban, R, Andolfato, L. and Zamparelli, M. Towards Model Re-Usability for the
Development of Telescope Control Systems. Munich : European Southern Orbservatory,
2009.
14. de Lange, Dorus, Guo, Jian and de Koning, Hans-Peter. Applicability of SysML to the
Early Definition Phase of Space Missions. 2011.
15. Saaty, T. L. Analytic Hierarchy Process. Encyclopedia of Biostatistics. 2005.
16. Expert Choice, The Analytic Sciences Corporation. An illustrated guide to the Analytic
Hierarchy Process. [Powerpoint in executable (*.exe)] Pittsburgh : s.n., 1999.
17. European Space Agency (ESA). Challenging Einstein on the ISS: ACES takes a step ahead.
ESA Technology Portal. [Online] [Cited: 15 03 2012.]
http://www.esa.int/esaMI/Technology/SEM7L2WNPBG_0.html.

Authors Biography
Julien Maurandy (TU Delft and EADS Astrium ST) has been prepared in the frame of his
MSc Final Thesis. He is a French international student who started off at Supméca, a French
"School of Engineering", which specializes in Mechanical Engineering. He joined TU Delft in
2009 to perform a double-degree programme, and to specialize in Aerospace Engineering. There,
he followed the Systems Engineering lectures of Prof. Eberhard Gill, who also supervised his
MSc Final Thesis. In 2010, he was given the opportunity to perform his MSc Final Thesis at
EADS Astrium, on the ACES project, where he got familiar with MBSE and SysML, and
performed its assessment.
Prof. Ebehard Gill (TU Delft, Chairholder of the Space Systems Engineering
department) holds a diploma in physics and a PhD in theoretical astrophysics of the Eberhard-
Karls-University Tübingen, Germany, as well as a Master of Space Systems Engineering of the
Delft University of Technology. He has been working at the German Aerospace Center (DLR) in
the field of precise satellite orbit determination, autonomous navigation and spacecraft formation
flying. He has been Co-Investigator on several international missions. Since 2007, he holds the
Chair of Space Systems Engineering of the Delft University of Technology.
Roland Stalford (EADS Astrium ST) is currently assigned to the Atomic Clock Ensemble
in Space Project (ACES) at Astrium Friedrichshafen where he is responsible for the Ground
Segment and for the procurement of one of the two key payload instruments, the Space
Hydrogen Maser. Previously he was Head of Systems Engineering at Galileo Industries in Rome,
assigned from EADS-Astrium Germany in 2001, where he was responsible for all aspects of the
Galileo Programme Space and Ground Infrastructure as well as overall systems performance and
the Test User Segment. Prior to this, he played a senior role in the GNSS-1 EGNOS Central
Processing Facility and has many years of experience in the technical management of large
international space and aircraft programmes as well as extensive experience in the development
and certification of safety critical software for Manned Space Programme and avionic inertial
navigation and flight control systems. He has in the past worked at the European Space Agency
(EURECA programme) and EUMETSAT (Meteosat Second Generation) and was attached to
CNES in Toulouse for the Hermes programme.
Achim Helm (EADS Astrium ST) is working as Systems Engineer for the ACES project.
Previously, he worked as a scientist at the Geo-research Centre in Potsdam in the field of GNSS
Remote Sensing and Satellite Radio Altimetry.