J. Parallel Distrib. Comput.

74 (2014) 30873097

Contents lists available at ScienceDirect

J. Parallel Distrib. Comput.

journal homepage: www.elsevier.com/locate/jpdc

Towards efficient and fair resource trading in community-based

cloud computing
Han Zhao, Xinxin Liu, Xiaolin Li
Scalable Software Systems Laboratory, University of Florida, Gainesville, FL 32608, USA

highlights

We create a distributed market-oriented framework for cloud resource trading.

We propose a multiagent technique for budget-free optimal resource allocation.
We propose a new directed hypergraph model for budget-limited resource trading.
We develop effective heuristic-based protocols for budget-limited resource trading.

article

info

Article history:
Received 16 June 2013
Received in revised form
9 July 2014
Accepted 29 July 2014
Available online 7 August 2014
Keywords:
Cloud computing
Resource trading
Social welfare
Envy-free
Hypergraph

abstract
In this paper, we investigate the resource trading problem in a community-based cloud computing setting
where multiple tenants communicate in a peer-to-peer (P2P) fashion. Enabling resource trading in a
community cloud unleashes the untapped cloud resources, thus presents a flexible solution for managing
resource allocation. However, finding an efficient and fair resource allocation is challenging mainly due
to the heterogeneity of tenants. Our work first develops a market-oriented model to support resource
negotiation and trading. Based on this model, we adopt a multiagent-based technique that allows a group
of autonomous tenants to reach an efficient and fair resource allocation. Further, when budget constraint
presents, we propose a directed hypergraph model to facilitate resource trading amongst heterogeneous
tenants. We analyze the application of the directed hypergraph model to trading decision making, and
design a series of heuristic-based resource trading protocols for both budget-unaware and budget-aware
scenarios. The performances of the proposed protocols are validated through simulations. The results are
in tune with the theoretical analysis and provide insights into practical application issues.
Published by Elsevier Inc.

1. Introduction
The ever-increasing demand for computing power motivates
recent advances in multi-core processing and high-speed networking, and drives the emergence of distributed computing platforms
that span a variety of heterogeneous devices across the Internet.
With the aid of hardware virtualization, these platforms are capable of offering rent-on-demand resource leasing services that
let end users instantly access a vast pool of resources, known as

The work presented in this paper is supported in part by National Science

Foundation (grants ACI 1245880, ACI 1229576, CCF-1128805, OCI-0904938, and
CNS-0709329).
Corresponding author.
E-mail addresses: han@cise.ufl.edu (H. Zhao), xinxin@cise.ufl.edu (X. Liu),
andyli@ece.ufl.edu, andyli.ece@gmail.com (X. Li).
URL: http://www.andyli.ece.ufl.edu/ (X. Li).

http://dx.doi.org/10.1016/j.jpdc.2014.07.005
0743-7315/Published by Elsevier Inc.

the cloud computing model. Current cloud computing model

is mostly vendor driven, with users having no control over the
data or the technology supported by the cloud. Such a vendordriven model, although convenient to use, also brings many issues
to light, e.g., failure of monocultures, tradeoff between convenience and control, and concerns about environmental impact [6].
To address these issues, researchers have proposed an alternative
model that provides a collaborative resource sharing platform
called community cloud [18,19,27]. Different from the centralized vendor model, community-based cloud computing leverages
under-utilized networked private resources for infrastructure support. Tenants within the same community cloud typically share
common security and compliance concerns, and may delegate
management to some trusted third-party organization.
Similar to the centralized vendor-driven model, the communitybased model provides computation and storage resources as metered services. Therefore, the design goal of the community cloud
should not only focus on the quality of computing service, but

3088

H. Zhao et al. / J. Parallel Distrib. Comput. 74 (2014) 30873097

should equally address the economic aspect such that tenants receive cost-effective cloud service provisioning. While managing resource allocation is relatively straightforward in the centralized
vendor-driven model (e.g., Amazons on-demand and spot instance
pricing), it is particularly challenging due to the heterogeneity in
the multitenancy environment. In a community cloud, we are facing a free market where tenants are only incentivized to accept
profitable resource exchange. As a result, a well designed multitenancy resource trading protocol is highly desirable to effectively
regulate the management of resource allocation.
In this paper, we study the distributed resource trading problem
in a community-based cloud computing environment, and propose
a set of multitenancy resource trading protocols to jointly optimize resource allocation efficiency and fairness. Specifically, better efficiency refers to the increased aggregate valuations of all
the tenants, and better fairness is interpreted as reduced envy between every pairwise combination of tenants. Our solution follows
a market-oriented design principle, and uses a directed hypergraph model to integrate these two seemingly conflicting design
objectives into one unified resource trading framework. It directly
extends Chevaleyre et al.s work [11], and further addresses the
challenge of budget limited resource trading. With systematic
analysis of the resource trading market, a set of heuristic-based distributed resource trading protocols are developed and evaluated.
The comprehensive study presented in this paper has broad
utility in the growing world of everything-as-a-service, because it characterizes the extent to which independent and selfinterested tenants interact with each other. Our analysis shows
that incentive preserving resource exchanges tend to benefit the
system, both from a global view of the overall service efficiency
and from a local view of the improved service quality valuation.
Moreover, the proposed resource trading approaches are complementary to the vendor-driven cloud computing. For example, consider user Alice rents a virtual machine from Amazon with reserved
instance pricing. After Alice finishes her job and before the lease
expires, Alice might sublease this virtual machine to user Bob in
order to partially compensate for her resource rental cost.
The contribution of this paper is primarily four-folded, and is
summarized as follows.

Create a distributed market-oriented framework for modeling

the multitenancy resource trading problem in community
cloud.
Leverage a multiagent-based technique to solve the optimal resource allocation for distributed budget-free resource trading.
Propose a novel directed hypergraph model to facilitate the
analysis of budget constrained resource trading.
Develop effective heuristic-based protocols for resource trading
given the presence of budget limitation.
The reset of the paper is organized as follows. Section 2 presents
an overview of the related work. In Section 3, we describe the problem setting and formulate the resource allocation objectives. In
Section 4, we introduce a multiagent-based technique to achieve
optimal resource trading efficiency and fairness. Section 5 further
investigates allocation strategies with limited budget. We propose
a novel directed hypergraph model and develop a series of distributed resource trading protocols based on heuristic approaches.
Section 6 shows simulation results and analyzes their implications.
Finally, in Section 7 we summarize our solution and conclude the
paper.

combinatorial multi-criteria optimization problems become more

attractive. Common optimization techniques include machine
learning [30], evolutionary algorithms [14], swarm intelligence
[29], and socialeconomy approaches [31,24]. All these approaches
share a common flavor that involves interacting entities evolving
towards the optimal solution (by following certain learning or negotiation rules). Our proposed approach falls into the category of
socialeconomy approaches. They are built based on the observation that resource management in distributed systems shares common features with commodity allocations driven by market power
in the economic study. It is widely adopted to create a computational economy for grid computing [1,7] and the emerging cloud
computing [8,34]. In an early study, Wolski et al. [37] presented
two different market strategies for controlling resource allocation,
namely commodities markets and auction. The commodities markets strategy treats disparate resources as interchangeable commodities, while auction requires orchestration from a centralized
auctioneer for collecting bids and determining winners. Our proposed resource trading framework is designed for a community
cloud environment, and belongs to the commodities market category. In particular, we propose a P2P resource trading market for
managing cloud resource allocation. Example research related to
this notion includes [13,36]. In [13], a P2P data replication system
was proposed to improve fault-tolerance of digital collections in library. In [36], the authors proposed a multiple currency economy
that any peer can issue its own currency. Different from their design, peers directly exchange resources in our distributed resource
trading design.
In this paper, two economic metrics are used to quantify the
quality of an allocation: efficiency in terms of overall social welfare,
and fairness in terms of envy-freeness. The metric of efficiency
is important to characterize the achievable system performance,
and was studied in a number of publications [20,39,4]. On the
other hand, the metric of fairness highlights individuals utility
such that each individual achieves the maximum contentment
of its allocated share [16]. Compared to efficiency, the envy-free
fairness has generally received far less attentions. A related work
targeting grid computing was found in [32]. Using game theory, the
authors tackled with a multicriteria optimization problem with the
aid of axiomatic theory of equity. The authors concluded that for
fair and feasible scheduling on global scale computational grid, a
strong community control is required. The research conducted in
this paper approaches the multicriteria optimization problem from
a different angle, and further investigates how to balance the two
metrics amongst budget-aware distributed tenants.
Our proposed protocols build on a directed hypergraph model.
A hypergraph is an extension of the graph concept that one edge
(called a hyperedge) can connect an arbitrary set of vertices rather
than two. A hypergraph model is flexible and informative to use
in algorithm design as it generalizes the graph. For that reason, it
becomes attractive to improve algorithm performance in various
research domains, e.g., page reputation computation for search
engines [3], cellular mobile communication [33] and memory
management [22]. For large-scale scientific computing, atalyrek
and Aykanat [10] proposed a multilevel partitioning approach
for mapping repeated sparse matrixvector computations to
multicomputers using hypergraph. Their approach significantly
reduces communication overheads while achieving drastically
improved mapping results. In their hypergraph model, hyperedges
represent affinity among subsets of the data, and the weights
reflect the strength of this affinity. We model the resource trading
problem in a similar manner that aims to optimize the aggregate
weights of the directed hypergraph model.

2. Related work
This paper presents distributed protocol design to jointly optimize resource trading efficiency and fairness. As the organization of distributed resource evolves towards a more hierarchical
architecture [23], distributed algorithms designed for solving

3. A distributed resource trading framework for community

cloud
This section presents the design overview of a distributed resource trading framework for the community cloud. In Section 3.1,

H. Zhao et al. / J. Parallel Distrib. Comput. 74 (2014) 30873097

3089

3.2. Problem statement

For each tenant, we assume a private valuation model, indicating that tenants are mutually blind to each other and evaluate individual allocation independently. pi s valuation to some resource
bundle is defined by the valuation function Vi (), Vi () = 0 and
Vi (Ai ) Vi (Ai ) for all Ai Ai (assuming the existence of virtual currency, Vi () is measured with the unit of this virtual currency). Moreover, we assume the valuation function is modular,
i.e., Vi (Ai Aj ) = Vi (Ai ) + Vi (Aj ) Vi (Ai Aj ) for all Ai , Aj A.
Our first goal for distributed resource trading protocol design
is to achieve optimal
nmatchmaking efficiency such that the social
welfare, i.e., =
i=1 Vi (Ai ), is maximized.
Definition 2. Efficiency: Let be the set of all possible matchmaking results, an efficient matchmaking is an allocation A =
{A1 , A2 , .
. . , An } that maximizes the social welfare: max =
maxA
pi P Vi (Ai ).

Fig. 1. Multitenancy resource trading: system model.

we depict the resource trading system model. In Section 3.2, we

clarify the problem assumptions, define the goals for resource trading, and formulate the problem.
3.1. System model
Consider a scenario where a number of highly autonomous
tenants connected in a P2P manner, each holding a set of indivisible
resources. A resource is an abstraction of hardware bundle or
software service, e.g., Virtual Machine (VM), computational time,
etc. These resources form a publicly accessible resource pool,
and they are completely allocated to all the tenants initially, as
described in Fig. 1. All tenants form a collaborative community
with common purposes and concerns. The underlying P2P
communication infrastructure ensures that every tenant is able to
talk to every other tenant within the same community (they may
not communicate directly, but there is at least one communication
path between every pairwise tenants on the topology). For this
study, we do not consider dynamic tenants join and leave. We also
assume that the distributed system is reliable. Any resource can be
assigned to any tenant, incurring certain benefit and cost that may
vary depending on the specific resource-tenant assignment. Each
tenant can be involved in any number of resource trading activities,
following the specific tenant negotiation protocol. The distributed
resource trading results in a remapping of resources to tenants. We
call each instance of such a resource remapping a matchmaking.
Tenants are incentivized to purchase under-utilized resources
from the tenants who currently hold them. As a result, the system
evolves towards better resource utilization in the long run.
Formally, let P = {p1 , . . . , pn } be the finite set of tenants, and
let R = {r1 , . . . , rm } be the finite set of indivisible resources.
Typically we have |R| > |P|. This, however, is not necessarily
always the case, i.e., some tenants may obtain empty allocation.
A matchmaking is defined as a mapping A: P 2R . More
specifically, we have the following definition:
Definition 1. Matchmaking: A matchmaking A = {A1 , A2 , . . . , An }
(Ai represents the resource
to pi ) is a mapping A:
bundle allocated

P 2R satisfying: Ai Aj = , and Ai = A.

The condition of
Ai = A ensures that the final matchmaking
result is a complete allocation.

The efficiency criterion reflects the overall system performance.

For example, suppose there are two resources, one has 2 cores + 1G
memory and the other has 1 core + 2G memory. Also assuming
user Alice has a CPU-bound job and user Bob has a memory-bound
job. Therefore, Alice has higher valuation for the first resource
while Bob prefers the second resource. By assigning the first
resource to Alice and the second to Bob, the aggregate valuation
is maximized, and the system features best job turnaround time.
We define a resource-bundle as a collection of one or more
resources held by any tenant pi , i.e., a resource-bundle is an nonempty subset of Ai . We define a Deal as the basic event in the
multitenancy resource trading framework. A deal represents the
process of resource-bundle transfer from one tenant to another.
In order to acquire resources from another tenant, certain amount
of compensation is necessary to complete the deal. A Payment
Function i,j defines this compensation amount pi pays to pj . If
i,j is negative, then pi receives money from pj . Each tenant keeps
a record of its payment history. Formally, we define pi s Balance
as the summation of
its withdrawals and deposits in all deals pi
is involved in: i =
i . All tenants are utility-driven that seek
to make profit at each deal. Formally, suppose after a deal, the
allocation of pi becomes Ai , a deal must be a Rational Deal (RD)
if and only if Vi (Ai ) Vi (Ai ) i,j for all pi P. Note that the
requirement of rational deal applies to both tenants involved in
the deal, thus is a bilateral constraint. The Utility of pi is given as
Ui (Ai ) = Vi (Ai ) i .
The second goal of our protocol design is to promote fairness
within the system. By associating the valuation and payment
function, fairness denotes to envy-free [5] amongst all tenants,
indicating that no tenant would get better off by swapping its
allocation with another peer though a rational deal. Specifically,
the definition of a fair allocation is given as follows.
Definition 3. Fairness: A matchmaking result is characterized as
fair iff there exists A = {A1 , A2 , . . . , An } such that: (a) pi , pj
P, pi and pj has direct connection; and (b) Vi (Ai ) i Vi (Aj ) j .
The fairness criterion is in line with the envy-free definition
given out in [11] that takes transferable utility into account. The
authors proved that a Efficient and Envy-Free (EEF) state always
exists. Here, we further extend their result by adding topology
constraint to the fairness definition. Our definition limits envy-free
states to neighboring tenants. This is justifiable as the underlying
communication topology might not be a fully connected network.
In addition, a common practice in distributed systems is to employ
a budget transfer mechanism to enforce incentives for community
control [25]. For example, in P2P and social networks, some
form of digital cash, or numerical reputations representing trust
relationships may be used for rewarding and punishing certain
actions. We formally define budget constraint as follows.

3090

H. Zhao et al. / J. Parallel Distrib. Comput. 74 (2014) 30873097

The reasoning behind Proposition 1 is fairly simple. Each RD

results in remapping of resources to tenants with higher interests. When no RD is possible with respect to the communication
topology, the system converges to a topology-wide efficient state.
Another implication is that the final state is independent of the
execution order of RDs. Now suppose after an execution of an RD,
Since the deal is bilaterally benethe current allocation becomes A.
ficial to both tenants involved in the deal, we calculate the payment
range with the following equations.
Fig. 2. A counter example of globally efficient allocation: solid lines represent
communication links and dashed lines represent envy relations.

Definition 4. Budget: Budget bti

able to offer after t deals. Let b0i

expresses maximum amount pi is

be the initial budget initially, we

have:
bti = b0i it .
Given any initial allocation, the goal of this study is to investigate to what extent efficiency and fairness can be achieved in the
multitenancy resource trading framework described above, and
to design resource trading protocols to guide tenant interactions
evolving towards system-wide efficiency and fairness. We analyze
situations with and without the budget limitation. From now on,
we label the scenario with budget constraint as budget-aware, and
refer to the later scenario as budget-unaware.
4. Budget-unaware resource trading protocol
In this section, we develop a resource trading protocol without
the presence of budget constraint. Our protocol design is based on
the multiagent-based resource allocation optimization framework
presented in [11].
4.1. Design preliminaries
By following certain payment rules, we will show that the
resource trading protocol is capable of reaching topology-wide
efficiency as well as envy-free fairness upon convergence. A
topology-wide efficient allocation is an allocation such that for every tenant, the allocation for the sub-topology consisting of that
tenant and its direct neighbors is efficient, i.e., the matchmaking achieves maximum social welfare on the sub-topology. We
introduce topology-wide efficiency because for a partially connected communication topology, a globally efficient matchmaking
is not guaranteed unless the order of resource trading is carefully
planned. This is illustrated by the example described in Fig. 2. Consider a system with three tenants and one resource initially assigned to p1 . Tenants have different valuations for this particular
resource. Communication links exist between {p1 , p2 } and {p1 , p3 }.
Both p2 and p3 envy p1 as they have higher valuation for the resource. Suppose they both propose for trade, and p1 accepts p3 s
proposal first. The final allocation is not globally efficient where
the resource should be assigned to p2 .
In the resource trading framework, each tenant completes
transactions with neighbors using only rational deals (RD), and obtains or loses resource bundle accordingly. An RD indicates that the
transaction is beneficial for pushing resources to tenants who value
them more. In fact, ANY sequence of RD executions will achieve
efficiency with regard to the underlying communication topology. This is due to the following observations: (1) RD increases
social welfare according to its definition; and (2) if no more RD is
possible, then the matchmaking must reach the maximum possible social welfare. Given modular valuation function, we have the
following proposition.
Proposition 1 (Convergence to Efficiency [17]). Any sequence of RD
involving any number of resource exchange will eventually yield to
topology-wide efficiency.

Vi (Ai ) Vi (Ai ) i,j

(1)
Vj (Aj ) Vj (Aj ) i,j .
By solving Eq. (1), the result of the payment function i,j falls
into the range of [Vj (Aj ) Vj (Aj ), Vi (Ai ) Vi (Ai )], which defines the
rational payment range.
4.2. A multiagent-based optimization approach for resource trading
This section introduces the theoretical foundation of our multitenancy resource trading protocol design. It is mainly based on
the theoretical framework developed by Chevaleyre et al. [11,12]
for multiagent systems. One central conclusion is that resource allocation efficiency and fairness can be simultaneously achieved in
a multiagent negotiation framework. To support this conclusion, a
proper payment function was selected to deal with the increased
social surplus (A ) (A) after each deal. In particular, a payment
function called Globally Uniform Payment Function (GUPF) was
proposed. Suppose A and A are allocations before and after an RD
execution, respectively, the GUPF is defined as follows.

[(A ) (A)]

.
(2)
n
Eq. (2) is labeled as globally uniform because this payment is
imposed on all tenants. For tenants who do not involved in the
deal, Vi (Ai ) Vi (Ai ) equals to zero, so each of them receives an
equal share of the social surplus created by the trading activity.
Note that GUPF is within the bound of rational payment (Eq. (1)), as
illustrated by the following analysis. First, it is obvious that social
surplus at least does not decrease after an RD occurs. Therefore
(A ) (A) 0. Next, we show that GUPF Vj (Aj ) Vj (Aj ).
According to the definition of social welfare we have:
GUPF : i = [Vi (Ai ) Vi (Ai )]

(A ) (A) =

Vk + Vi (Ai ) + Vj (Aj )

k=i,j

Vk + Vi (Ai ) + Vj (Aj )

k=i,j

= [Vi (Ai ) Vi (Ai )] [Vj (Aj ) Vj (Aj )].

As a result, the following inequality stands.

(A ) (A)
[Vi (Ai ) Vi (Ai )] [Vj (Aj ) Vj (Aj )]
.
n

Rearranging this inequality gives GUPF Vj (Aj ) Vj (Aj ). Finally

we have:
GUPF [Vj (Aj ) Vj (Aj ), Vi (Ai ) Vi (Ai )].
In addition to GUPF, an one-off payment amount at initial is
introduced. The initial payment amount, called initial equitability
(A0 )

payment, is defined as: 0 = Vi (A0i ) n . The main purpose

for this payment function is to level the playing field. The next
two theorems show that imposing initial equitability payment and
GUPF for resource trading leads to efficient and fair matchmaking.
We first show the following invariant for individual utility holds
after every RD.

H. Zhao et al. / J. Parallel Distrib. Comput. 74 (2014) 30873097

3091

Theorem 1. If each tenant pays initial equitability payments at start

and pays GUPF after each RD executes, then all tenants share the same
(A)
utility: Ui (Ai ) = n after each RD.
Proof Sketch. The balance i of pi remains invariant after each
deal [11]. This is true for the base case, where equitability payment is used. Next suppose at allocation A, pi s balance i equals
(A)
to Vi (Ai ) n , then after allocation changes to Ai , adding GUPF

[(Vi (Ai ) Vi (Ai )) (

i = Vi (A ) (A) .

(A )
n

(A)
n

Fig. 3. Deal negotiation.

)] to pi s balance at A leads to

Now with this balance invariant, the utility of pi at allocation A

is computed as:
Ui (Ai ) = Vi (Ai ) i

(A)
= Vi (Ai ) Vi (Ai )
n

(A)

n
Therefore, after each RD, the utility of each tenant presents an
equal share of the overall social welfare. 
With this invariant, we prove the following theorem. Note that
our version is slightly different from that presented in [11], as we
target at topology-wide efficiency and use a more strict assumption of modular domain.
Theorem 2 (Convergence to Efficiency and Fairness [11]). When all
valuations are modular and budget limitation is not a concern, paying
initial equitability payment at start and GUPF after each RD for every
pi P will converge to a matchmaking state that achieves both
topology-wide efficiency and envy-free fairness.
Proof. First, according to Proposition 1, when no RD is possible, the
final matchmaking A is in the topology-wide efficient state. Next,
we show that this topology-wide efficient allocation is also in the
state of envy-free fairness. Recall that two tenants having envy relationship only if there exists a communication link between them.
Now suppose pi and pj be any two tenants directly connected. According to the definition of topology-wide efficiency, transferring
pj s allocation to pi will not increase (A ). Therefore,
Vi (Ai ) + Vj (Aj ) Vi (Ai Aj ).
As valuations are modular, Vi (Ai Aj ) = Vi (Ai ) + Vi (Aj ) Vi (Ai
Aj ). Since Ai Aj = , we have:
Vj (Aj ) Vi (Aj )

Vi (Ai ) Vi (Ai )

(A )

(A )

Vi (Aj ) Vj (Aj )
.

From Theorem 1, Vi (Ai )

(A )
n

equals to i and Vj (Aj )

(A )
n

Therefore, we have Vi (Ai ) i

equals to j at final allocation A.
Vi (Aj ) j for every combination of pairwise tenants. 
4.3. Protocol implementation
We develop a hybrid centralized/distributed resource trading
protocol based on the theoretically optimal negotiation framework. From this point, we refer this protocol Budget-unaware
Multitenancy Resource Trading (BuMRT). In BuMRT, each tenant
has its own execution cycle. Since after each RD, all tenants are imposed to pay a cost defined by GUPF, we also need a bank to redistribute the wealth across the system. This role can be assigned
to some server node, referred to as the bank node. The bank node
is also in charge of other coordinating activities such as charging
initial equitability payment and detecting convergence.

Fig. 4. Payment message broadcast along a broadcast tree.

4.3.1. Local view establishment

For each tenant, the first step in the execution cycle is to establish the local view of envy relationship with other tenants. Specifically, each tenant sets up an agent process and delegates resource
trading to it. The agent process will periodically probe neighboring
tenants who respond with messages encapsulating the true valuation of the holding resources. When the agent notices some resource intrigues more interest than its neighbor, this information
will be recorded by adding the neighbor to a trading partner list.
Note that our current design does not prevent malicious agent process from reporting untruthful valuation of the resource bundle.
How to design incentive-compatible mechanisms to suppress such
behaviors is still an open question to be explored.
4.3.2. Deal negotiation
Each tenant runs two separate processes, each with different
purposes. After the local view is installed, one process is in charge
of initiating resource trading requests to those tenants in the
trading partner list, and the other process uses a proposal queue
to actively handle incoming resource trading requests, as shown in
Fig. 3. When an agreement is established, the two tenants start to
transfer resources and associated payments. The payment amount
determined by GUPF is also broadcasted to others. The trading
process is atomic, i.e., in the event of failure, the trade is rolled back
as if it never happens. When transferring resources, the associated
authentication key to access the resources is transferred. There are
other ways (e.g., [38]) to migrate virtual resources but they are out
of the scope of this paper.
4.3.3. Message broadcasting
Based on the analysis presented in Section 4.2, each RD generates a payment value calculated by GUPF, and this value needs
to be disseminated to all the non-participating tenants in the network. Therefore, after an RD is completed, we designate the trade
proposer to broadcast the corresponding GUPF result to other tenants except for the two involved in the transaction. In order to not
flood the network, we adopt a spanning-tree protocol [2] for broadcasting, as exemplified by Fig. 4. When this process is done, all
tenants communicate with the bank node for redistributing of the
total wealth within the system. When there are multiple RDs to be
completed, there is no need to schedule their executions in a specific order as any sequence of RD can lead to convergence (Proposition 1). An RD can start instantly between pairwise tenants without
waiting for other RDs complete. An RD is completed as long as the
associated payment accounting and bookkeeping are finished.

3092

H. Zhao et al. / J. Parallel Distrib. Comput. 74 (2014) 30873097

Combining the allocation matrix and the envy matrix, we

are ready to unifying allocation and envy relationships into one
directed hypergraph model. We first create a three-dimensional
space, m n n, as shown in the left side of Fig. 5. A directed
hypergraph H = (V , E ) is composed of a finite non-empty set V
of hypernodes and a finite non-empty set E of hyperarcs. Using the
coordinates of the hyperspace, we define the hypernode as follows.
A hypernode v (Eq. (5)) is a three-tuple (x, y, z ), where x R
represents the resource, y R represents the tenant currently
holding x, and z is some tenant has envy relationship with y, i.e.,
z P, y,z = 1. A hyperarc e (Eq. (6)) is a pair T , h, where
T V is the tail of e and h V \ T is its head. The tail set T
includes those hypernodes whose host tenants involved in envy
relationships with the host of the head.
Hypernode: A hypernode is a three-tuple:
Fig. 5. A directed hypergraph model. The proposed directed hypergraph model
derives from an mnn hyperspace. A hypernode is a point mapping allocation and
envy relationship on the hyperspace. A hyperarc connects a hypernode v V with
a set of other hypernodes belonging to a common tenant and has envy relationship
with the v s host tenant. An example directed hypergraph is shown on the right side.

4.3.4. Convergence detection

A trading agent seeking for potential transactions is marked as
active. When no RD is possible, the agent turns to a dormant state.
The event of receiving a trade proposal will trigger the agent to become active again. When all agents become dormant, the matchmaking result achieves topology-wide efficiency and envy-free
fairness simultaneously. Detecting the convergence to this optimal
state poses a classical problem of termination detection that has a
rich literature [28]. We adopt a simple version of the distributed
termination detection algorithm based on that described in [15].

s.t.

v = (x, y, z ) V
xR
y P, and x,y = 1
z P, and y,z = 1.

(5)

Hyperarc: A hyperarc e E is an ordered pair T , h iff:

s.t.

e = T , h E
h = (x1 , y1 , z1 ) V
v = (x2 , y2 , z2 ) T V
y2 = z1 .

(6)

5. Budget-aware resource trading protocol

Each tenant can establish a local view of the directed hypergraph by following the procedure presented in Section 4.3.1. The
hyperarcs imply potential transactions to be negotiated. In a distributed environment, when one transaction is accomplished using an RD, resource allocation changes which might affect other
resource trading activities. Building a directed hypergraph is thus
helpful to evaluate the quality of trading selections.

5.1. Modeling resource trading using a directed hypergraph

5.2. Optimal structures in directed hypergraph

When budget constraint is imposed, the convergence to the optimal matchmaking state might not exist. In this section, we develop a directed hypergraph model for community-based cloud
resource trading. A hypergraph is a generalization of the 2D graph
that an edge can connect a set of vertices. If the hypergraph is directional, an edge (a.k.a. a hyperarc) connects a hypernode (head)
with a set of hypernodes (tail set). The motivation behind the directed hypergraph model lies in its implication for one-to-many relationship. A 2D graph merely models connectivity among tenants,
but cannot represent task allocation and envy relationship among
them. A directed hypergraph is more informative, succinctly capturing the scenario that a resource is held by some tenant, but inspires more interest from some other tenants each holding a set of
resources.
We propose two matrices to build up a hyperspace. The first
matrix is an Allocation Matrix (AM). It is an m n matrix that
takes binary values, representing current resource matchmaking
state for all tenants. Each entry i,j in AM is defined as follows.

When the local view of the directed hypergraph is established,

each tenant has two actions to perform when interacting with others: propose deal and acknowledge deal. When multiple proposals
arrive, the resource can only be traded with one proposer. From a
global point of view, we are interested in finding an optimal structure in the directed hypergraph model that represents the potential impact of a tenants trading selection strategy to successive
transactions. If we associate each hyperarc with certain weight
(e.g., valuation gain or envy degree drop, as will be introduced
later), we would like to find out some optimal structure that minimizes/maximizes the aggregate weights with regard to the specific
structure. For example, one possible optimal structure is a minimum/maximum spanning hypertree. Specifically, let R be the root
set of a hypertree, we define a hypertree TR = (R N , E ) rooted
at R following three conditions: (1) cycle-free; (2) R N = ; and
(3) each v N has exactly one entering hyperarc and no hyperarc
has a node of R as its head. With this description of a hypertree, we
define a spanning hypertree as follows.
Spanning Hypertree: A spanning hypertree of H = (V , E ) is
defined as:

i,j =

1
0

tenant j holds resource i

otherwise.

(3)

The second matrix is an Envy Matrix (EM) representing current

matchmaking unfairness (or envy relationship). Suppose we have
two tenants, Alice and Bob. Bob is said to envy Alice when Bob
has higher valuation for some resource currently allocated to Alice.
Again, we use binary values to represent the envy relationships.
Formally, An Envy Matrix is a n n matrix defined as follows.

i,j =

1
0

pi is envies pj
otherwise.

(4)

s.t.

TR = (V , ET )
ET E
(Te he ) R,

e E \ ET .

(7)

An example of a spanning hypertree is depicted in Fig. 6. If a

hyperarc e exists such that (Te he ) R, we can use it to span
the root set R, and decrease the cardinality of R by one. Therefore,
it guarantees the minimality of the root set R. Finding the optimal
spanning hypertree is helpful to evaluate the quality of the set of
trading decisions with regard to every hypernode in the directed

H. Zhao et al. / J. Parallel Distrib. Comput. 74 (2014) 30873097

3093

Protocol 1: (a) V-BaMRT (b) E-BaMRT (c) P-BaMRT

begin
for pi P do
Scanning neighboring peers, put envious peers into
envy set T ;

// --Trade Proposal Process-while T = do

Fig. 6. (Left) directed hypergraph; (Right) spanning hypertree.

hypergraph model. Another possible application of the directed

hypergraph model is to find the minimum edge cover within
the established hypergraph for optimal trading analysis from a
global point of view. We will not proceed with the hypergraph
algorithm development, as many applications can be found in
prior literature, e.g., finding an optimal spanning hypertree is
NP-complete in general [35]. In the next section, we will develop a
set of heuristic-based protocols that are both effective and simpler
to implement for budget-aware resource trading in a community
cloud environment.
5.3. Budget-aware Multitenancy Resource Trading (BaMRT) protocol
When proposing for resource trade, a tenant rationally calculates its payment amount. When the budget limitation bi is imposed on pi P, the rational payment amount i,j for trade
proposal is in the range of:

i,j [Vj (Aj ) Vj (Aj ), min{Vi (Ai ) Vi (Ai ), bi }].

(8)

According to analysis in Section 4.2, resource allocation in the

community cloud evolves towards efficient and fair state when
tenants pay initial equitability 0 and GUPF in BuMRT. However,
when budget limitation presents, tenants do not always abide by
these routine payments. Therefore, we are interested in investigating the transition of resource allocation states, when tenants
pay different amounts as long as the amounts fall in the range of
Eq. (8). In this section, we propose a series of heuristic-based BaMRTs. These protocols confine the trading activities of each tenant
to neighboring peers, allowing them to conduct local negotiations.
However, They are different with each other in terms of trading selection criterion. The complete description of the proposed BaMRT
protocols are illustrated in Protocol 1.
Tenants delegate trading controls to trading agents who perform two basic operations periodically: proposing trade and selecting offer. When proposing a trade, the agent simply selects the
neighboring peer who he envies most as the trading partner. In
order to quantify the matchmaking unfairness between pairwise
trading partners, we use the following equation to define the envy
degree on a particular hyperarc.

i,j = max{Ui (Ai , i ) Ui (Ai , i ), 0}.

(9)

The trading agent may select any payment amount within the
rational range. A tenant can set up a predefined payment policy
for the trading agent. For example, a conservative policy results
in resource acquisition with low cost, while an aggressive policy
helps funding peer tenants to conduct further trades, and might
benefit more in return. We will evaluate different payment policies
in performance evaluation. When multiple offers arrive, each
trading agent needs to carefully evaluate trading decisions with
a local view of the directed hypergraph model. This is especially
important when offers conflict with each other since the resource
can only be granted to one neighbor. In our design, each trading

1) Calculate envy degrees for every pj T ;

2) Tentatively trade with every pj T and
calculate envy degree decrease;
3) Sorts deals based on envy degree decrease;
4) Selects pj with the highest envy degree
decrease;
5) Selects payment within the range defined by
Equation 8;
if pj accepts offer then
Make payments;
Removes pj from T

// --Offer Selection Process-while offer(s) available (blocks on the offer monitoring

process) do
if single offer then
Accepts offer;
Receives payments and updates local envy list
T;
if multiple offers then
Selects offer with;

(a) highest social welfare gain, or

(b) largest envy degree decrease, or
(c) highest transaction profits

Accepts offer;
Receives payments and updates local envy list
T;

agent employs a hill climbing technique to negotiate resource

trading with neighboring peers. The hill climbing algorithm is fast
and effective in finding a local optimal matchmaking. The local
optimal offer selection decision must be rational, as the payment
amounts conforming to RD increase the overall social welfare
(Proposition 1). In other words, if a trade occurs, the allocation
efficiency is reinforced, and the corresponding envy relationship
between the trading parties is eliminated.
We propose three versions of BaMRT in favor of different trading selection criterion. Each of them follows different paths to
reach the local minimum. The first version labeled as Valuation
oriented BaMRT (V-BaMRT), let trading agents select trades with
the highest social welfare gain. In the second version, each agent
selects the neighboring peer who he envies most as the trading partner. We label this version of BaMRT as Envy oriented
BaMRT (E-BaMRT). Finally, we propose Profit oriented BaMRT
(P-BaMRT), in which agents select offers that will bring in the highest transaction profits (defined as the difference of payment and
gained valuation). These protocols work similarly to BuMRT except
that it does not require message broadcasting to redistribute social
wealth within the community.
6. Performance evaluation
In this section, we investigate the performance of the proposed
protocols through three different sets of simulations. First, we
implement BuMRT and validate its efficiency and fairness. The

3094

H. Zhao et al. / J. Parallel Distrib. Comput. 74 (2014) 30873097

(a) Measurement of efficiency.

(b) Measurement of fairness: envy degree.

(c) Measurement of fairness: envious pair.

Fig. 7. Performance evaluation for budget-unaware case.

simulation results justify our analysis presented in Section 4. In

the second set of simulations, three versions of BaMRT described in
Protocol 1 are compared in various norms. It serves as a guideline to
choose the appropriate heuristic with regard to different desirable
metric. Finally, we evaluate the performance impact of different
payment selection policy and initial budget for BaMRT. In terms
of profit earned from all deals, we show that the aggregate
aggressiveness in payment tends to increase the profit. Moreover,
we show the trend of convergence time of BaMRT influenced by
different initial budget settings in our experiment.
6.1. Simulation settings
We instantiate the matchmaking framework to a generalized
distributed computing environment, and implement the resource
trading protocols using SimGrid [9]. The core scheduling and
communication functions are implemented using the applicationlevel simulation interfaces provided by the MSG module of
SimGrid. The main abstractions in the MSG module are hosts and
tasks. Hosts represent machines on which tasks execute, and tasks
represent amount of work to do and data to exchange. SimGrid
provides predefined APIs to define the simulation environment
(computing and networking capabilities are described in XML
files). One can implement the task scheduling algorithm on top
of this simulation infrastructure and evaluate its effects. Mapping
to our framework, we use hosts to simulate tenants and tasks to
simulate resources to be allocated. In our simulation, we create
a community cloud platform with 20 tenants and 800 resource
units using SimGrid. The key question is how to differentiate
these tenants to simulate valuation heterogeneity for different
resource bundles. To be specific, we need a generic valuation
function to quantify the heterogeneous interests of all tenants.
Due to its subjective nature, there is no closed-form formula for
the valuation. However, using functions conforming to modularity
should be sufficient to serve our purpose in the simulation. We pick
a concave, strictly increasing, and continuous valuation function,
Vi (x) = c xr , to quantify the utility of tenant i obtaining x units of
tasks. The concavity assumption is suitable under a wide range of
circumstances, as it points out that the marginal valuation should
diminish when the allocation increases [26]. We randomly pick c to
10 and r in the range of (0.2, 0.6). In theory, any modular valuation
function should be valid to use in evaluating our algorithms. In
addition, as our algorithms are implemented on top of SimGrid, we
will use nodes and task units hereafter to refer to tenants and
resource units, respectively.
We primarily use four metrics to evaluate the performance of
the proposed protocols. First, we use social welfare to quantify the
allocation efficiency. Next, in order to validate fairness, the total envy degree amongst all nodes is recorded after each transaction. In addition, two nodes that envy each other form an envious
pair. The total number of envious pairs is also counted throughout the negotiation process. Finally, we measure system profit as an

indication of systems side utility. For each transaction, the profit

earned is the difference of buyers valuation and the associated
payment amount. The system profit is thus defined as the cumulative profit earned in all transactions.
6.2. Evaluation of BuMRT
In the first set of simulations, nodes communicate with each
other using BuMRT until convergence is reached. The results are
plotted in Fig. 7. At the start of each simulation, 800 task units
are randomly allocated to 20 nodes. We generate three topology
profiles representing different network configurations. The first
topology profile (labeled as fully connected) describes a fully
connected mesh network, and the rest profiles describe two relatively sparse network topologies. The fully connected topology
has a total node degree of 20 19 = 380. The node degrees
of the other two profiles are normalized relative to the fully connected profile. We use these normalized values, 0.45 and 0.72, to
represent the connectivity of both profiles. In order to validate efficiency, we also implement a self-adaptive auction algorithm [39]
that achieves maximum social welfare when tasks are allocated.
This result, labeled as optimal in Fig. 7(a), defines the global optimal social welfare. From Fig. 7(a), we observe that in all topology profiles, the overall social welfare increases all the time and
converges after around 24 transactions. In addition, for the fully
connected network, the final allocation achieves the maximum social welfare when converges. Fig. 7(b) and (c) show that all simulations converge to fair state where all envy relations are eliminated.
Note that after each transaction, both envy degree and envious pair
number do not necessarily decrease. This can be explained as follows: although the overall unfairness will be eliminated eventually, each single transaction only eliminates envy between the two
trading partners, but may create envy relationship between other
pairs. Another interesting observation for Fig. 7 is that the initial
matchmaking unfairness is closely related to the network connection degree. This is because envy relation is more likely to present
if more nodes are connected. Moreover, more connected network
also means more opportunities for tasks to be assigned to nodes
who value them more. Therefore, the achievable local efficiency is
more likely to increase as the network becomes more connected.
6.3. Convergence analysis
To analyze the convergence rate of the budget-unaware resource trading process, we conduct multiple simulations for varying problem scales. The topology is fixed to a fully connected
network, and we use the same initial allocation for fair comparison. Because BuMRT has no fixed order of RD execution, we average the results of 20 simulation runs for each data point. Initially,
the number of nodes is fixed to 20, and the task unit number is
increased from 50 to 80 with step size of 5. Next, we fix the task
units to 80 and increase the node number from 10 to 70. The result
is displayed in Fig. 8. We observe that while the convergence rate

H. Zhao et al. / J. Parallel Distrib. Comput. 74 (2014) 30873097

3095

Similarly, Fig. 9(c) and (d) show that E-BuMRT performs better in
promoting fairness. And not surprisingly, the overall profits gain is
in favor of U-BuMRT, as shown in Fig. 9(b).
6.5. Sensitivity analysis

Fig. 8. Convergence analysis for budget-unaware case.

is hardly affected by the task unit number (left of Fig. 8), it steadily
increases as system scales up (right of Fig. 8). Hence, we reach the
conclusion that the convergence rate is not dependent on the resources to be allocated, but rather affected by how many tenants
participating in the resource trading process.
6.4. Evaluation of BaMRT
Next, we add budget limitation to each node and compare the
performance of different versions of BaMRT presented in Section 5.
The node and the task unit number are set to be 20 and 800,
respectively. Based on the analysis of the average transaction
payment range, we assign each node an initial budget of 100. When
a transaction is completed, the node making payment will deduct
the corresponding amount from its balance. Conversely, its trade
partner will add the same amount to its balance. For the purpose
of fair comparison, all simulations are conducted using the same
setting (valuation functions and initial allocation). All simulations
use a same fully-connected network. The comparison results are
exhibited in Fig. 9. From these results, we draw the conclusion
that the performance of each protocol is primarily influenced by
the offer selection strategy. In V-BaMRT, the offer brings the most
social welfare growth is selected. Therefore in Fig. 9(a) we observe
that V-BaMRT leads to the highest local efficiency when converged.

In this section, we investigate the impact of different payment

selection strategies and initial budget settings. The configuration
parameters are kept the same as in Section 6.4.
As analyzed in Section 5.3, each node can set up arbitrary payment policy for the trading agent. A conservative policy results in
resource acquisition with low cost, while an aggressive policy helps
funding other tenants to conduct more trading activities. Which
policy gives better result depends on the offer selection strategies
and initial budget distribution. We modify the simulation code to
let each node select payment amount within the allowed range
deterministically. Specifically, let the payment selection range be
(low, high), we devise three deterministic payment selection strategies for evaluation:

Aggressive: payment = lowsec : budget-unaware-protocol +

0.75 (high low)
Modest: payment = low + 0.5 (high low)
Conservative: payment = low + 0.25 (high low).
We compare the aggregate profits of the system in Fig. 10. Each
value is the average result of 20 simulation runs. The result suggests that more aggressive bidding behavior will result in higher
system profits at convergence. This can be explained that if all
nodes offer higher at each deal, more nodes will get funded that
lead to more transactions. As a result, the micro-economy of the
small computing community is boosted.
Finally, we alter the initial budget assignment and measure its
impact to the system envy degree. Taking initial budget of 100 to
be the base case (marked as 1X), the startup fund for each node
is altered from 0.5 to 2 times of 100. Again we average the result

(a) Efficiency isec:budget-unaware-protocolmprovement.

(b) Profits gain.

(c) Fairness improvement: envy degree.

(d) Fairness improvement: envious pair.

Fig. 9. Performance evaluation for budget-aware case.

3096

H. Zhao et al. / J. Parallel Distrib. Comput. 74 (2014) 30873097

implemented using the Zookeeper services [21]. Zookeeper implements a Paxos state machine, and a deployment of the Zookeeper
provides services for highly reliable distributed coordination. Although BuMRT does not require RD serialization, it can still benefit
from many useful functions, e.g., atomic broadcast and leader election. Implementing BuMRT and deploy the protocol to realistic P2P
environment are part of our future research plan.
References

Fig. 10. Impact of different payment selection strategies for budget-aware case.

Fig. 11. Impact of initial budget assignment for budget-aware case.

of 20 simulation runs. The comparison is visualized in Fig. 11. We

observe that for the case of abundant initial fund assignment, the
convergence value is close to that achieved by BuMRT. When the
initial budget reaches 200, all protocols converge to an envy degree of 0 as if there are no budget constraint. On the contrary, for
a poorly funded computing community, the trading activities are
more likely to freeze due to lack of budget, resulting in potential
longer convergence time and higher degree of unfairness.
7. Conclusion and future work
In this paper, we have studied the resource trading problem in a
community-based cloud computing environment. The goal of this
research is to investigate the interactions among independent and
rational tenants, and to establish effective and easy-to-implement
negotiation protocols for system-wide allocation efficiency and
fairness. Towards this goal, we first adopted a multiagent-based
optimization framework and analyzed the optimal results without concerning about budget limitation. Next, we proposed a novel
directed hypergraph model that combines allocation and envy relationship in a three-dimensional hyperspace. This model effectively captured the impact of trading selection decisions from a
global point of view. When budget limitation is imposed on each
tenant, we developed a set of distributed resource trading protocols based on heuristic approaches. Simulation results show that
the proposed protocols perform well in a wide range of settings.
We expect that the comprehensive study presented in this paper
would open new vistas for designing effective resource management strategies in utility computing environments.
In Section 4.3, we discussed the implementation details of
BuMRT. This hybrid resource trading protocol is suitable to be

[1] D. Abramson, R. Buyya, J. Giddy, A computational economy for grid computing

and its implementation in the Nimrod-G resource broker, Future Gener.
Comput. Syst. 18 (2002) 10611074.
[2] B. Awerbuch, Optimal distributed algorithms for minimum weight spanning
tree, counting, leader election, and related problems, in: Proceedings of the
Nineteenth Annual ACM Symposium on Theory of Computing, STOC 1987,
1987, pp. 230240.
[3] K. Berlt, E.S.D. Moura, A.L.D.C. Carvalho, M. Cristo, N. Ziviani, T. Couto, A
hypergraph model for computing page reputation on web collections, in:
Brazilian Symposium on Databases, 2007, pp. 3549.
[4] H. Boche, S. Naik, T. Alpcan, Characterization of non-manipulable and
pareto optimal resource allocation strategies for interference coupled
wireless systems, in: Proceedings of the 29th Conference on Information
Communications, INFOCOM 2010, 2010, pp. 24172425.
[5] S.J. Brams, A.D. Taylor, Fair Division: From Cake-Cutting to Dispute Resolution,
Cambridge University Press, 1996.
[6] G. Briscoe, A. Marinos, Digital ecosystems in the clouds: towards community
cloud computing, in: 3rd IEEE International Conference on Digital Ecosystems
and Technologies, DEST 2009, 2009, pp. 103108.
[7] R. Buyya, D. Abramson, S. Venugopal, The grid economy, Proc. IEEE 93 (3)
(2005) 698714.
[8] R. Buyya, S. Pandey, C. Vecchiola, Cloudbus toolkit for market-oriented cloud
computing, in: Cloud Computing, in: Lecture Notes in Computer Science,
vol. 5931, 2009, pp. 2444.
[9] H. Casanova, A. Legrand, M. Quinson, SimGrid: a generic framework for largescale distributed experiments, in: 10th IEEE International Conference on
Computer Modeling and Simulation, UKSIM 2008, 2008.
[10] U. atalyrek, C. Aykanat, Hypergraph-partitioning-based decomposition for
parallel sparse-matrix vector multiplication, IEEE Trans. Parallel Distrib. Syst.
10 (7) (1999) 673693.
[11] Y. Chevaleyre, U. Endriss, S. Estivie, N. Maudet, Reaching envy-free states
in distributed negotiation settings, in: Proceedings of the Twentieth International Joint Conference on Artificial Intelligence, IJCAI 2007, 2007,
pp. 12391244.
[12] Y. Chevaleyre, U. Endriss, N. Maudet, Allocating goods on a graph to eliminate
envy, in: Proceedings of the 22nd AAAI Conference on Artificial Intelligence,
AAAI 2007, 2007, pp. 700705.
[13] B. Cooper, H. Garcia-Molina, Peer-to-peer resource trading in a reliable
distributed system, in: The First International Workshop on Peer-to-Peer
Systems, IPTPS 2001, 2002, pp. 319327.
[14] A. di Costanzo, M.D. de Assuncao, R. Buyya, Harnessing cloud technologies for
a virtualized distributed computing infrastructure, IEEE Internet Comput. 13
(2009) 2433.
[15] Distributed Termination Detection, URL: http://www.risc.jku.at/software/daj/
TermDetect/.
[16] J. Edmonds, K. Pruhs, Balanced allocations of cake, in: Proceedings of the 47th
Annual IEEE Symposium on Foundations of Computer Science, FOCS 2006,
2006, pp. 623634.
[17] S. Estivie, Y. Chevaleyre, U. Endriss, N. Maudet, How equitable is rational
negotiation? in: Proceedings of the Fifth International Joint Conference
on Autonomous Agents and Multiagent Systems, AAMAS 2006, 2006,
pp. 866873.
[18] Eucalyptus Community Cloud, URL: http://www.eucalyptus.com/eucalyptuscloud/community-cloud.
[19] Federal Community Cloud, URL: http://www-304.ibm.com/industries/
publicsector/us/en/contentemplate1/!!/xmlid=207581.
[20] P. Gradwell, Distributed combinatorial resource scheduling, in: 1st International Workshop on Smart Grid Technologies, 2005.
[21] P. Hunt, M. Konar, F.P. Junqueira, B. Reed, Zookeeper: wait-free coordination
for Internet-scale systems, in: Proceedings of the 2010 USENIX Conference on
USENIX Annual Technical Conference, 2010, pp. 1111.
[22] S. Krishnamoorthy, U. atalyrek, J. Nieplocha, A. Rountev, P. Sadayappan,
Hypergraph partitioning for automatic memory hierarchy management, in:
Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC 2006,
2006.
[23] Y.-K. Kwok, K. Hwang, S. Song, Selfish grids: game-theoretic modeling and
NAS/PSA benchmark evaluation, IEEE Trans. Parallel Distrib. Syst. 18 (2007)
621636.
[24] K. Lai, L. Rasmusson, E. Adar, L. Zhang, B.A. Huberman, Tycoon: an
implementation of a distributed, market-based resource allocation system,
Multiagent Grid Syst. 1 (2005) 169182.
[25] N. Laoutaris, L.J. Poplawski, R. Rajaraman, R. Sundaram, S.-H. Teng, Bounded
budget connection (bbc) games or how to make friends and influence people,
on a budget, in: Proceedings of the Twenty-Seventh ACM Symposium on
Principles of Distributed Computing, PODC 2008, 2008, pp. 165174.

H. Zhao et al. / J. Parallel Distrib. Comput. 74 (2014) 30873097

[26] R.T. Ma, D.M. Chiu, J.C. Lui, V. Misra, D. Rubenstein, On resource management
for cloud users: a generalized kelly mechanism approach, Tech. Rep., Electrical
Engineering, May 2010.
[27] A. Marinos, G. Briscoe, Community cloud computing, CoRR abs/0907.2485.
[28] J. Matocha, T. Camp, A taxonomy of distributed termination detection
algorithms, J. Syst. Softw. 43 (3) (1998) 207221.
[29] S. Pandey, L. Wu, S.M. Guru, R. Buyya, A particle swarm optimizationbased heuristic for scheduling workflow applications in cloud computing
environments, in: Proceedings of the 24th IEEE International Conference
on Advanced Information Networking and Applications, AINA 2010, 2010,
pp. 400407.
[30] J. Perez, C. Germain-Renaud, B. Kgl, C. Loomis, Multi-objective reinforcement
learning for responsive grids, J. Grid Comput. 8 (2010) 473492.
[31] O. Regev, N. Nisan, The POPCORN market. Online markets for computational
resources, Decis. Support Syst. 28 (12) (2000) 177189.
[32] K. Rzadca, D. Trystram, A. Wierzbicki, Fair game-theoretic resource management in dedicated grids, in: Proceedings of the Seventh IEEE International Symposium on Cluster Computing and the Grid, CCGrid 2007, 2007,
pp. 343350.
[33] S. Sarkar, K. Sivarajan, Hypergraph models for cellular mobile communication
systems, IEEE Trans. Veh. Technol. 47 (2) (1998) 460471.
[34] B. Song, M. Hassan, E.-N. Huh, A novel cloud market infrastructure for
trading service, in: Proceedings of the 2009 International Conference on
Computational Science and its Applications, ICCSA 2009, 2009, pp. 4450.
[35] I. Tomescu, M. Zimand, Minimum spanning hypertrees, Discrete Appl. Math.
54 (1) (1994) 6776.
[36] D.A. Turner, K.W. Ross, A lightweight currency paradigm for the P2P resource
market, in: Seventh International Conference on Electronic Commerce
Research, 2004.
[37] R. Wolski, J.S. Plank, J. Brevik, T. Bryan, G-commerce: market formulations
controlling resource allocation on the computational grid, in: Proceedings of
the 15th International Parallel & Distributed Processing Symposium, IPDPS
2001, 2001, pp. 4653.
[38] M. Zhao, R.J. Figueiredo, Experimental study of virtual machine migration
in support of reservation of cluster resources, in: Proceedings of the
2nd International Workshop on Virtualization Technology in Distributed
Computing, VTDC07, 2007, pp. 5:15:8.
[39] H. Zhao, X. Li, Efficient grid task-bundle allocation using bargaining based
self-adaptive auction, in: Proceedings of the 9th IEEE/ACM International
Symposium on Cluster Computing and the Grid, CCGrid 2009, 2009,
pp. 411.

3097

Han Zhao is currently a senior research engineer at Qualcomm Research. He received the Ph.D. in Computer and Information Science and Engineering at University of Florida,
and the B.E. degree in Software Engineering from Jilin University, China, and the M.Sc. degree in Computer Science
from Oklahoma State University. His research interests include parallel and distributed computing, autonomic systems and P2P systems. He is a student member of ACM and
IEEE.

Xinxin Liu is currently a software engineer at Google. She

received her Ph.D. Computer and Information Science and
Engineering from University of Florida, and her B.E. degree
in Software Engineering from Jilin University, China. Her
research interests include wireless sensor networks and
resource management. She is a student member of IEEE.

Xiaolin Li is Associate Professor in Department of Electrical and Computer Engineering at University of Florida.
His research interests include Parallel and Distributed Systems, CyberPhysical Systems, and Network Security. His
research has been sponsored by National Science Foundation (NSF), Department of Homeland Security (DHS), and
other funding agencies. He is an associate editor of several international journals and a program chair or co-chair
for over 10 international conferences and workshops. He
is on the executive committee of IEEE Technical Committee on Scalable Computing (TCSC) and served as a panelist
for NSF. He received a Ph.D. degree in Computer Engineering from Rutgers University, USA. He is the founding director of the Scalable Software Systems Laboratory
(http://www.s3lab.ece.ufl.edu). He received the National Science Foundation CAREER Award in 2010. He is a member of IEEE and ACM.