Towards improving resource discovery in Grid computing using semantic and multi-agent

methodology

Abdul Khalique Shaikh1, Saadat M. Alhashmi2, Rajendran Parthiban3

School of IT, Monash University, Sunway Campus, Malaysia,

1
shaikh@infotech.monash.edu.my
2
saadat.m.alhashmi@infotech.monash.edu.my
3
School of Engineering, Monash University, Bandar Sunway, Malaysia,

rajendran.parthiban@eng.monash.edu.my

Abstract
=====================================================================

Grid computing can be used to aggregate physical and logical resources for executing compute-intensive
applications. However, due to the heterogeneous and dynamic nature of computational Grid systems, efficient
resource discovery is a very challenging issue. Agents can be used to enhance resource discovery processes, since
they are distributed in nature and autonomous and intelligent in behavior. Furthermore, agents can interact with
neighboring agents based on local knowledge and move along predetermined routes autonomously on behalf of a
user. Ontology can provide benefits in a resource discovery mechanism. These benefits include enhancement of
interoperability, quality of information, accuracy and efficiency. In this paper, we propose a comprehensive solution
for a decentralized resource discovery process by improving the ontological framework. This proposed work is
based on extending the combined semantic and agent approaches. The aim of this research is to extend and develop
the decentralized agent and semantic-based resource discovery process to improve the performance and maximize
the utilization of resources in a computational Grid. Preliminary simulations carried out using the proposed approach
show more tasks being accepted for complex jobs. Consequently, this approach could be useful in increasing the
utilization of resources in a Grid Computing environment.

Keywords: Grid Computing, Resource Discovery, Semantic Technology, Multi-agent system, Ontology
1. Introduction
Grid computing is a distributed computing
environment aimed at sharing geographically
distributed resources [1]. Resource discovery is a
function of Resource Management System (RMS)
which is a central part of Grid computing and
responsible for collaboration of these distributed
resources in an efficient way [2]. As the resources in
Grid computing environment are dispersed and
dynamic in nature, the maximal utilization and
collaboration of these resources can be challenging.
Furthermore, these resources are managed
independently by various organizations with their
own local policies [3]. Such kind of behaviors of a
Grid system affects the resource discovery process
radically.
Centralized resource discovery approach is
preferred in small and medium size Grids which
provides easier control and management. However, it
is not scalable and causes poor performance in a large
scale Grid system. On the other hand, a distributed
mechanism is more successful than centralized
because it possesses characteristics of scalability,
dynamism, load balancing, and heterogeneity and
fault resilience. However, a syntactical distributed
approach is based on exact keyword matching and
provides limited results. To overcome these issues,
semantic technology has been introduced. In the
semantic approach, semantic metadata and ontology
are used to describe the resources which are very
helpful to discover and monitor Grid computing
resources. Agents with semantic technology can
improve the resource discovery process as they are
distributed in nature and can move to predetermined
routes autonomously. By integrating agent and
semantic-based approach, the resource discovery
process in Grid computing is significantly improved.
We propose a comprehensive solution for a
decentralized resource discovery process by
enhancing the ontological framework for which, we
develop sub-domain specific ontologies for each Grid
resource. We also use agents for avoiding overhead
of negotiation where resource agents need to know
neighbor connections and semantic descriptions of
neighboring resource agents. Based on local
knowledge, the resource agent interacts semantically
with neighboring agents and forms resource chain for
a task. This proposed work is based on extending the
integration of semantic and agent approaches.
The remaining sections of this paper are
organized as follows:
Section 2 describes reviews and outlines
limitations in existing resource discovery systems.
Section 3 explains multi-agent and semantic
technologies. Section 4 presents implementation and
results. Finally section 5 provides conclusion and
possible future work.
2. Related work
Significant research has been done to
improve resources discovery component of Grid
computing in both academia and industry. This
section discusses some of the state of the art in this
domain before proposing how it can further be
improved.
Condor [4] uses centralized queries for
resource discovery which works on the principle of a
matchmaking approach [5]. The matchmaking is
based on an idea that resource providing services and
clients requesting service advertise their
characteristics and requirements using classified
advertisements (ClassAds). The matchmaking
framework includes several components of a resource
discovery mechanism which provides stateless
matching services [6]. MDS [7] is a suite of
components for monitoring and discovering resources
and services which offered by Globus. It uses
standard interfaces defined within the WS-Resource
Framework (WSRF) and WS-Notification (WS-N)
specifications to provide query and subscription
interfaces. Both Condor and MDS follow centralized
and hierarchical approaches framework respectively
and work fine for a small to a medium scale Grid.
However, for large-scale Grids, these are not scalable
and cause poor performance. Also it poses single
point of failure [8], and do not guarantee scalability
and fault tolerance [9].
On the other hand, distributed approach in a
resource discovery process is very helpful in terms of
self-organization, adaption, load balancing and fault
tolerance but it has a number of limitations such as
searching mechanisms rely on string matching [1].
This mechanism is based on fixed query method
which gives very limited results. A DHT (Distributed
Hash Table) based search algorithm is introduced in
[10] where the author has designed and implemented
SWORD, a Scalable Resource Discovery service for
Wide-area distributed system. Author has claimed
that DHTs are important distribution application
building blocks because a service built on a top of
DHT automatically inherits the DHT‟s self
configuration, self healing and scalability. However
the paper [6] mentions that SWORD provides
network coordinates as a location information but the
latency is not sufficient for band-width demanding
applications. The focus of SWORD is only to provide
scalability in terms of querying [11]. In all the above
syntactical distributed approaches, it is possible to
miss relatively capable resources because it is based
on exact key words matching and current schedulers
in Grid Computing do not possess the capability to
infer the relationship among resources [12]. To
overcome these issues, a semantic approach has been
introduced.
A peer-to-peer semantic-based system Atlas
proposed in [13] for distributed storage and retrieval
of RDFs data. This is built on top of the DHT
bamboo and supports pull and push query scenario. It
is being used currently to realize the metadata service
of S-OGSA (Semantic Open Grid Service
Architecture) in a fully distributed and scalable way.
Authors claim that resource discovery services for
semantic grid can be made scalable, fault tolerant,
robust and adaptive. However, the process of query
evolution, measurement and experimental results are
missing.
A novel semantic supported agent-based
decentralized Grid resource discovery mechanism has
been proposed in [14]. This heuristic algorithm has
been developed to find out the neighbors resources
and introducing semantic similarity of domain
ontology using the decentralized approach. The
experimental results show that the success probability
of resource discovery increases with decrease in the
similarity threshold. The level of the task complexity
is inversely proportional to success probability.
Authors claim that the algorithm has a flexibility to
efficiently and dynamically discover resources.
However, their experiment results show that the task
success probability is very low under average task
complexity and average semantic threshold values. In
this experiment authors use a general domain specific
ontology with artificial generated similarity threshold
values. However, in reality, each type of resource is
different in terms of function. This is the reason we
use sub-domain specific ontologies for each resource
separately.
 To make a comprehensive resource
discovery model, we propose extending the combined
agent and semantic approach for accurate and
autonomous resource discovery. The semantic
matching is very flexible and it can enhance
interpretability in a Grid environment [15]. Agents
are extremely helpful to provide methodologies for
consuming, processing, creating and disseminating of
information and knowledge within a distributed
framework. These objectives could be achieved by
improving ontological frameworks using sub-domain
specific ontology for each type of resources
separately. Semantic threshold values must be
computed by an appropriate formula.
3. Multi-agent and Semantic
Technology
This section presents features of a Multi-
agent system and a semantic technology. It also
explains how these can be supportive in resource
discovery process in Grid computing.
3.1 Multi-agent Technology
Agents can be useful for analysis, design
and realization of a distributed environment such as
Grid. Agents have ability to manage dynamic
properties [16, 17]. They provide better solutions in
terms of adding autonomy, adaptability and control
intelligence to the Grid services [18]. To search for
appropriate resources, agents can move by
predetermined routes autonomously and process a job
in a very flexible way. An individual agent can query
neighbor nodes without need of global knowledge
and can semantically interact with neighbor agent-
based on local knowledge [14]. In our experiment
there are 64 agents on provider side and four types of
resources are uniformly distributed among these
agents. Each agent has its own label i.e. A1,Q2, Ab2
etc. Each agent is randomly connected with some
other agents who are called neighbors of that agent.
Those neighbor agents keep information about other
agents which play an important role in making
resource chain for a task.
3.2 Semantic Technology
The performance of resource discovery
depends on the quality of information. Semantic
technology provides meta information which
describes data semantic and supports interoperability
[19]. Semantic similarity threshold values can be
calculated with the help of ontologies of Grid
resources.
3.2.1 Semantic Similarity:
Semantic similarity is defined as how
concepts of ontology are related to each other. The
similarity of concepts represents the degree of
commonality between concepts. The authors of paper
[20] derive conceptual similarity using the notion of
“similarity graph” which is a subpart of the ontology
represented as a graph with subset of concepts as
nodes and relations connecting these as edges. Also
authors of this paper derive a function sim (x,y) that
measures the degree of similarity proportional to how
much concept x and y share or how close they are.
We compute semantic similarity based on the
following formula which is published in the same
paper.
Equation 1
where is a factor that determines the degree of
influence of generalization of ontology concepts. The
value of „ ‟ lies in between 0 and 1. If value of ‟ ‟ is
1 that means perfect generalization where each and
every concept is defined and 0 means very poor
generalization. We use the value of P = 2/3 (0.67)
which is used to compute the semantic similarity
threshold in paper [20] with the same degree of
generalization, is the set of nodes (upwards)
reachable from x and is the reachable
nodes shared by x and y. The lowest values
means not similar and the highest
value of means fully similar i.e
We use four types of resources i.e Processor,
Memory, Storage and Operating System in our
experiment. To compute semantic similarity, we use
two existing ontologies for resource „Processor‟ and
„Operating System‟ from paper [21] and developed
ontologies for other two types of resources i.e
„Memory‟ and „Storage‟. The Ontology for
„Memory‟ resources is as follows:
Figure 1: Ontology for ‘Memory’ Resource
With the help of four sub-domain specific
ontologies and Equation 1, we compute the values of
semantic similarity threshold between concepts of
resources. The semantic similarity threshold values of
„Memory‟ resource concepts are as follows:
 4.1 Searching Mechanism
Suppose a user selects two tasks and set the
task complexity value 0.25 with the semantic
similarity threshold 0.4, it means each task requires 4
resources to get the task done with semantic
similarity equal or greater than 0.4 values. We
assume that task 1 requires one „Processor‟, One
„Memory‟, One „Storage‟, and one „Operating
system‟. However, name of resources for this task 1
are randomly generated from ontology concepts for
respective resource types i.e „Processor32‟ „RAM‟
„HardDisk‟ and „MacOs‟ respectively. The task 2
requires three „Memory‟ and one „Operating system‟.
To make a complete service chain of required
resources for these two tasks separately, searching
mechanism compiles the following steps.
First of all, search algorithm works for the
Table 1: Semantic Similarity Threshold Values task 1 required resources i.e „Processor32‟ „RAM‟
of ‘Memory’ Resource „HardDisk‟ and „MacOs‟. Search algorithm selects
an agent „K1‟ randomly who has resource
3.2.2 Task Complexity
„Processor‟ now this agent compares its resource
Task complexity is measured on
with one of the required resources i.e „Processor32‟
the basis of length of the tasks which means
„RAM‟ „HardDisk‟ and „MacOs‟ based on semantic
resources required by tasks are different. If task
similarity threshold. In this case, the similarity
requires maximum number of resources then the task
threshold between „Processor‟ and „Processor32‟ is
complexity is high and if task requires minimum
0.78 and user has set 0.4 threshold it means
resources then the task complexity is low. We define
semantic similarity of the agent is greater than the
the length of tasks 4, 8, 12 and 16 for each task. If
required resource, then algorithm checks if
task complexity is set 0.25 it means one task needs 4
neighbors of K1 that hold resources i.e (Processor,
resources and 0.5 means 8 resources, 0.75 means 12
Storage, Memory, OS) has at least one of the
resources and 1 means one task requires 16 resources.
required resource other than randomly picked agent
resource i.e (RAM, HardDisk and MacOs ) and their
status should be free. In this case it meets above
4. Implementation conditions and randomly picked node K1 who has
We extend the decentralized resource agent- resource „Processor‟ is committed for the task 1.
based semantic resource discovery algorithm used in In case of the semantic similarity threshold
paper [14]. To implement resource discovery model, is not matched, this step repeats until and unless we
we opted to work on Netlogo simulation and get the right resource. In 2nd step, randomly
modeling tool. selected another node i.e „AP3‟ who has resource
We evaluate the performance of the „Storage‟, algorithm requires the agent AP3
resource discovery model in terms of task success
compares its own resource i.e „Storage‟ with one of
probability. We observe the task success probability
the required resources i.e „RAM‟ „HardDisk‟ and
under different network and under different
„MacOs‟ on basis of semantic threshold. In this case
semantic threshold. Our objective is to extend this
work to make resource discovery accurate and semantic similarity threshold between „Storage‟ and
autonomous by improving ontological framework. „Harddisk‟ is 0.78 which is greater than pre-set
We use sub-domain specific ontology for each type value of semantic threshold. It means condition is
of resource separately. In this implementation, we matched, then checks if neighbor of AP3 node
use realistic semantic similarity threshold values resources i.e OS, Memory Storage and Processor
which are calculated using Equation 1. We have has at least one resource which is already
created 64 agents on provider side and uniformly committed, in this case „Processor‟ is matched
distributed four types of resources among these because it is already committed. Now randomly
agents. picked node which has resource „Storage‟ are
committed for the task 1. At this stage we have two
committed resources i.e Processor, HardDisk and
two required resources RAM and MacOs. Repeats
 randomly selecting one another node Bj4 who has
resource „OS‟ first it compares the semantic
similarity threshold with one of required resource, in
this case semantic similarity threshold between OS
and MacOS is 0.78 which is greater than required
semantic similarity threshold then checks neighbors
of BJ4 has resources „Processor‟ „Storage‟ „OS‟
„Memory‟, one of these resource node is already
committed with the task 1 so resource „OS‟ is
committed on behalf of „MacOS‟, for last resource
of the task 1, randomly selecting one another node
„Ad2‟ who has resource „Memory‟, the algorithm
requires compares this resource with one of the
required resource i.e. „RAM‟ on basis of semantic
similarity threshold. The value is 0.78 between
„Memory‟ and „RAM‟ which is greater than pre set
value of semantic similarity threshold, then checks
if neighbors of Ad2 who has resources (Storage
Processor and OS) and one of these should be
committed for the task 1 then finally the randomly
picked node who has resource „OS‟ is committed.
Now the resource chain for the task 1 has completed
and it is ready to work with required resources.
Subsequently 2nd task, once again, follow the
similar steps and so on. If any task doesn‟t fulfill the
algorithm steps then it is rejected and algorithm
works for next job.
5. Results
We have run 200 simulations with different
values of the task complexity, the number of tasks
and the semantic similarity threshold. The results of
these simulations are as follows:
Figure 2: Result of simulations under task
complexity 0.25
Figure 3: Result of simulations under task
complexity 0.50
Above graphs show the relationship between
the task success probability and the semantic
similarity under different levels of the task
complexity and the different number of tasks. The
results show that the task success probability is
higher when using low semantic similarity threshold
and the success probability is lower when high task
complexity is set. The results show that the greater
number of tasks affects the low task success
probability. We have compared our results with paper
[14] and found our results better than that. The reason
of getting better task success probability is that we
have used sub domain of specific ontology for each
type of resource and we have computed semantic
threshold values by appropriate formula. However
under 0.8 and 1.0 semantic threshold our results show
0 task success probability. The reason is, a
classification widens the similarity gap. Hence, when
the semantic values are increased, the task success
probability in our experiment is almost 0.
Our overall results indicate that the extending the
integration of semantic and agent approach by
enhancing ontological framework can be useful in
increasing the utilization of resources and improving
the performance of resource discovery process which
is beneficial for both providers and users.
6. Conclusion and Future Directions
It can be concluded that a resource searching
mechanism in a Grid Computing environment must
be flexible and effective so that tasks can be run with
appropriate required resources in a given timeframe.
An extension of a combined agent and semantic-
based decentralized resource discovery model has
been presented and implemented in this paper. The
proposed model allows individual resource agents to
find neighboring agents-based on local knowledge
rather than on global information. We have used sub-
domain specific ontologies for each resource
separately and computed semantic threshold values
on the basis of ontological concepts. We compared
our results with existing work and found our results
to be slightly better in terms of task success
probability. Our experimental findings show that the
success probability of resource discovery increases
with the decrease in similarity threshold. The level of
the task complexity is inversely proportional to the
task success probability. As for future work, we are
planning to generate detailed ontologies for each grid
computing resource which include synonyms,
hyponyms and hypernyms. Subsequently we will
extend this resource discovery model into large scale
to check the scalability of the system.
References
1. J. Li, “Grid resource discovery based on
semantically linked virtual organizations,”
Future Generation Computer Systems, vol. 26,
no. 3, 2009, pp. 361-373.
2. M. Beltrán, et al., “Resource Dissemination
Policies on Grids,” On the Move to Meaningful
Internet Systems 2004: OTM 2004 Workshops,
2004, pp. 135-143.
3. A. Caminero, et al., “Extending GridSim with an
architecture for failure detection,” Proc. Parallel
and Distributed Systems, 2007 International
Conference on, 2007, pp. 1-8.
4. K. Krauter, et. al., “A taxonomy and survey of
grid resource management systems for
distributed computing,” Software: Practice and
Experience, vol. 32, no. 2, 2002, pp. 135-164.
5. A. Sharma and S. Bawa, “An Improved
Resource Discovery Approach Using P2P Model
for Condor: A Grid Middleware,” Proc. World
Academy of Science, Engineering And
Technology, 2006.
6. J. Kim, et al., “Using Data Accessibility for
Resource Selection in Large-Scale Distributed
Systems,” IEEE Transactions on Parallel and
Distribution System, vol. 20, no. 6, 2009, pp.
788-801.
7. J. Schopf, et al., “Monitoring and discovery in a
web services framework: Functionality and
performance of the globus toolkit's mds4,”
Argonne National Laboratory, Preprint
ANL/MCS-P1248-0405, 2005.
8. Hauswirth and R. M. Schmidt, “An overlay
network for resource discovery in Grids,” Proc.
Sixteenth International Workshop on Database
and Expert Systems Applications, 2005, pp. 343
- 348
9. P. Trunfio, et al., “Peer-to-Peer resource
discovery in Grids: Models and systems,” Future
Generation Computer Systems, vol. 23, no. 7,
2007, pp. 864-878.
10. J. Albrecht, et al., “Design and implementation
trade-offs for wide-area resource discovery,”
ACM Trans. Internet Technol., vol. 8, no. 4,
2008, pp. 1-44.
11. M. Cardosa and A. Chandra, “Resource Bundles:
Using Aggregation for Statistical Wide-Area
Resource Discovery and Allocation,” Proc. The
28th International Conference on Distributed
Computing Systems IEEE Computer Society,
2008.
12. B.R. Amarnath, et al., “Ontology-based Grid
resource management,” Software: Practice and
Experience, vol. 39, no. 17, 2009, pp. 1419-
1438.
13. Z. Kaoudi, et al., “Semantic Grid Resource
Discovery in Atlas,” Knowledge and Data
Management in GRIDs, 2007, pp. 185-199.
14. H. Liangxiu and D. Berry, “Semantic-Supported
and Agent-Based Decentralized Grid Resource
Discovery,” Future Generation Computer
Systems, vol. 24, no. 8, 2008, pp. 806-812.
15. L. Chen and F. Tao, “An Intelligent
Recommender System for Web Resource
Discovery and Selection,” Intelligent Decision
and Policy Making Support Systems, 2008, pp.
113-140.
16. Z. Li and M. Parashar, “Rudder: An agent-based
infrastructure for autonomic composition of grid
applications,” Multiagent and Grid Systems, vol.
1, no. 3, 2005, pp. 183-195.
17. Y. Jian, et al., “Grid Resource Management
Based on Mobile Agent,” Proc. International
Conference on Computational Inteligence for
Modelling Control and Automation and
International Conference on Intelligent Agents
Web Technologies and International Commerce
(CIMCA'06), 2006, pp. 255-255.
18. J.W. Yin, et al., “Weaving an agent-based
Semantic Grid for distributed collaborative
manufacturing,” International Journal of
Production Research, vol. 48, no. 7, 2009, pp.
2109 - 2126.
19. D. Fensel, Ontologies: a silver bullet for
knowledge management and electronic
commerce, Springer Verlag, 2004.
20. T. Andreasen, et al., “From ontology over
similarity to query evaluation,” Proc. 2nd
CologNET-ElsNET Symposium-Questions and
Answer: Theoretical and Applied Perspective,
2003, pp. 39-50.
21. A.C.T. Vidal, et al., “Applying semantics to
grid middleware,” Concurr. Comput. : Pract.
Exper., vol. 21, no. 13, 2009, pp. 1725-1741.