1

IntelligentGrid: Rapid Deployment of Grid Compute Nodes forImmediate Execution of Batch and Parallel Applications

Amril Nazir, Hong Ong, Sidek SallehMIMOS Bhd

Technology Park Malaysia57000 Kuala Lumpur, Malaysia

amrilnurman.nazir,ong.h@mimos.my

Thamarai Selvi, Rajendar KDepartment of Computer Technology

Madras Institute of TechnologyAnna University

stselvi@annauniv.edu,k.rajendar@gmail.com

Abstract—In this paper, we present IntelligentGrid – a proto-type system that enables a rapid resource provisioning of Gridcompute nodes for immediate execution of queued high priorityjobs in production Grid environments. It uses virtual machine(VM) to technology to deploy additional Grid compute nodes forthe execution of serial and parallel jobs. We have built a prototypeof IntelligentGrid using a well-known gLite Grid middleware andXen hypervisor, and our experiments confirm the feasibility ofthis approach.

I. INTRODUCTION

A computational Grid enables the aggregation of multipleclusters for dynamic sharing, selection, and management ofresources. Examples of Grid middleware such as gLite [2],Globus [6] and Unicore [5] provide protocols and function-alities to enable the dynamic of sharing resources. Generally,they combine existing cluster management systems such asCondor [8], SGE [9], LSF [17], and PBS [10] job managementsystems to manage the computational needs of sequential andparallel applications.

Although much work has been done to improve the variousaspects of Grid software, the current Grid and resource man-agement systems have several shortcomings. First, it is oftendifficult to obtain as many resources as a job may need. Insuch situation, the user job is forced to wait in the queue untilsuitable resources are released by other running jobs that havecompleted their execution. In the worst case, the user job maynot even run at all if the job requirements cannot be met by allaccessible physical servers. Second, it is difficult to obtain theresources when one needs them at sudden spikes in demand.In particular, at high load, it is often difficult to find physicalresources that exactly match the number of processors as wellas software specification requested by the user. As a result,the job cannot be executed. Third, the number of resourcesthat can be deployed at any given time is restricted to thenumber of physical machines; only one job can be executedon a single physical server node at any one time regardlessof the number of processors available. Therefore, given theabove limitations, there is a need to design a mechanism foron-demand deployment of resources when they are neededmost; specifically, this implies that resources have to be made

available immediately with very little advance notice. Thisrequirement is especially crucial for high-priority jobs.

This paper describes IntelligentGrid, a system for rapidprovisioning of Grid worker nodes for pending, urgent jobs.IntelligentGrid uses virtual machine (VM) technology to dy-namically provision Grid worker nodes. The virtual machinetechnology is based on the Xen hypervisor [3] and the Gridinfrastructure is based on the well-known gLite middleware[4]. The gLite middleware is used by the Enabling Grids forthe E-Science Grid (EGEE), which is currently the world’slargest production Grid with collaborative efforts across 319sites with a total number of 100,945 CPUs. Similarly, Xen isa popular open-source high performance virtualization tech-nology originally developed at the University of Cambridge.With the combination of both technologies and our proposedresource management, IntelligentGrid was developed to realizethe need for the creation of a customized Grid environmentfor the efficient management of users’ tasks.

II. RELATED WORK

Several efforts have been made towards applying virtu-alization concepts within a Grid environment. First, VirtualWorkspace [12] represents an approach for the integration ofGrid and virtualization. The approach allows users to definea virtual environment on top of a Grid environment. Thebasic idea is to allow Globus middleware to run on top ofthe Xen hypervisor. From this work, [14] and [7] proposeda virtual cluster that could be formed for the managementof an execution environment using virtual machines. Thisapproach also extends virtual workspace and investigates theperformance issues related to the execution of applications ona virtual cluster. A middleware system, Violin [11] proposed avirtual Internet-working infrastructure which manages virtualmachines and virtual network technologies to create virtualdistributed environments. This also enables the execution ofdistributed applications which require customized execution,network environments and security. The In-Vigo project [1]was built using components that allow for the virtualizationof Grid resources and user interfaces. Additionally, the In-Vigo has a distributed virtual file system to facilitate data

2011 IEEE Conference on Open Systems (ICOS2011), September 25 - 28, 2011, Langkawi, Malaysia

978-1-61284-931-7/11/$26.00 ©2011 IEEE 180

2

transfer across Grid resources; virtual machines provide isola-tion, resource integrity, legacy software support, environmentencapsulation and customization.

Another effort has been made to incorporate virtual ma-chine technology in the Globus Grid infrastructure [15]. Thisarchitecture deploys virtual machines across the Grid siteand manages jobs using the Gridway scheduler. A solutionfor controlling and managing a Grid infrastructure is alsodescribed in [13] where the elements of a gLite Grid mid-dleware can be incorporated into virtual machines, whichthen can be deployed throughout the Grid infrastructure asneeded, reducing significantly the cost of system maintenanceand management. The CARE resource broker [16] also usesvirtualization technology to deploy virtual machines acrossthe Globus Grid site. The CARE broker deploys the requirednumber of virtual machines in the Globus Grid infrastructureto meet the application requirements. CARE can create virtualclusters dynamically where each virtual machine is configuredwith the required software execution environment for facilitat-ing application execution on demand.

All these research efforts address the benefits of virtualiza-tion in a Grid environment. Our work is similar in that we also,in effect, make use of virtualization technology to provide theexecution environment for running Grid jobs. However, ourwork differs in that:

1) We propose an alternative system which manages thejobs across a Grid infrastructure without the need todefine additional interfaces;

2) In our implementation, there is no need to modify theexisting configuration of Grid systems; and

3) Most importantly, unlike prior works, our architecturemakes it possible to automatically and dynamicallydeploy additional Grid worker nodes using virtualizationtechnology based on workload demands without userintervention.

III. ARCHITECTURAL OVERVIEW

This section presents an architectural overview of oursystem. The system relies on virtualization technology todynamically provision additional worker nodes for pendingjobs. At the high level, users may submit their jobs either to themeta-scheduler (MS) level or directly to the Local ResourceManager (LRM).

Figure 1. High Level Architectural Overview of IntelligentGrid Grid inrelation to Grid Infrastructure.

Figure 1 shows the architectural overview of how the In-telligentGrid is integrated with the overall Grid infrastructure.

At the high level, the meta-scheduler or broker performs thenecessary resource matching process for the allocation of eachjob to the most appropriate Resource Manager. As can be seenfrom the figure, the IntelligentGrid component resides betweenthe LRM and hardware levels. All pending jobs that cannot beallocated by the MS (either due to an insufficient amount ofnodes or software unavailability) are redirected to the LRMs,which are supported by the IntelligentGrid, for execution.

IV. IMPLEMENTATION

The IntelligentGrid prototype uses gLite middleware toconstruct the Grid infrastructure and Xen hypervisor forthe virtualization technology. The Local Resource Manager(LRM) is based on the Open PBS batch scheduler. The gLiteGrid architecture consists of the following major components:User Interface (UI), Workload Management System (WMS),Computing Element (CE) and Worker node (WN). The UserInterface (UI) resides at the top level and enables users tosubmit jobs. Jobs which are submitted from the UI are sent tothe Workload Management System (WMS). The WMS has alist of all available Local Resource Managers which are knownas Computing Element (CEs). Based on the job requirements,the WMS determines the most appropriate CE to submit thejob to. If there are sufficient and suitable worker nodes (WNs)to run the job, the job is executed immediately; otherwise,the job is held in the queue until sufficient worker nodesare available. The IntelligentGrid prototype is implementedusing Java and UNIX shell scripts. The LRMS Queue Monitorperiodically pulls information on queued status jobs at theCE by using the ‘qstat –a’ command and extracts the noderequirement.

Algorithm 1 Pseudo code for VM resource provisioning.1) Get high-priority, queued job J from LRM queue.2) Process K jobs on available physical worker nodes.3) Calculate the number of virtual machines N which

can be created using the information about physicalmachines.

4) Get the number of jobs J-K which can be processed bythe creation of M (M<=N) virtual machines.

5) Put P jobs on ‘held’ state and invoke the deploymentprocess of virtual machines.

6) Monitor the VM creation process and, after deployment,configure necessary settings and submit job.

7) Track job state and after successful execution removevirtual machines after completion of job.

8) Repeat steps from (1) until all jobs have been processed.

When all the relevant parameters are obtained, the VMCreator starts to prepare and configure required number ofvirtual machines from a VM image file. A VM image file isa pre-configured image file that is comprised of the ScientificLinux version 4 OS with gLite’s Worker Node service. It isalso configured with the relevant MPICH library for supportingthe execution of parallel jobs in Grid environments. To ensurethat cross-communication can be established between existingworker nodes and newly deployed nodes, the VM Creator

2011 IEEE Conference on Open Systems (ICOS2011), September 25 - 28, 2011, Langkawi, Malaysia

181

3

appends the IP addresses of newly created VMs to boththe CE and the MPICH configuration files. Similarly, eachtime a worker node is removed upon job completion, theVM Terminator removes the corresponding IP addresses fromthe CE and the MPICH configuration files as appropriate.The pseudo code for VM resource provisioning is shown inAlgorithm 1.

V. PERFORMANCE

Performance evaluation was conducted to answer the fol-lowing questions: a) How long would it take to deploy Gridworker nodes with IntelligentGrid, i.e., taking into consider-ation deployment and configuration latency? Specifically, weare interested in finding out how the delays in our approachcompare to those of current Grid approaches; b) What is theimpact of deployment request and task allocation costs on thepractical aspect of resource provisioning strategies employedby IntelligentGrid?

First, we assume a typical gLite Grid environment where aGrid broker (meta-scheduler) is used to co-ordinate heteroge-neous and geographically distributed resources from multipleLRMs. Figure 2 shows an overview of our experimental test-bed. The test-bed comprises three physical servers namely,vmhx1, vmhx2, and vmhx3, each equipped with four processorswith Intel(R) Xeon(R) 2.66 GHz or Intel(R) Xeon(R) CPU2.80 GHz processors, RAM from 2 GB to 30 GB, and runningOpenSUSE 11.1 with Xen hypervisor.

Figure 2. Experimental Testbed

The first experiment is to measure (1) the amount of timeit takes for a job to be served and (2) the amount of time ittakes for a node to be allocated with a rented node as soon asthe rental request is initiated. Timings are measured from twopoints: the time it takes for the LRM Queue Monitor to identifya queued job and to initiate a resource provisioning decisionas soon as the job is queued due to insufficient resources,and the time it takes for the Resource Identifier to discoverphysical servers for VM deployment and the VM Creator toconfigure a new worker node so that it is ready for allocation.We timed certain major IntelligentGrid operations, measuringthe cost of individual operations, as well as the total overheadtime required to make provisioning requests. In particular, wemeasure the invocation costs of the LRM Queue Monitor, the

Resource Identifier, and the VM Creator, as well as the VMRemoval invocation operations.

Table I presents the total overhead time taken by theIntelligentGrid to deploy worker nodes with increasing jobsize, i.e., number of requested nodes for each job. For eachexperiment, we submit a dummy parallel job that runs forexactly 5 minutes. Each experiment is repeated with the samejob but for different job sizes. We then measure the totaltime it takes for the worker nodes to be deployed for eachjob. As indicated in Table I, it would take an average of 7.4minutes to deploy a Grid worker node, and the time it takesto provision new worker nodes increases linearly as the jobsize increases. For instance, it would take approximately 16.4minutes to deploy a parallel job with a job size of 4.

Table II further shows the breakdown of the total overheadtime for a job with size of 4. We can observe that thetime overheads for queuing time and allocation time by theLRM Queue Monitor are negligible, but the VM Creatortakes a considerable time to deploy virtual machines onphysical servers. However, we would argue that long-runningjobs without IntelligentGrid may be queued for several hours(which is currently quite common for batch jobs). Hence, anadditional delay of 7~20 minutes for scheduling is insignif-icant. Nonetheless, we feel that the deployment time can beimproved by further optimizing the VM deployment policy,which is necessary for supporting execution of interactive jobs.We conclude that even though the VM deployment process stillneeds to be refined further, IntelligentGrid is fairly scalable interms of provisioning more than one VM simultaneously whencompared to a conventional Grid setting with no deploymentmechanism.

Parallel Job Size Total Overhead (mins)1 7.4 (0.9)2 11.3 (0.8)3 14.5 (0.4)4 16.4 (1.6)8 21.8 (1.8)

Table ITOTAL OVERHEAD FOR INTELLIGENTGRID WITH INCREASING JOB SIZE.

SMALL FIGURES IN PARENTHESES ARE STANDARD DEVIATIONS.

IntelligentGrid Components Time (mins)LRM Queue Monitor 2.4

VM Creator 12.2Resource Identifier 1.8

Total time 16.4 (1.6)

Table IIBREAK DOWN OF THE TOTAL OVERHEAD TIME FOR INTELLIGENTGRIDWITH PARALLEL JOB SIZE OF 4. SMALL FIGURES IN PARENTHESES ARE

STANDARD DEVIATIONS.

Next, we carried out a set of experiments to compare theperformance of the IntelligentGrid in comparison to a standardGrid system. Again, the same test-bed configurations are used,which comprise three physical servers. Figure 3 presents ourresults in terms of throughput. Throughput is computed asthe number of completed job per minute over time, as more

2011 IEEE Conference on Open Systems (ICOS2011), September 25 - 28, 2011, Langkawi, Malaysia

182

4

jobs get completed. For each experiment, the bench-markingmeasurements consist of the execution of 30 jobs wherebyeach job runs a computation for 10 minutes on average on aworker node. The results show that the IntelligentGrid systemoutperforms the standard Grid system. As can be seen, theIntelligentGrid achieves significantly higher throughput, anincrease of up to 50% in comparison to the standard Gridsystem. This high throughput is due to the efficiency of theIntelligentGrid in reducing the number of pending jobs inthe queue by deploying additional worker nodes. As morejobs are being served at any one time, the system is able tocomplete more jobs. It is interesting to note that 17 additionalworker nodes are being deployed on the three physical nodesat the end of our experiments for all 30 jobs. This has asignificant impact on performance because up to 8 jobs canbe executed simultaneously. Additionally, these results showthat high throughput can be achieved without the need to usean additional physical server. Nonetheless, it is envisaged thatwe can obtain a much higher improvement in throughput byadding an increasing number of physical nodes to the LRM.

Figure 3. Experimental performance of Intelligent vs. conventional gLiteGrid system.

VI. CONCLUSION

We have described the IntelligentGrid system that providesthe mechanisms whereby resource nodes can be automaticallyprovisioned in a dynamic and incremental fashion based onuser workloads. In particular, we have focused on the designand implementation of general, extensible abstractions ofIntelligentGrid components. IntelligentGrid uses virtualisationtechnology to resolve the problem of long waiting timesof high priority queued jobs due to insufficient resources.Virtualisation technology offers effective resource manage-ment mechanisms such as isolated, secure job scheduling, andutilisation of computing resources.

IntelligentGrid is the first system that presents a full fledgeddynamic provisioning system without the need for configura-tion changes to existing Grid and cluster environments. Thesystem can be deployed immediately at without the need to

reconfigure existing Grid and/or Cluster infrastructure. Theonly requirement is to have physical servers which are VM-enabled. The rest of the configuration happens dynamically,through the interactions of various IntelligentGrid components.By simplifying configuration, IntelligentGrid makes it possi-ble to automatically and dynamically deploy additional GridCompute Nodes based on workload demands without userintervention.

REFERENCES

[1] S. Adabala, V. Chadha, P. Chawla, R. Figueiredo,J. Fortes, I. Krsul, A. Matsunaga, M. Tsugawa, J. Zhang,M. Zhao, et al. From virtualized resources to virtual com-puting grids: the In-VIGO system. Future GenerationComputer Systems, 21(6):896–909, 2005.

[2] P. Andreetto, S. Andreozzi, G. Avellino, S. Beco, A. Cav-allini, M. Cecchi, V. Ciaschini, A. Dorise, F. Giacomini,A. Gianelle, et al. The gLite workload managementsystem. In Journal of Physics: Conference Series, volume119, page 062007. IOP Publishing, 2008.

[3] Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand,Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, andAndrew Warfield. Xen and the art of virtualization.In Proceedings of the nineteenth ACM symposium onOperating systems principles, SOSP ’03, pages 164–177,New York, NY, USA, 2003. ACM.

[4] G Codispoti, C Grandi, A Fanfani, D Spiga, M Cinquilli,F Farina, V Miccio, F Fanzago, A Sciaba, Caprara,S Belforte, D Bonacorsi, A Sartirana, D Dongiovanni,D Cesini, S Wakefield, J Hernandez, SLemaitre, M Lit-maath, Y Calas, and E Roche. Use of the glite-wmsin cms for production and analysis. Journal of Physics:Conference Series, 219(6):062007, 2010.

[5] Dietmar Erwin. Unicore - a grid computing environment.In Lecture Notes in Computer Science, pages 825–834.Springer-Verlag, 2001.

[6] I. Foster and C. Kesselman. Globus: A metacomput-ing infrastructure toolkit. The International Journalof Supercomputer Applications and High PerformanceComputing, 11(2):115–128, Summer 1997.

[7] Ian T. Foster, Timothy Freeman, Katarzyna Keahey,D. Scheftner, B. Sotomayor, and Xuehai Zhang. VirtualClusters for Grid Communities. In CCGRID, pages 513–520. IEEE Computer Society, 2006.

[8] James Frey, Todd Tannenbaum, Ian Foster, Miron Livny,and Steve Tuecke. Condor-G: A computation manage-ment agent for multi-institutional grids. Cluster Comput-ing, 5:237–246, 2002.

[9] W. Gentzsch. Sun grid engine: Towards creating acompute power grid. In CCGRID ’01: Proceedings ofthe 1st International Symposium on Cluster Computingand the Grid, page 35, Washington, DC, USA, 2001.IEEE Computer Society.

[10] Robert L. Henderson. Job scheduling under the portablebatch system. In IPPS ’95: Proceedings of the Workshopon Job Scheduling Strategies for Parallel Processing,pages 279–294, London, UK, 1995. Springer-Verlag.

2011 IEEE Conference on Open Systems (ICOS2011), September 25 - 28, 2011, Langkawi, Malaysia

183

5

[11] X. Jiang and D. Xu. Violin: Virtual internetworking onoverlay infrastructure. Parallel and Distributed Process-ing and Applications, pages 937–946, 2005.

[12] K. Keahey, I. Foster, T. Freeman, and X. Zhang. Virtualworkspaces: Achieving quality of service and quality oflife in the Grid. Sci. Program., 13(4):265–275, 2005.

[13] M.C. Montes, J.P.G. Callejón, M.R. de Solar, and R.R.Pollán. Management of a grid infrastructure in GLITEwith Virtualization. In IbergridŠ2007 First IberianGrid Infrastructure Conference, Santiago de Compostela,Spain, pages 313–321, 2007.

[14] J. Moore, D. Irwin, L. Grit, S. Sprenkle, and J. Chase.Managing mixed-use clusters with Cluster-on-Demand.Duke University Department of Computer Science Tech-nical Report, 2002.

[15] AJ Rubio-Montero, E. Huedo, RS Montero, andIM Llorente. Management of virtual machines on globusGrids using GridWay. In Parallel and Distributed Pro-cessing Symposium, 2007. IPDPS 2007. IEEE Interna-tional, pages 1–7. IEEE, 2007.

[16] T. Selvi Somasundaram, B.R. Amarnath, R. Kumar,P. Balakrishnan, K. Rajendar, R. Rajiv, G. Kannan,G. Rajesh Britto, E. Mahendran, and B. Madusudhanan.CARE Resource Broker: A framework for schedulingand supporting virtual resource management. FutureGeneration Computer Systems, 26(3):337–347, 2010.

[17] Ming Q. Xu. Effective metacomputing using lsf multi-cluster. ccgrid, 00:100, 2001.

2011 IEEE Conference on Open Systems (ICOS2011), September 25 - 28, 2011, Langkawi, Malaysia

184