Job Scheduling: History and Evolution

November 17, 2010

Welkin Associates

Agenda

Job Scheduler Terminology and Components

The Scheduler Study Overview
Brief History of Job Scheduling
Job Scheduler Functions
Job Scheduler Architectures
Evaluation of Job Schedulers and Vendors
Q/A

Job Scheduler Terminology and Components

Terminology
Job: An executable unit. Can be a systems command, a script, an
application program, or a collection of executable units. Jobs can have one
or more steps, each containing an executable unit.
Job-Dependencies: Logical events or conditions that must be satisfied in
order for jobs to be scheduled for execution. Predecessor and Successor.
Job-Flow: A collection of two or more jobs related by job-dependencies.
Schedule: The current active set of scheduled jobs and job-flows ready to
be released for execution and monitored by the Master Scheduler.

Components
Master Scheduler: Central software component that defines, modifies,
stores job and job-flow definitions, dependencies, security information,
events, and schedules. Provides automatic submission of executables,
monitors status of all jobs and job-flows, provides alerts, controls starting,
stopping, restarting of jobs and job-flows. Creates Audit trail of all events.
Agent: A program that resides on each server that is controlled by the
Master. The agent monitors jobs on its server, communicates with the
Master about the status of jobs, and executes commands relayed by the
Master.
GUI Console: Console display for Administrators and Operators.
Relational Database: The central repository used by the Master to store all
job, job-flow, dependencies, security information, schedules and audit trails.

Job Scheduler Terminology and Components (Cont.)

Administrator

Operator
Agent
Job-Flows

Administrator
GUI console

Operator GUI
console
Agent

Job-Flows

Master Scheduler

Agent

Logs

RDMS

Job-Flows

Scheduler Study Overview

OPUS: Legacy Job Scheduler used for Satellite data

processing
Developed by Space Telescope Science Institute (STScI ) in 1994 for HST
Very fast and efficient pipeline processor. Powerful job-flow features and
GUI console
Adopted and customized for NOAA as a Job Scheduler on AIX p-series for
migration from z/OS Mainframe (2005)
OPUS requires replacement by a COTS Scheduler:
STScI no longer supports OPUS
OPUS GUI written in Java has compatibility issues with latest version of
Java

The Scheduler Study

Evaluated using two in-house Schedulers (NDE and STARS)
Evaluated COTS Schedulers and vendors.

Brief History of Job Scheduling

In the beginning there were 80 column Hollerith cards

The term batch was derived from stacks of punched cards
The first true Job Schedulers were people and card readers
Therefore, originally Job Scheduling known as batch processing
1964 IBM OS/360 JCL presented first job dependency functions

Brief History of Job Scheduling (Cont.)

Early Mainframe era: Batch processing using card decks

Later Mainframe era: JCL on IBM Mainframes.

Jobs: Dependencies/Priorities/Queues/TOD-Calendar/Alerts/Reruns

Evolution of 3rd party vendor Mainframe Job Schedulers

IBM Tivoli Workload Scheduler (TWS)
CA 7 Mainframe Scheduler
BMC Control-M
ASG Zeke Scheduler

Unix systems: crons

Today: Job Scheduling has greatly matured, with a number of

software vendors that have developed and marketed their Job
Scheduler products for over 25 years

Brief History of Job Scheduling (Cont.)

Workload Automation Maturity Pyramid

Brief History of Job Scheduling (Cont.)

Gartner: The future automation model will be IT Workload Automation Broker (ITWAB)

Job Scheduler Functions

Scheduler Functions:
Create and edit jobs, job-flows, users, groups, schedules
Calendar scheduling
File-Watchers (Trigger file events)
Job parameters within job-flows available
Job Queues, Classes, Priority
Virtual resources to control jobs and files
High Availability / Failover capability
Reliable and fast Scheduler database with open schema
Scheduler commands to start, stop, suspend, resume, and restart jobs
Alerts generated for job errors and sent to Operators (GUI console, email, cell)
Reporting tools
GUI console:
Windows, Web-based, Unix/Linux
Flexible and configurable for Administrators and Operators, easy to use
Graphical display of jobs and job-flows. Click and drag highly preferable
Detailed display of job-flow and job status, dependencies, errors/alerts

Job Scheduler Architectures

Architecture: Master-Agent vs. Peer-to-Peer

Job Scheduler Architectures (Cont.)

Architecture: Master Agent

Research/Advisory Firms Job Scheduler Evaluations

EMA Criteria and Methodology for Evaluating Job Schedulers

Research/Advisory Firms Job Scheduler Evaluations (Cont.)

EMA Radar Report for Job Schedulers 01/2010

Research/Advisory Firms Job Scheduler Evaluations (Cont.)

November 2009 Market Overview: Workload Automation, Q3 2009
Workload Automation Vendors

Forrester List of Evaluated Job Schedulers

Research/Advisory Firms Job Scheduler Evaluations (Cont.)

The Gartner Magic Quadrant is copyrighted

(04/28/2009) by Gartner, Inc., and is reused with
permission. The Magic Quadrant is a graphical
representation of a marketplace at and for a specific
time period. It depicts Gartners analysis of how certain
vendors measure against criteria for that marketplace,
as defined by Gartner. Gartner does not endorse any
vendor, product or service depicted in the Magic
Quadrant, and does not advise technology users to
select only those vendors placed in the Leaders
quadrant. The Magic Quadrant is intended solely as a
research tool, and is not meant to be a specific guide
to action. Gartner disclaims all warranties, express or
implied, with respect to this research, including any
warranties of merchantability or fitness for a particular
purpose.
Source: Gartner, Inc.
Magic Quadrant for Job Scheduling

28 April 2009, Report Id Number: G00166953

Milind Govekar, Gartner Analyst

Gartner Job Scheduler Magic Quadrant Report 04/2009

Q/A

Additional Slides for reference

18

Workload Automation Example

19

Future Growth of WLA and Job Scheduling

Forrester: Forecast for Growth of WLA and Job Scheduling

Evaluation of Job Schedulers and Vendors

Vendor

Scheduler Product Name

ASCI
Absyss
ActivMindz
APM Software GmbH
Apple
Argent
ASG
ASG
Automation Anywhere
Ayehu
BMC
Camellia Software

ActiveBatch
Visual TOM
TaskForest

Job Scheduler
Zena for Distributed Systems
Zeke for z/OS Mainframes
Automation Anywhere
eyeShare
Control-M
Batch Job Server (BJS)

Open Source,
Freeware,
COTS
COTS
COTS
Open Source
COTS
COTS
COTS
COTS
COTS
COTS
COTS
COTS
COTS

CISCO

Tidal Enterprise Scheduler (TES)

COTS

Cluster Resources

Maui Cluster Scheduler

Open Source

Computer Associates

AutoSys

COTS

APX/PCC
XGRID

Evaluation of Job Schedulers and Vendors

Vendor

Scheduler Product Name

Open Source,
Freeware,
COTS

Corporate Practical Solutions

Bqplus

COTS

Cronless.com
David G. Walker
Exact
Exspans
Flux Corp.

Open Source
Free Ware
COTS
COTS
COTS

Grand Logic

cronless
php JobScheduler
JobBOSS
Automan
Flux
JobServer

Help/Systems
Honico
Hux Software

Robot/SCHEDULE Enterprise
Batchman
HQ Scheduler

COTS
COTS
COTS

IBM

COTS

COTS
Open Source

ISE

Tivoli Workload Automation Suite

BICsuite Scheduler
EnterpriseSCHEDULE

JOX IT
LXI

JOX
LXI Scheduler

COTS
COTS

IndependIT

COTS

Evaluation of Job Schedulers and Vendors

Vendor

Scheduler Product Name

MJTNet
MVP Systems
Neteject.com
Network Automation
Nguyen Anthuan
OAR Team
Open Systems Mgmt
OpsWise
ORSYP
OW2 Consortium
Platform Computing
PTC Software
Redwood Software
ROC Software
Runtime Design
Automation
SMA
SoftTree Tech

Macro Scheduler
JAMS
VisualCron
Automate7
SetCronJob
OAR
Cosbatch
Automation Center
Dollar Universe
ProActive Parallel Suite
Load Sharing Facility (LSF)
PTC Scheduler
Cronacle
Maestro Scheduler

Open Source,
Freeware,
COTS
COTS
COTS
COTS
COTS
Free Ware
Open Source
COTS
COTS
COTS
Open Source
COTS
COTS
COTS
COTS

Network Computer (NC)

COTS

OpCon/xps
24x7 Scheduler

COTS
COTS

Evaluation of Job Schedulers and Vendors

Open Source,
Freeware,
COTS
Open Source

Vendor

Scheduler Product Name

Sourceforge

Jcrontab

Stonebranch

Indesca
N1 Grid Engine

COTS
COTS

Tenth Generation

Job2Do
Job Scheduler for Windows

Terracotta

Quartz Enterprise Job Scheduler

Open Source

The Coens Group

Jobstream
Workload Automation Suite

Open Source
COTS

University of Warwick

Grid MP
TITAN Grid Scheduler

Open Source

University of Wisconsin

Condor

Open Source

Vinzant

Global ECS (GECS)

COTS

Xi Software

GNUbatch

Open Source

XuiS Software

EnterpriseSchedule

COTS

Sun Microsystems
Sypsoft

UC4
Univa UD

COTS
COTS

COTS