Managing Data Sources Within the

Blackboard GUI
Mark Spurlock, Carl Hardin, C. Lemarinier
Innovative Technology Center
University of Tennessee Knoxville
Intended Audience
Sysadmins at institutions using the snapshot and dsm
tools who would like to automate and expand the
functionality of each.
Would-be building block developers who would like to
see some working JSP code using the Bb API.
Helpful skills and knowledge: experience with Bb batch
data integration, a little understanding of Perl and
relational databases including Blackboards back end,
JSP.
Team background and contact info:
Mark Spurlock; Manager, Technical Systems
Development; 10 years experience with Bb;
spurlock@utk.edu
Carl Hardin; Web Application Developer; 2 years
experience with Bb and JSP; chardin@utk.edu
Celine Marinier; Web Application Developer;
experienced Java and JSP programmer;
clemarin@utk.edu
Web site: http://itc.utk.edu
University of Tennessee Profile
Bb/CourseInfo school for 10+ years.
ASP hosted since 2002.
Approximately 30,000 active users during academic
year.
Approximately 56,000 accumulated course sites. (We
retain course materials for two years after the end of
the course term.)
800 gigs of data in Blackboard.
90 percent of students enrolled in at least one Bb site.
Problem description: Command-line snapshot and dsm
Lots of switches and complex command line
Example:
./snapshot_override.sh -
Ddata.source.key=bb_group.dat f CRS_SNPSHT t
/usr/local/blackboard/apps/snapshot/data/bb_group.d
ata C
/usr/local/blackboard/apps/snapshot/data/snapshot.pr
operties V utk.blackboard.com
Isolates snapshot from GUI administration, requiring
additional skill set (Linux command line).
As is, requires lots of human intervention.
Typical means of control:
Hand-editing of snapshot.properties
External controller script, written in some language
like Perl
Custom-designed controller perhaps provided by
Blackboard Global Services
Typical functions and process of snapshot controller
Logging
Transfer feed files (ftp) from SIS.
Validate and prep files for the snapshot client.
Call snapshot client for each file received.
Cleanup of work files and archive.
Email notice of completion along with errors.
Other nice to have functions:
Quality-control checks (e.g. rollback, error tracking).
Calendar-aware automation.
System maintenanceimplementation of institutions
data retention standards.
Sysadmin control and correction through Bb APIs and
GUI, rather than disjunction of the two.
How weve done it in the past
Lots of HTML forms
Lots of Perl scripts
Lots of Cron jobs
Lots of human memory because lots of pieces were in
lots of places.
In some ways, this was necessary because we had to
build up our knowledge and experience of what we
needed the system to do for us. A better way had to
evolve from experience.
Example: Course deletion
Old policy: Keep everything forever or at least until
faculty wanted to delete it (absolutely, positively):
1) Instructor requests via online form
2) Request stored in database, email warning
sent to every instructor of course.
3) Cron job processes request after two-week
grace period.
4) Course reassigned to TO_DELETE data
source and disabled (hidden)
5) Six months latergot to be sure!course
actually deleted/purged.
Course deletion problems
Not automated.
Not integrated (with business processes).
Faculty didnt delete.
Customer service and policy frequently in conflict.
Process mysterious to everyone but me.
Worst feature: disjunction between customer face and
backend
Support (even GUI sysadmin) has little understanding
of how things work.
Limits customer service.
Process control not in the hands of those most
responsible for its accuracy, consequences.
Flip side, difficulty of ad hoc change not understood
by those requesting it.
Touching one piece can break something somewhere
else.
Little redundancy in knowledgeable personnel.
Problem Solution: Give more control to the GUI admin
Command line becomes point and click.
GUI admin knows instantly what the snapshot will do
and when.
Though requiring technical expertise to set up,
requires less technical expertise to maintain and
operate.
Existing data source building blocks
Smart DSM Limited to dsm.sh functionality.
Located here:
http://www.blackboard.com/extend/b2.aspx?Extensio
nID=10242
bb-datasource-manager Limited to dsm.sh
functionality. Could not get to run with Bb version 8.
Located here: http://code.google.com/p/bb-datasource-
manager/
Smart DSM interface
Smart DSM interface, part 2
Functional correspondence of dsm.sh Smart DSM
dsm.sh f LIST v host
dsm.sh f CREATE b datasource v host
dsm.sh f MODIFY b datasource v host
dsm.sh f COUNTS(...) b datasource v host
dsm.sh f DISABLE(...) b datasource v host
dsm.sh f PURGE(...) b datasource v host
dsm.sh f REMOVE b datasource v host
See /usr/local/blackboard/apps/snapshot/bin/readme.txt
Shortcomings of Smart DSM
dsm onlyno snapshot functionality
Still manual and immediate
Not customizable (relies entirely on built-in
Blackboard data)
To address these, we knew we needed a database.
Design decision: How to implement the database
Modify the existing Bb database or...
Build an extended data_source table in a separate
database.
Modifying the existing Bb database seemed bad
because it might get destroyed by upgrades, not sure
it was within our license, and could raise support
issues.
MySQL database
Building block requires extra fields.
Desire not to alter Bb database.
Already used MySQL database for other building
blocks (for example, student textbooks).
Means the snapshot controller and client can run from
any machine.
Con: Requires installation of MySQL driver on
application server. (Note: If you provide instructions to
ASP, you TSM should be willing to do this.)
Alternatively, you can embed MySQL with the building
block
Download the latest Connector/ODBC - MySQL
ODBC driver from http://www.mysql.com/downloads/.
Unzip the package.
Locate the driver/jar file (i.e. mysql-connector-java-
5.0.5-bin.jar).
Copy the jar file to the following folder in the building
block: WEB-INF/lib/.
Overview of utk-ds-tool
Building Block Sysadmin GUI
Bb Oracle
Database
Custom
MySQL DB
B
b

A
P
I
M
y
S
Q
L
J
a
v
a

D
r
i
v
e
r
P
e
r
l

D
B
I
Snapshot
Controller
dsm snapshot
utk-ds-tool interface part 1
utk-ds-tool interface, part 2
utk-ds-tool interface, part 3
Overview of snapshot controller integration
Controller selects active records for processing.
Checks todays date and sets values in feed files
appropriately (example: AVAILABLE_IND = Y or
AVAILABLE_IND=N).
Uses dsType field to know which snapshot function to
call for each data source. Example:
snapshot_override.sh "-Ddata.source.key=$batchUid" -
V $host -f ${dsType}__SNPSHT -t $dir/$batchUid -C
$dir/snapshot.properties;
Snapshot controller and data retention
GUI Sysadmin sets data retention and course
availability based on university policies and academic
calendar.
Snapshot automatically cycles courses based on
those calendar settings and values in the database.
No manual intervention and everyone knows when
things are going to happen.
Time for the nitty-gritty
Thats the 1,000 foot view.
Now for some coding details.
Overview of our development environment
Blackboard 8
MyEclipse 6.0.1
Reference API's
bb-cms-admin.jar
bb-platform.jar
bb-taglibs.jar
Tomcat 5.5
RHEL 3 (Red Hat Linux)
Oracle 10.4
MySQL 5
Perl 5+
Components of the UTK Data Source Tool (GUI)
Link available only on the "System Admin tab.
view.jsp (home)
ConfirmDisable.jsp
DisableDataSource.jsp
ConfirmPurge.jsp
PurgeDataSource.jsp
modDataSource.jsp (modify form)
processUpdate.jsp
addDataSource.jsp (insert form)
processInsert.jsp
Database code part 1 -- MySQL
Connecting MySQL within the JSP files
Since we knew wed be using the MySQL database in
several JSP files, we created an include file:
<%@ include file="../include/ConnMySQL.inc" %>
Contents of include file
<%
Connection Conn = null;
try {
Conn =
DriverManager.getConnection("jdbc:mysql://your.dbhost.ip:3306
/databasename?user=********&password=********");
}
catch (Exception E) {
out.println("Unable to load driver.");
E.printStackTrace(new PrintWriter(out));
}
Statement statement = null;
ResultSet rs = null;
%>
Database code, part 2 Oracle (virtual host)
<%
// Begin: Initialize the BbServiceManager and set the context
try {
blackboard.platform.BbServiceManager.init("/usr/local/blackboard/config/service-config-snapshot-
jdbc.properties");
// The virtual host is needed to establish the proper database context.
VirtualInstallationManager vm =
(VirtualInstallationManager)BbServiceManager.lookupService(VirtualInstallationManager.class);
String vhostUID = System.getProperty("dbhost","bb_bb60");
VirtualHost vhost = vm.getVirtualHost(vhostUID);
if(vhost == null) {
throw new Exception("Virtual Host '" + vhostUID + "' not found.");
}
// Now that the vhost is set we can set the context based on that vhost
ContextManager cm =
(ContextManager)BbServiceManager.lookupService(ContextManager.class);
Context context = cm.setContext(vhost);
} catch(Exception e) {
System.out.println("Exception trying to init the BbPersistenceManager\n " + e.toString() +
"..exiting.\n");
System.exit(0);
}
// End: Initialize the BbServiceManager and set the context
%>
Code to load data from each database (view.jsp)
<%@page import="blackboard.base.BbList"%>
<% try {
// Loading all data sources
BbList dsList = DataSourceLoader.Default.getInstance().loadAll();
GenericFieldComparator compBatchUid = new
GenericFieldComparator(BaseComparator.ASCENDING, "getBatchUid", DataSource.class);
GenericFieldComparator compDescription = new
GenericFieldComparator(BaseComparator.ASCENDING, "getDescription",
DataSource.class);
%>
<bbUI:list collection="<%=dsList %>" collectionLabel="Available Data Sources"
objectId="eachDataSource" className="blackboard.admin.data.datasource.DataSource"
resultsPerPage="-1">
<bbUI:listElement width = "10%" label= "Data Source (BatchUid)" name="BatchUid" comparator
= "<%=compBatchUid%>">
<%=eachDataSource.getBatchUid() %>
</bbUI:listElement>
<bbUI:listElement width = "40%" label= "Description" name="Description" comparator =
"<%=compDescription%>">
<%=eachDataSource.getDescription() %>
</bbUI:listElement>
...
Code, continued (view.jsp)
<%
String strType="";
try {
String s_batchuid = eachDataSource.getBatchUid();
// Run select statement for MySQL (snapshot/data_source)
// statement and rs are both declared in connMySQL.inc
statement = Conn.createStatement();
// Clear the MySQL recordset rs for each record passed
// rs = null;
// MySQL query to check custom data_source table
statement.executeQuery("SELECT batchUid, dsType FROM data_source WHERE batchUid =
'"+s_batchuid+"'");
rs = statement.getResultSet();
// Check for a matching record in data_source (MySQL)
if (rs.next()) {
strType = rs.getString("dsType");
// If a record is found in MySQL, verify that dsType is not null
if (strType != null) {
%>
Code, continued (view.jsp)
<bbUI:listElement width = "5%" label= "" name="Modify" align="center">
<A class="inlineAction"
HREF="modDataSource.jsp?BatchUid=<%=eachDataSource.getBatchUid()
%>">Modify </A>
</bbUI:listElement>
<bbUI:listElement width = "5%" label= "" name="Disable" align="center">
<A class="inlineAction"
HREF="confirmDisableRequest.jsp?BatchUid=<%=eachDataSource.getBatchUid()
%>">Disable </A>
</bbUI:listElement>
<bbUI:listElement width = "5%" label= "" name="Purge" align="center">
<A class="inlineAction"
HREF="confirmPurgeRequest.jsp?BatchUid=<%=eachDataSource.getBatchUid()
%>">Purge </A>
</bbUI:listElement>
<%
Code, continued (view.jsp)
} //Closing if (rs.dsType != null)
else {
// We have a match in both databases, but dsType is null
// Data sources with dsType equal to null (in MySQL) are not allowed to disable,
purge, or modify
// The list elements are still inserted, but no buttons are made available in the
GUI
%>
<bbUI:listElement width = "5%" label= "" name="Modify" align="center">
</bbUI:listElement>
<bbUI:listElement width = "5%" label= "" name="Disable" align="center">
</bbUI:listElement>
<bbUI:listElement width = "5%" label= "" name="Purge" align="center">
</bbUI:listElement>
<%
} // Closing else
} // Closing if (rs.next())
Code, continued (view.jsp)
else {
// There was no matching record in MySQL
// The list elements are still inserted, but no buttons are made available in the GUI
%>
<bbUI:listElement width = "5%" label= "" name="Modify" align="center">
</bbUI:listElement>
<bbUI:listElement width = "5%" label= "" name="Disable" align="center">
</bbUI:listElement>
<bbUI:listElement width = "5%" label= "" name="Purge" align="center">
</bbUI:listElement>
<%
} // Closing else (no matching record found in MySQL)
} catch (Exception exception) {
exception.printStackTrace();
throw new ServletException("Error : " + exception.getMessage());
}
// Close database connections
rs.close ();
statement.close ();
%>
</bbUI:list>
Sample code for disabling data source
(DisableDataSource.jsp)
<%
// Begin: Purging data source by BatchUid
try {
DataSourcePersister dsPersister =
blackboard.admin.persist.datasource.DataSourcePersister.Default.getInstance();
dsPersister.disableAllAdminObjects(strBatchUid, date); // Note: date has been set
to null for our purpose
} catch(Exception e) {
System.out.println("Exception trying to disable the data source by BatchUid\n " +
e.toString() + "..exiting.\n");
System.exit(0);
}
// End: Disabling data source by BatchUid
%>
Sample code for purging data source
(DisableDataSource.jsp)
<%
// Begin: Purging data source by BatchUid
try
{
DataSourcePersister dsPersister =
blackboard.admin.persist.datasource.DataSourcePersister.Default.getInstance();
dsPersister.purgeAllAdminObjects(strBatchUid, date); // Note: date has been set to
null for our purpose
}
catch(Exception e)
{
System.out.println("Exception trying to purge the data source by BatchUid\n " +
e.toString() + "..exiting.\n");
System.exit(0);
}
// End: Purging data source by BatchUid
%>
Sample code for updating a data source
(processUpdate.jsp)
This code shows how to update the MySQL database.
if(request.getParameter("ds_isActive")!=null){
isActive=request.getParameter("ds_isActive");
}
String s_query="UPDATE data_source SET
term='"+request.getParameter("ds_term")
+"', active='"+isActive
+"', dsType='"+request.getParameter("ds_type")
+"', makeAvailDate=DATE('"+request.getParameter("ds_avail")
+"'),
makeUnavailDate=DATE('"+request.getParameter("ds_unavail")
+"'), disableDate=DATE('"+request.getParameter("ds_disable")
+"'), purgeDate=DATE('"+request.getParameter("ds_purge")
+"') WHERE batchUid='"+s_batchuid+"'";
Code, continued (processUpdate.jsp)
<%@ include file="../include/ConnMySQL.inc" %>
<%
try {
statement = Conn.createStatement();
int res=statement.executeUpdate(s_query);
if (res==1) conf_message = "ok";
else conf_message = "ko";
statement.close ();
}

%>
Code, continued (processUpdate.jsp)
This code shows how to update the Oracle database.
<%
try {
DataSource ds_selected =
DataSourceLoader.Default.getInstance().loadByBatchUid(s_batchuid);
ds_selected.setDescription(s_description);
DataSourcePersister dsPersister =
blackboard.admin.persist.datasource.DataSourcePersister.Default.getI
nstance();
dsPersister.modify(ds_selected);
}
Sample code for inserting a new data source
(processInsert.jsp)
This code shows how to update the MySQL database.
String s_dtmodified=java.util.Calendar.getInstance().toString();
String s_description=request.getParameter("ds_description");
String mysql_query="INSERT into data_source (batchUid, term, active,
dsType, makeAvailDate, makeUnavailDate, disableDate, purgeDate)
values("
+ "'" + s_batchuid
+ "', '" + s_term
+ "', '" + s_isactive
+ "', '" + s_type
+ "', DATE('" + s_avail
+ "'), DATE('" + s_unavail
+ "'), DATE('" + s_disable
+ "'), DATE('" + s_purge
+ "'))";
Code, continued (processInsert.jsp)
<%@ include file="../include/ConnMySQL.inc" %>
<%
statement = Conn.createStatement();
boolean res=statement.execute(mysql_query);
if (res==true) conf_message = "ok";
else conf_message = "ko";
statement.close ();
%>
Code, continued (processInsert.jsp)
This code shows how to insert new data sources to the
Oracle database.
<%
try {
DataSource new_ds = new DataSource();
new_ds.setBatchUid(s_batchuid);
new_ds.setDescription(s_description);
new_ds.setDataSourceId(Id.newId(DataSource.DATA_TYPE));
DataSourcePersister dsPersister =
blackboard.admin.persist.datasource.DataSourcePersister.Default.getI
nstance();
dsPersister.create(new_ds);
}
Pitfalls encountered during development
Reference API's in war files conflict with installation.
Blackboard drops the MySQL database driver
connection when Bb services are restarted.
Javascript form validations need special attention.
Reference APIs in MyEclipse (solution)
The Blackboard APIs are extremely helpful while you
are developing JSP files within MyEclipse.
Exporting these APIs in your war file will cause your
building block to fail when it is installed in Blackboard.
Simply remove these reference APIs from your build
path before creating your war file with MyEclipse:
bb-cms-admin.jar
bb-platform.jar
bb-taglibs.jar
Blackboard drops MySQL connection (solution)
MySQL work around to get a new instance of the database driver
when Bb services are restarted.
<%
// registering the MySQL JDBC driver
try {
// The newInstance() call is a work around for some
// broken Java implementations
Class.forName("com.mysql.jdbc.Driver").newInstance();
}
catch (Exception E) {
out.println("Unable to load driver.");
E.printStackTrace(new PrintWriter(out));
}
%>
Javascript form validations (solution)
<%@page import="blackboard.platform.plugin.PlugInUtil"%>
<%PlugInUtil.authorizeForSystemAdmin(request, response);%>
<script language="Javascript">
<!--
function ValidateForm() {
if (document.update_form.ds_type.value == '' || document.update_form.ds_type.value==null) {
alert('Please enter a type.');
document.update_form.ds_type.focus();
return false;
}
if (document.update_form.ds_term.value == '' || document.update_form.ds_term.value==null) {
alert('Please enter a term.');
document.update_form.ds_term.focus();
return false;
}
.
.
.
}
-->
</script>
<form name="update_form" action="<%=PlugInUtil.getUri("utk", "utk_dsm","module/processUpdate.jsp")%>"
onSubmit="return ValidateForm();">
.
.
.
</form>
Future enhancements
Choose database location and driver type in building
block properties.
Add vhost field to database table: many installations
could run one snapshot client.
Building of snapshot.properties file through retrieval of
stored database values (for example, specification of
which fields are snapshot controlled).
Conclusion and questions?