Looking for Commitment, Part 1

Published: March 11, 2009

by Paul Tuohy In this article I will discuss what commitment control is, why you may want to use it, and the basic requirements for commitment control. In subsequent articles we will look more closely at some of the considerations for using commitment control effectively: how commitment control works; different ways in which it can be implemented; and how it works in ILE. Throughout these articles I will be using the SQL terminology of schema, table, row, and index as opposed to library, physical file, record, and logical file. However, be aware that commitment control is applicable to both SQL-defined tables and DDS-defined physical files, regardless of which terminology is used. Background Commitment control is a powerful, but seldom used, feature that helps ensure the integrity of a database. Commitment control gives you the ability to ensure that a set of related changes across one or more tables in a database are not permanently applied until a program actually commits them. Alternatively, a program may roll back any of the changes instead of committing them. Many would consider commitment control a necessity in the maintenance of a database. Any programmer who has ever used embedded SQL knows that SQL (a database language) assumes you are making use of commitment control. How often have you changed the COMMIT parameter on the CRTSQLxxxI command? But if commitment control is such a necessity, why aren't more applications using this powerful option? There are three main reasons: 1. Commitment control requires the use of journals. In the early days of the system, the unreasonable fear that the use of journals might consume too much disk space or inadvertently affect system performance meant that many software vendors were reluctant to force their clients to use journals. Until the advent of RPG IV, it was cumbersome to make optional use of commitment control in programs. Commitment control used to work only at a job level. When a program issued a commit, it committed all open transactions for the job rather than just for the program. You had to be extremely careful to ensure that a program did not inadvertently commit a transaction that it wasn't meant to.

2. 3.

So what has changed to make the use of commitment control more acceptable? Systems are now bigger and faster, and there is much less resistance to the use of journals. RPG IV allows optional use of commitment control in programs. ILE provides a means of limiting the scope of commitment control within a job.

Why Use Commitment Control When a program is processing a transaction that consists of writing, updating, and deleting one or more rows across multiple tables in a database, commitment control offers a means of treating the transaction as a single unit. For example, commitment control provides a way for a program to identify and process an order as a single transaction-even though that order is spread over many rows on many tables, and adding the order involves updating columns in many rows on other tables (e.g., stock figures and customer balance information). Imagine what would happen if your system suddenly lost power while a program (or a number of programs) was processing a transaction. When the system performs an initial program load (IPL), the databases would now contain a number of incomplete transactions. However, if the programs were using commitment control, the system would automatically roll back any uncommitted rows from the databases. Now imagine what would happen if a program processing a transaction failed because of a bug. (I know, it's hard to imagine.) Again, there would be incomplete transactions on the database. Of course, if the program were using commitment control, you would have a means of rolling back the incomplete transactions. Therefore, commitment control should be a consideration for any transaction that consists of more than one row on one or more tables. Requirements--Journals and Journal Receivers Commitment control requires that the tables involved in the transaction being committed are all attached to the same journal. The basic concept behind journals is to offer a means of database recovery up to a specific point in time. When a table is attached to a journal, the database manager records a copy of every row in the table that is added updated or

deleted. This means that the system has a copy of every change that was made to an attached table so, in the case of data loss, the system has a means of recovering all changes that were made to that table. The journal process consists of two parts: a journal and a journal receiver. Journals are created using the Create Journal (CRTJRN) command and journal receivers are created using the Create Journal Receiver (CRTJRNRCV) command. When a journal is created, it is attached to a journal receiver. You can specify that changes made to a table be recorded in a journal by using the Start Journal Physical File (STRJRNPF) command. You may choose to record before images, after images or both before and after images.

Figure 1: The Journal Process The journal process is shown in Figure 1. The journal is depicted as a funnel and a bucket represents the journal receiver. As changes are made to tables, a copy of the changes is sent to the journal, which then drops the copies into the attached journal receiver. When a journal receiver is full, a new journal receiver can be created and attached to the journal in place of the original. This is a simple means of managing the amount of space occupied by the journal receivers. As you will see in a later article, the commitment control process uses special journal entries to enforce commitment of a transaction. If your tables are already attached to journals, there is nothing else you need to do to implement commitment control--apart from the applying the required code in your programs. If you are using journals for database recovery (or high availability), you more then likely have a strategy where all tables (regardless of application) are attached to a single journal. If you do not require journals for database recovery, you may want to consider having separate journals per application or per schema (library), as is the default when you create a schema with SQL.

Looking for Commitment, Part 2

Published: March 18, 2009

by Paul Tuohy In this article I will take a look at the basic rules and coding requirements for using commitment control within RPG programs. A Database

Note: The code show above is available for download here. Figure 1 shows the SQL used to create a schema named COMMIT, which contains two related tables, called HEADER and DETAILS. The main points to note are as follows (refer to the corresponding letters in Figure 1): A. Creating the schema COMMIT results in a library named COMMIT that contains a journal named QSQJRN and a corresponding journal receiver named QSQJRN0001. All tables created in the schema will be automatically attached to the journal QSQJRN. B. The table HEADER consists of a key, a text field, and a numeric field containing the number of corresponding DETAIL rows for the key. HEADER has a primary key constraint based on the key field. C. The table DETAILS consists of a key (which corresponds to the key on the header table), a sequence number, and a text field. DETAILS has a primary key constraint based on the key field and the sequence number. D. An index is created for each table (HEADER1 and DETAILS1). The index keys correspond to the definition of the primary key constraint for each table. RPG programs will use these indexes to access the rows in the tables. E. A foreign key constraint is defined between the DETAILS and HEADER tables. The constraint contains a cascade delete rule which means that when a row is deleted from the HEADER table all corresponding rows (based on the key fields) are automatically deleted from the DETAILS table. A cascade delete is only possible when the files are attached to the same journal. Since this is a schema the HEADER and DETAILS tables are automatically attached to the journal QSQJRN. If you intend to implement commitment control on an existing database, you must ensure that each table (or physical file) is attached to the same journal. Having the journal in the same schema as the tables is not a requirement for commitment control; the only requirement for commitment control is that the tables are attached to the same journal. A Program

Figure 2 - A test program to demonstrate commitment control. Note: The code show above is available for download here. Figure 2 shows the source of an RPG program used to populate the database and test the commitment control process. This program prompts for a key, writes one record to the HEADER table and three records to the DETAILS table for the key value entered. Every time a row is added to the DETAILS table, the Number of Rows on the corresponding HEADER row is incremented. The program then prompts to commit, roll back, or ignore the transaction. The main points to note are as follows (refer to the corresponding numbers in Figure 2): A. The program uses the indexes HEADER1 and DETAILS1 to access the database. Both tables specify the COMMIT keyword to indicate that commitment control may be used on the tables. The SomeTimes indicator controls whether (*ON) or not (*OFF) commitment control is active when the files are opened.

B. The indicator SomeTimes is passed as a parameter when the program is called. C. The program requests the entry of a key value. D. If a key value is entered, the program adds a record to the header table and three corresponding records to the details table. E. The Number of Rows on the HEADER rows is incremented for each row added to DETAILS. F. The program prompts for an entry to determine what should be done with the rows just added to the tables. G. If the entry was "c" for commit, then the rows are committed. Note that the SomeTimes indicator also conditions the commit operation. Issuing a commit or roll back operation when commitment control is not active results in runtime error RNQ0802:
COMMIT or ROLBK operation attempted when commitment control was not active (C G D F).

H. If the entry was "r" for roll back, then the rows just written are removed from the tables. As with the commit operation, the roll back operation is also conditioned by the SomeTimes indicator. I. No action takes place if the entry was "i" for "ignore." Remember that tables opened while commitment control is enabled must be attached to the same journal. If either of the tables is not attached to a journal, or if the tables are not attached to the same journal, the system will issue the following message:
Member *N not journaled to journal *N.

A Normal Call Let's start by calling the program without commitment control (the indicator parameter must be in single quotes):
CALL PGM(COMMITRPG1) PARM('0')

When prompted, enter a value of 'aa' for the key. It is irrelevant what value you enter for the commitment option because the program is not performing commitment control. By using the run query command to look at the contents of the two tables, you will see that they have the values shown in Figure 3. A single row was added to the HEADER table and three corresponding rows were added to the DETAILS table.

Figure 3 - Contents of tables after call with no commitment control. Note: The code show above is available for download here. A Committed Call Now let's try to call the program with commitment control enabled by entering the command:
CALL PGM(COMMITRPG1) PARM('1')

Disaster! The program fails with the message:

Commitment definition *N not valid for open of DETAILS1

The second level message text for the error provides more information, but the issue is that you forgot to say you were using commitment control in the job. It is not enough to simply have the COMMIT keyword on the file specifications in the RPG program--you must also specify that you are using commitment control in the job. Enter the Start Commitment Control (STRCMTCTL) command as follows:
STRCMTCTL LCKLVL(*CHG) CMTSCOPE(*JOB) TEXT('Test Commitment Control')

The command specifies a lock level of *CHG and a commit scope of *JOB. Note that *JOB is not the default value for the commit scope parameter; the default value is *ACTGRP for activation group. We will look more closely at the lock level and commit scope parameters (along with the rest of the STRCMTCTL parameters) in my next article. Putting Commitment Control to Work Now let's look at your three main options when using commitment control (specify a parameter value of '1' on all calls to the program). Call the program and enter a value of "bb" for the key and a value of "c" for the commitment option. The tables should now contain the values shown in Figure 4. An additional row has been added to the HEADER table and three corresponding rows were added to the DETAILS table.

Figure 4 - Contents of tables after call with commitment control. Note: The code show above is available for download here. Call the program and enter a value of "cc" for the key and a value of "r" (roll back) for the commitment option. When you look at the contents of the two tables you will see that they remain unchanged, the new rows have not been added and the contents are as they were in Figure 4. The ROLBK operation in the program removed the four rows that had been added to the two tables. Call the program and enter a value of "dd" for the key and a value of "i" (ignore) for the commitment option. What do you expect to find on the tables? Will the new rows appear in the table or won't they? At first glance, it appears that the new rows are on the tables, as shown in Figure 5. Even if you signed into another job and viewed the contents of the tables, the new rows would appear to be there.

Figure 5 - Contents of tables after call with commitment control; no commit or rollback issued. Note: The code show above is available for download here. But this is not the full story. Although the new rows appear in the tables, they are only available to other programs if the tables are open for input only. Use the Display Record Locks (DSPRCDLCK) command to see that the three newly inserted rows are all locked, as shown in Figure 6. These rows are not available for update to any other programs. Although the rows have been physically added to the tables, they have not yet had a commit or a roll back instruction issued to them.

Figure 6 - Rows locked while awaiting a commit or rollback. Note: The code show above is available for download here. Enter the command ROLLBACK. This means that the pending changes have been removed and will not appear if you view the contents of the tables. Alternatively, you could have entered the command COMMIT to have the pending changes applied. Isn't it nice to know that there are commands for COMMIT and ROLLBACK? Ending Commitment Control At any stage in a job you can end commitment control by issuing the End Commitment Control (ENDCMTCTL) command. There are no parameters for this command. If there are pending changes when you end commitment control you will receive the message:
ENDCMTCTL requested with changes pending. (RB C CM)

Entering a value of RB (the default reply) indicates that a roll back should be performed. Entering a value of CM indicates that a commit should be performed. Entering a value of C indicates that the End Commitment Control command should be cancelled. If there are pending changes when you end a job, the implicit roll back is performed before the job ends. All For Now. . .

Looking for Commitment, Part 3

Published: March 25, 2009

by Paul Tuohy In this article I will take a closer look at how commitment control works by looking at the journal entries for commitment control. I will also discuss the LCKLVL and CMTSCOPE parameters on the STRCMTCTL command. Commitment Control and Journals Commitment control is dependant upon the use of a journal. A journal is used in conjunction with commitment control as follows: When you start commitment control (using the STRCMTCTL command), a commitment boundary entry is placed in the journal. As a program inserts, updates, and deletes rows in a table, the rows are actually inserted, updated, and deleted, and the corresponding entries are made in the journal. If the program issues a commit, then another commitment boundary entry is placed in the journal. If the program issues a roll back, then all entries in the journal revert back to the previous commitment boundary or are removed from the tables.

Journal Entries Each journal entry contains a sequence number, a journal code, and a journal entry type. The journal entry type is dependent on the journal code. There are many journal codes, but for the purpose of learning about commitment control, we are only interested in the following two: 1. 2. C--Commitment control operation R--Record level operation

The journal entry types for a journal code of C are:

Type BA BC CM CN

Description
Commit block in use at abnormal end Commitment control environment begun Set of record changes committed End rollback

DB EC LW PC RB R1 SB SC SQ SU

Internal entry Commitment control environment ended Logical unit of work ended Prepare commit block Set of record changes rolled back Rollback started Start save point Commit cycle started Release save point

Rollback save point

The journal entry types for a journal code of R are:

Type BR DL DR IL PT PX UB UP UR

Description
Before image of record updated for rollback Record deleted from physical file member Record deleted for rollback Increment record limit Record added to physical file member Record added directly to physical file member Before image of record updated in physical file member After image of record updated in physical file member After image of record updated for rollback

Journal Entries for a Committed Transaction Let's look at what happens behind the scenes when a transaction is committed. To make it easier to decipher the journal entries generated, enter the following command to attach a new receiver to the QSQJRN journal:
CHGJRN JRN(COMMIT/QSQJRN) JRNRCV(*GEN) SEQOPT(*RESET)

The value *RESET for the SEQOPT parameter indicates that sequencing restarts at 1, as opposed to continuing from the last entry in the previous journal receiver. When you attach a new journal receiver to a journal, the last journal entry in the original journal receiver identifies the new journal receiver. The first journal entry in the new journal receiver identifies the original journal receiver. Enter the Start Commitment Control (STRCMTCTL) command as follows:
STRCMTCTL LCKLVL(*CHG) CMTSCOPE(*JOB) TEXT('Test Commitment Control')

Remember that *JOB is not the default value for the Commit Scope (CMTSCOPE) parameter.*ACTGRP is the default (more about this later in the article). Call the program we used in the previous article using the command:
CALL PGM(COMMITRPG1) PARM('1')

When prompted, enter a value of "h1" for the key and a value of "c" (commit) for the commitment option. Enter the following command to view a list of journal entries, as shown below:
DSPJRN JRN(COMMIT/QSQJRN) Sequence Code Type Object 1 J PR 2 C BC 3 C SC 5 R PX HEADER 7 R PX DETAILS Library Job COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA Time 11:12:39 11:13:52 11:14:05 11:14:05 11:14:05

COMMIT COMMIT

8 9 10 11 12 13 14 15 16

R R R R R R R R C

UB UP PX UB UP PX UB UP CM

HEADER HEADER DETAILS HEADER HEADER DETAILS HEADER HEADER

COMMIT COMMIT COMMIT COMMIT COMMIT COMMIT COMMIT COMMIT

COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA

11:14:05 11:14:05 11:14:05 11:14:05 11:14:05 11:14:05 11:14:05 11:14:05 11:14:07

Note: When referring to the QSQJRN journal in any command (e.g., CHGJRN or DSPJRN) it is recommended that you qualify the object name with the library name. QSQJRN is a commonly used name for journals and, if you depend on the library list as opposed to a qualified name, you may find that you are inadvertently changing or displaying the wrong journal. The main points to note about the journal entries above are as follows (refer to the corresponding numbers in the Sequence column and check the Journal Code and Journal Type in the previous tables): 2--Commitment control is started. This is not as a result of running the STRCMTCTL command. Rather, the journal entry is created when the first file is opened with commitment control enabled 3--The start of a commit cycle. This is the starting point for inserting, updating, or deleting related rows under commitment control. In other words, this entry identifies the start of a transaction 5--A row is inserted in the HEADER table 7--A row is inserted in the DETAILS table 8--The before image for the HEADER row update 9--The after image for the HEADER row update 10 to 15--The second and third DETAILS rows are added and the HEADER row is updated for each 16--A commit is issued

Here you clearly see how commitment control is implemented. A journal entry (Sequence 5) marks the beginning of a transaction, and a corresponding journal entry (Sequence 16) marks the end of a transaction. The interceding journal entries identify the transaction. Of course, the format is easy to see here because there is only one program currently accessing the database. If there were two or more programs accessing the database at the same time, there would be journal entries for each program interspersed throughout the list. How can you identify which record level journal entry relates to which commitment control entry? It is identified in the details of the record level entries. Use option 5 to view an entry, and press F10 to display the entry details. The entry details for the before image of the HEADER row prior to the first update (Sequence 8) is shown below in Figure 1. The Commit cycle ID (Sequence 3) is the sequence number of the corresponding Commit Cycle Started journal entry. All of the record level journal entries for a transaction will have the same Commit cycle ID.

Figure 1 - Entry details for the HEADER row before image.

Since the DSPJRN command arranges the list by sequence number, it can be difficult to discern which journal entries belong to which job or program--especially when multiple programs and/or jobs are accessing the tables at the same time. In this case, you can either identify the required job in the Job parameter on the DSPJRN command or direct the output from the DSPJRN command to a database and access the output with QUERY or SQL, and sequence the journal entries by sequence number within Commit cycle ID. Journal Entries for a Pending Transaction Let's see what happens with the journal entries when you have a pending transaction. Call the program again and enter a value of "h2" for the key and a value of "i" (ignore) for the commitment option. As you will remember from the previous article, the new rows are physically placed in the HEADER and DETAIL tables, but are not available for update by any other job (they are available for input, however). Issuing the DSPJRN command again will display the journal entries shown in the code below. A Commit Cycle Started entry (Sequence 17) marks the beginning of the transaction, and is followed by the record level entries for the transaction. There is no corresponding commit entry (Code C, Type CM) since the program did not issue a commit operation.
Sequence 17 19 21 22 23 24 25 26 27 28 29 Code C R R R R R R R R R R Type SC PX PX UB UP PX UB UP PX UB UP Object HEADER DETAILS HEADER HEADER DETAILS HEADER HEADER DETAILS HEADER HEADER Library COMMIT COMMIT COMMIT COMMIT COMMIT COMMIT COMMIT COMMIT COMMIT COMMIT Job COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA Time 12:51:03 12:51:03 12:51:03 12:51:03 12:51:03 12:51:03 12:51:03 12:51:03 12:51:03 12:51:03 12:51:03

At this point, if the job ended abnormally for any reason, an automatic rollback would be performed to remove the pending entries from the tables. Issuing a COMMIT command at the command line places a commit journal entry in the journal (at Sequence 30). Journal Entries for Rolling Back a Transaction Finally, let's see what happens when you issue a rollback. Call the program again and enter a value of "h3" for the key and a value of "r" (rollback) for the commitment option. Issue the DSPJRN command to view the journal entries shown in the code below. A Commit Cycle Started entry (Sequence 31) marks the beginning of the transaction and is followed by the record level entries for the transaction (Sequence 33 to 43). Although they have not been committed, rows on the HEADER and DETAILS tables have been physically changed. When you issue a rollback operation, the row is changed back to its original values. This is reflected in the corresponding journal entries (Sequence 44 to 53). These entries are basically the reverse of the entries in Sequence 33 to 43. For example, the entry in Sequence 44 (the before image) corresponds to the entry in Sequence 43 (after image). The entry in Sequence 45 (after image) corresponds to entry in Sequence 42 (before image). An original Insert operation will have a corresponding Delete operation, and an original Delete operation will have a corresponding Insert operation.
Sequence 31 33 35 36 37 38 39 40 41 42 43 44 45 46 47 48 Code C R R R R R R R R R R R R R R R Type SC PX PX UB UP PX UB UP PX UB UP BR UR DR BR UR Object HEADER DETAILS HEADER HEADER DETAILS HEADER HEADER DETAILS HEADER HEADER HEADER HEADER DETAILS HEADER HEADER Library COMMIT COMMIT COMMIT COMMIT COMMIT COMMIT COMMIT COMMIT COMMIT COMMIT COMMIT COMMIT COMMIT COMMIT COMMIT Job COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA Time 13:09:01 13:09:01 13:09:01 13:09:01 13:09:01 13:09:01 13:09:01 13:09:01 13:09:01 13:09:01 13:09:01 13:09:04 13:09:04 13:09:04 13:09:04 13:09:04

49 50 51 52 53 54

R R R R R C

DR BR UR DR DR RB

DETAILS HEADER HEADER DETAILS HEADER

COMMIT COMMIT COMMIT COMMIT COMMIT

COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA COMCONPTA

13:09:04 13:09:04 13:09:04 13:09:04 13:09:04 13:09:04

A rollback journal entry (Sequence 54) marks the end of the transaction. All of the record level entries (for the original transaction and the rollback entries) have the same Commit cycle ID of 31. Ending Commitment Control Ending commitment control--by using the ENDCMTCTL command or ending a job normally--results in a "Commitment control environment ended" journal entry, as shown in the following code. This is the corresponding end entry for the original start entry (Sequence 2 in the first piece journal list) when commitment control was started.
Sequence 55 Code C Type EC Object Library Job COMCONPTA Time 13:29:50

Locking Level and Scoping When you start commitment control, using the STRCMTCTL command, you must specify the locking level for rows in tables opened under commitment control. Your options are: *CHG--Every row read for update is locked. Rows that are changed, added, or deleted, remain locked until the transaction is committed or rolled back. Unchanged rows are unlocked. *CS--Every row accessed is locked. Rows that are changed, added, or deleted, remain locked until the transaction is committed or rolled back. Unchanged rows are unlocked. *ALL--Every row accessed is locked until the transaction is committed or rolled back.

As mentioned previously, the default value for the CMTSCOPE parameter on the STRCMTCTL command is *ACTGRP. This means that commitment control only applies to tables that are open within the activation group in which the COMMIT or ROLLBACK is issued. Using a value of *ACTGRP means that the STRCMTCTL and ENDCMTCTL commands must be issued from within the required activation group. The value of *JOB (as I have been using in the examples) means that commitment control applies to tables that are open within the job, regardless of activation group, in which the COMMIT or ROLLBACK is issued. The default use of *ACTGRP provides a means of minimizing the chance of unintentionally committing or rolling back a pending transaction within a job. All For Now. . . There you have it. You should now have enough information to start playing with commitment control.