RMVJRNCHG is a nice little line command. The first time IBM i administrators see it, they can’t help but think “Great! Down with backups!” …ok, ok, maybe they don’t – there’s more to backups than a simple ability to roll back database updates to a point in the past.

But let’s think about it. Imagine that you are an application or database administrator on IBM i and the system is in the middle of executing a long-running daily batch. Then, as sometimes happens, the batch crashes. You call in an expert, they find the cause of the problem and fix it. Now it’s time to restart the batch. But how do you do it?

Although some applications are intelligent enough to be restarted from a point of failure, most are not. More importantly, even the most intelligent of applications can hardly be expected to be able to restart after ANY error.  This is exactly where backups come in: you restore and rerun. The issue is, this may take quite a long time, presenting a problem for the business.

In an ideal world, RMVJRNCHG should help solve such problems. Naturally, you would need to make sure that your application objects are properly journaled, but that’s quite common these days. Then, you would have to find the target journal entry, i.e. the point in time that you would like to recover the system to. It could be either a moment immediately preceding the kick-off of the batch, or a checkpoint that the batch can restart from.

Finding the related journal entry may be a non-trivial task, given that there could be millions of them. But suppose you find it, select the objects for the rollback, type the RMVJRNCHG command and press Enter. Well, I can think of at least two reasons you may not want to do this. Firstly, there’s a chance that the RMVJRNCHG command would fail, leaving you with no option but to restore from backup. This is because your batch may include unrecoverable commands, such as CLRPFM, CPYF with MBROPT(*REPLACE) parameter, RGZPFM, or similar.  Each of those leaves only a simple audit entry in the journal, crippling the potential for a subsequent rollback. The second reason is performance-related. If your batch is well-designed and capable of using the multiprocessor architecture of IBM POWER servers, RMVJRNCHG, being a single-threaded process, could spend up to n times longer to roll back than it took the batch to reach the point of failure. The n here is the number of cores in the configuration. In the end, you may still be better off restoring from backups.

So, the “to use or not to use” dilemma in relation to the RMVJRNCHG command is very real.

However, there is a way to avoid this dilemma altogether by using a piece of software called iSTREAM. It is a multifunctional tool, and the full range of its functionality will be explored in other articles, but for now, let’s focus on how it can help make the RMVJRNCHG command more usable.

iSTREAM can define a special job execution mode, which gives system administrators more control over the way CL commands are executed in any selected group of jobs (a good technical description of the iSTREAM mode can be found in the iSTREAM CL Command Transformer (CCT) Guide). This mode can, for example, be defined as follows:

/* Configure and start iSTREAM mode */

STRISTMOD UNIT(TST) CTLLIB(LIB1) JRN(LIB1/JRN) LIBLIST(LIB1 LIB2 LIB3)

TST here is the name of the configuration, LIB1, LIB2 and LIB3 are application libraries, and JRN is the name of the journal used by the application to record both before and after images of files and data areas used by the application.

For a job executing in the iSTREAM mode, commands of the above “compromising” type, e.g. CLRPFM, CPYF with MBROPT(*REPLACE) parameter, or RGZPFM would be intercepted and replaced with their recoverable equivalents. For CLRPFM, for example, this would be a function explicitly deleting all the file records. There is much more to this command transformation than could be addressed in a short article, but, hopefully, the concept is clear: iSTREAM helps prevent commands such as those above to disable the RMVJRNCHG functionality.

Symbolic checkpoints could be defined to make the definition of targets for the rollback easier:

/* Create checkpoint for rollback */

CRTCKPRLB UNIT(TST) CKPT(BATCH)

The performance problem related to the batch multiprocessing on IBM i could also be resolved, if for the rollback the administrator made use of the related iSTREAM command as follows:

/* Rollback to checkpoint */

ISTSSYS/RLBTOCKP UNIT(TST) CKPT(BATCH) STREAMS(10)

The above command quotes the name of the earlier created configuration, defines the target checkpoint for the rollback, and breaks the rollback into 10 parallel streams (multistreaming of the rollback process for a single file/table is also supported).

The bottom line is, the long-standing RMVJRNCHG dilemma, which is often resolved negatively by  IBM i application administrators, may well be overturned with the help of additional instrumental software.

The BigData hype has largely passed the IBM i by. There are several obvious reasons for this. Firstly, IBM i is not the natural platform of choice for connecting multiple data-generating IoT devices. Secondly, the term BigData is traditionally applied to the processing of unstructured data, whereas IBM i is more of a structured data processing engine.

Strictly speaking, however, the term BigData simply refers to data sets, structured or unstructured, that are too large or complex to be dealt with by traditional data-processing application software. Purists may debate this, of course, but it is clear that large, structured data sets on the IBM i platform present a processing challenge similar in many ways to those addressed by high-throughput computing or MapReduce architectures.

Let’s consider an example of a simplified daily commercial transaction processing batch. It involves two tables, TRANSACT and ACCOUNTS, the former potentially including millions of rows of data accumulated during the current business day. Each transaction must be processed, with the value of the transaction used to update the related account balance. The algorithm for processing a single transaction can be represented by the following sequence of steps:

01 READ TRANSACTION RECORD
02 LOCATE RELATED ACCOUNT RECORD
03 RETRIEVE LAST KNOWN BALANCE
04 CALCULATE UPDATED BALANCE
05 UPDATE ACCOUNT RECORD
06 UPDATE TRANSACTION RECORD MARKING IT “PROCESSED”

The program that is usually written to perform the daily batch processing includes a loop constructed over the above steps:

01 LOOP UNTIL EOF
02 READ TRANSACTION RECORD
03 LOCATE RELATED ACCOUNT RECORD
04 RETRIEVE LAST KNOWN BALANCE
05 CALCULATE UPDATED BALANCE
06 UPDATE ACCOUNT RECORD
07 UPDATE TRANSACTION RECORD MARKING IT “PROCESSED”
08 END-OF-LOOP

The resulting code is obviously extremely inefficient, for it makes use of a single-streamed process to update millions of records, and is thus unable to leverage the multi-processor architecture of the server. Although implementing the above algorithm in SQL may help with performance due to the built-in intelligence of the optimizer, the latter has its limits, and as a result, most financial platforms (on IBM i or servers of other architectures) suffer from long batch runtimes.

Attempts at optimization, e.g. by breaking the processing into multiple parallel streams, yield mixed results. This is due to the lack of consistency, hard-coding of stream controls, and problems with complex code maintenance.

This challenge is not specific to IBM i-based systems, but some advanced features of IBM i can help greatly with its solution. These advanced and in some ways unique features, e.g. ADDLIBE, OVRDBF, OPNQRYF, are often taken for granted by weathered IBM i gurus, but they can form the foundation of an elegant streaming solution for the platform, helping address the long batch runtime problems for practically any existing IBM i ILE/OPM application.

In a nutshell, the idea is simple: a long-running batch processing program is split into several streams, with each stream able to view and, consequently, process only a subset of the TRSANSACTION table rows. Schematically, it can be represented as follows:

In order to multistream a program with the name DAILYBATCH stored in OLD library, a CL script (program) with the same name is created and added to the library NEW. The script submits multiple stream jobs invoking STREAMCTL program each. The latter issues an OPNQRYF command for the TRANSACT file (or builds a temporary SQL view), selecting a subset of the TRANSACT file records depending on the number of the stream, overrides (OVRDBF) the file in the original DAILYBATCH program to the query file (or the temporary view), and invokes the original DAILYBATCH.

Conceptually, this approach can be very efficient and seems straightforward, but in practice, it’s not so easy to implement, as writing the above framework from scratch would take time and effort, and different batch processes may require different split methods. For example, whilst splitting the TRANSACT file into multiple RRN ranges may be sufficient in some cases, if the DAILYBATCH program were slightly more sophisticated than the example above, say, calculating branch or location subtotals for reporting, this could require a split by branch or location range, rather than by relative record numbers.

However, there is a tool, that can be used to significantly simplify this problematic task.

The tool is called iSTREAM, and as suggested by the name, it helps creating programs based on the pattern of streaming. iSTREAM differs significantly from tools like Hadoop or CouchDB: whilst the latter simply provide frameworks to develop products using MapReduce or Stream Processing, iSTREAM can help configure the existing legacy batch processes to use the multistreaming paradigm – with no additional programming whatsoever.

In order to turn the sample program DAYLYBATCH into a multistreamed batch the following framework commands would have to be executed:

01 DFNSPTPRM PROGRAM(DAILYBATCH)
MASTERFILE((OLD/TRANSACT *FILE *FIRST *N *ALL 10 *ALLIO *CALC (*RRN)))
/* multistreaming definition with stream breakdown by relative record numbers of TRANSACT */

02 INZSPTRNG PROGRAM(DAILYBATCH) STREAMS(10)
/* stream definition for 10 streams */

03 ISTSSYS/CMPSPTDFN PROGRAM(DAILYBATCH)
/* compilation of the customized programs of the framework */

04 STRISTMOD HOTLIB(NEW)
/* entering “iSTREAM” mode of operation in a job */

05 CALL DAILYBATCH
/* invocation of the batch */

As a result, multiple streams of DAILYBATCH will be submitted for execution, each for its own range of records from TRANSACT. The main job will be put in WAIT state until the stream processing ends.

Clearly, this is just one basic example of iSTREAM’s capabilities. iSTREAM supports multiple data breakdown methods, e.g. by field/column values of one or multiple files/tables, by field/column values with yet another breakdown for each given range of values, or by automatically balancing numbers of records in each range. The framework can be used with any OPM and ILE program irrespective of the data access interface used, native or SQL.

iSTREAM LP includes seven functional options, of which the multistreaming toolkit is just one. All seven options help optimize existing software or systems being developed, each of them using a flavour of the parallel programming paradigm. Among them are multistreamed versions of CPYF and RMVJRNENT commands, a configurable automated backup using a DR system, and a file sync utility. This has become possible mainly due to the unique IBM i features still missing on other platforms considered contemporary and modern.