Hints and best practices

This section provides more details with regard to different aspects relating to job authoring.

Use of job properties

Job properties are an integral part of designing and implementing jobs and have already been discussed at length in previous sections. This section summarises where job properties can be defined and applied.

Definition of job properties can take place:

• In a PropertyGroup section and will thus have global scope within the job

• Through a CreateProperty task and will have target scope until the target completes execution where after the property will become globally available

• Through converting a task output parameter to a job property. This takes place through the Output element enclosed within a task definition

• Through specification at execution time, either from the user interface executes and schedule dialogs (see sections Execute a job or Schedule a job) or through the JobRunner.exe command line (see the section on Debugging).

Use of job items

Occasionally a job needs to work with lists of information, e.g. lists of input data files or lists of time series. This is not easily achieved with the job constructs discussed so far. For assistance with this, we have job items with the following definitions:

• Items are input to the job processing and typically represent lists of information values and are grouped into item types, based on their element names. Item types are named lists of items that can be used as parameters for tasks. The tasks use the item values to perform the steps of the job execution.

• An ItemGroup is a collection of user-defined Item elements. Every item used in a job must be specified as a child of an ItemGroup element.

This explanation is perhaps a bit abstract but the example shown in Listing 16 will make this easier to understand. The example shows how to import all the time series files found in a folder to a time series group within the Workbench database.

Listing 16 Loop over files

Note the following in the listing:

• At point 1 a list named dfs0 of files is created. The *.dfs``0 part of the Include statement implies all files with the .dfs0 file extension. Other wildcard possibilities comprise ‘?’ which matches a single character and ‘**’ which matches a partial path[^2]. It is also possible to exclude part of the files in the Include list through an Exclude attribute. Adding Exclude="a*.dfs0" to the dfs0 item definition would result in importing all dfs0-files except those starting with an ‘a’.

• At point 2 the same list is being used for importing the time series to the Workbench database. The ‘%’ character in %/dfs0.Identity) informs the job executor to call the ImportTimeseries task for each element in the dfs0 list and passing to it the identity of the element the name.

Another operator other than '%' operating on items is '@' which combines the items together in a semi-colon separated property value. This is demonstrated in Listing 17

Listing 17 Batching items

Running the above job snippet will result in the output shown to the right of the job code. It can be seen in fact, that the first Message task has been executed three times and the second just once by combining the items into a single list.

In some cases it might be required to convert a property e.g. from a task output parameter to an item, in order to execute a task on each of the elements in the item. This can be performed through the CreateItem as depicted in Listing 18.

Listing 18 Conversion of a property to an item

This example defines a property Test as a semi-colon separated list of characters and at point 2 transforms it into an item. Note how the use of the CreateItem task requires an embedded Output element in order to capture the task output as an item.

The Job Manager provides a special task JobHelper for transforming individual elements from an item into named properties. An example of this is demonstrated in Listing 19 where the job uses the standard RunScript task for executing a custom script MyScript. Now assume MyScript returns time series meta-data in the form of "min-value;max-value;mean-value;null-value-count" and the mean-value shall be used as input in a later task.

Listing 19 JobHelper example

Note in this example how:

• At point 1 the RunScript has just one output parameter named Result. This example script (MyScript) returns a text string with a semi-colon separated list of values

• At point 2 the JobHelper tasks convert the semi-colon separated output string from the RunScript task into an item collection

• At point 3 the JobHelper tasks return the value of the item at index 2. The embedded Output element converts this to a property named MeanValue

• At point 4 the mean value is printed.

See the section JobHelper task in Appendix A Standard tasks for more detailed information on the JobHelper task.

Debugging

Creating jobs can be compared to writing software programs; sometimes the jobs function differently to what was anticipated. This can be due to tasks behaving differently than imagined, errors with passing information between tasks or targets or job properties are wrongly spelled or initialised. In such cases, it can often be beneficial to "debug" the jobs.

Debugging jobs is best performed from a command line outside the MIKE Workbench environment. Jobs are scheduled or always directly executed using the DHI.Solutions. JobManager.JobRunner.exe (hereinafter JobRunner) command line tool.

• In the event that the job is executed directly from within the Job Manager, that component will spawn the JobRunner as a detached and hidden process.

• If the job is executed through a scheduled process, the Windows Task Scheduler will execute the JobRunner according to the defined schedule.

Executing the JobRunner from a command line requires the user to:

• Create a command line using e.g. Windows button+R, specify cmd.exe and click Open

• Have the Windows PATH environment variable including the Workbench installation folder by issuing the following command path %PATH%;C:\Program Files (x86)\DHI\<version>\MIKE OPERATIONS.

Now the JobRunner can be executed in the following way:

Listing 20 JobRunner command line

The command line arguments imply the following:

• -c shall be followed by the connection name, i.e. the same connection name that is being used when logging onto the Workbench.
• -w shall be followed by the workspace name
• -u shall be followed by the user name
• -p shall be followed by the password corresponding the specified user name
• -j shall be followed by a reference to the job that shall be executed.

This can be provided in one of two ways:

1. If the job has been exported as an XML file, the reference can be the path to the file. E.g. –j c:\temp\Example.job
2. If the job resides in the database, the reference shall be specified as e.g. in –j dss://Example1

Other useful command line arguments include:

• -a key-value, where key-value shall be specified as MyKey=MyValue. The JobRunner will convert these key/value pairs into job properties. The command line can take multiple –a arguments.

When debugging from the command line, it is often a good idea to insert a number of Message tasks with information like target name and value of various properties. Listing 21 shows a slightly changed version of Listing 1 where a number of Message tasks have been added and the original WriteLinesToFile task has been substituted by a Message task.

Listing 21 Changed version of Example1

The figure below shows the execution of this job from a command line.

Figure 2 Executing a job from a command line

The red highlighted text in the figure indicates the output from the three added Message tasks. Also note the log information printed for each of the tasks.

Date and time specification

Many of the tasks provided by the MIKE Workbench takes a time stamp as input parameter and in order to ensure that jobs can be shared across different computers with potential different regional settings including date format, it is advisable to always use the so-called invariant format for dates. Listing 22 shows how to specify a time stamp according to the invariant format.

Listing 22 Invariant time stamp format.

The job Manager also supports the time stamp format defined through the regional setting on the computer but as already mentioned, it is advisable to use the invariant format.

There are a number of useful standard tasks for handling time stamps. These include:

• MakeTimeStamp which can generate a time stamp based on an offset from "now".

• SetTimeStamp which can write a time stamp to a specified place in the registry or update a file's last modified time

• GetTimeStamp which can read a time stamp from a specified place in the registry or from a file's last modified time.