Scenario Manager Tools

Confidence Intervals

This will generate error models from the collected data of the collection spreadsheet (simulated and observed data).

Tool Info

Tool Info
Display Name	Confidence Intervals
API Name	Confidence Intervals
Tools Explorer	/Statistics/Confidence Intervals
NuGet Package	DHI.MikeOperations.ScenarioManager.Tools.RStatistics
Assembly name (.dll)	DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.dll
API Reference	DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.IRStatisticsConfidenceIntervalTool
Input Items	No input items required
Output Items	A time series

Tool Properties

Display Name	API Name	Description
Options
Action	Action	Gets or sets the action to perform. Execute - Gets the analysis results and returns them in the output items. DataCollection - Collects data for the data analysis. Analysis - Runs the data Analysis using the R-script.
Data Collection
From Date	FromDate	Gets or sets the time of forecast date from where simulations of the data collection scenario are taken. Note that the property is only available when: Action=Analysis,DataCollection
Mask	Mask	Gets or sets the mask for finding output time series for mapping. If the mask is not specified, no time series is found for mapping. Regular expressions are used for finding matching output time series. Note that the property is only available when: Action=DataCollectionUpdateMappingSheet=True
Period Interval	CollectionPeriodInterval	Gets or sets the time period used when collecting data. This interval can only be specified when new data collection spreadsheets are created. For existing spreadsheets, the collection interval is specified by the spreadsheet. Seconds - Time period Seconds. Minutes - Time period Minutes. Hours - Time period Hours. Days - Time period Days. Months - Time period Months. Years - Time period Years. . Note that the property is only available when: Action=DataCollection
Period Length	CollectionPeriodLength	Gets or sets the length of the period interval used when collection data. This interval can only be specified when new data collection spreadsheets are created. For existing spreadsheets, the collection interval is specified by the spreadsheet. Note that the property is only available when: Action=DataCollection
Scenario Path	DataCollectionScenarioPath	Gets or sets the path of the scenario for collecting data. Note that the property is only available when: Action=DataCollection
Spreadsheet Path	DataCollectionSpreadsheetPath	Gets or sets the full path to the data collection spreadsheet. Note that the property is only available when: Action=Analysis,DataCollection
To Date	ToDate	Gets or sets the time of forecast date to where simulations of the data collection scenario are taken. Note that the property is only available when: Action=Analysis,DataCollection
Update Mapping Sheet	UpdateMappingSheet	Gets or sets a value indicating whether the mapping sheet should be updated, so that a row in the mapping sheet and a data collection sheet is present for all output time series of the scenario. Note that the property is only available when: Action=DataCollection
Results
Analysis Name	AnalysisName	Gets or sets the name of the analysis. The name is used for updating error model in the document manager. Note that the property is only available when: Action=Analysis
Confidence Intervals	ConfidenceIntervals	Gets or sets a comma separated string with confidence intervals to use for calculating the error model. The list must use decimal points. Default: "0.05,0.125,0.5,0.875,0.95". Note that the property is only available when: Action=Analysis,Execute
Error Model Path	ErrorModelPath	Gets or sets the path in the document manager to the error model to use for generating confidence intervals. Note that the property is only available when: Action=Execute
Last Error	LastError	Gets or sets the last error when taking initial error into consideration. Last error is the difference between the observed last observed value and a previous simulation (obs.value - sim.value). Note that the property is only available when: Action=Execute
Model Type	ModelType	Gets or sets the model type (1-4). Note that the property is only available when: Action=Analysis
Time Series Path	TimeSeriesPath	Gets or sets the path to the time series to generate confidence intervals for. Note that the property is only available when: Action=Execute
Use Weights	UseWeights	Gets or sets a value indicating whether weights should be used. Note that the property is only available when: Action=Analysis

Tool Methods

• GetAnalysisMeasures(): Gets an array of measures supported by the analysis.

Code Sample

IronPythonCPythonC#

import clr
clr.AddReference("DHI.Solutions.ScenarioManager.Tools.RStatisticsTool")
from DHI.Solutions.ScenarioManager.Tools.RStatisticsTool import *

# Get the tool.
tool = app.Tools.CreateNew("Confidence Intervals")
# Add input items.
tool.InputItems.Add(<No input items required>)
# Set tool properties before executing the tool.
tool.Execute()
# Read the output from the list of output items.
output = tool.OutputItems

import clr
clr.AddReference("DHI.Solutions.ScenarioManager.Tools.RStatisticsTool")
from DHI.Solutions.ScenarioManager.Tools.RStatisticsTool import *

# Get the tool.
tool = DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.IRStatisticsConfidenceIntervalTool(app.Tools.CreateNew("Confidence Intervals"))
# Add input items.
tool.InputItems.Add(<No input items required>)
# Set tool properties before executing the tool.
tool.Execute()
# Read the output from the list of output items.
output = tool.OutputItems

// Get the tool.
var tool = application.Tools.CreateNew("Confidence Intervals") as DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.IRStatisticsConfidenceIntervalTool;
// Add input items.
tool.InputItems.Add(<No input items required>);
// Set tool properties before executing the tool.
tool.Execute();
// Read the output from the list of output items.
output = tool.OutputItems;

Runtime.config

What tools to load during application startup, is specified in the XML file Runtime.config in the plugins section of the manager.

The XML below shows the tool plugin information found in the Runtime.config file.

<Plugin
  Name="DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.RStatisticsConfidenceIntervalTool"
  Type="DHI.Solutions.Generic.ITool"
  Assembly="DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.dll" />

Requirements

The confidence interval tool is based on R-scripts and requires R to be installed. The R installer can be downloaded from CRAN website. Packages used in the scripts (i.e. quantreg, abind, MASS, quantregGrowth) can be found on the CRAN website

The tools will support data collection, writing input files for the scripts, executing the scripts and parsing the results into MIKE Workbench entities.

Note

The R tools described in this chapter, are supported using R libraries before R version 3.5.

Create the model

An error model is created for each model object variable. The error models are saved in the Document Manager under the same relative path named: /R-Statistics/<Error model name>/<Model Object Name>/<Model object variable name> (See image below).

In the spreadsheet manager, a spreadsheet will be created, holding information about the error model created.

Action must be set to Analysis Then the following properties should be specified:

• Spreadsheet Path
  The full path to the data collection spreadsheet.
• From Date
  Simulations with time of forecast between From Date and To Date are collected. If left blank all simulations are be collected.
• To Date
  Simulations with time of forecast between From Date and To Date are collected. If left blank all simulations are be collected.
• Analysis name
  The name of the analysis. The name is used for updating error model in the document manager.
• Confidence Intervals
  A comma separated string with quantiles included in the calculation of the error model. The list must use decimal points. Default: 0.05,0.175,0.5,0.875,0.95
• Model Type
  Model Type value can be 1, 2, 3 or 4.
  1) Quantile regression model
  2) Quantile regression model using initial error at time of forecast.
  3) Quantile regression model with NQT transformation
  4) Quantile regression model with NQT transformation and using initial error at time of forecast
• Use Weights
  If weights should be used in quantile calibration. Each simulation is given a weight by its magnitude rank in the dataset rank of data/number values. Lowest value in a size 10 data set would get 1/10 as weight and largest 10/10.

Execute the model

Action should be set to Execute This will create an ensemble time series representing the uncertainty in the deterministic forecast at the quantiles specified during the generation of the error model.

The following properties must be specified.

• Confidence Intervals
  A comma separated string specifying the quantiles that will be calculated.
  The list must use decimal points. Default: 0.05,0.175,0.5,0.875,0.95.
  The confidence intervals specified must be supported by the error model as specified during the creation in the range ]0 ; 1[. It can be a subset for the confidence of the supported confidence intervals.
• Error Model Path
  The full path to the error model in the document manager.
• Last Error
  The last error when taking initial error into consideration.
  Last error is the difference between the observed last observed value and a previous simulation (obs.value - sim.value).
  If Last Error is set to 0, it is not taken into account.
• Time Series Path
  The full path of the time series to generate confidence intervals for.

Goodness of Fit

This tool measures intends to summarize how well two continuous datasets fit each other. They might compare distributional parameters (statistical moments, IQR, Peak time frequency, etc.), residual errors (RMSE, ME, bias, NRMSE, etc.), data patterns (correlation coefficient, NSE) or concurrent comparisons (linear regression, peak difference, peak percentage error, etc.).

The Goodness of Fit tool calculates goodness of fit measures based on a collected simulations and observations. The tool will generate a result spreadsheet containing goodness of fit measures for each station (Model object and model object variable) and for each lead time in the forecast period.

Tool Info

Tool Info
Display Name	Goodness of Fit
API Name	Goodness of Fit
Tools Explorer	/Statistics/Goodness of Fit
NuGet Package	DHI.MikeOperations.ScenarioManager.Tools.RStatistics
Assembly name (.dll)	DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.dll
API Reference	DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.IRStatisticsGoodnessOfFitTool
Input Items	No input items required
Output Items	A time series

Tool Properties

Display Name	API Name	Description
Options
Action	Action	Gets or sets the action to perform. Execute - Gets the analysis results and returns them in the output items. DataCollection - Collects data for the data analysis. Analysis - Runs the data Analysis using the R-script.
Data Collection
From Date	FromDate	Gets or sets the time of forecast date from where simulations of the data collection scenario are taken. Note that the property is only available when: Action=Analysis,DataCollection
Mask	Mask	Gets or sets the mask for finding output time series for mapping. If the mask is not specified, no time series is found for mapping. Regular expressions are used for finding matching output time series. Note that the property is only available when: Action=DataCollectionUpdateMappingSheet=True
Period Interval	CollectionPeriodInterval	Gets or sets the time period used when collecting data. This interval can only be specified when new data collection spreadsheets are created. For existing spreadsheets, the collection interval is specified by the spreadsheet. Seconds - Time period Seconds. Minutes - Time period Minutes. Hours - Time period Hours. Days - Time period Days. Months - Time period Months. Years - Time period Years. . Note that the property is only available when: Action=DataCollection
Period Length	CollectionPeriodLength	Gets or sets the length of the period interval used when collection data. This interval can only be specified when new data collection spreadsheets are created. For existing spreadsheets, the collection interval is specified by the spreadsheet. Note that the property is only available when: Action=DataCollection
Scenario Path	DataCollectionScenarioPath	Gets or sets the path of the scenario for collecting data. Note that the property is only available when: Action=DataCollection
Spreadsheet Path	DataCollectionSpreadsheetPath	Gets or sets the full path to the data collection spreadsheet. Note that the property is only available when: Action=Analysis,DataCollection
To Date	ToDate	Gets or sets the time of forecast date to where simulations of the data collection scenario are taken. Note that the property is only available when: Action=Analysis,DataCollection
Update Mapping Sheet	UpdateMappingSheet	Gets or sets a value indicating whether the mapping sheet should be updated, so that a row in the mapping sheet and a data collection sheet is present for all output time series of the scenario. Note that the property is only available when: Action=DataCollection
Results
Model Object Name	ModelObjectName	Gets or sets the model object name to get results for. Note that the property is only available when: Action=Execute
Model Object Variable	ModelObjectVariable	Gets or sets the model object variable to get results for. Note that the property is only available when: Action=Execute
Spreadsheet Path	ResultSpreadsheetPath	Gets or sets the full path of the spreadsheet for the analysis results. Note that the property is only available when: Action=Analysis,Execute

Tool Methods

• GetAnalysisMeasures(): Gets an array of measures supported by the analysis.

Code Sample

IronPythonCPythonC#

import clr
clr.AddReference("DHI.Solutions.ScenarioManager.Tools.RStatisticsTool")
from DHI.Solutions.ScenarioManager.Tools.RStatisticsTool import *

# Get the tool.
tool = app.Tools.CreateNew("Goodness of Fit")
# Add input items.
tool.InputItems.Add(<No input items required>)
# Set tool properties before executing the tool.
tool.Execute()
# Read the output from the list of output items.
output = tool.OutputItems

import clr
clr.AddReference("DHI.Solutions.ScenarioManager.Tools.RStatisticsTool")
from DHI.Solutions.ScenarioManager.Tools.RStatisticsTool import *

# Get the tool.
tool = DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.IRStatisticsGoodnessOfFitTool(app.Tools.CreateNew("Goodness of Fit"))
# Add input items.
tool.InputItems.Add(<No input items required>)
# Set tool properties before executing the tool.
tool.Execute()
# Read the output from the list of output items.
output = tool.OutputItems

// Get the tool.
var tool = application.Tools.CreateNew("Goodness of Fit") as DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.IRStatisticsGoodnessOfFitTool;
// Add input items.
tool.InputItems.Add(<No input items required>);
// Set tool properties before executing the tool.
tool.Execute();
// Read the output from the list of output items.
output = tool.OutputItems;

Runtime.config

What tools to load during application startup, is specified in the XML file Runtime.config in the plugins section of the manager.

The XML below shows the tool plugin information found in the Runtime.config file.

<Plugin
  Name="DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.RStatisticsGoodnessOfFitTool"
  Type="DHI.Solutions.Generic.ITool"
  Assembly="DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.dll" />

Requirements

The goodness of fit measures tool is based on R-scripts and requires R to be installed. The R installer can be downloaded from CRAN website. Packages used in the scripts (i.e. hydroGOF, abind) can be found on the CRAN website

The tools will support data collection, writing input files for the scripts, executing the scripts and parsing the results into MIKE Workbench entities.

Note

The R tools described in this chapter, are supported using R libraries before R version 3.5.

Analysis

Running Action=Analysis requires the following properties.

• Spreadsheet Path
  The full path to the data collection spreadsheet.
• From Date
  Simulations with time of forecast between From Date and To Date are collected. If left blank all simulations are be collected.
• To Date
  Simulations with time of forecast between From Date and To Date are collected. If left blank all simulations are be collected.
• Results Spreadsheet Path
  The full path of the results spreadsheet.

The results contain in the rows different performance measures and in the columns for each lead times in ascending order. The table below describes the measures supported by the Goodness of Fit tool. Refer to the hydroGOF CRAN package for more information.

• MeanError
  Mean error or bias between simulation and observation. $$ ME = \frac{1}{n}\sum_{t=1}^n (Q_{sim}^t - Q_{obs}^t) $$ where \({Q_{obs}^t}\) is observation and \({Q_{sim}^t}\) simulation at time t.

• MeanAbsoluteError
  Mean absolute error calculates the mean of the absolute error: $$ MAE = \frac{1}{n}\sum_{t=1}^n (Q_{sim}^t - Q_{obs}^t) $$ where \({Q_{obs}^t}\) is observation and \({Q_{sim}^t}\) simulation at time t.

• MeanSquaredError
  Mean Square Error MSE is the mean square error or the residual mean square. It is a biased estimator in contrast to the variance which is unbiased. $$ MSE=\dfrac{SSE}{n} $$ Where SSE is the sum of squared errors and n the number of observation and estimation pairs. A value closer to 0 indicates a fit that is more useful for prediction.

• RootMeanSquareError
  Root Mean Square Error (RMSE) is also known as the fit standard error and the standard error of the regression.
  It is an estimate of the standard deviation of the random component in the data, and is defined as the square root of the Mean Squared Error. $$ RMSE = \sqrt{\frac{1}{n}\sum_{t=1}^n {(Q_{sim}^t - Q_{obs}^t)}^2} $$ Where \({Q_{obs}^t}\) is observation and \({Q_{sim}^t}\) simulation at time t.
  Just as with the mean squared error, an RMSE value closer to 0 indicates a fit that is more useful for prediction.

• RootMeanSquareErrorNormalized
  A normalized version of the Root mean square error that has been divided by the range of the dataset. Range from -100% to 100%.

• PercentBias
  Percent bias measures the average tendency of the simulated values to be larger or smaller than their observed ones.
  The optimal value of PBIAS is 0.0, with low-magnitude values indicating accurate model simulation. Positive values indicate overestimation bias, whereas negative values indicate model underestimation bias.

• RootMeanSquareErrorRatio
  Root Mean Square Error Ratio.

• StandardDeviationRatio
  The ratio between the standard deviation of the simulation and observation: $$ rSD=\frac{std(sim)}{std(obs)} $$

• NashSutcliffeEfficiency
  The Nash–Sutcliffe model efficiency coefficient is used to assess the predictive power of hydrological models. It is defined as: $$ NSE=1-\frac{\sum_{t=1}^T{(Q_{obs}^t - Q_{sim}^t)}^2}{\sum_{t=1}^T{(Q_{obs}^t-\overline{Q}_{obs})}^2} $$ where \({Q_{obs}}\) is the mean of observed discharges, and \({Q_{sim}}\) is modeled discharge. \({Q_{obs}^t}\) is observed discharge and \(Q_{sim}^t\) simulated discharge at time t.
  [1] Nash–Sutcliffe efficiency can range from −∞ to 1. An efficiency of 1 (E = 1) corresponds to a perfect match of modeled discharge to the observed data. An efficiency of 0 (E = 0) indicates that the model predictions are as accurate as the mean of the observed data, whereas an efficiency less than zero (E < 0) occurs when the observed mean is a better predictor than the model or, in other words, when the residual variance (described by the numerator in the expression above), is larger than the data variance (described by the denominator). Essentially, the closer the model efficiency is to 1, the more accurate the model is.

• NashSutcliffeEfficiencyModified
  Same as the NashSutcliffeEfficiency but using absolute errors instead of squared errors. $$ NSE_1=1-\frac{\sum_{t=1}^T{|Q_{obs}^t-Q_{sim}^t|}}{\sum_{t=1}^T{|Q_{obs}^t-\overline{Q}_{obs}|}} $$ where \(Q_{obs}\) is the mean observation, \(Q_{obs}^t\) observation and \(Q_{sim}^t\) simulation at time t.
  Making it less sensitive to extreeme values.

• NashSutcliffeEfficiencyRelative
  Same as the NashSutCliffeEfficiency but relative to the mean of the observation.

  \[ rNSE=1-\frac{\sum_{t=1}^T{(\dfrac{Q_{obs}^t-Q_{sim}^t}{\overline{Q}_{obs}})}^2}{\sum_{t=1}^T{(\dfrac{Q_{obs}^t-\overline{Q}_{obs}}{\overline{Q}_{obs}})}^2} \]

  where \(Q_{obs}\) is the mean observation. \(Q_{obs}^{t}\) is observation and \(Q_{sim}^{t}\) simulation at time t.

• IndexAgreement
  The Index of Agreement (d) developed by Willmott (1981) as a standardized measure of the degree of model prediction error and varies between 0 and 1.
  A value of 1 indicates a perfect match, and 0 indicates no agreement at all (Willmott, 1981).
  It is defined by:
  
  where Q ̅_0, is the mean observation. Qot is observed discharge and Qmt simulated discharge at time t. d varies from 0 to 1 whit increasing values meaning better agreement between simulation and observations.
  The index of agreement can detect additive and proportional differences in the observed and simulated means and variances; however, it is overly sensitive to extreme values due to the squared differences (Legates and McCabe, 1999).

• IndexAgreementModified
  In the modified index of agreement method, the squaring has been removed in order to make it less sensitive to extreme values.
  
  where Q ̅_0 is the mean observation. Qot is observed discharge and Qmt simulated discharge at time t. It varies from 0 to 1 whit increasing values meaning better agreement between simulation and observations. The modified version limits the inflation from squaring the terms.

• IndexAgreementRelative
  Same as the modified index of agreement but relative to the observation.
  The relative index of agreement is defined by:
  
  where Q ̅_0 is the mean observation. Qot is observed discharge and Qmt simulated discharge at time t.
  d varies from 0 to 1 whit increasing values meaning better agreement between simulation and observations.

• PersistenceIndex
  Compares the squared error of the simulation with the squared error if the previous observation was used as forecase.
  
  Where Qm is simulation and Qot is is observation at time t. Qot-1 is observation at the previous time step at time t-1.
  Nb. This score does not seem to be apropriate when applied on lead times as the persistence would be the observed value from the previous forecast.

• PearsonCorrelationCoefficient
  The Pearson correlation coefficient is the covariance of the observed and simulated values divided by the product of their standard deviations:
  
  Where ρis the pearson correlation coefficient, cov the covariance, σ the standard diviation, Qm the simulation and Qo the observation.

• CoefficientDetermination
  Coefficient of determination denoted r2 is the square value of the coefficient of correlation:
  
  Where Q ̅_0 is the mean observation, Q ̅_m the mean simulation. Qot is observed discharge and Qmt simulated discharge at time t.
  r^2 range between 0 and 1 and describes how much of the observed dispersion is explained by the simulation.

• CoefficientDeterminationMultiplied
  Same as Coefficient of determination but multiplied by the regression line between simulation and observation to penalize for systematic errors in forecast as the correlation of coefficient only compares dispersion.
  
  Where b is the slope b is the slope of the regression line between sim and obs.

• KlingGuptaEfficiency
  Kling-Gupta efficiency (KGE) between simulation and observation.
  This goodness-of-fit measure was developed by Gupta et al. (2009) to provide a diagnostically interesting decomposition of the Nash-Sutcliffe efficiency (and hence MSE), which facilitates the analysis of the relative importance of its different components (correlation, bias and variability) in the context of hydrological modelling.
  Kling et al. (2012), proposed a revised version of this index, to ensure that the bias and variability ratios are not cross-correlated.
  In the computation of this index, there are three main components involved:
  

  1. r: The Pearson product-moment correlation coefficient. Ideal value is r=1
  2. Beta: The ratio between the mean of the simulated values and the mean of the observed ones. Ideal value is Beta=1
  3. vr: Variability ratio, which could be computed using the standard deviation (Alpha) or the coefficient of variation (Gamma) of simulation and observation, depending on the value of method.
     
     • Alpha: The ratio between the standard deviation of the simulated values and the standard deviation of the observed ones. Ideal value is Alpha=1.
       
     • Gamma: The ratio between the coefficient of variation (CV) of the simulated values to the coefficient of variation of the observed ones. Ideal value is Gamma=1.
       KGE can range from −∞ to 1 and improves as it becomes closer to 1.

• VolumetricEfficiency
  The Volumetric efficiency is given by:
  
  Where Qot is observed discharge and Qmt simulated discharge at time t.
  VE ranges between 0 and 1 where simulation improves as VE increases.

• InterQuantileRange
  Is a measure of spread in the data and is defined as the difference between the upper and lower quartile:
  
  

• PeakTimeFrequencyObserved
  Frequency of Peak in observed time series.

• PeakTimeFrequencySimulated
  Frequency of peak in simulated time series.

• MedianSimulated
  The median value of the simulation.

• MedianObserved
  The median value of the observation.

Execution

Action=Execute will simply list the content of the spreadsheet in a table.

Specify the result spreadsheet and the model object name and variable of the station to display.

Run to List or to Chart.

Hierarchy Configuration

The Hierarchy tool allows the user to define the input timeseries of a scenario in a robust and user-friendly way.

Tool Info
Display Name	Hierarchy Configuration
API Name	Hierarchy Configuration
Tools Explorer	/Scenario Tools/Hierarchy Configuration
NuGet Package	DHI.MikeOperations.ScenarioManager.Tools.HierarchyConfiguration
Assembly name (.dll)	DHI.Solutions.ScenarioManager.UI.Tools.HierarchyDefinitionTool.dll
API Reference	DHI.Solutions.ScenarioManager.UI.Tools.HierarchyDefinitionTool.IHierarchyDefinitionTool
Input Items	A single scenario or a single scenario input time series
Output Items	No output items

Tool Properties

Display Name	API Name	Description
Tool Settings
Update	Update	Gets or sets a value indicating whether to update the XML on the ScenarioInputTimeSeriesDefinition table in the database. Default=true. Always false for input items of type ISimulationInputTimeseries. Note that the property is only available when: SimulationInputTS=False

Code Sample

IronPythonCPythonC#

import clr
clr.AddReference("DHI.Solutions.ScenarioManager.UI.Tools.HierarchyDefinitionTool")
from DHI.Solutions.ScenarioManager.UI.Tools.HierarchyDefinitionTool import *

# Get the tool.
tool = app.Tools.CreateNew("Hierarchy Configuration")
# Add input items.
tool.InputItems.Add(<A single scenario or a single scenario input time series>)
# Set tool properties before executing the tool.
tool.Execute()
# Read the output from the list of output items.
output = tool.OutputItems

import clr
clr.AddReference("DHI.Solutions.ScenarioManager.UI.Tools.HierarchyDefinitionTool")
from DHI.Solutions.ScenarioManager.UI.Tools.HierarchyDefinitionTool import *

# Get the tool.
tool = DHI.Solutions.ScenarioManager.UI.Tools.HierarchyDefinitionTool.IHierarchyDefinitionTool(app.Tools.CreateNew("Hierarchy Configuration"))
# Add input items.
tool.InputItems.Add(<A single scenario or a single scenario input time series>)
# Set tool properties before executing the tool.
tool.Execute()
# Read the output from the list of output items.
output = tool.OutputItems

// Get the tool.
var tool = application.Tools.CreateNew("Hierarchy Configuration") as DHI.Solutions.ScenarioManager.UI.Tools.HierarchyDefinitionTool.IHierarchyDefinitionTool;
// Add input items.
tool.InputItems.Add(<A single scenario or a single scenario input time series>);
// Set tool properties before executing the tool.
tool.Execute();
// Read the output from the list of output items.
output = tool.OutputItems;

Runtime.config

What tools to load during application startup, is specified in the XML file Runtime.config in the plugins section of the manager.

The XML below shows the tool plugin information found in the Runtime.config file.

<Plugin
  Name="DHI.Solutions.ScenarioManager.UI.Tools.HierarchyDefinitionTool.HierarchyDefinitionTool"
  Type="DHI.Solutions.Generic.ITool"
  Assembly="DHI.Solutions.ScenarioManager.UI.Tools.HierarchyDefinitionTool.dll" />

Longitudinal Profile

This tool creates a longitudinal profile time series based on simulation output time series. It supports the calculation points, structures, cross sections and boundaries of MIKE11 model, the calculation points, structures and cross sections of MIKE HYDRO River model, and the nodes of MIKE+ model.

Tool Info
Display Name	Longitudinal Profile
API Name	Longitudinal Profile
Tools Explorer	/Scenario Tools/Longitudinal Profile
NuGet Package	DHI.MikeOperations.ScenarioManager.Tools.LongitudinalProfile
Assembly name (.dll)	DHI.Solutions.ScenarioManager.Tools.SimulationTimeSeriesProfileTool.dll
API Reference	DHI.Solutions.ScenarioManager.Tools.SimulationTimeSeriesProfileTool.ISimulationTimeSeriesProfileTool
Input Items	One or more simulation model objects
Output Items	An X-Y data series

Tool Properties

Display Name	API Name	Description
Tool Settings
Additional time series	AdditionalTimeSeries	Gets or sets the value of AdditionalTimeSeries
Time relative to ToF	TimeRelativeToTOF	Gets or sets the value of TimeRelativeToTOF (seconds)
Time Series Information	TimeSeriesInformation	Gets or sets the value about the simulation output time series to include in the longitudinal profile.

Code Sample

IronPythonCPythonC#

import clr
clr.AddReference("DHI.Solutions.ScenarioManager.Tools.SimulationTimeSeriesProfileTool")
from DHI.Solutions.ScenarioManager.Tools.SimulationTimeSeriesProfileTool import *

# Get the tool.
tool = app.Tools.CreateNew("Longitudinal Profile")
# Add input items.
tool.InputItems.Add(<One or more simulation model objects>)
# Set tool properties before executing the tool.
tool.Execute()
# Read the output from the list of output items.
output = tool.OutputItems

import clr
clr.AddReference("DHI.Solutions.ScenarioManager.Tools.SimulationTimeSeriesProfileTool")
from DHI.Solutions.ScenarioManager.Tools.SimulationTimeSeriesProfileTool import *

# Get the tool.
tool = DHI.Solutions.ScenarioManager.Tools.SimulationTimeSeriesProfileTool.ISimulationTimeSeriesProfileTool(app.Tools.CreateNew("Longitudinal Profile"))
# Add input items.
tool.InputItems.Add(<One or more simulation model objects>)
# Set tool properties before executing the tool.
tool.Execute()
# Read the output from the list of output items.
output = tool.OutputItems

// Get the tool.
var tool = application.Tools.CreateNew("Longitudinal Profile") as DHI.Solutions.ScenarioManager.Tools.SimulationTimeSeriesProfileTool.ISimulationTimeSeriesProfileTool;
// Add input items.
tool.InputItems.Add(<One or more simulation model objects>);
// Set tool properties before executing the tool.
tool.Execute();
// Read the output from the list of output items.
output = tool.OutputItems;

Runtime.config

What tools to load during application startup, is specified in the XML file Runtime.config in the plugins section of the manager.

The XML below shows the tool plugin information found in the Runtime.config file.

<Plugin
  Name="DHI.Solutions.ScenarioManager.Tools.SimulationTimeSeriesProfileTool.SimulationTimeSeriesProfileTool"
  Type="DHI.Solutions.Generic.ITool"
  Assembly="DHI.Solutions.ScenarioManager.Tools.SimulationTimeSeriesProfileTool.dll" />

R-Statistics Tool

Execution

Tool for executing R-Statistics scripts.

Tool Info
Display Name	R-Statistics
API Name	R-Statistics
Tools Explorer	/Statistics/R-Statistics
NuGet Package	DHI.MikeOperations.ScenarioManager.Tools.RStatistics
Assembly name (.dll)	DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.dll
API Reference	DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.IRStatisticsTool
Input Items	No input items required
Output Items	No output items

Tool Properties

Display Name	API Name	Description
Parameters
Arguments	ArgumentString	Gets or sets the argument string to use for the R-execution.
R version	RVersion	Gets or sets the value of R version
Use 64 Bit	Use64Bit	Gets or sets a value indicating whether the 64 bit version of should be used. If the 64 bit execution file does not exist, 32 bit execution will be used.
Working Folder	WorkingFolder	Gets or sets the working folder for the script execution.

Code Sample

IronPythonCPythonC#

import clr
clr.AddReference("DHI.Solutions.ScenarioManager.Tools.RStatisticsTool")
from DHI.Solutions.ScenarioManager.Tools.RStatisticsTool import *

# Get the tool.
tool = app.Tools.CreateNew("R-Statistics")
# Add input items.
tool.InputItems.Add(<No input items required>)
# Set tool properties before executing the tool.
tool.Execute()
# Read the output from the list of output items.
output = tool.OutputItems

import clr
clr.AddReference("DHI.Solutions.ScenarioManager.Tools.RStatisticsTool")
from DHI.Solutions.ScenarioManager.Tools.RStatisticsTool import *

# Get the tool.
tool = DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.IRStatisticsTool(app.Tools.CreateNew("R-Statistics"))
# Add input items.
tool.InputItems.Add(<No input items required>)
# Set tool properties before executing the tool.
tool.Execute()
# Read the output from the list of output items.
output = tool.OutputItems

// Get the tool.
var tool = application.Tools.CreateNew("R-Statistics") as DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.IRStatisticsTool;
// Add input items.
tool.InputItems.Add(<No input items required>);
// Set tool properties before executing the tool.
tool.Execute();
// Read the output from the list of output items.
output = tool.OutputItems;

Runtime.config

What tools to load during application startup, is specified in the XML file Runtime.config in the plugins section of the manager.

The XML below shows the tool plugin information found in the Runtime.config file.

<Plugin
  Name="DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.RStatisticsTool"
  Type="DHI.Solutions.Generic.ITool"
  Assembly="DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.dll" />

This tool calls R-Statistics in a separate process with the specified parameters.

This tool is used by the Confidence Interval, performance and Skill score tools.

Runtime.config

What tools to load during application startup, is specified in the Runtime.config file as plugins.

The XML below shows the tool plugin information found in the Runtime.config file.

<Plugin
  Name="DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.RStatisticsTool"
  Type="DHI.Solutions.Generic.ITool"
  Assembly="DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.dll" />

Tool Properties

Display Name	Name	Description
Arguments	ArgumentString	Gets or sets the argument string to use for the R-execution.
R version	RVersion	Gets or sets the value of R version
Use 64 Bit	Use64Bit	Gets or sets a value indicating whether the 64 bit version of should be used. If the 64 bit execution file does not exist, 32 bit execution will be used.
Working Folder	WorkingFolder	Gets or sets the working folder for the script execution.

Code Sample

C#IronPythonCPython

// Get the tool.
var tool = application.Tools.CreateNew("R-Statistics") as DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.IRStatisticsTool;

# Get the tool.
tool = app.Tools.CreateNew("R-Statistics")

# Get the tool.
tool = DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.IRStatisticsTool(app.Tools.CreateNew("R-Statistics"))

Requirements

The R-Statistics tool is based on R-scripts and requires R to be installed. The R installer can be downloaded from CRAN website.

The tools will support data collection, writing input files for the scripts, executing the scripts and parsing the results into MIKE Workbench entities.

Note

The R tools described in this chapter, are supported using R libraries before R version 3.5.

Data collection

The R-statistics tools Confidence intervals, Goodness of Fit and Skill Scores use the data collection spreadsheet in the format generated by using the DataCollection option.

This means that all the tools use the same data collection spreadsheet for carrying out analysis.

Data collection will collect observed and simulated data.

• Observed data are collected from observation time series specified in the mapping sheet.
• Simulated data are collected from approved simulations on the specified scenario. Only data after Time of Forecast are collected.

When collecting data, the following properties are available in the tools:

• Update Mapping Sheet
  A value indicating whether the mapping sheet should be created and/or updated, so that a row in the mapping sheet and a data collection sheet is present for all output time series included in the specified scenario.
  If the spreadsheet does not already exist, it will be created in the path specified.
• Mask
  Mask for finding output time series for mapping. If the mask is not specified, no time series is found for mapping.
  Regular expressions are used for finding matching output time series.
  The mask is shown only when updating the mapping sheet.
• Spreadsheet Path
  The full path to the data collection spreadsheet.
• Scenario Path
  The path of the scenario from where data is collected.
• From Date
  Simulations with time of forecast between From Date and To Date are collected. If left blank all simulations are be collected.
• To Date
  Simulations with time of forecast between From Date and To Date are collected. If left blank all simulations are be collected.
• Period Interval
  The time unit used when collecting data.
  This interval can only be specified when new data collection spreadsheets are created.
  For existing spreadsheets, the collection interval is specified by the spreadsheet.
• Period Length
  The length of a period interval used when collection data.
  This interval can only be specified when new data collection spreadsheets are created.
  For existing spreadsheets, the collection interval is specified by the spreadsheet.

The image below shows a sample of a data collection spreadsheet.

The data collection spreadsheet contains 3 types of sheets.

1. Configuration
   A single sheet containing Information about what scenario to collect data from and a description of intervals.
   

   • Data Collection Scenario Path - Full path to the scenario containing forecasted data.
   • Period Interval - Period interval for time steps when collecting data (seconds, minutes, hours, days, months or years).
   • Period Interval Length - The length of each time step when collecting data.

2. Mapping
   A single sheet containing information about stations where data should be collected and what output- and observations time series to use as well as thresholds for the station.
   

   • Sheet Name - The name of the sheet where data from the station is collected.
   • Model Object Name - Name of the model object containing the output time series.
   • Model Object Variable - Variable of the time series.
   • Observation Time Series - Path to observation time series in the time series explorer corresponding to the simulation point. This column must be manually updated after the data collection spreadsheet has been generated.
   • Thresholds - Threshold defined for the simulation point. Note that the thresholds should be in descending order.

3. Collected data
   A sheet per station where data is collected containing forecasted and observed time series.

Scenario Input Time series Generation

Scenario Input Time series Generation Tool for creating time series using prepared scripts of Scenario Input Timew Series Definitions. The output type of the prepared scripts, should be a single time series. The tool will be available when selecting a scenario in the scenario explorer.

Tool Info
Display Name	Scenario Input Time series Generation
API Name	Scenario Input Time series Generation
Tools Explorer	/Scenario Tools/Scenario Input Time series Generation
NuGet Package	DHI.MikeOperations.ScenarioManager.Tools.InputTimeseriesGeneration
Assembly name (.dll)	DHI.Solutions.ScenarioManager.Tools.dll
API Reference	DHI.Solutions.ScenarioManager.Tools.ISITSTool
Input Items	No input items required
Output Items	A time series

Tool Properties

Display Name	API Name	Description
Tool Settings
Execution Log	ExecutionLog	Gets or sets a execution log option. None - Will not execute log. Document - Log will be save as a document. File - Log will be save as a file.
Input Time series Definition	InputTSName	Gets or sets the input time series definitions to generated time series for.
Log Specification	LogSpecification	Gets or sets the path of the log when using the execution log option. Note that the property is only available when: ExecutionLog=Document,File
Output Folder	OutputFolder	Gets or sets the time series group where the generated time series are added to.

Code Sample

IronPythonCPythonC#

import clr
clr.AddReference("DHI.Solutions.ScenarioManager.Tools")
from DHI.Solutions.ScenarioManager.Tools import *

# Get the tool.
tool = app.Tools.CreateNew("Scenario Input Time series Generation")
# Add input items.
tool.InputItems.Add(<No input items required>)
# Set tool properties before executing the tool.
tool.Execute()
# Read the output from the list of output items.
output = tool.OutputItems

import clr
clr.AddReference("DHI.Solutions.ScenarioManager.Tools")
from DHI.Solutions.ScenarioManager.Tools import *

# Get the tool.
tool = DHI.Solutions.ScenarioManager.Tools.ISITSTool(app.Tools.CreateNew("Scenario Input Time series Generation"))
# Add input items.
tool.InputItems.Add(<No input items required>)
# Set tool properties before executing the tool.
tool.Execute()
# Read the output from the list of output items.
output = tool.OutputItems

// Get the tool.
var tool = application.Tools.CreateNew("Scenario Input Time series Generation") as DHI.Solutions.ScenarioManager.Tools.ISITSTool;
// Add input items.
tool.InputItems.Add(<No input items required>);
// Set tool properties before executing the tool.
tool.Execute();
// Read the output from the list of output items.
output = tool.OutputItems;

Runtime.config

What tools to load during application startup, is specified in the XML file Runtime.config in the plugins section of the manager.

The XML below shows the tool plugin information found in the Runtime.config file.

<Plugin
  Name="DHI.Solutions.ScenarioManager.Tools.SITSTool"
  Type="DHI.Solutions.Generic.ITool"
  Assembly="DHI.Solutions.ScenarioManager.Tools.dll" />

Skill Scores/Categorical forecast

The skill of a forecast expresses the relative accuracy of a set of forecasts compared to a reference forecast. Often the value of a given measure in itself is hard to assess but by referencing to another forecast the relative improvement is found which is easier to evaluate.

Skill scores are commonly used in this assessment and are formulated: $$ SS_{ref}=\dfrac{A - A_{ref}}{A_{perf} - A_{ref}}\cdot100\% $$ Where A is the accuracy of the forecast, Aref the reference and Aperf the value a forecast would take if it was perfect. The skill score therefore measure the improvement over a reference forecast compared to that of a perfect forecast set. Any quality measure can be used as accuracy, both goodness of fit measures and quality attributes for the joint distributions.

• \(if A = A_{ref} then SS_{ref} = 0\%\)
• \(if A = A_{perf} then SS_{ref} = 100\%\)
• \(if A < A_{ref} then SS_{ref} < 0\%\)

The reference forecast could for example be persistency or the climatology estimated from observations. It might be of interest to calculate how much a set of forecasts from a recalibrated model compare to the old model calibration using this as reference.

The Skill Score tool will generate a result spreadsheet containing skill score results for each station (Model object and model object variable).

Skill scores can be deriving from the forecast attributes in a contingency table making the attributes relative to a reference. A further advantage is that the approach can be extended to multi categorical contingency tables if accuracy measures are considered and a single scalar value thereby can represent the accuracy of the complete sample joint probability. By accuracy we mean comparing the correct against the incorrect forecasts disregarding the type of error.

The skill scores Heidke skill score(HSS) and Peirce skill score(PSS) are commonly used but do not consider the rarity of the event therefore it is recommended to use the Gerrity skill score(GS). The forecast categories need to be ordinal as the GS rewards correct forecast of rare events higher than for an event that often occur and also the distance between forecast and observed category. For a flood forecasting where the categories range from high to low a flood warning issued as high is penalized harder if the following observed event is low than if it was medium.

Tool info

Tool Info
Display Name	Skill Scores
API Name	Skill Scores
Tools Explorer	/Statistics/Skill Scores
NuGet Package	DHI.MikeOperations.ScenarioManager.Tools.RStatistics
Assembly name (.dll)	DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.dll
API Reference	DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.IRStatisticsSkillScoresTool
Input Items	No input items required
Output Items	A data table

Tool Properties

Display Name	API Name	Description
Options
Action	Action	Gets or sets the action to perform. Execute - Gets the analysis results and returns them in the output items. DataCollection - Collects data for the data analysis. Analysis - Runs the data Analysis using the R-script.
Data Collection
From Date	FromDate	Gets or sets the time of forecast date from where simulations of the data collection scenario are taken. Note that the property is only available when: Action=Analysis,DataCollection
Mask	Mask	Gets or sets the mask for finding output time series for mapping. If the mask is not specified, no time series is found for mapping. Regular expressions are used for finding matching output time series. Note that the property is only available when: Action=DataCollectionUpdateMappingSheet=True
Period Interval	CollectionPeriodInterval	Gets or sets the time period used when collecting data. This interval can only be specified when new data collection spreadsheets are created. For existing spreadsheets, the collection interval is specified by the spreadsheet. Seconds - Time period Seconds. Minutes - Time period Minutes. Hours - Time period Hours. Days - Time period Days. Months - Time period Months. Years - Time period Years. . Note that the property is only available when: Action=DataCollection
Period Length	CollectionPeriodLength	Gets or sets the length of the period interval used when collection data. This interval can only be specified when new data collection spreadsheets are created. For existing spreadsheets, the collection interval is specified by the spreadsheet. Note that the property is only available when: Action=DataCollection
Scenario Path	DataCollectionScenarioPath	Gets or sets the path of the scenario for collecting data. Note that the property is only available when: Action=DataCollection
Spreadsheet Path	DataCollectionSpreadsheetPath	Gets or sets the full path to the data collection spreadsheet. Note that the property is only available when: Action=Analysis,DataCollection
To Date	ToDate	Gets or sets the time of forecast date to where simulations of the data collection scenario are taken. Note that the property is only available when: Action=Analysis,DataCollection
Update Mapping Sheet	UpdateMappingSheet	Gets or sets a value indicating whether the mapping sheet should be updated, so that a row in the mapping sheet and a data collection sheet is present for all output time series of the scenario. Note that the property is only available when: Action=DataCollection
Results
Crossing Time Tolerance	CrossingTimeTolerance	Gets or sets the crossing time toloerance. Note that the property is only available when: Action=Analysis
Max Lead Time Steps	MaxLeadTimeSteps	Gets or sets the max lead time steps. Note that the property is only available when: Action=Analysis
Model Object Name	ModelObjectName	Gets or sets the model object name to get results for. Note that the property is only available when: Action=Execute
Model Object Variable	ModelObjectVariable	Gets or sets the model object variable to get results for. Note that the property is only available when: Action=Execute
Spreadsheet Path	ResultSpreadsheetPath	Gets or sets the full path of the spreadsheet for the analysis results. Note that the property is only available when: Action=Analysis,Execute

Tool Methods

• GetAnalysisMeasures(): Gets an array of measures supported by the analysis.

Code Sample

IronPythonCPythonC#

import clr
clr.AddReference("DHI.Solutions.ScenarioManager.Tools.RStatisticsTool")
from DHI.Solutions.ScenarioManager.Tools.RStatisticsTool import *

# Get the tool.
tool = app.Tools.CreateNew("Skill Scores")
# Add input items.
tool.InputItems.Add(<No input items required>)
# Set tool properties before executing the tool.
tool.Execute()
# Read the output from the list of output items.
output = tool.OutputItems

import clr
clr.AddReference("DHI.Solutions.ScenarioManager.Tools.RStatisticsTool")
from DHI.Solutions.ScenarioManager.Tools.RStatisticsTool import *

# Get the tool.
tool = DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.IRStatisticsSkillScoresTool(app.Tools.CreateNew("Skill Scores"))
# Add input items.
tool.InputItems.Add(<No input items required>)
# Set tool properties before executing the tool.
tool.Execute()
# Read the output from the list of output items.
output = tool.OutputItems

// Get the tool.
var tool = application.Tools.CreateNew("Skill Scores") as DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.IRStatisticsSkillScoresTool;
// Add input items.
tool.InputItems.Add(<No input items required>);
// Set tool properties before executing the tool.
tool.Execute();
// Read the output from the list of output items.
output = tool.OutputItems;

Runtime.config

What tools to load during application startup, is specified in the XML file Runtime.config in the plugins section of the manager.

The XML below shows the tool plugin information found in the Runtime.config file.

<Plugin
  Name="DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.RStatisticsSkillScoresTool"
  Type="DHI.Solutions.Generic.ITool"
  Assembly="DHI.Solutions.ScenarioManager.Tools.RStatisticsTool.dll" />

Requirements

The Skill scores tool is based on R-scripts and requires R to be installed. The R installer can be downloaded from CRAN website. Packages used in the scripts (i.e., verification, abind) can be found on the CRAN website

The tools will support data collection, writing input files for the scripts, executing the scripts and parsing the results into MIKE Workbench entities.

Note

The R tools described in this chapter, are supported using R libraries before R version 3.5.

Analysis

Running Action=Analysis requires the following properties.

Analysis will run time R package verification and create or update the results spreadsheet specified.

• Spreadsheet Path
  The full path to the data collection spreadsheet.
• From Date
  Simulations with time of forecast between From Date and To Date are collected. If left blank all simulations are be collected.
• To Date
  Simulations with time of forecast between From Date and To Date are collected. If left blank all simulations are be collected.
• Results Spreadsheet Path
  The full path of the results spreadsheet.
• Crossing Time Tolerance
  Maximum time steps allowed between observed and forecasted crossing of warning threshold.
  If exceeded forecasted warning level is considered a miss.
• Max Lead Time Steps
  Time steps after the maximum lead time step value are not included in calculations. Default -1 means that all time steps are included.

The result spreadsheet will contain a sheet per station. The columns will contain the results for each skill score, the rows will contain the thresholds.

Execution

Action=Execute will simply list the content of the spreadsheet in a table.

Specify the result spreadsheet and the model object name and variable of the station to display.

The table below shows the available skill scores. Please refer to the CRAN package verification for more information.

• PercentCorrect
  The percent correct is the percent of forecasts that are correct.
• BiasScore
  Bias score is the bias in frequency between forecasted and observed events.
• CriticalSuccessIndex
  The Critical Success Index (CSI) is also called the Threat Score.
  Its range is 0 to 1, with a value of 1 indicating perfect forecast. The CSI is relatively frequently used, with good reason. Unlike the POD and the FAR, it takes into account both false alarms and missed events, and is therefore a more balanced score. The CSI is somewhat sensitive to the climatology of the event, tending to give poorer scores for rare events.
• HeidkeSkillScore
  The Heidke Skill score (HSS) is in the usual skill score format.
  Skill = (score value – score for reference forecast) / (perfect score – score for reference forecast)
  For the HSS, the score" is the number correct or the proportion correct. The "reference forecast" is usually the number correct by chance or the proportion correct by chance.
  The HSS measures the fractional improvement of the forecast over the reference forecast. Like most skill scores, it is normalized by the total range of possible improvement over the reference, which means Heidke Skill scores can safely be compared on different datasets. The range of the HSS is -∞ to 1. Negative values indicate that the chance forecast is better, 0 means no skill, and a perfect forecast obtains a HSS of 1.
  The HSS is a popular score, partly because it is relatively easy to compute and perhaps also because the standard forecast, chance, is relatively easy to beat.
• PeirceSkillScore
  Peirce skill score (PSS) is similar to Heidke Skill score (HSS), except the reference hit rate in the denominator is random and unbiased forecasts.
• GerrityScore
  The Gerrity skill score (GSS) is a weighted sum of elements in the contingency table of possible combinations of the forecast and observed categories, where the a penalty matrix favor forecasts closer to the observed categories and reward forecasts of rare events. $$ GSS = \sum_{i=1}^k\sum_{j=1}^k P_{ij} S_{ij} $$ Here P are the entrances in the contingency table and S a penalty matrix.
  The GSS takes values between -infinity and 1, where values over 0 indicate that the forecast model is more skillful than climatology and a value of 1 indicates a perfect forecast.
• PercentCorrectCategory
  Percent correct by category (vector).
• ProbabilityOfDetection
  Probability of Detection (POD) is also called the hit score (H). The range of H is 0 to 1 and the score is positively oriented. A perfect score is 1. Since the formula for H contains reference to misses and not to false alarms, the hit rate is sensitive only to missed events rather than false alarms. This means that one can always improve the hit rate by forecasting the event more often, which usually results in higher false alarms and, especially for rare events, is likely to result in an over-forecasting bias.
• ProbabilityOfFalseDetection
  Probability of False Detection.
• FalseAlarmRatio
  False Alarm Ratio (FAR) is the fraction of the forecasts of the event associated with non-occurrences.
  The FAR can be controlled by deliberately under-forecasting the event; such a strategy risks increasing the number of missed events, which is not considered in the FAR. For this reason, the POD and the FAR should both be considered for a better understanding of the performance of the forecast.

It is advisable to use several scores in the assessment as they reveal different aspects of the forecast performance.