Compare Alternative Models

When you use MBC Model Fitting app to fit statistical models to experimental data, the app automatically fits models to your data. The default models can usually produce a good fit, but consider assessing fits and exploring alternative models. After assessing the initial model fit, use the app tools to create more models to search for the best fit.

To create alternative models, in the Model Browser model views, use the Common Tasks links. After you create models to compare, you see an Alternative Models list in the model view. To help you assess each model and decide which model to choose as best, use the plots and diagnostic statistics of the Model Browser.

After you use the Model Template dialog box to create child nodes, the toolbox selects the best model from the alternatives (indicated with a blue icon) based on the selection criteria you choose. For local nodes, the best child node of each response feature is chosen.

Assess all the fits in the Alternative Models list in case you want to choose an alternative as a better fit.

Compare Fits Using Model Plots

To compare the plots and statistics and select the best model, select each model in the list of alternative models. To determine the best fit, you examine both the plots and statistics. Start with a visual inspection of the model plots. In the Model Browser, use the Response Model plots at the test plan node. For more information about using plots to compare fits, see Assess High-Level Model Trends.

Compare Fits Using Statistics

Use the summary table to assess the currently selected model. When you have a list of alternative models, use the same statistics in the list to compare the models and select the best.

The summary table can include the following information.

Statistic	Description	Assess Model Fits
Observations	Number of observations used to estimate model.	NA
Parameters	Number of parameters in model. Gaussian process models display the effective number of parameters, so it can be noninteger.	Look for models with fewer parameters than observations. If the quantities are close, this indicates possible overfitting.
PRESS RMSE	Root mean squared error of predicted errors. PRESS RMSE is a measure of the predictive power of your models (for linear models only). The divisor used for PRESS RMSE is the number of observations. The residuals are in untransformed values to enable comparison between alternative models with different Box-Cox transformations.	Look for lower PRESS RMSE values to indicate better fits without overfitting. Compare PRESS RMSE with RMSE. If the value of PRESS RMSE is much bigger than the RMSE, then you are overfitting.
RMSE	Root mean squared error. The divisor used for RMSE is the number of observations minus the number of parameters. The residuals are in untransformed values, to enable comparison between alternative models with different Box-Cox transformations.	Look for lower RMSE values to indicate better fits, but beware of overfitting. See PRESS RMSE.
AICc	Information criteria. Not available for Gaussian process models.	Look for lower values of AICc. Only the difference between the AICc values for two models is meaningful, not the absolute value: if this difference is greater than about 10, then discard the worse model.
Box-Cox	Power transform used for box-cox transformation. 1 indicates no transform. 0 indicates a log transform is used.	1 indicates no transform. 0 indicates a log transform is used.
Validation RMSE (one-stage models only)	Root mean squared error between the one-stage model and the validation data.	Look for lower values of the validation RMSE.

For linear models, use the stepwise functions (choose a Stepwise option during model setup) to refine your models and remove less useful model terms. Make sure that you examine outliers, but do not automatically remove them without good reason.

After you evaluate your models and make your selections, export your models to CAGE for optimized calibration. From the test plan node, click Generate calibration in the Common Tasks pane.

Summary Statistics

To help you evaluate models, you can select additional statistics in the Summary Statistics dialog box. The standard summary statistics are PRESS RMSE (for linear models only) and RMSE.

1. To open the Summary Statistics dialog box:

   • From any global model node, select Model > Summary Statistics.

   • From the test plan, right-click the global model block and select Summary Statistics. If you want the summary statistics to apply to all the models within the test plan, use this option before building models. When you create a child node, the summary statistics inherit the test plan node or the parent node statistics.

2. Choose additional statistics by selecting the check boxes.

3. Click OK. Changes made from a global model node are applied immediately to the Summary table and the Models list pane.

Available Statistics	Description	Assess Model Fits
GCV Weighted PRESS -2LogL R^2 R^2 adj PRESS R^2 DW	Names and formula are in the dialog box.	In general, look for lower values. This table provides general guidance. Statistic Value Model Fit R^2 R^2 adj PRESS R^2 Less than 0.5 Poor Greater than 0.5 and less than 0.9 Moderate Greater than 0.9 and less than 1.0 Good Close to 1.0 Potentially overfit
cond(J)	Condition indicator.	High values (e.g., > 108) can be a sign of numerical instability.
AIC AICc (small sample) BIC	Information criteria. Available for comparison only if the same data is used for all models. Not available for Gaussian process models.	The differences between two information criteria are of interest, not the absolute value. As a rule of thumb, a difference of 2 or less means models are roughly the same quality of fit. All the information criteria impose a penalty for overfitting, which increases linearly with the number of parameters in the model. As a rough guide, typical models favoured by BIC are less complicated than those chosen by AIC or AICc. AICc (small sample) is designed for cases where parameters/observations < 40. Use AIC or AICc, but not both.
Durbin-Watson	Correlation of adjacent observations, usually the residuals of a time series regression.	Value of 2.0 suggests that there is no time correlation.

Note

• For ordinary least squares cases, 'p' is the number of parameters, but for non-ordinary least squares cases (rols and ridge least squares) 'p' is the effective number of parameters (p = N-df).

• The total sum of squares (SST) calculation depends on a model constant. If your model has a constant, the total sum of squares (SST) calculation depends on the mean (SST=sum(y-ymean)^2). If your model does not have a constant, the calculation does not depend on the mean (SST=sum(y)^2).

Other Model Assessment Windows

To view the information listed in the table, you can other model assessment windows.

To	Select
View a read-only version of the inputs, predicted, and actual responses.	View > Modeling Data to open the Data Editor.
Displaying the fit relative to the selected data.	Model > Evaluate to open a wizard that helps you select the data, and then opens the Model Evaluation window.
Plot multiple models on the same plot, or a table view.	Model > Selection Window to open the Model Selection window.