# regularize

Find weights to minimize resubstitution error plus penalty term

## Syntax

```ens1 = regularize(ens) ens1 = regularize(ens,Name,Value) ```

## Description

`ens1 = regularize(ens)` finds optimal weights for learners in `ens` by lasso regularization. `regularize` returns a regression ensemble identical to `ens`, but with a populated `Regularization` property.

`ens1 = regularize(ens,Name,Value)` computes optimal weights with additional options specified by one or more `Name,Value` pair arguments. You can specify several name-value pair arguments in any order as `Name1,Value1,…,NameN,ValueN`.

## Input Arguments

 `ens` A regression ensemble, created by `fitrensemble`.

### Name-Value Pair Arguments

Specify optional comma-separated pairs of `Name,Value` arguments. `Name` is the argument name and `Value` is the corresponding value. `Name` must appear inside quotes. You can specify several name and value pair arguments in any order as `Name1,Value1,...,NameN,ValueN`.

 `'lambda'` Vector of nonnegative regularization parameter values for lasso. For the default setting of `lambda`, `regularize` calculates the smallest value `lambda_max` for which all optimal weights for learners are `0`. The default value of `lambda` is a vector including `0` and nine exponentially-spaced numbers from `lambda_max/1000` to `lambda_max`. Default: `[0 logspace(log10(lambda_max/1000),log10(lambda_max),9)]` `'MaxIter'` Maximum number of iterations allowed, specified as a positive integer. If the algorithm executes `MaxIter` iterations before reaching the convergence tolerance, then the function stops iterating and returns a warning message. The function can return more than one warning when either `npass` or the number of `lambda` values is greater than 1. Default: `1e3` `'npass'` Maximal number of passes for lasso optimization, a positive integer. Default: `10` `'reltol'` Relative tolerance on the regularized loss for lasso, a numeric positive scalar. Default: `1e-3` `'verbose'` Verbosity level, either `0` or `1`. When set to `1`, `regularize` displays more information as it runs. Default: `0`

## Output Arguments

 `ens1` A regression ensemble. Usually you set `ens1` to the same name as `ens`.

## Examples

expand all

Regularize an ensemble of bagged trees.

Generate sample data.

```rng(10,'twister') % For reproducibility X = rand(2000,20); Y = repmat(-1,2000,1); Y(sum(X(:,1:5),2)>2.5) = 1;```

You can create a bagged classification ensemble of 300 trees from the sample data.

```bag = fitrensemble(X,Y,'Method','Bag','NumLearningCycles',300); ```

`fitrensemble` uses a default template tree object `templateTree()` as a weak learner when `'Method'` is `'Bag'`. In this example, for reproducibility, specify `'Reproducible',true` when you create a tree template object, and then use the object as a weak learner.

```t = templateTree('Reproducible',true); % For reproducibiliy of random predictor selections bag = fitrensemble(X,Y,'Method','Bag','NumLearningCycles',300,'Learners',t);```

Regularize the ensemble of bagged regression trees.

`bag = regularize(bag,'lambda',[0.001 0.1],'verbose',1);`
```Starting lasso minimization for Lambda=0.001. Initial MSE=0.109923. Lasso minimization completed pass 1 for Lambda=0.001 MSE = 0.086912 Relative change in MSE = 0.264768 Number of learners with non-zero weights = 15 Lasso minimization completed pass 2 for Lambda=0.001 MSE = 0.0670602 Relative change in MSE = 0.296029 Number of learners with non-zero weights = 34 Lasso minimization completed pass 3 for Lambda=0.001 MSE = 0.0623931 Relative change in MSE = 0.0748019 Number of learners with non-zero weights = 51 Lasso minimization completed pass 4 for Lambda=0.001 MSE = 0.0605444 Relative change in MSE = 0.0305348 Number of learners with non-zero weights = 70 Lasso minimization completed pass 5 for Lambda=0.001 MSE = 0.0599666 Relative change in MSE = 0.00963517 Number of learners with non-zero weights = 94 Lasso minimization completed pass 6 for Lambda=0.001 MSE = 0.0598835 Relative change in MSE = 0.00138719 Number of learners with non-zero weights = 105 Lasso minimization completed pass 7 for Lambda=0.001 MSE = 0.0598608 Relative change in MSE = 0.000379227 Number of learners with non-zero weights = 113 Lasso minimization completed pass 8 for Lambda=0.001 MSE = 0.0598586 Relative change in MSE = 3.72856e-05 Number of learners with non-zero weights = 115 Lasso minimization completed pass 9 for Lambda=0.001 MSE = 0.0598587 Relative change in MSE = 6.42954e-07 Number of learners with non-zero weights = 115 Lasso minimization completed pass 10 for Lambda=0.001 MSE = 0.0598587 Relative change in MSE = 4.53658e-08 Number of learners with non-zero weights = 115 Completed lasso minimization for Lambda=0.001. Resubstitution MSE changed from 0.109923 to 0.0598587. Number of learners reduced from 300 to 115. Starting lasso minimization for Lambda=0.1. Initial MSE=0.109923. Lasso minimization completed pass 1 for Lambda=0.1 MSE = 0.104917 Relative change in MSE = 0.0477191 Number of learners with non-zero weights = 12 Lasso minimization completed pass 2 for Lambda=0.1 MSE = 0.0851031 Relative change in MSE = 0.232821 Number of learners with non-zero weights = 30 Lasso minimization completed pass 3 for Lambda=0.1 MSE = 0.081245 Relative change in MSE = 0.0474877 Number of learners with non-zero weights = 40 Lasso minimization completed pass 4 for Lambda=0.1 MSE = 0.0796749 Relative change in MSE = 0.0197067 Number of learners with non-zero weights = 53 Lasso minimization completed pass 5 for Lambda=0.1 MSE = 0.0788411 Relative change in MSE = 0.0105746 Number of learners with non-zero weights = 64 Lasso minimization completed pass 6 for Lambda=0.1 MSE = 0.0784959 Relative change in MSE = 0.00439793 Number of learners with non-zero weights = 81 Lasso minimization completed pass 7 for Lambda=0.1 MSE = 0.0784429 Relative change in MSE = 0.000676468 Number of learners with non-zero weights = 88 Lasso minimization completed pass 8 for Lambda=0.1 MSE = 0.078447 Relative change in MSE = 5.24449e-05 Number of learners with non-zero weights = 88 Completed lasso minimization for Lambda=0.1. Resubstitution MSE changed from 0.109923 to 0.078447. Number of learners reduced from 300 to 88. ```

`regularize` reports on its progress.

Inspect the resulting regularization structure.

`bag.Regularization`
```ans = struct with fields: Method: 'Lasso' TrainedWeights: [300x2 double] Lambda: [1.0000e-03 0.1000] ResubstitutionMSE: [0.0599 0.0784] CombineWeights: @classreg.learning.combiner.WeightedSum ```

Check how many learners in the regularized ensemble have positive weights. These are the learners included in a shrunken ensemble.

`sum(bag.Regularization.TrainedWeights > 0)`
```ans = 1×2 115 88 ```

Shrink the ensemble using the weights from `Lambda = 0.1`.

`cmp = shrink(bag,'weightcolumn',2)`
```cmp = CompactRegressionEnsemble ResponseName: 'Y' CategoricalPredictors: [] ResponseTransform: 'none' NumTrained: 88 Properties, Methods ```

The compact ensemble contains `87` members, less than 1/3 of the original `300`.

expand all