Does the function ClassificationTree.fit automatically prune?

Dear All,

I am currently trying to construct a classification tree for a variable Y using different explanatory variables X. I want to use CART and therefore try to use the function Classification.Tree.fit(Y,X) in MATLAB.

The thing is that my variable Y has two categories, 's' and 'n', where 'n' is very 'rare', meaning only ~5% of data is of this certain class. This means that the majority of the Ys are of the class 's'.

When constructing the tree, I get about 8-10 levels, where the terminal nodes have very few (or not many) predicted observations. Now, let the grown tree be denoted tree, so if I do the following: [~,~,~,bestLevel]=cvLoss(tree,'subtrees','all');

I get that bestLevel is the root (!), meaning every future predicted value would be of just one class... Could it be that my prediction values in X are bad, or am I doing something very wrong here?

I was also wondering: when constructing the initial tree - does the function Classification.Tree.fit() automatically prune the tree to an "optimal size" before returning it, or does it make a big a tree as possible and leaves this to the user to prune afterwards?

[eli seri]

do you know which pruning algorithm is being used (Cost Complexity Pruning, Rule Post Pruning, Pessimist Error Pruning, Error Bases Pruning,....)

Answer 1

[Ilya]

I described strategies for learning on imbalanced data in this post http://www.mathworks.com/matlabcentral/answers/11549-leraning-classification-with-most-training-samples-in-one-category The easiest thing to do is set 'prior' to 'uniform'.

The optimal pruning level could be equal to the largest pruning level for some data. This is not necessarily an indication that something went wrong.

Take a look at 'MergeLeaves' and 'Prune' parameters in the doc for ClassificationTree.fit. The doc for 'Prune' says that ClassificationTree computes the optimal sequence of pruned subtrees. The tree is not pruned; just the optimal sequence is computed. The doc for 'MergeLeaves' says that ClassificationTree merges leaves that originate from the same parent node, and that give a sum of risk values greater or equal to the risk associated with the parent node. That is, ClassificationTree applies a minimal amount of pruning, just for the leaves. If the tree prunes by classification error (default), this amounts to merging leaves that share the most popular class per leaf.

[Niklas Axelsson]

Thanks for the answer! That makes sense. I am still not sure why the pruning algorithm decides to get rid of all the branches and just leave the root, very weird.

At the moment I am just creating my tree using: tree = ClassificationTree.fit(Y,X,'PredictorNames',{'...'})

I do want to use CART. Do you think it will make a difference of using 'fitensemble' or maybe 'TreeBagger' instead? And what is the difference? Is it better?

[Ilya]

Please refer to the doc for Statistics Toolbox for a description of fitensemble and TreeBagger. The User Guide section should answer your question.

Also, if you found my answer useful, please accept it.

[Michael]

So would there be anyway to have the learners in fitensemble use pruned trees? It doesn't seem like there is to me.

[Ilya]

You don't want to do that. The predictive power of an ensemble comes from collaboration of its weak learners (trees), not so much from the accuracy of individual learners. Pruning trees in an ensemble only tends to increase the training time without improving the ensemble accuracy.

But if you want to do this nevertheless, you can. Take a look at the doc page for fitensemble. Click on the learners link to navigate to the respective section of the page. Then click on one of the template methods. This brings you to this page. Make a template using

tmp = ClassificationTree.template('prune','on','mergeleaves','on')

and pass that template to fitensemble.

By the way, when you ask a new question, it is best to start a new thread.

[Andrew]

Is this really true? It seems to me the "MergeLeaves" functionality is not doing the same thing as Pruning. Nor does specifying pruning as "on" actually prune the tree: it merely computes the PruneList. Somewhat telling is that your proposed solution does not allow the user to set the prune level in any way. I tried this suggestion and did not get significantly smaller trees using it.

So I have not yet found a way to actually get a bag of pruned trees, either at creation time or after the fact by calling "prune". Your comment about this being unnecessary to improve the ensemble accuracy notwithstanding, perhaps a reason one would want this functionality (as do I) is to create a smaller ensemble with similar performance to the original. Note that I am NOT talking about an ensemble with fewer weak learners in this case, but one in which the individual weak learners' sizes can be pruned, not just limited via MinLeaf/MinParent, which is a different approach.

So it would be nice if the prune method was supported, both on an ensemble and on individual CompactClassificationTrees. (One can produce the latter with a PruneList, using 'prune' set to 'on' as you suggested, so why not provide the capability to actually do the pruning?)