CompactClassificationSVM

Compact support vector machine (SVM) for one-class and binary classification

Description

CompactClassificationSVM is a compact version of the support vector machine (SVM) classifier. The compact classifier does not include the data used for training the SVM classifier. Therefore, you cannot perform some tasks, such as cross-validation, using the compact classifier. Use a compact SVM classifier for tasks such as predicting the labels of new data.

Creation

Create a CompactClassificationSVM model from a full, trained ClassificationSVM classifier by using compact.

Properties

SVM Properties

Alpha — Trained classifier coefficients

numeric vector

This property is read-only.

Trained classifier coefficients, specified as an s-by-1 numeric vector. s is the number of support vectors in the trained classifier, sum(Mdl.IsSupportVector).

Alpha contains the trained classifier coefficients from the dual problem, that is, the estimated Lagrange multipliers. If you remove duplicates by using the RemoveDuplicates name-value pair argument of fitcsvm, then for a given set of duplicate observations that are support vectors, Alpha contains one coefficient corresponding to the entire set. That is, MATLAB^® attributes a nonzero coefficient to one observation from the set of duplicates and a coefficient of 0 to all other duplicate observations in the set.

Data Types: single | double

Beta — Linear predictor coefficients

numeric vector

This property is read-only.

Linear predictor coefficients, specified as a numeric vector. The length of Beta is equal to the number of predictors used to train the model.

MATLAB expands categorical variables in the predictor data using full dummy encoding. That is, MATLAB creates one dummy variable for each level of each categorical variable. Beta stores one value for each predictor variable, including the dummy variables. For example, if there are three predictors, one of which is a categorical variable with three levels, then Beta is a numeric vector containing five values.

If KernelParameters.Function is 'linear', then the classification score for the observation x is

$$f\left(x\right)=\left(x/s\right)\prime \beta +b.$$

Mdl stores β, b, and s in the properties Beta, Bias, and KernelParameters.Scale, respectively.

To estimate classification scores manually, you must first apply any transformations to the predictor data that were applied during training. Specifically, if you specify 'Standardize',true when using fitcsvm, then you must standardize the predictor data manually by using the mean Mdl.Mu and standard deviation Mdl.Sigma, and then divide the result by the kernel scale in Mdl.KernelParameters.Scale.

All SVM functions, such as resubPredict and predict, apply any required transformation before estimation.

If KernelParameters.Function is not 'linear', then Beta is empty ([]).

Data Types: single | double

Bias — Bias term

scalar

This property is read-only.

Bias term, specified as a scalar.

Data Types: single | double

KernelParameters — Kernel parameters

structure array

This property is read-only.

Kernel parameters, specified as a structure array. The kernel parameters property contains the fields listed in this table.

Field	Description
Function	Kernel function used to compute the elements of the Gram matrix. For details, see 'KernelFunction'.
Scale	Kernel scale parameter used to scale all elements of the predictor data on which the model is trained. For details, see 'KernelScale'.

To display the values of KernelParameters, use dot notation. For example, Mdl.KernelParameters.Scale displays the kernel scale parameter value.

The software accepts KernelParameters as inputs and does not modify them.

Data Types: struct

SupportVectorLabels — Support vector class labels

s-by-1 numeric vector

This property is read-only.

Support vector class labels, specified as an s-by-1 numeric vector. s is the number of support vectors in the trained classifier, sum(Mdl.IsSupportVector).

A value of +1 in SupportVectorLabels indicates that the corresponding support vector is in the positive class (ClassNames{2}). A value of –1 indicates that the corresponding support vector is in the negative class (ClassNames{1}).

If you remove duplicates by using the RemoveDuplicates name-value pair argument of fitcsvm, then for a given set of duplicate observations that are support vectors, SupportVectorLabels contains one unique support vector label.

Data Types: single | double

SupportVectors — Support vectors

s-by-p numeric matrix

This property is read-only.

Support vectors in the trained classifier, specified as an s-by-p numeric matrix. s is the number of support vectors in the trained classifier, sum(Mdl.IsSupportVector), and p is the number of predictor variables in the predictor data.

SupportVectors contains rows of the predictor data X that MATLAB considers to be support vectors. If you specify 'Standardize',true when training the SVM classifier using fitcsvm, then SupportVectors contains the standardized rows of X.

If you remove duplicates by using the RemoveDuplicates name-value pair argument of fitcsvm, then for a given set of duplicate observations that are support vectors, SupportVectors contains one unique support vector.

Data Types: single | double

Other Classification Properties

CategoricalPredictors — Categorical predictor indices

vector of positive integers | []

This property is read-only.

Categorical predictor indices, specified as a vector of positive integers. CategoricalPredictors contains index values indicating that the corresponding predictors are categorical. The index values are between 1 and p, where p is the number of predictors used to train the model. If none of the predictors are categorical, then this property is empty ([]).

Data Types: double

ClassNames — Unique class labels

categorical array | character array | logical vector | numeric vector | cell array of character vectors

This property is read-only.

Unique class labels used in training, specified as a categorical or character array, logical or numeric vector, or cell array of character vectors. ClassNames has the same data type as the class labels Y. (The software treats string arrays as cell arrays of character vectors.) ClassNames also determines the class order.

Data Types: single | double | logical | char | cell | categorical

Cost — Misclassification cost

numeric square matrix

This property is read-only.

Misclassification cost, specified as a numeric square matrix.

• For two-class learning, the Cost property stores the misclassification cost matrix specified by the Cost name-value argument of the fitting function. The rows correspond to the true class and the columns correspond to the predicted class. That is, Cost(i,j) is the cost of classifying a point into class j if its true class is i. The order of the rows and columns of Cost corresponds to the order of the classes in ClassNames.

• For one-class learning, Cost = 0.

Data Types: double

ExpandedPredictorNames — Expanded predictor names

cell array of character vectors

This property is read-only.

Expanded predictor names, specified as a cell array of character vectors.

If the model uses dummy variable encoding for categorical variables, then ExpandedPredictorNames includes the names that describe the expanded variables. Otherwise, ExpandedPredictorNames is the same as PredictorNames.

Data Types: cell

Mu — Predictor means

numeric vector | []

This property is read-only.

Predictor means, specified as a numeric vector. If you specify 'Standardize',1 or 'Standardize',true when you train an SVM classifier using fitcsvm, the length of Mu is equal to the number of predictors.

MATLAB expands categorical variables in the predictor data using dummy variables. Mu stores one value for each predictor variable, including the dummy variables. However, MATLAB does not standardize the columns that contain categorical variables.

If you set 'Standardize',false when you train the SVM classifier using fitcsvm, then Mu is an empty vector ([]).

Data Types: single | double

PredictorNames — Predictor variable names

cell array of character vectors

This property is read-only.

Predictor variable names, specified as a cell array of character vectors. The order of the elements in PredictorNames corresponds to the order in which the predictor names appear in the training data.

Data Types: cell

Prior — Prior probabilities

numeric vector

This property is read-only.

Prior probabilities for each class, specified as a numeric vector.

For two-class learning, if you specify a cost matrix, then the software updates the prior probabilities by incorporating the penalties described in the cost matrix.

• For two-class learning, the software normalizes the prior probabilities specified by the Prior name-value argument of the fitting function so that the probabilities sum to 1. The Prior property stores the normalized prior probabilities. The order of the elements of Prior corresponds to the elements of Mdl.ClassNames.

• For one-class learning, Prior = 1.

Data Types: single | double

ScoreTransform — Score transformation

character vector | function handle

Score transformation, specified as a character vector or function handle. ScoreTransform represents a built-in transformation function or a function handle for transforming predicted classification scores.

To change the score transformation function to function, for example, use dot notation.

• For a built-in function, enter a character vector.

  Mdl.ScoreTransform = 'function';

  This table describes the available built-in functions.

  Value	Description
  'doublelogit'	1/(1 + e–2x)
  'invlogit'	log(x / (1 – x))
  'ismax'	Sets the score for the class with the largest score to 1, and sets the scores for all other classes to 0
  'logit'	1/(1 + e–x)
  'none' or 'identity'	x (no transformation)
  'sign'	–1 for x < 0 0 for x = 0 1 for x > 0
  'symmetric'	2x – 1
  'symmetricismax'	Sets the score for the class with the largest score to 1, and sets the scores for all other classes to –1
  'symmetriclogit'	2/(1 + e–x) – 1

• For a MATLAB function or a function that you define, enter its function handle.

  Mdl.ScoreTransform = @function;

  function must accept a matrix (the original scores) and return a matrix of the same size (the transformed scores).

Data Types: char | function_handle

Sigma — Predictor standard deviations

[] (default) | numeric vector

This property is read-only.

Predictor standard deviations, specified as a numeric vector.

If you specify 'Standardize',true when you train the SVM classifier using fitcsvm, the length of Sigma is equal to the number of predictor variables.

MATLAB expands categorical variables in the predictor data using dummy variables. Sigma stores one value for each predictor variable, including the dummy variables. However, MATLAB does not standardize the columns that contain categorical variables.

If you set 'Standardize',false when you train the SVM classifier using fitcsvm, then Sigma is an empty vector ([]).

Data Types: single | double

Object Functions

compareHoldout	Compare accuracies of two classification models using new data
discardSupportVectors	Discard support vectors for linear support vector machine (SVM) classifier
edge	Find classification edge for support vector machine (SVM) classifier
fitPosterior	Fit posterior probabilities for compact support vector machine (SVM) classifier
gather	Gather properties of Statistics and Machine Learning Toolbox object from GPU
incrementalLearner	Convert binary classification support vector machine (SVM) model to incremental learner
lime	Local interpretable model-agnostic explanations (LIME)
loss	Find classification error for support vector machine (SVM) classifier
margin	Find classification margins for support vector machine (SVM) classifier
partialDependence	Compute partial dependence
plotPartialDependence	Create partial dependence plot (PDP) and individual conditional expectation (ICE) plots
predict	Classify observations using support vector machine (SVM) classifier
shapley	Shapley values
update	Update model parameters for code generation

Examples

Reduce Size of SVM Classifier

Reduce the size of a full support vector machine (SVM) classifier by removing the training data. Full SVM classifiers (that is, ClassificationSVM classifiers) hold the training data. To improve efficiency, use a smaller classifier.

Load the ionosphere data set.

load ionosphere

Train an SVM classifier. Standardize the predictor data and specify the order of the classes.

SVMModel = fitcsvm(X,Y,'Standardize',true,...
    'ClassNames',{'b','g'})

SVMModel = 
  ClassificationSVM
             ResponseName: 'Y'
    CategoricalPredictors: []
               ClassNames: {'b'  'g'}
           ScoreTransform: 'none'
          NumObservations: 351
                    Alpha: [90x1 double]
                     Bias: -0.1343
         KernelParameters: [1x1 struct]
                       Mu: [0.8917 0 0.6413 0.0444 0.6011 0.1159 0.5501 0.1194 0.5118 0.1813 0.4762 0.1550 0.4008 0.0934 0.3442 0.0711 0.3819 -0.0036 0.3594 -0.0240 0.3367 0.0083 0.3625 -0.0574 0.3961 -0.0712 0.5416 -0.0695 0.3784 ... ] (1x34 double)
                    Sigma: [0.3112 0 0.4977 0.4414 0.5199 0.4608 0.4927 0.5207 0.5071 0.4839 0.5635 0.4948 0.6222 0.4949 0.6528 0.4584 0.6180 0.4968 0.6263 0.5191 0.6098 0.5182 0.6038 0.5275 0.5785 0.5085 0.5162 0.5500 0.5759 0.5080 ... ] (1x34 double)
           BoxConstraints: [351x1 double]
          ConvergenceInfo: [1x1 struct]
          IsSupportVector: [351x1 logical]
                   Solver: 'SMO'

SVMModel is a ClassificationSVM classifier.

Reduce the size of the SVM classifier.

CompactSVMModel = compact(SVMModel)

CompactSVMModel = 
  CompactClassificationSVM
             ResponseName: 'Y'
    CategoricalPredictors: []
               ClassNames: {'b'  'g'}
           ScoreTransform: 'none'
                    Alpha: [90x1 double]
                     Bias: -0.1343
         KernelParameters: [1x1 struct]
                       Mu: [0.8917 0 0.6413 0.0444 0.6011 0.1159 0.5501 0.1194 0.5118 0.1813 0.4762 0.1550 0.4008 0.0934 0.3442 0.0711 0.3819 -0.0036 0.3594 -0.0240 0.3367 0.0083 0.3625 -0.0574 0.3961 -0.0712 0.5416 -0.0695 0.3784 ... ] (1x34 double)
                    Sigma: [0.3112 0 0.4977 0.4414 0.5199 0.4608 0.4927 0.5207 0.5071 0.4839 0.5635 0.4948 0.6222 0.4949 0.6528 0.4584 0.6180 0.4968 0.6263 0.5191 0.6098 0.5182 0.6038 0.5275 0.5785 0.5085 0.5162 0.5500 0.5759 0.5080 ... ] (1x34 double)
           SupportVectors: [90x34 double]
      SupportVectorLabels: [90x1 double]

CompactSVMModel is a CompactClassificationSVM classifier.

Display the amount of memory used by each classifier.

whos('SVMModel','CompactSVMModel')

  Name                 Size             Bytes  Class                                                 Attributes

  CompactSVMModel      1x1              31227  classreg.learning.classif.CompactClassificationSVM              
  SVMModel             1x1             141317  ClassificationSVM                                               

The full SVM classifier (SVMModel) is more than four times larger than the compact SVM classifier (CompactSVMModel).

To label new observations efficiently, you can remove SVMModel from the MATLAB® Workspace, and then pass CompactSVMModel and new predictor values to predict.

To further reduce the size of the compact SVM classifier, use the discardSupportVectors function to discard support vectors.

Train and Cross-Validate SVM Classifier

Load the ionosphere data set.

load ionosphere

Train and cross-validate an SVM classifier. Standardize the predictor data and specify the order of the classes.

rng(1);  % For reproducibility
CVSVMModel = fitcsvm(X,Y,'Standardize',true,...
    'ClassNames',{'b','g'},'CrossVal','on')

CVSVMModel = 
  ClassificationPartitionedModel
    CrossValidatedModel: 'SVM'
         PredictorNames: {'x1'  'x2'  'x3'  'x4'  'x5'  'x6'  'x7'  'x8'  'x9'  'x10'  'x11'  'x12'  'x13'  'x14'  'x15'  'x16'  'x17'  'x18'  'x19'  'x20'  'x21'  'x22'  'x23'  'x24'  'x25'  'x26'  'x27'  'x28'  'x29'  'x30'  'x31'  'x32'  'x33'  'x34'}
           ResponseName: 'Y'
        NumObservations: 351
                  KFold: 10
              Partition: [1x1 cvpartition]
             ClassNames: {'b'  'g'}
         ScoreTransform: 'none'

CVSVMModel is a ClassificationPartitionedModel cross-validated SVM classifier. By default, the software implements 10-fold cross-validation.

Alternatively, you can cross-validate a trained ClassificationSVM classifier by passing it to crossval.

Inspect one of the trained folds using dot notation.

CVSVMModel.Trained{1}

ans = 
  CompactClassificationSVM
             ResponseName: 'Y'
    CategoricalPredictors: []
               ClassNames: {'b'  'g'}
           ScoreTransform: 'none'
                    Alpha: [78x1 double]
                     Bias: -0.2210
         KernelParameters: [1x1 struct]
                       Mu: [0.8888 0 0.6320 0.0406 0.5931 0.1205 0.5361 0.1286 0.5083 0.1879 0.4779 0.1567 0.3924 0.0875 0.3360 0.0789 0.3839 9.6066e-05 0.3562 -0.0308 0.3398 -0.0073 0.3590 -0.0628 0.4064 -0.0664 0.5535 -0.0749 0.3835 ... ] (1x34 double)
                    Sigma: [0.3149 0 0.5033 0.4441 0.5255 0.4663 0.4987 0.5205 0.5040 0.4780 0.5649 0.4896 0.6293 0.4924 0.6606 0.4535 0.6133 0.4878 0.6250 0.5140 0.6075 0.5150 0.6068 0.5222 0.5729 0.5103 0.5061 0.5478 0.5712 0.5032 ... ] (1x34 double)
           SupportVectors: [78x34 double]
      SupportVectorLabels: [78x1 double]

Each fold is a CompactClassificationSVM classifier trained on 90% of the data.

Estimate the generalization error.

genError = kfoldLoss(CVSVMModel)

genError = 0.1168

On average, the generalization error is approximately 12%.

References

[1] Hastie, T., R. Tibshirani, and J. Friedman. The Elements of Statistical Learning, Second Edition. NY: Springer, 2008.

[2] Scholkopf, B., J. C. Platt, J. C. Shawe-Taylor, A. J. Smola, and R. C. Williamson. “Estimating the Support of a High-Dimensional Distribution.” Neural Computation. Vol. 13, Number 7, 2001, pp. 1443–1471.

[3] Christianini, N., and J. C. Shawe-Taylor. An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge, UK: Cambridge University Press, 2000.

[4] Scholkopf, B., and A. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization and Beyond, Adaptive Computation and Machine Learning. Cambridge, MA: The MIT Press, 2002.

Extended Capabilities

C/C++ Code Generation

Generate C and C++ code using MATLAB® Coder™.

Usage notes and limitations:

• The predict and update functions support code generation.

• To integrate the prediction of an SVM classification model into Simulink^®, you can use the ClassificationSVM Predict block in the Statistics and Machine Learning Toolbox™ library or a MATLAB Function block with the predict function.

• When you train an SVM model by using fitcsvm, the following restrictions apply.

  • The value of the 'ScoreTransform' name-value pair argument cannot be an anonymous function. For generating code that predicts posterior probabilities given new observations, pass a trained SVM model to fitPosterior or fitSVMPosterior. The ScoreTransform property of the returned model contains an anonymous function that represents the score-to-posterior-probability function and is configured for code generation.

  • For fixed-point code generation, the value of the 'ScoreTransform' name-value pair argument cannot be 'invlogit'. Also, the value of the 'KernelFunction' name-value pair argument must be 'gaussian', 'linear', or 'polynomial'.

  • For fixed-point code generation and code generation with a coder configurer, the following additional restrictions apply.

    • Categorical predictors (logical, categorical, char, string, or cell) are not supported. You cannot use the CategoricalPredictors name-value argument. To include categorical predictors in a model, preprocess them by using dummyvar before fitting the model.

    • Class labels with the categorical data type are not supported. Both the class label value in the training data (Tbl or Y) and the value of the ClassNames name-value argument cannot be an array with the categorical data type.

For more information, see Introduction to Code Generation.

GPU Arrays

Accelerate code by running on a graphics processing unit (GPU) using Parallel Computing Toolbox™.

Usage notes and limitations:

• The following object functions fully support GPU arrays:

  • discardSupportVectors

  • fitPosterior

  • gather

• The following object functions offer limited support for GPU arrays:

  • compareHoldout

  • edge

  • loss

  • margin

  • partialDependence

  • plotPartialDependence

  • predict

For more information, see Run MATLAB Functions on a GPU (Parallel Computing Toolbox).

Version History

Introduced in R2014a

R2022a: Cost property stores the user-specified cost matrix

Starting in R2022a, the Cost property stores the user-specified cost matrix, so that you can compute the observed misclassification cost using the specified cost value. The software stores normalized prior probabilities (Prior) that do not reflect the penalties described in the cost matrix. To compute the observed misclassification cost, specify the LossFun name-value argument as "classifcost" when you call the loss function.

Note that model training has not changed and, therefore, the decision boundaries between classes have not changed.

For training, the fitting function updates the specified prior probabilities by incorporating the penalties described in the specified cost matrix, and then normalizes the prior probabilities and observation weights. This behavior has not changed. In previous releases, the software stored the default cost matrix in the Cost property and stored the prior probabilities used for training in the Prior property. Starting in R2022a, the software stores the user-specified cost matrix without modification, and stores normalized prior probabilities that do not reflect the cost penalties. For more details, see Misclassification Cost Matrix, Prior Probabilities, and Observation Weights.

Some object functions use the Cost and Prior properties:

• The loss function uses the cost matrix stored in the Cost property if you specify the LossFun name-value argument as "classifcost" or "mincost".

• The loss and edge functions use the prior probabilities stored in the Prior property to normalize the observation weights of the input data.

If you specify a nondefault cost matrix when you train a classification model, the object functions return a different value compared to previous releases.

If you want the software to handle the cost matrix, prior probabilities, and observation weights in the same way as in previous releases, adjust the prior probabilities and observation weights for the nondefault cost matrix, as described in Adjust Prior Probabilities and Observation Weights for Misclassification Cost Matrix. Then, when you train a classification model, specify the adjusted prior probabilities and observation weights by using the Prior and Weights name-value arguments, respectively, and use the default cost matrix.