Does "feature selection" consider the interdependence between multiple parameters, e.g. quotient, difference, etc., as meaningful combinations for identification?

We want to identify important features in our data set. Using the Statistics and Machine Learning Toolbox R2018a we found suitable functions for feature selection. However, in the available feature selection methods, the following case is in our opinion not considered:

"2 Features A and B are not meaningful for distinguishing the important features individually, but the quotient A / B is very good."

Does feature selection consider the interdependence between multiple parameters, e.g. quotient, difference, etc., as meaningful combinations for identification?

Answer 1

[MathWorks Support Team]

In general we refer to this as the interdependence or joint ability of predictors to influence classification even when individual predictors are not useful. If you want to generate specific additional predictors based on original predictors – such as A./B based on A and B – then you can do this easily at the command line. Possible methods are:

FSCNCA and FSRNCA 

• Methods that select features using neighborhood component analysis (NCA).
• They return a set of weights for each specified feature, and indicate the important features using NCA.

Functions fscnca and fsrnca can detect such interaction or interdependence – the exact form of useful interdependence (A/B, A-B etc.) does not need to be specified. An example to illustrate this concept is attached (see “testfscna” as ”.m” and “.html”). 

FITLM, FITGLM and linear model variants (e.g. FITLME) 

• Methods that accept a formula through which one can specify checking for interactions 

This Example on “Fit Linear Regression Using Specified Model Formula” gives a good overview on the required steps. Additional information on the formula for model specification can be found in the following documentation.Please run the below command in the MALTAB R2018a command window to get the release specific documentation:

 web(fullfile(docroot, 'stats/fitlm.html'))

Decision Trees with interaction test 

• Very good method for detecting bivariate combinations of predictors.

For instance, the following will find powerful predictors in input data X and class labels y:

>> bag = fitcensemble(X, y, 'Method', 'Bag', 'Learner', templateTree('PredictorSelection', 'interaction-curvature'))
>> oobPermutedPredictorImportance(bag)

If you are interested in regression, replace fitcensemble with fitrensemble. You can also learn a model using all variables, take out a specific combination of variables and see how much removing these variables affects the model accuracy. An exhaustive list of variable combinations can be generated using the combnk function.

FSCNCA and FSRNCA with manually augmented features

• Method to manually handle formulas which can specify checking for interactions

Augment the features with a new feature that represents the interaction between other features (e.g. A/B) and pass this augmented set to fscnca to see the importance weight for that new feature.You can use the function x2fx for creating this new augmented matrix, as explainedPlease run the below command in the MALTAB R2018a command window to get the release specific documentation:

 web(fullfile(docroot, 'stats/x2fx.html'))

Note that feature selection only accepts a numeric predictor matrix, it does not accept categorical . If we want to use categorical variables, we can specify them for x2fx and x2fx returns the correct matrix with dummy variables for categorical predictors.Please follow the below link to search for the required information regarding the current release:https://www.mathworks.com/help/