isanomaly
Syntax
Description
finds anomalies in the table tf
= isanomaly(LOFObj
,Tbl
)Tbl
using the LocalOutlierFactor
object LOFObj
and returns the logical array tf
,
whose elements are true
when an anomaly is detected in the corresponding
row of Tbl
. You must use this syntax if you create
LOFObj
by passing a table to the lof
function.
specifies options using one or more name-value arguments in addition to any of the input
argument combinations in the previous syntaxes. For example,
tf
= isanomaly(___,Name=Value
)
causes
scoreThreshold
=0.5isanomaly
to identify observations with scores above
0.5
as anomalies.
[
also returns an anomaly score, which is a local
outlier factor value, for each observation in tf
,scores
] = isanomaly(___)Tbl
or
X
. A score value less than or close to 1 indicates a normal
observation, and a value greater than 1 can indicate an anomaly.
Examples
Detect Novelties
Create a LocalOutlierFactor
object for uncontaminated training observations by using the lof
function. Then detect novelties (anomalies in new data) by passing the object and the new data to the object function isanomaly
.
Load the 1994 census data stored in census1994.mat
. The data set consists of demographic data from the US Census Bureau to predict whether an individual makes over $50,000 per year.
load census1994
census1994
contains the training data set adultdata
and the test data set adulttest
. The predictor data must be either all continuous or all categorical to train a LocalOutlierFactor
object. Remove nonnumeric variables from adultdata
and adulttest
.
adultdata = adultdata(:,vartype("numeric")); adulttest = adulttest(:,vartype("numeric"));
Train a local outlier factor model for adultdata
. Assume that adultdata
does not contain outliers.
[Mdl,tf,s] = lof(adultdata);
Mdl
is a LocalOutlierFactor
object. lof
also returns the anomaly indicators tf
and anomaly scores s
for the training data adultdata
. If you do not specify the ContaminationFraction
name-value argument as a value greater than 0, then lof
treats all training observations as normal observations, meaning all the values in tf
are logical 0 (false
). The function sets the score threshold to the maximum score value. Display the threshold value.
Mdl.ScoreThreshold
ans = 28.6719
Find anomalies in adulttest
by using the trained local outlier factor model.
[tf_test,s_test] = isanomaly(Mdl,adulttest);
The isanomaly
function returns the anomaly indicators tf_test
and scores s_test
for adulttest
. By default, isanomaly
identifies observations with scores above the threshold (Mdl.ScoreThreshold
) as anomalies.
Create histograms for the anomaly scores s
and s_test
. Create a vertical line at the threshold of the anomaly scores.
h1 = histogram(s,NumBins=50,Normalization="probability"); hold on h2 = histogram(s_test,h1.BinEdges,Normalization="probability"); xline(Mdl.ScoreThreshold,"r-",join(["Threshold" Mdl.ScoreThreshold])) h1.Parent.YScale = 'log'; h2.Parent.YScale = 'log'; legend("Training Data","Test Data",Location="north") hold off
Display the observation index of the anomalies in the test data.
find(tf_test)
ans = 0x1 empty double column vector
The anomaly score distribution of the test data is similar to that of the training data, so isanomaly
does not detect any anomalies in the test data with the default threshold value. You can specify a different threshold value by using the ScoreThreshold
name-value argument. For an example, see Specify Anomaly Score Threshold.
Specify Anomaly Score Threshold
Specify the threshold value for anomaly scores by using the ScoreThreshold
name-value argument of isanomaly
.
Load the 1994 census data stored in census1994.mat
. The data set consists of demographic data from the US Census Bureau to predict whether an individual makes over $50,000 per year.
load census1994
census1994
contains the training data set adultdata
and the test data set adulttest
.
Remove nonnumeric variables from adultdata
and adulttest
.
adultdata = adultdata(:,vartype("numeric")); adulttest = adulttest(:,vartype("numeric"));
Train a local outlier factor model for adultdata
.
[Mdl,tf,scores] = lof(adultdata);
Plot a histogram of the score values. Create a vertical line at the default score threshold.
h = histogram(scores,NumBins=50,Normalization="probability"); h.Parent.YScale = 'log'; xline(Mdl.ScoreThreshold,"r-",join(["Threshold" Mdl.ScoreThreshold]))
Find the anomalies in the test data using the trained local outlier factor model. Use a different threshold from the default threshold value obtained when training the local outlier factor model.
First, determine the score threshold by using the isoutlier
function.
[~,~,U] = isoutlier(scores)
U = 1.1567
Specify the value of the ScoreThreshold
name-value argument as U
.
[tf_test,scores_test] = isanomaly(Mdl,adulttest,ScoreThreshold=U); h = histogram(scores_test,NumBins=50,Normalization="probability"); h.Parent.YScale = 'log'; xline(U,"r-",join(["Threshold" U]))
Plot Contours of Anomaly Scores
Generate a sample data set that contains outliers. Compute anomaly scores for the points around the sample data by using the isanomaly
function, and create a contour plot of the anomaly scores. Then, check the performance of the trained local outlier model by plotting the precision-recall curve.
Use a Gaussian copula to generate random data points from a bivariate distribution.
rng("default") rho = [1,0.05;0.05,1]; n = 1000; u = copularnd("Gaussian",rho,n);
Add noise to 5% of randomly selected observations to make the observations outliers.
noise = randperm(n,0.05*n); true_tf = false(n,1); true_tf(noise) = true; u(true_tf,1) = u(true_tf,1)*5;
Train a local outlier factor model by using the lof
function. Set the fraction of anomalies in the training observations to 0.05. For better performance, you can also modify the local outlier factor algorithm options by specifying name-value arguments, such as SearchMethod
, NumNeighbors
, and Distance
. In this case, specify the number of nearest neighbors to use as 40.
[LOFObj,tf,scores] = lof(u,ContaminationFraction=0.05,NumNeighbors=40);
Compute anomaly scores for 2-D grid coordinates around the training observations by using the trained local outlier factor model and the isanomaly
function.
l1 = linspace(min(u(:,1),[],1),max(u(:,1),[],1)); l2 = linspace(min(u(:,2),[],1),max(u(:,2),[],1)); [X1,X2] = meshgrid(l1,l2); [~,scores_grid] = isanomaly(LOFObj,[X1(:),X2(:)]); scores_grid = reshape(scores_grid,size(X1,1),size(X2,2));
Create a scatter plot of the training observations and a contour plot of the anomaly scores. Flag true outliers and the outliers detected by lof
.
idx = setdiff(1:1000,noise); scatter(u(idx,1),u(idx,2),[],[0.5 0.5 0.5],".") hold on scatter(u(noise,1),u(noise,2),"ro","filled") scatter(u(tf,1),u(tf,2),60,"kx",LineWidth=1) contour(X1,X2,scores_grid,"ShowText","on") legend(["Normal Points" "Outliers" "Detected Outliers"],Location="best") colorbar hold off
Check the performance of the trained local outlier factor model by plotting the precision-recall curve and computing the area under the curve (AUC) value. Create a rocmetrics
object. rocmetrics
computes the false positive rates and the true positive rates (or recall) by default. Specify the AdditionalMetrics
name-value argument to additionally compute the precision values (or positive predictive values).
rocObj = rocmetrics(true_tf,scores,true,AdditionalMetrics="PositivePredictiveValue");
Plot the curve by using the plot
function of rocmetrics
. Specify the y-axis metric as precision (or positive predictive value) and the x-axis metric as recall (or true positive rate). Display a filled circle at the model operating point corresponding to LOFObj.ScoreThreshold
. Compute the area under the precision-recall curve using the trapezoidal method of the trapz
function, and display the value in the legend.
r = plot(rocObj,YAxisMetric="PositivePredictiveValue",XAxisMetric="TruePositiveRate"); hold on idx = find(rocObj.Metrics.Threshold>=LOFObj.ScoreThreshold,1,'last'); scatter(rocObj.Metrics.TruePositiveRate(idx), ... rocObj.Metrics.PositivePredictiveValue(idx), ... [],r.Color,"filled") xyData = rmmissing([r.XData r.YData]); auc = trapz(xyData(:,1),xyData(:,2)); legend(join([r.DisplayName " (AUC = " string(auc) ")"],""),"true Model Operating Point") xlabel("Recall") ylabel("Precision") title("Precision-Recall Curve") hold off
Input Arguments
LOFObj
— Trained local outlier factor model
LocalOutlierFactor
object
Trained local outlier factor model, specified as a LocalOutlierFactor
object.
Tbl
— Predictor data
table
Predictor data, specified as a table. Each row of Tbl
corresponds to one observation, and each column corresponds to one predictor variable.
Multicolumn variables and cell arrays other than cell arrays of character vectors are
not allowed.
If you train LOFObj
using a table, then you must provide
predictor data by using Tbl
, not X
. All
predictor variables in Tbl
must have the same variable names and
data types as those in the training data. However, the column order in
Tbl
does not need to correspond to the column order of the
training data.
Data Types: table
X
— Predictor data
numeric matrix
Predictor data, specified as a numeric matrix. Each row of X
corresponds to one observation, and each column corresponds to one predictor
variable.
If you train LOFObj
using a matrix, then you must provide
predictor data by using X
, not Tbl
. The
variables that make up the columns of X
must have the same order as
the training data.
Data Types: single
| double
Name-Value Arguments
Specify optional pairs of arguments as
Name1=Value1,...,NameN=ValueN
, where Name
is
the argument name and Value
is the corresponding value.
Name-value arguments must appear after other arguments, but the order of the
pairs does not matter.
Before R2021a, use commas to separate each name and value, and enclose
Name
in quotes.
Example: CacheSize=5000,ScoreThreshold=0.3
uses a Gram matrix of size
5000 megabytes and identifies observations with scores exceeding 0.3 as
anomalies.
CacheSize
— Size of Gram matrix in megabytes
1000
(default) | positive scalar | "maximal"
Size of the Gram matrix in megabytes, specified as a positive scalar or
"maximal"
. For the definition of the Gram matrix, see Algorithms. The
isanomaly
function can use a Gram matrix when the
Distance
name-value argument of the lof
function "fasteuclidean"
.
When CacheSize
is "maximal"
,
isanomaly
attempts to allocate enough memory for an entire
intermediate matrix whose size is MT
-by-MX
,
where MT
is the number of rows of the training data in
LOFObj
and MX
is the number of rows of the
input data, X
or Tbl
.
CacheSize
does not have to be large enough for an entire
intermediate matrix, but must be at least large enough to hold an
MX
-by-1 vector. Otherwise, isanomaly
uses
the "euclidean"
distance.
If Distance
is "fasteuclidean"
and
CacheSize
is too large or "maximal"
,
isanomaly
might attempt to allocate a Gram matrix that
exceeds the available memory. In this case, MATLAB® issues an error.
Example: CacheSize="maximal"
Data Types: double
| char
| string
ScoreThreshold
— Threshold for anomaly score
LOFObj.ScoreThreshold
(default) | nonnegative scalar
Threshold for the anomaly score, specified as a nonnegative scalar.
isanomaly
identifies observations with scores above the
threshold as anomalies.
The default value is the ScoreThreshold
property value of LOFObj
.
Example: ScoreThreshold=0.5
Data Types: single
| double
Output Arguments
tf
— Anomaly indicators
logical column vector
Anomaly indicators, returned as a logical column vector. An element of
tf
is true
when the observation in the
corresponding row of Tbl
or X
is an anomaly,
and false
otherwise. tf
has the same length as
Tbl
or X
.
isanomaly
identifies observations with
scores
above the threshold (the
ScoreThreshold
value) as anomalies.
scores
— Anomaly scores (local outlier factor values)
numeric column vector
Anomaly scores (local outlier factor values), returned as a
numeric column vector whose values are nonnegative. scores
has the
same length as Tbl
or X
, and each element of
scores
contains an anomaly score for the observation in the
corresponding row of Tbl
or X
. A score value
less than or close to 1 can indicate a normal observation, and a value greater than 1
can indicate an anomaly.
More About
Local Outlier Factor
The local outlier factor (LOF) algorithm detects anomalies based on the relative density of an observation with respect to the surrounding neighborhood.
The algorithm finds the k-nearest neighbors of an observation and computes the local reachability densities for the observation and its neighbors. The local outlier factor is the average density ratio of the observation to its neighbor. That is, the local outlier factor of observation p is
where
lrdk(·) is the local reachability density of an observation.
Nk(p) represents the k-nearest neighbors of observation p. You can specify the
IncludeTies
name-value argument astrue
to include all the neighbors whose distance values are equal to the kth smallest distance, or specifyfalse
to include exactly k neighbors. The defaultIncludeTies
value oflof
isfalse
for more efficient performance. Note that the algorithm in [1] uses all the neighbors.|Nk(p)| is the number of observations in Nk(p).
For normal observations, the local outlier factor values are less than or close to 1,
indicating that the local reachability density of an observation is higher than or similar
to its neighbors. A local outlier factor value greater than 1 can indicate an anomaly. The
ContaminationFraction
argument of lof
and the ScoreThreshold
argument of isanomaly
control the threshold for the local outlier
factor values.
The algorithm measures the density based on the reachability distance. The reachability distance of observation p with respect to observation o is defined as
where
dk(o) is the kth smallest distance among the distances from observation o to its neighbors.
d(p,o) is the distance between observation p and observation o.
The algorithm uses the reachability distance to reduce the statistical fluctuations of d(p,o) for the observations close to observation o.
The local reachability density of observation p is the reciprocal of the average reachability distance from observation p to its neighbors.
The density value can be infinity if the number of duplicates is greater than the number of
neighbors (k). Therefore, if the training data contains duplicates, the
lof
and isanomaly
functions use the weighted
local outlier factor (WLOF) algorithm. This algorithm computes the weighted local outlier
factors using the weighted local reachability density (wlrd).
where
and w(o) is the number of duplicates for observation o in the training data. After computing the weight values, the algorithm treats each set of duplicates as one observation.
Algorithms
To compute the local outlier factor values (
scores
) for each observation inTbl
orX
,isanomaly
finds the k-nearest neighbors among the training observations stored in theX
property of aLocalOutlierFactor
object.isanomaly
considersNaN
,''
(empty character vector),""
(empty string),<missing>
, and<undefined>
values inTbl
andNaN
values inX
to be missing values.isanomaly
does not use observations with missing values.isanomaly
assigns the anomaly score ofNaN
and anomaly indicator offalse
(logical 0) to observations with missing values.
References
[1] Breunig, Markus M., et al. “LOF: Identifying Density-Based Local Outliers.” Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, 2000, pp. 93–104.
Version History
Introduced in R2022bR2023b: "fasteuclidean"
distance support
The isanomaly
function supports the "fasteuclidean"
Distance
algorithm. This algorithm usually computes distances faster
than the default "euclidean"
algorithm when the number of variables in a
data point exceeds 10. The algorithm uses extra memory to store an intermediate Gram matrix
(see Algorithms). Set the size of this
memory allocation using the CacheSize
name-value argument.
MATLAB-Befehl
Sie haben auf einen Link geklickt, der diesem MATLAB-Befehl entspricht:
Führen Sie den Befehl durch Eingabe in das MATLAB-Befehlsfenster aus. Webbrowser unterstützen keine MATLAB-Befehle.
Select a Web Site
Choose a web site to get translated content where available and see local events and offers. Based on your location, we recommend that you select: .
You can also select a web site from the following list:
How to Get Best Site Performance
Select the China site (in Chinese or English) for best site performance. Other MathWorks country sites are not optimized for visits from your location.
Americas
- América Latina (Español)
- Canada (English)
- United States (English)
Europe
- Belgium (English)
- Denmark (English)
- Deutschland (Deutsch)
- España (Español)
- Finland (English)
- France (Français)
- Ireland (English)
- Italia (Italiano)
- Luxembourg (English)
- Netherlands (English)
- Norway (English)
- Österreich (Deutsch)
- Portugal (English)
- Sweden (English)
- Switzerland
- United Kingdom (English)