• Accuracy is not sufficient metric for evaluation

• Recall:
• Precision
• F-measure
• Recall (detection rate)
• ratio between the number of correctly detected anomalies and the total number of anomalies
• False alarm (false positive)
• ratio between the number of data records from normal class that are misclassified as anomalies and the total number of data records from normal class.
• ROC curve: a tradeoff between detection rate and false alarm rate.
• Area under the ROC curve (AUC) is computed using a tapeziod rule.

4. Taxonomy

5. Classification Based Techniques

• Main idea: build a classification model for normal (and anomalous (rare)) events based on labeled training data, and use it to classify each new unseen event.
• Classification models must be able to handle skewed (imbalanced) class distribution
• Categories
• supervised learning techniques
• require knowledge of both normal and anomaly class
• build classifier to distinguish between normal and known anomalies
• semi-supervised classification techniques
• require knowledge of normal class only
• use modified classification model to learn the normal behavior and then detect any deviations from normal behavior as anomalous
• Advantage
• supervised learning techniques
• models that can be easily understood
• high accuracy in detecting many kinds of known anomalies
• semi-supervised classification techniques
• models that can be easily  understood
• normal behavior can be accurately learned
• Drawbacks
• supervised learning techniques
• require both labels from both normal and anomaly class
• cannot detect unknown and emerging anomalies
• semi-supervised classification techniques
• require labels from normal class
• possible high false alarm rate
• previous unseen data records may be recognized as anomalies

6. Nearest Neighbor Based

• Key assumption
• normal points have close neighbors while anomalies are located far from other points
• Two approach
• compute neighborhood for each data record
• analyze the neighbor to determine whether data record is anomaly or not
• Categories
• distance-based methods
• anomalies are data points most distant from other points
• density-based methods
• anomalies are data points in low density regions
• Advantage
• can be used in unsupervised or semi-supervised setting (do not make any assumptions about data distribution)
• Drawbacks
• if normal points do not have sufficient number of neighbors the techniques may fail
• computationally expensive
• in high dimensional spaces, data is sparse and the concept of similarity may not be meaningful anymore
• due to the sparseness, distances between any two data records may become quite similar
• each data record may be considered as potential outlier