User:Shex1627/sandbox

Not to be confused with latent Dirichlet allocation.

LDA of IRIS dataset

Linear discriminant analysis (LDA) is a generalization of Fisher's linear discriminant, a method used in statistics, pattern recognition and machine learning to find a linear combination of features that characterizes or separates two or more classes of objects or events. The resulting combination may be used as a linear classifier, or, more commonly, for dimensionality reduction before later classification.

LDA is closely related to analysis of variance (ANOVA) and regression analysis, which also attempt to express one dependent variable as a linear combination of other features or measurements. However, ANOVA uses categorical independent variables and a continuous dependent variable, whereas discriminant analysis has continuous independent variables and a categorical dependent variable (i.e. the class label). Logistic regression and probit regression are more similar to LDA than ANOVA is, as they also explain a categorical variable by the values of continuous independent variables. These other methods are preferable in applications where it is not reasonable to assume that the independent variables are normally distributed, which is a fundamental assumption of the LDA method.

LDA is also closely related to principal component analysis (PCA) and factor analysis in that they both look for linear combinations of variables which best explain the data. LDA explicitly attempts to model the difference between the classes of data. PCA on the other hand does not take into account any difference in class, and factor analysis builds the feature combinations based on differences rather than similarities. Discriminant analysis is also different from factor analysis in that it is not an interdependence technique: a distinction between independent variables and dependent variables (also called criterion variables) must be made.

LDA works when the measurements made on independent variables for each observation are continuous quantities. When dealing with categorical independent variables, the equivalent technique is discriminant correspondence analysis.

Introduction to LDA as a Bayes Classifier and Finding Posterior Probabilities

Consider an observation ${\vec {x}}$, a p-vector(vector of length p) where each element of ${\vec {x}}$ is the value of a feature(also called predictor or variable). In other words, ${\textstyle {\vec {x}}=<{\text{feature1}},{\text{feature2}},...{\text{feature p}}>}$, where $p$ is the number of features). The classification problem is to assign $x$ to a class $y$, which is from one of the $K$ possible classes(i.e $y\in \{1,...,K\}$).

As a Bayes classifier, LDA approaches classification through assigning $x$ a posterior probability for each class. According to Bayes theorem, $p(y=k|{\vec {x}})={\frac {p({\vec {x}}|y=k)*p(y=k)}{\sum _{l=1}^{K}p({\vec {x}}|y=l)*p(y=l)}}$, where $p({\vec {x}}|y=k)$ is the likelihood and $p(y=k)$ is the prior probability, which represents you or the classifier's initial knowledge about the classes. The prior probability $p(y=k)$ is often denoted as $\pi _{k}$. A popular way of assigning prior probability to a class is to use its proportion in your data(also called dataset, trainining set, samples or observations). In other words, $\pi _{k}={\frac {\text{Obversations that are in class K}}{\text{Total number of observations}}}$ For instance, to classify if student A would turn in his homework on time without any inside information about his schedule, it is natural to look at the proportion of times he did submit the homework on time.

LDA assumes each class is multivariate Gaussian. To compute the likelihood $p(x|y=k)$, which is also the probability density of a multivariate Gaussian, we find the mean ${{\vec {\mu }}_{k}}$ and the covariance $\Sigma$ from the data. Note that ${\mu _{k}}$ is the vector of means of the features for the $k^{\text{th}}$ class(i.e ${\textstyle {\vec {\mu }}_{k}=<\mathbb {E} [{\text{feature1}}|y=k],\mathbb {E} [{\text{feature2}}|y=k],...\mathbb {E} [{\text{featureP}}|y=k]>}$). And $\Sigma$ is the covariance matrix of all the features and LDA makes the assumption that all classes share the same $\Sigma$.(QDA, quadratic discriminant analysis, assumes a different covariance matrix for each class instead)

$$f_{k}({\vec {x}})={\frac {1}{(2\pi )^{p/2}|\Sigma |^{1/2}}}exp(-{\frac {1}{2}}({\vec {x}}-{\vec {\mu }}_{k})^{T}\Sigma ^{-1}({\vec {x}}-{\vec {\mu }}_{k}))\quad {\textrm {and}}\quad f_{k}({\vec {x}})={\frac {1}{(2\pi )^{p/2}|\Sigma |^{1/2}}}e^{-{\frac {1}{2}}({\vec {x}}-{\vec {\mu }}_{k})^{T}\Sigma ^{-1}({\vec {x}}-{\vec {\mu }}_{k})}$$

To find the posterior probability $p(y=k|{\vec {x}})$, calculating the denominator of the Bayes formula is often omitted because it is the same across all classes. In addition, the log posterior probability is often used rather than the raw probability(detailed rationales in Log Probability). So applying log to $p({\vec {x}}|y=k)*\pi _{k}$,

$$\delta _{k}({\vec {x}})=\log(p({\vec {x}}|y=k)*\pi _{k})={\vec {x}}^{T}\Sigma ^{-1}{\vec {\mu }}_{k}-{\frac {1}{2}}{\vec {\mu }}_{k}^{T}\Sigma ^{-1}{\vec {\mu }}_{k}+\log(\pi _{k})$$

LDA for two classes(Edited Section for clarity)

Building on the previous section, consider the situation which observation ${\vec {x}}$ can only be either class 0 or class 1. LDA classifies ${\vec {x}}$ belongs to class 0 if the log of the likelihood ratios is bigger than some threshold T(i.e $\log {\frac {p(y=0|{\vec {x}})}{p(y=1|{\vec {x}})}}>T$), so that

${\begin{aligned}\log {\frac {p(y=0|{\vec {x}})}{p(y=1|{\vec {x}})}}&\ >\ T\\\log p(y=0|{\vec {x}})-\log p(y=1|{\vec {x}})&\ >\ T\\({\vec {x}}-{\vec {\mu }}_{0})^{T}\Sigma ^{-1}({\vec {x}}-{\vec {\mu }}_{0})+\ln |\Sigma |-({\vec {x}}-{\vec {\mu }}_{1})^{T}\Sigma ^{-1}({\vec {x}}-{\vec {\mu }}_{1})-\ln |\Sigma |&\ >\ T\\({\vec {x}}-{\vec {\mu }}_{0})^{T}\Sigma ^{-1}({\vec {x}}-{\vec {\mu }}_{0})-({\vec {x}}-{\vec {\mu }}_{1})^{T}\Sigma ^{-1}({\vec {x}}-{\vec {\mu }}_{1})&\ >\ T\\\end{aligned}}$

because $\Sigma$ is Hermitian,

${\vec {x}}^{T}{\Sigma }^{-1}{\vec {\mu }}_{i}={{\vec {\mu }}_{i}}^{T}{\Sigma }^{-1}{\vec {x}}$

and the above decision criterion becomes a threshold on the dot product

${\vec {w}}\cdot {\vec {x}}>c$

for some threshold constant c, where

${\vec {w}}=\Sigma ^{-1}({\vec {\mu }}_{1}-{\vec {\mu }}_{0})$

$c={\frac {1}{2}}(T-{{\vec {\mu }}_{0}}^{T}\Sigma _{0}^{-1}{{\vec {\mu }}_{0}}+{{\vec {\mu }}_{1}}^{T}\Sigma _{1}^{-1}{{\vec {\mu }}_{1}})$

This means that the criterion of an input ${\vec {x}}$ being in a class $y$ is purely a function of this linear combination of the known observations.

It is often useful to see this conclusion in geometrical terms: the criterion of an input ${\vec {x}}$ being in a class $y$ is purely a function of projection of multidimensional-space point ${\vec {x}}$ onto vector ${\vec {w}}$ (thus, we only consider its direction). In other words, the observation belongs to $y$ if corresponding ${\vec {x}}$ is located on a certain side of a hyperplane perpendicular to ${\vec {w}}$. The location of the plane is defined by the threshold c.

Canonical discriminant analysis for k classes

Canonical discriminant analysis (CDA) finds axes (k − 1 canonical coordinates, k being the number of classes) that best separate the categories. These linear functions are uncorrelated and define, in effect, an optimal k − 1 space through the n-dimensional cloud of data that best separates (the projections in that space of) the k groups. See “Multiclass LDA” for details below.

Fisher's linear discriminant

The terms Fisher's linear discriminant and LDA are often used interchangeably, although Fisher's original article actually describes a slightly different discriminant, which does not make some of the assumptions of LDA such as normally distributed classes or equal class covariances.

Suppose two classes of observations have means ${\vec {\mu }}_{0},{\vec {\mu }}_{1}$ and covariances $\Sigma _{0},\Sigma _{1}$. Then the linear combination of features ${\vec {w}}\cdot {\vec {x}}$ will have means ${\vec {w}}\cdot {\vec {\mu }}_{i}$ and variances ${\vec {w}}^{T}\Sigma _{i}{\vec {w}}$ for $i=0,1$. Fisher defined the separation between these two distributions to be the ratio of the variance between the classes to the variance within the classes:

$S={\frac {\sigma _{\text{between}}^{2}}{\sigma _{\text{within}}^{2}}}={\frac {({\vec {w}}\cdot {\vec {\mu }}_{1}-{\vec {w}}\cdot {\vec {\mu }}_{0})^{2}}{{\vec {w}}^{T}\Sigma _{1}{\vec {w}}+{\vec {w}}^{T}\Sigma _{0}{\vec {w}}}}={\frac {({\vec {w}}\cdot ({\vec {\mu }}_{1}-{\vec {\mu }}_{0}))^{2}}{{\vec {w}}^{T}(\Sigma _{0}+\Sigma _{1}){\vec {w}}}}$

This measure is, in some sense, a measure of the signal-to-noise ratio for the class labelling. It can be shown that the maximum separation occurs when

${\vec {w}}\propto (\Sigma _{0}+\Sigma _{1})^{-1}({\vec {\mu }}_{1}-{\vec {\mu }}_{0})$

When the assumptions of LDA are satisfied, the above equation is equivalent to LDA.

Be sure to note that the vector ${\vec {w}}$ is the normal to the discriminant hyperplane. As an example, in a two dimensional problem, the line that best divides the two groups is perpendicular to ${\vec {w}}$.

Generally, the data points to be discriminated are projected onto ${\vec {w}}$; then the threshold that best separates the data is chosen from analysis of the one-dimensional distribution. There is no general rule for the threshold. However, if projections of points from both classes exhibit approximately the same distributions, a good choice would be the hyperplane between projections of the two means, ${\vec {w}}\cdot {\vec {\mu }}_{0}$ and ${\vec {w}}\cdot {\vec {\mu }}_{1}$. In this case the parameter c in threshold condition ${\vec {w}}\cdot {\vec {x}}>c$ can be found explicitly:

$c={\vec {w}}\cdot {\frac {1}{2}}({\vec {\mu }}_{0}+{\vec {\mu }}_{1})={\frac {1}{2}}{\vec {\mu }}_{1}^{T}\Sigma _{1}^{-1}{\vec {\mu }}_{1}-{\frac {1}{2}}{\vec {\mu }}_{0}^{T}\Sigma _{0}^{-1}{\vec {\mu }}_{0}$.

Otsu's method is related to Fisher's linear discriminant, and was created to binarize the histogram of pixels in a grayscale image by optimally picking the black/white threshold that minimizes intra-class variance and maximizes inter-class variance within/between grayscales assigned to black and white pixel classes.

Multiclass LDA

In the case where there are more than two classes, the analysis used in the derivation of the Fisher discriminant can be extended to find a subspace which appears to contain all of the class variability. This generalization is due to C. R. Rao. Suppose that each of C classes has a mean $\mu _{i}$ and the same covariance $\Sigma$. Then the scatter between class variability may be defined by the sample covariance of the class means

$\Sigma _{b}={\frac {1}{C}}\sum _{i=1}^{C}(\mu _{i}-\mu )(\mu _{i}-\mu )^{T}$

where $\mu$ is the mean of the class means. The class separation in a direction ${\vec {w}}$ in this case will be given by

$S={\frac {{\vec {w}}^{T}\Sigma _{b}{\vec {w}}}{{\vec {w}}^{T}\Sigma {\vec {w}}}}$

This means that when ${\vec {w}}$ is an eigenvector of $\Sigma ^{-1}\Sigma _{b}$ the separation will be equal to the corresponding eigenvalue.

If $\Sigma ^{-1}\Sigma _{b}$ is diagonalizable, the variability between features will be contained in the subspace spanned by the eigenvectors corresponding to the C − 1 largest eigenvalues (since $\Sigma _{b}$ is of rank C − 1 at most). These eigenvectors are primarily used in feature reduction, as in PCA. The eigenvectors corresponding to the smaller eigenvalues will tend to be very sensitive to the exact choice of training data, and it is often necessary to use regularisation as described in the next section.

If classification is required, instead of dimension reduction, there are a number of alternative techniques available. For instance, the classes may be partitioned, and a standard Fisher discriminant or LDA used to classify each partition. A common example of this is "one against the rest" where the points from one class are put in one group, and everything else in the other, and then LDA applied. This will result in C classifiers, whose results are combined. Another common method is pairwise classification, where a new classifier is created for each pair of classes (giving C(C − 1)/2 classifiers in total), with the individual classifiers combined to produce a final classification.

Incremental LDA

The typical implementation of the LDA technique requires that all the samples are available in advance. However, there are situations where the entire data set is not available and the input data are observed as a stream. In this case, it is desirable for the LDA feature extraction to have the ability to update the computed LDA features by observing the new samples without running the algorithm on the whole data set. For example, in many real-time applications such as mobile robotics or on-line face recognition, it is important to update the extracted LDA features as soon as new observations are available. An LDA feature extraction technique that can update the LDA features by simply observing new samples is an incremental LDA algorithm, and this idea has been extensively studied over the last two decades. Catterjee and Roychowdhury proposed an incremental self-organized LDA algorithm for updating the LDA features. In other work, Demir and Ozmehmet proposed online local learning algorithms for updating LDA features incrementally using error-correcting and the Hebbian learning rules. Later, Aliyari et al. derived fast incremental algorithms to update the LDA features by observing the new samples.

Practical use

In practice, the class means and covariances are not known. They can, however, be estimated from the training set. Either the maximum likelihood estimate or the maximum a posteriori estimate may be used in place of the exact value in the above equations. Although the estimates of the covariance may be considered optimal in some sense, this does not mean that the resulting discriminant obtained by substituting these values is optimal in any sense, even if the assumption of normally distributed classes is correct.

Another complication in applying LDA and Fisher's discriminant to real data occurs when the number of measurements of each sample (i.e., the dimensionality of each data vector) exceeds the number of samples in each class. In this case, the covariance estimates do not have full rank, and so cannot be inverted. There are a number of ways to deal with this. One is to use a pseudo inverse instead of the usual matrix inverse in the above formulae. However, better numeric stability may be achieved by first projecting the problem onto the subspace spanned by $\Sigma _{b}$. Another strategy to deal with small sample size is to use a shrinkage estimator of the covariance matrix, which can be expressed mathematically as

$\Sigma =(1-\lambda )\Sigma +\lambda I\,$

where $I$ is the identity matrix, and $\lambda$ is the shrinkage intensity or regularisation parameter. This leads to the framework of regularized discriminant analysis or shrinkage discriminant analysis.

Also, in many practical cases linear discriminants are not suitable. LDA and Fisher's discriminant can be extended for use in non-linear classification via the kernel trick. Here, the original observations are effectively mapped into a higher dimensional non-linear space. Linear classification in this non-linear space is then equivalent to non-linear classification in the original space. The most commonly used example of this is the kernel Fisher discriminant.

LDA can be generalized to multiple discriminant analysis, where c becomes a categorical variable with N possible states, instead of only two. Analogously, if the class-conditional densities $p({\vec {x}}\mid c=i)$ are normal with shared covariances, the sufficient statistic for $P(c\mid {\vec {x}})$ are the values of N projections, which are the subspace spanned by the N means, affine projected by the inverse covariance matrix. These projections can be found by solving a generalized eigenvalue problem, where the numerator is the covariance matrix formed by treating the means as the samples, and the denominator is the shared covariance matrix. See “Multiclass LDA” above for details.

Applications

In addition to the examples given below, LDA is applied in positioning and product management.

Bankruptcy prediction

In bankruptcy prediction based on accounting ratios and other financial variables, linear discriminant analysis was the first statistical method applied to systematically explain which firms entered bankruptcy vs. survived. Despite limitations including known nonconformance of accounting ratios to the normal distribution assumptions of LDA, Edward Altman's 1968 model is still a leading model in practical applications.

Face recognition

In computerised face recognition, each face is represented by a large number of pixel values. Linear discriminant analysis is primarily used here to reduce the number of features to a more manageable number before classification. Each of the new dimensions is a linear combination of pixel values, which form a template. The linear combinations obtained using Fisher's linear discriminant are called Fisher faces, while those obtained using the related principal component analysis are called eigenfaces.

Marketing

In marketing, discriminant analysis was once often used to determine the factors which distinguish different types of customers and/or products on the basis of surveys or other forms of collected data. Logistic regression or other methods are now more commonly used. The use of discriminant analysis in marketing can be described by the following steps:

1. Formulate the problem and gather data—Identify the salient attributes consumers use to evaluate products in this category—Use quantitative marketing research techniques (such as surveys) to collect data from a sample of potential customers concerning their ratings of all the product attributes. The data collection stage is usually done by marketing research professionals. Survey questions ask the respondent to rate a product from one to five (or 1 to 7, or 1 to 10) on a range of attributes chosen by the researcher. Anywhere from five to twenty attributes are chosen. They could include things like: ease of use, weight, accuracy, durability, colourfulness, price, or size. The attributes chosen will vary depending on the product being studied. The same question is asked about all the products in the study. The data for multiple products is codified and input into a statistical program such as R, SPSS or SAS. (This step is the same as in Factor analysis).
2. Estimate the Discriminant Function Coefficients and determine the statistical significance and validity—Choose the appropriate discriminant analysis method. The direct method involves estimating the discriminant function so that all the predictors are assessed simultaneously. The stepwise method enters the predictors sequentially. The two-group method should be used when the dependent variable has two categories or states. The multiple discriminant method is used when the dependent variable has three or more categorical states. Use Wilks’s Lambda to test for significance in SPSS or F stat in SAS. The most common method used to test validity is to split the sample into an estimation or analysis sample, and a validation or holdout sample. The estimation sample is used in constructing the discriminant function. The validation sample is used to construct a classification matrix which contains the number of correctly classified and incorrectly classified cases. The percentage of correctly classified cases is called the hit ratio.
3. Plot the results on a two dimensional map, define the dimensions, and interpret the results. The statistical program (or a related module) will map the results. The map will plot each product (usually in two-dimensional space). The distance of products to each other indicate either how different they are. The dimensions must be labelled by the researcher. This requires subjective judgement and is often very challenging. See perceptual mapping.

Biomedical studies

The main application of discriminant analysis in medicine is the assessment of severity state of a patient and prognosis of disease outcome. For example, during retrospective analysis, patients are divided into groups according to severity of disease – mild, moderate and severe form. Then results of clinical and laboratory analyses are studied in order to reveal variables which are statistically different in studied groups. Using these variables, discriminant functions are built which help to objectively classify disease in a future patient into mild, moderate or severe form.

In biology, similar principles are used in order to classify and define groups of different biological objects, for example, to define phage types of Salmonella enteritidis based on Fourier transform infrared spectra, to detect animal source of Escherichia coli studying its virulence factors etc.

Earth science

This method can be used to separate the alteration zones. For example, when different data from various zones are available, discriminant analysis can find the pattern within the data and classify it effectively.

See also

• Data mining
• Decision tree learning
• Factor analysis
• Kernel Fisher discriminant analysis
• Logit (for logistic regression)
• Multidimensional scaling
• Pattern recognition
• Perceptron
• Preference regression
• Quadratic classifier

Further reading

• Duda, R. O.; Hart, P. E.; Stork, D. H. (2000). Pattern Classification (2nd ed.). Wiley Interscience. ISBN 0-471-05669-3. MR 1802993.
• Hilbe, J. M. (2009). Logistic Regression Models. Chapman & Hall/CRC Press. ISBN 978-1-4200-7575-5.
• Mika, S.; et al. (1999). "Fisher Discriminant Analysis with Kernels". IEEE Conference on Neural Networks for Signal Processing IX: 41–48. doi:10.1109/NNSP.1999.788121.
• H. Richard McFarland and Donald St. P. Richards, “Exact Misclassification Probabilities for Plug-In Normal Quadratic Discriminant Functions. I. The Equal-Means Case” Journal of Multivariate Analysis, 2001, vol. 77, issue 1, pages 21–53 link to article
• H. Richard McFarland and Donald St. P. Richards, “Exact Misclassification Probabilities for Plug-In Normal Quadratic Discriminant Functions. II. The Heterogeneous Case” Journal of Multivariate Analysis, 2002, vol. 82, issue 2, pages 299-330 link to article

External links

• Discriminant Correlation Analysis (DCA)
• ALGLIB contains open-source LDA implementation in C# / C++ / Pascal / VBA.
• Psychometrica.de open-source LDA implementation in Java
• LDA tutorial using MS Excel
• Biomedical statistics. Discriminant analysis
• StatQuest: Linear Discriminant Analysis (LDA) clearly explained on YouTube