Dimension reduction is a strategy with the help of which, data from high dimensional space can be converted to low dimensional space. This can be achieved using any one of the two dimension reduction techniques :
Linear Discriminant Analysis(LDA)
Principal Component Analysis(PCA)
Linear Discriminant Analysis(LDA)
Linear discriminant analysis i.e. LDA is one of the dimension reduction techniques which is capable of discriminatory information of the class.
The major advantage of using LDA strategy is, it tries to obtain directions along with classes which are best separated.
Scatter within class and Scatter between classes, both are considered when LDA is used.
Minimizing the variance within each class and maximizing the distance between the means are the main focus of LDA.
Algorithm for LDA
Let the number of classes be “c” and u_{i }be the mean vector of class i, where i=1,2,3,.. .
Let N_{i }be the number of samples within class i, where i=1,2,3…C.
Total number of samples, N=∑ Ni.
Number of samples within Class Scatter Matrix.
Scatter Within Class
Number of samples between Class Scatter Matrix.
Scatter Between Class
Advantages : Linear Discriminant Analysis
Suitable for larger data set.
Calculations of scatter matrix in LDA is much easy as compared to co-variance matrix.
Disadvantages : Linear Discriminant Analysis
More redundancy in data.
Memory requirement is high.
More Noisy.
Applications : Linear Discriminant Analysis
Face Recognition.
Earth Sciences.
Speech Classification.
Principal Component Analysis(PCA)
Principal Component Analysis i.e. PCA is the other dimension reduction techniques which is capable of reducing the dimensionality of a given data set along with ability to retain maximum possible variation in the original data set.
PCA standouts with the advantage of mapping data from high dimensional space to low dimensional space.
Another advantage of PCA is, it is able to locate most accurate data representation in low dimensional space.
In PCA, Maximum variance is the direction in which data is projected.
Algorithm For PCA
Let d_{1},d_{2}, d_{3},…,d_{d }be the whole data set consisting of d-dimensions.
Calculate the mean vector of these d-dimensions.
Calculate the covariance matrix of data set.
Calculate Eigen values(λ_{1},λ_{2,}λ_{3,…,}λ_{d}) and their corresponding Eigen vectors (e_{1}, e_{2}, e_{3},….e_{d}).
Now, Sort the Eigen vectors in descending order and then choose “p” Eigen vectors having largest values in order to generate a matrix “A” with dimensions p*d.
i.e. A = d * p.
Using the matrix “A” (i.e. A = d * p) in order to transform samples into new subspace with the help of:
y = A^{T} * x
Where,A^{T} Transpose matrix of “A”
Advantages : Principal Component Analysis
Less redundancy in data.
Lesser noise reduction.
Efficient for smaller
Disadvantages : Principal Component Analysis
Calculation of exact co-variance matrix is very difficult.