Modifying maximum likelihood test for solving singularity and outlier problems in high dimensional cases

The maximum likelihood (ML) test in the structural covariance analysis is an effective tool in statistical analysis of multivariate test. However, the performance of the classical location and scatter estimators is usually flawed by singularity and outliers’ problems in
high dimensional data sets. The study aimed to modify the existing ML test by incorporating the Cholesky banded regularization methods and thresholding sample covariance matrix to resolve singularity problems and incorporating the L₁-median with Weiszfeld’s algorithm (MLw) covariance matrix to solve outliers’ problem in high dimensional data sets. This study suggested to replace the classical estimators with MLw estimator due to its good properties. On the other hand, several shortcomings such as inconsistency under normal distribution, based on small sample size with large variables in high dimensions were discovered. To improve the ML estimators and to maintain the high breakdown point while having high dimensional data sets, a new robust estimator of banded Cholesky and L₁-median with Weiszfeld algorithm, MLѡвсн was suggested. The performance of MLѡвсн
and four more modified ML-tests, namely MLtests with banded Cholesky estimator MLвсн with Thresholding MLтн with Weiszfeld Algorithm MLѡ and with Weiszfeld’s Algorithm Estimator and Thresholding MLѡтн had been examined using Type I error and power of test values in a simulation study. The MLвсн outperformed other modified ML-tests and presents a robust estimator with high breakdown points and affine equivariance with better computational efficiency in resolving the singularity and outlier problems. The validation of this robust modified ML-test was conducted in a real application on the Pakistan microeconomic system and turned out with good performance. The study contributes to improve the performance of the sample covariance matrices in relation to singularity and outlier problem in high dimensional data cases.