Can I obtain the covariance matrix of a stochastic process with plenty of measurements?

3 views (last 30 days)
Hello everyone.
I have implemented the Karhunen-Loève expansion as per this question.
I have tested it with samples of data obtaining possitive results.
However, now I am dealing with a matrix of dimensions 211302*50, meaning that I have roughly 200000 observations of 50 random variables.
If I try to calculate the covariance matrix of these using the observations as columns (as per the previus link), the program crashes and the error returned is:
Error using *
Requested 211302x211302 (332.7GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take
a long time and cause MATLAB to become unresponsive.
Error in cov (line 155)
c = (xc' * xc) ./ denom;
Related documentation
Is there a way to do what I want or is the matrix just too big?
Best regards.
  1 Comment
Sharmin Kibria
Sharmin Kibria on 25 Jun 2021
As the error suggested, the covariance matrix you are trying to build is too big. Your solution needed to allocate memory that was larger than what is allowed for array storage. That is why MATLAB crashed.

Sign in to comment.

Answers (1)

Bjorn Gustavsson
Bjorn Gustavsson on 25 Jun 2021
To me it seems that you have misunderstood the dimensions. Your covariance-matrix should, to my understanding be 50x50 when you have 50 random variables observed 200000 times. With a call like this:
tic,C = cov(randn(211302,50));toc
I get a 50-by-50 covariance-matrix C in ~0.22 s.




Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by