Performing matrix subtraction for huge data

2 Ansichten (letzte 30 Tage)
MSDataSpl1
MSDataSpl1 am 11 Aug. 2017
Bearbeitet: José-Luis am 15 Aug. 2017
I have a matrix say A = [2,5,7]' and another matrix B = [2,7,9]', now i want to subtract B from A in such a way like each element of B will get subtracted from all elements of A, like c = A - B then c should be like (Expected outcome) = [0 3 5, -5 -2 0, -7 -4 -2]
Its like each element of B will get deducted from all elements of A and result will be stored in row wise in new matrix row wise.I dont want to use any loops but only vectorization.So i have used bsxfun but it is good for small matrix but if the dimension of A is 230600 * 1 and B is 1400000 * 1 then bsxfun show out of memory error so how to solve this issue without using loops??
  4 Kommentare
Stephen23
Stephen23 am 11 Aug. 2017
Bearbeitet: Stephen23 am 11 Aug. 2017
@MSDataSpl1: what optimization? You asked about an arithmetic operation (which will be impossible for any desktop today PC to perform), not an optimization.
José-Luis
José-Luis am 11 Aug. 2017
Bearbeitet: José-Luis am 11 Aug. 2017
You could perform that operation, but a more appropriate question is: should you?
How about just performing the actual operation you need only when you need it? No need to have such a big array in memory.
When you start talking about this much data then you need a database. Loop and save to a database every now and then.
I really really doubt that you need to keep such an array in memory or in a database. I would go out of my way to avoid that. It's just a waste of disk space.

Melden Sie sich an, um zu kommentieren.

Antworten (3)

José-Luis
José-Luis am 11 Aug. 2017
Bearbeitet: José-Luis am 11 Aug. 2017
The problem is not bsxfun, it is the sheer amount of data:
dataSize = 230600 * 1400000 * 8 (size of a double in bytes)
more or less equals
2.6e12 Bytes
2.6e9KB
2.6e6MB
2.6e3GB
That strikes me as quite a bit of data for the kind of computers that exist nowadays.
You need to divide your problem in smaller chunks. Better yet, you need to carefully evaluate if there are better ways of achieving what you are trying to do.
  4 Kommentare
MSDataSpl1
MSDataSpl1 am 11 Aug. 2017
@John & @Jose could you answer this one,
https://in.mathworks.com/matlabcentral/answers/351074-i-want-to-find-the-minimum-value-of-the-matrices-so-i-am-using-min-function-but-the-problem-is-the-d
José-Luis
José-Luis am 15 Aug. 2017
Bearbeitet: José-Luis am 15 Aug. 2017
I had a look at that question. Still not clear to me what you want to achieve. Please learn to format your post. If it's hard to read, people might be less willing to help.
Please accept the answer that best solves your problem.

Melden Sie sich an, um zu kommentieren.


Jan
Jan am 11 Aug. 2017
Explain, what the actual problem is you are working on. It is not likely that you really need to create this output matrix explicitly, because it is such easy to access the output dynamically: The element at the index [n,m] is A(n) - B(m). Then you can and should omit the creation of C = A.' - B. Where ever you want to write C(n,m) write A(n) - B(m) instead.
I'm sure there will be further methods to reduce the need to create C explicitly. If you want to search the minimum only, or what ever. So please explain the mathematical problem you want to solve.
  1 Kommentar
MSDataSpl1
MSDataSpl1 am 11 Aug. 2017
@Jan Simon I am trying to take the difference of the time stamps successive traces(messages) which are coming in a particular time gap so like in 1 hr 1500000 messages comes so i want to take the difference of the timestamps and then calculate the minimum and plot the data. like x = trace(i) - trace(i-1) where i = 1500000.
or
you can answer my this question then it will also help,
https://in.mathworks.com/matlabcentral/answers/351074-i-want-to-find-the-minimum-value-of-the-matrices-so-i-am-using-min-function-but-the-problem-is-the-d

Melden Sie sich an, um zu kommentieren.


alice
alice am 11 Aug. 2017
You can avoid to loop using the minus operator comportment:
A = [2,5,7]';
B = [2,7,9]';
result = A'-B;
  1 Kommentar
José-Luis
José-Luis am 11 Aug. 2017
And that still won't solve the problem even if OP is sitting on top of a Google data center.

Melden Sie sich an, um zu kommentieren.

Kategorien

Mehr zu Matrix Computations finden Sie in Help Center und File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by