Performing matrix subtraction for huge data

I have a matrix say A = [2,5,7]' and another matrix B = [2,7,9]', now i want to subtract B from A in such a way like each element of B will get subtracted from all elements of A, like c = A - B then c should be like (Expected outcome) = [0 3 5, -5 -2 0, -7 -4 -2]

Its like each element of B will get deducted from all elements of A and result will be stored in row wise in new matrix row wise.I dont want to use any loops but only vectorization.So i have used bsxfun but it is good for small matrix but if the dimension of A is 230600 * 1 and B is 1400000 * 1 then bsxfun show out of memory error so how to solve this issue without using loops??

[Stephen23]

Your final matrix would have 230600*1400000 = 3.2284e11 elements in it, and would require atleast 230600*1400000*8 = 2.5827e+012 = 2.349 tebibytes of memory just for MATLAB to be able to have it in the workspace. Do you have that much memory in your computer?

[MSDataSpl1]

No i dont have that much of memory but by any other means can we perform this optimization.

[Stephen23]

@MSDataSpl1: what optimization? You asked about an arithmetic operation (which will be impossible for any desktop today PC to perform), not an optimization.

[José-Luis]

You could perform that operation, but a more appropriate question is: should you?

How about just performing the actual operation you need only when you need it? No need to have such a big array in memory.

When you start talking about this much data then you need a database. Loop and save to a database every now and then.

I really really doubt that you need to keep such an array in memory or in a database. I would go out of my way to avoid that. It's just a waste of disk space.

Answer 1

[José-Luis]

The problem is not bsxfun, it is the sheer amount of data:

dataSize = 230600 * 1400000 * 8 (size of a double in bytes)

more or less equals

 2.6e12 Bytes
 2.6e9KB
 2.6e6MB
 2.6e3GB

That strikes me as quite a bit of data for the kind of computers that exist nowadays.

You need to divide your problem in smaller chunks. Better yet, you need to carefully evaluate if there are better ways of achieving what you are trying to do.

[John D'Errico]

+1. Regardless, tomorrow, or next week, month or year, if and when someone gets a large enough computer to solve the current problem, then they will decide they need to to solve a problem that is 1000 times as large.

John's law: Computational problems always expand to be larger than the capabilities of the available computing resources.

The art of computing and of mathematics in general is to solve intractable problem not by brute force, but by more subtle means. There will always be many who will choose to use brute force anyway. :(

[José-Luis]

I like John's law. :)

[MSDataSpl1]

@John & @Jose could you answer this one,

https://in.mathworks.com/matlabcentral/answers/351074-i-want-to-find-the-minimum-value-of-the-matrices-so-i-am-using-min-function-but-the-problem-is-the-d

[José-Luis]

I had a look at that question. Still not clear to me what you want to achieve. Please learn to format your post. If it's hard to read, people might be less willing to help.

Please accept the answer that best solves your problem.

Answer 2

[Jan]

Explain, what the actual problem is you are working on. It is not likely that you really need to create this output matrix explicitly, because it is such easy to access the output dynamically: The element at the index [n,m] is A(n) - B(m). Then you can and should omit the creation of C = A.' - B. Where ever you want to write C(n,m) write A(n) - B(m) instead.

I'm sure there will be further methods to reduce the need to create C explicitly. If you want to search the minimum only, or what ever. So please explain the mathematical problem you want to solve.

[MSDataSpl1]

@Jan Simon I am trying to take the difference of the time stamps successive traces(messages) which are coming in a particular time gap so like in 1 hr 1500000 messages comes so i want to take the difference of the timestamps and then calculate the minimum and plot the data. like x = trace(i) - trace(i-1) where i = 1500000.

or

you can answer my this question then it will also help,

https://in.mathworks.com/matlabcentral/answers/351074-i-want-to-find-the-minimum-value-of-the-matrices-so-i-am-using-min-function-but-the-problem-is-the-d

Answer 3

[alice]

You can avoid to loop using the minus operator comportment:

A = [2,5,7]';
B = [2,7,9]';
result = A'-B;

[José-Luis]

And that still won't solve the problem even if OP is sitting on top of a Google data center.