Normal Distribution - using 24x45x65 matrix. - How to do this?

Hello

I have a set of data in a matrix of: 24x45x65 --> hours x days x customers i want to make a normal distribution plot for each hour of lets say 1 customer on y-axis. so there will be 24 Graphs. then i can show the mean of it..lower and upper boundry on the plot. its data of some customers using energy in different times of the day..(before going to Work and coming back from the Work) example(used pain

t :P)

is it possible to do so? i will include my data to this file..so its easier to understand :)

thanks in advance, help will be appreciated alot.

Answer 1

[Star Strider]

This seems to work:

load x_weekday
xwkd = x_weekday;

for k1 = 1:size(xwkd,1)
    hrmx = xwkd(k1,:,:);            % All data for hour ‘k1’
    hrmn(k1) = mean(hrmx(:));       % Mean
    hrsd(k1) = std(hrmx(:));        % Standard deviation
end
N = numel(hrmx);

figure(2)                           % Plot Mean ± Standard Deviation
errorbar([1:24], hrmn, hrsd)
grid
xlabel('Hour')
ylabel('kWh ± SD') 

figure(3)                           % Plot Mean ± Standard Error
errorbar([1:24], hrmn, hrsd/sqrt(N))
grid
xlabel('Hour')
ylabel('kWh ± SE')

It plots the data for all customers. To plot data for a particular customer, say Customer #10, change this line:

hrmx = xwkd(k1,:,:);            % All data for hour ‘k1’

to:

hrmx = xwkd(k1,:,10);           % All data for Customer 10 for hour ‘k1’

[awda]

Hi Star.

thanks for helping. i appreciate it..but is it possible to make it look like the example i made with paint. where it is made using standard deviation normal graph. with sigma 1 and so on.. this plot only shows me the same as normal plot of 1 customers load just with boundrys..

thanks your help is appreciated :), im stuck on this :(

[Star Strider]

Would a 3D surface plot of the normally-distributed data with interpolation do what you want? I’ll work on that, since the same code can also be used for other plots like contour.

Figure 1 plots usage and ±1 standard deviation error bars. Figure 2 plots usage and standard error error bars, actually more statistically meaningful and useful. Since the means have an inherent error most accurately described by the standard error (that is normally distributed) rather than the standard deviation, I will plot the standard errors and their distributions.

A FEW HOURS LATER ...

load x_weekday
xwkd = x_weekday;

for k1 = 1:size(xwkd,1)
    hrmx = xwkd(k1,:,:);            % All data for hour ‘k1’
    hrmn(k1) = mean(hrmx(:));       % Mean
    hrsd(k1) = std(hrmx(:));        % Standard deviation
end
n = numel(hrmx);
hrse = hrsd/sqrt(n);
Npdf = @(x,mu,s) exp(-(0.5.*(x-mu)/s).^2) / (s*sqrt(2*pi));

x = 1:size(xwkd,1);
y = linspace(0,5);
[X,Y] = meshgrid(x,y);

z = zeros(length(y),length(x));
for k1 = 1:length(x)
    Z(:,k1) = Npdf(y, hrmn(k1), hrsd(k1));
end

figure(4)
surfc(X, Y, Z)
xlabel('Hour')
ylabel('kWh')
zlabel(' Probability')
view(-160, 60)
grid on

You can rotate the plot you generate interactively to view it from different angles. You can also substitute contour for surfc if you want a contour plot. If you want to see more of the kWh axis, change the upper limit in this statement from 5 to something else:

y = linspace(0,5);

[awda]

Hi Star :)

first i must say i am very grateful, your a genius :). I am trying to understand this 100%. how is the first 2 figures related to the normal distribution curve (cause no guassian function is used) link:

http://en.wikipedia.org/wiki/File:Standard_deviation_diagram.svg

and what do i see in the 3-D plot that differs from the 2 other plots?

Sorry for being not so understanding with this thanks for helping again :)

[Star Strider]

My pleasure!

The distributions I plotted are based on the ‘standard error of the mean’, not standard deviation, since we are plotting Hourly means over Days and Customers. (Wikipedia explains Standard error of mean versus standard deviation clearly.) The Gaussian function (as characterised by the Normal distribution) is given in the Npdf line, the same as in the Statistics Toolbox normpdf function. I used that rather than the t-distribution because with these numbers ( n = 2925 ), they are essentntially the same.

The 3D plot displays the distributions of the estimated means and standard errors, essentially as you depicted in your original post. It simply does it in 3D because that is more meaningful. The 2D errorbar plot displays the same essential information in a different format. (You can also experiment with the ribbon plot.)

P.S. — I just realised that I uploaded the wrong image last night. I edited my previous comment, uploading the correct one.

[awda]

aha ok. so first 2D plot is standard error of the mean. while the 3rd 3D plot is the answer to the original question. which i draw in paint, just in 3D?

By the way, is it also possible to make it plot in Gaussian curve, like the one i linked earlier.. where mu is in mid and there is minus sigma and plus sigma in the right side ..and so on:) or this 3D plot sufices it.. You also mentioned it has standard Error in the 3D plot. which colors is it..or what does the colors say at all :P.. and what about the probability 1 to 4.

sorry for being so bad :(. just curious to know from you :P

[Star Strider]

Yes!

They are Gaussian curves, but (correctly) using the standard error of the mean rather than the standard deviation. The Normal distribution curve is symmetrical, but deliberately cuts off at zero kWh, since unless your customers have solar panels or windchargers and are feeding power back into the electrical power grid, all usage will be greater than zero. [There are probability distributions that will always be greater than zero (the lognormal being one), but the distribution has to be used in context, and unless you know that the data are best described by another distribution, using the normal distribution for the standard error is dictated by the Central Limit Theorem. The CLT probably applies anyway, since the number of observations, 2925 in each Hour, is likely sufficient.]

The colors are a convenient way of depicting the relative magnitude of the function at various locations in the plot. You can redefine them to be a different colorbar if you want to.

I do not understand ‘what about the probability 1 to 4’. If you mean Hours 1 to 4, I suspect that everyone is at home asleep and power usage is minimal and relatively consistent among the Customers and Days, so the standard errors are lowest then. (The same pattern evolves between Hours 20 through 24.) That is simply a guess on my part, but seems logical.

Not ‘bad’ at all! I will do my best to explain what I do in my code, and why, to the the best of my knowledge and ability.

[awda]

Hi, sorry i meant the probability in the figure from 0-1 or sometimes 0-0.4..how do i read the 3D graph using X-Y-Z. and what i find out from it. what does the "probability" show.

last: if i want to make RSE(relative standard error) in percentage. how do i make it.its simple i guess..but dno which code should i change to get it in plot. maybe pie form plot

(by the way: the 3D plot xwkd(k1,:,5), since k1 is =24..does it mean i only get for hour 24, or all hours.) cause when i in command type: xwkd(k1,:,5) it shows me only data for hour 24.

thanks..and sorry for disturbing so much :P

[Star Strider]

I plotted the probability density, not the cumulative probability. That is simply the probability of a particular amount of electricity consumption at the particular hour, expressed as a function of the standard error of the calculated sample mean for each Hour for all Customers and Days.

I would calculate the RSE for each of the 24 Hours as:

for k1 = 1:size(xwkd,1)
    rse(k1) = 100*hrse(k1)/hrmn(k1);
end

I calculated and plotted the usage values for all Customers and all Days for each of the 24 Hours. You have data for all Hours.

The xwkd array are your original data. If you want the information from the code I wrote, you need to use the variables I created: hrmx, hrmn, hrsd, hrse, and for the probability densities for each hour, Z. (I documented all but Z with comments. I forgot to comment Z, since it was late and I was tired. Sorry!)

If you want a contour plot of the probabilities, create:

figure(5)
contour(X, Y, Z)
xlabel('Hour')
ylabel('kWh')
title('Probability')

[awda]

ok thanks alot :)

[Star Strider]

Again, my pleasure!