How to fill a large gap in a sine wave?

Hi, this is a follow up to a previous question of mine. I am trying to fill a gap in a sine wave using the fillgaps function as described here - https://uk.mathworks.com/help/signal/examples/reconstructing-missing-data.html#ReconstructingMissingDataExample-6

I have tried to do this but as you can see in my image below, it is not working for me. It is drawing a straight line instead of correctly predicting the wave.

Below is an image of the code I am using in matlab:

Could someone please advise me on what I might be doing wrong? Do I need to use another function instead? Thanks in advance for your help.

MATLAB release version: 9.5.0.1298439 (R2018b) update 7

[KSSV]

It is a long gap between the data.....you cannot fill that long data gap.

[carolina]

I see, thanks for your reply. Do you know what the maximum gap that can be filled is?

[KSSV]

If the data is periodic ... one can replace the missing data with existing ...

Answer 1

[Roger J]

You can try something like the following:

temp_reconstructed = fillgaps(tablecomplete.table_temperature,80,50);

Where I have maximum prediction-sequence length of 80 samples and a model order of 50. I don't know what you will use for your data, but you can play around with it.

[neil jerome]

to elaborate on this: inserting a gap of 2000 points, better answers were found using larger input params for fillgaps. the attached shows to add in a gap of your own size (there are no gaps in your original file?!) and then brute force a sensible answer for fillgaps when you know the true values.

% gaps in sine wave file?
%%
file = 'TimeSineGapData.xlsx';
readfile = readtable(file);
table = readfile(1:12545,1:2);
tableunique = unique(table, 'rows');
table_temp = str2double(tableunique.TEMPERATURE);
table_temp_true = str2double(tableunique.TEMPERATURE);
table_temp(2000:4000) = NaN; % insert gap in data
table_time = tableunique.TIME;
table_time.Format = 'yyyy/MM7dd:HH:mm:ss';
table_all = timetable(table_time, table_temp, table_temp_true);

date_start = datetime('2018-07-07', 'Format', 'yyyy/MM7dd:HH:mm:ss');
date_end = datetime('2018-10-02', 'Format', 'yyyy/MM7dd:HH:mm:ss');
xq = (date_start:minutes(10):date_end);
tablecomplete = retime(table_all, xq);

%%
figure; hold on;
temp_recon = fillgaps(tablecomplete.table_temp,800,600); % filled!
plot(tablecomplete.table_time, temp_recon, 'r'); % recon in red
plot(tablecomplete.table_time, tablecomplete.table_temp, 'b'); % orig in blue

% grid search params for fillgaps
AA = 10:10:1000; nAA = length(AA); % number of points
BB = 10:20:600; nBB = length(BB); % order

res = NaN(nAA,nBB);

for aa = 1:nAA
    for bb = 1:nBB
        if BB(bb) <= AA(aa)
            temp_recon = fillgaps(tablecomplete.table_temp, AA(aa),BB(bb));
            diff = abs(tablecomplete.table_temp_true - temp_recon);
            res(aa, bb) = nansum(diff);
        end
    end
end
%%
figure; imagesc(res); colorbar;
% looks like using big numbers is useful :)

[carolina]

Thanks for your help. I have some long gaps with 8000 samples covering approximately two months from some weather stations. I specified the maximum number of samples and the model order with the function fillgaps() and it does work with my datasets. I decided not to interpolate weather stations with long gaps of more than 8000 samples.

I have another question about when I specify the model order - how can I determine it correctly when I have a long gap and a short gap with few samples in the same signal?

[neil jerome]

this will get a bit philosophical! for a small and large gap in the same data, if you find an acceptable answer for the long gap then it should also fill the short gap satisfactorily, although this would be vastly overpowered compared to what you would need if there were only the small gap. but you should probably use the same parameters for both gaps, since you are (probably) implicitly assuming that the data has the same underlying properties throughout. but some might argue that you should try to use the minimum necessary assumptions to create an answer, which would mean a reduced order/points for small gaps. again, i would use known data, deliberately blank out sections of your choice, and then try to recreate using different strategies and then compare back to the truth. but then you would ideally try this for all combinations of missing data and gap size, so you can easily get into a mess. the important thing is to explicitly recognise which assumptions you're making :)

so when you say 'correctly' determine, that might be a wild goose chase, so be careful. the bottom line is that the missing data simply isn't there, so you cannot ever know what is actually correct. something strange might have happened in the weather that day and you will never know! what you are doing is artificially creating data that 'doesnt look wrong', but not more than that! it doesn't - cannot - add more information than you already have, and in the worst case could be misleading (if something weird really did happen that day, and you recreated a normal day to sit in its place). if your analysis can't cope with missing data, adding in artificial data that is based on the extant data can only ever mimic the information already there, so what are you really adding? anyway, you see what i mean about getting philosophical! the criteria for 'success' are yours to decide, so in the end it depends what your goal is, and as long as you state exactly what you did (and why).

just my thoughts :)