Automatic Download from a Url when the file of the date is inputted by the user

I have written a code to download a specific data from cosmic satellite website, the code is as follows

url_https='https://data.cosmic.ucar.edu';
dataUrl=strcat(url_https,"/gnss-ro/cosmic2/nrt/level1b/2022/004/podTc2_nrt_2022_004.tar.gz");
dataFile="podTc2_nrt_2022_005.tar.gz";
FileFullPath=websave(dataFile,dataUrl);

Files=untar("F:\podtc2_data\podTc2_nrt_2022_005.tar.gz",'podTc2_nrt_2022_005');

Now this code only works for one day of the data that is provided by the data portal. is there a way such that I could create a loop that modifies a string of characters corresponent to the desired date that I am looking to work with, then the web function can read the string automatically created by MATLAB.

I was wondering if a web crawler or something similar has been done on matlab?

Overall, I have currently created a code that download a specific date of data, I am looking to code a set of loops where if I enter a specific date the code would automatically crawl the web and download the specific file that I am looking for.

I have been stuck on this section for a quite a long time, hopefully you guys could give me some advice and ideas on this matter.

[Jan]

Which part of the file name do you want to modify and how? The more details you mention, the smaller is the chance of a wrong guess.

[Tianchu Lu]

Thank you very much for your answer benearth, so the part that should be modified would be the part of _005, as 005 means 5th of Jan 2022 and hence 001 is 1st of Jan and so on and so forth. So the pusedocode that I had in mind was that the user( myself) would enter a specific date that the I would like to investigate into, and then a loop where matlab would urlread (https://data.cosmic.ucar.edu/gnss-ro/cosmic2/nrt/level1b/2022/) and identify the date that the user want and would then download the specific zipped file of that date. so a loop where it would urlread and identify the date inputted and download the file correspond to the inputted information. But I will defo have a play with the code that you submitted below and let you know how I got on.

Answer 1

[Jan]

This can be implemented with sprintf() or compose() easily.

url_https='https://data.cosmic.ucar.edu';
for year = 2020:2022  % ??? Bold guess
    for day = 1:365   % ??? Bold guess
        dataUrl  = sprintf('%s/gnss-ro/cosmic2/nrt/level1b/%d/%03d/podTc2_nrt_%d_%03d.tar.gz', ...
                    url_https, year, day, year, day);
        dataFile = sprintf('podTc2_nrt_%d_%03d.tar.gz', year, day);
        FileFullPath = websave(dataFile, dataUrl);        
    end
end

[Tianchu Lu]

after a few modifications to the code, it actualy works like magic! Thank you so much!

[Débora Rodrigues]

Hello!

Sorry to bother, but i have a similar problem. I am trying to download a .grib preciptation data, from this website Precipitation. I saw this discussion and tried to follow.

However i have an error:

Error using websave (line 95)

The '' protocol specified in URL, 'MERGE_CPTEC_20100101.grib2', is not supported. Specify the URL with the protocol 'http://' or 'https://'.

Error in download_precipitacao (line 19)

data = websave(data_url,data_file);

my code is:

url = 'http://ftp1.cptec.inpe.br/modelos/tempo/MERGE/GPM/DAILY'; 

for ano= 2010:2021
    for mes= 1:12 
        for dia=1:30
            data_url = sprintf('%s/%3d/%02d/MERGE_CPTEC_%d%02d%02d.grib2', ...
                       url, ano, mes, ano, mes, dia);
            
            data_file= sprintf('MERGE_CPTEC_%d%02d%02d.grib2', ano, mes, dia);
            data = websave(data_url,data_file);
        end
   end
end

if someone can help me, i would be very grateful, thanks!

[Débora Rodrigues]

Hello! So i solved my problem!

I changed the function websave for urlwrite, and it worked! I am living it in here in case someone with similar problens appear haha.

url = 'http://ftp1.cptec.inpe.br/modelos/tempo/MERGE/GPM/DAILY'; 

for ano= 2010:2021
    for mes= 1:12 
        for dia=1:30
            data_url = sprintf('%s/%3d/%02d/MERGE_CPTEC_%d%02d%02d.grib2', ...
                       url, ano, mes, ano, mes, dia);
            
            data_file= sprintf('MERGE_CPTEC_%d%02d%02d.grib2', ano, mes, dia);
            %data = websave(data_url,data_file);
            a= urlwrite(data_url,data_file);
        end
   end
end