Is there a way to use matlab to automate the download files from a web site

I want to copy files from a geographic database hosed on a website. I don't want to read the web page-just make a list of the files on the page and download them to my local machine 1 by 1 . I don't believe websave or webread is the appropriate tool.

[Sean de Wolski]

webread/websave sound right to me. Can you share the website?

[Stephen Forczyk]

The website is

https://dds.cr.usgs.gov/srtm/version2_1/SRTM3/North_America/

I could not get webread or websave to do what I wanted . Instead I went to a free program called JDownLoader2. There are thousands of files on each site. I needed a way to created an automated list and then download the files one after the other

[Rita Cognion]

I am having the same problem you had, 2 years on. I'm afraid you were right: websave and webread don't support it, and probably MATLAB doesn't enable simple file download. If anyone came up with a solution that uses MATLAB, I'd love to hear it.

[Jan]

It would be useful if you post, how you have tried it with webread. Currently it is hard to suggest a workaround without knowing, what you are doing.

Loading thousands of files from a webpage sounds strange. Usually such data sets are made available using an FTP service. But if the page is really designed to make it as hard as possible to grab the data, I'd use the FireFox add-on "DownloadThemAll".

[Stephen Forczyk]

I typically download lots of files from AWS and NOAA websites. I can do this with Python but have failed when I tried this with webread. I have studied up on the use of webread and might still get it to work. FTP does not appear to be possible with the AWS S3 bucket but there exists a companion Gov't website that I can FTP the data. Unfortunately they do enforce daily limits whic is lower than I want by about a factor of 50

[Jan]

@Stephen Forczyk: Do you want to post how you have tried it with websave? Maybe you get a useful suggestion from the forum.

Did you contact the FTP-admin already? The limits are set to prevent millions of users to access all data. If you have a scientific interest, the admins usually give you a higher volume.