How to scrape values from a dynamically loaded web page?

The issue was addressed by finding the API webpage of the main website. Then, I requsted an API-Key via an online form. They provided me an API key for accessing some basic data. However, more data needed a paid membership.

[Vinod]

Rather than scrape the dynamic page, do they have an API you can use? I looked it up and it appears barchart.com has an API.

[Mehran Rahmani]

Thanks Vinod for your helpful hint. I submitted a form on their webpage requesting access to their API. I assume they will send me an API Key, correct? I am currently awaiting their response and will comment when I tried their Key to fetch the data.

Answer 1

[Ameer Hamza]

Text Analytics toolbox provides an html parser: https://www.mathworks.com/help/textanalytics/ug/parse-html-and-extract-text-content.html. However, if you don't have a toolbox, then you may need to use regexp or textscan().

[Mehran Rahmani]

Thanks for your response. Similar to "urlfiler()" and "webread()", the textscan() does not read the numeric values dynamically loaded to the page. As Vinod noted above, the values should be fetched through the webpage API. I am yet to to try it.