Will MATLAB provide Bert word embedding support?

Hi

During workingon in a project, I came to know about GLOVE, word2vec and fast textword embedding. Few days back, in a group meeting, another word embedding model has came up which has reportedly shown good perforemance, called Bert.

Will Matlab provide(or plan to provide) Bert embedding support in the system?

Thanks

Answer 1

[Jon Cherrie]

BERT is available via this GitHub repository

• https://github.com/matlab-deep-learning/transformer-models

[hongjing lu]

Thanks for this. Could you please clarify how to donwload the supportfiles? Keep getting the download error for

\supportfiles\nnet\data\networks\bert\uncased_L12_H768_A12\parameters.mat

[Jon Cherrie]

This is a guess at the issue: maybe the download was started once but failed (e.g., a conection issue or CTRL+C). If that's the case then you might have a corrupt parameters.mat on your user path. You can manually delete it to get things moving again.

The parameters file should be in a location like this:

• C:\Users\username\Documents\MATLAB\Examples\R2021a\supportfiles\nnet\data\networks\bert\uncased_L12_H768_A12

[hongjing lu]

Thanks Jon for the prompt reply. I did a bit more digging. Not sure whether this comes from the setting in my local Matlab (2020b). The problem is from passing string array to matlab.internal.examples.downloadSupportFile().

I changed line 17-18 in getSupportFilePath.m to change the input string array to charater array. Then the download works.

fileURL = char(strjoin([directory,fileName],"/"));

filePath = matlab.internal.examples.downloadSupportFile('nnet',fileURL);

[Jon Cherrie]

Good to know that you resolved that issue!

For BERT, you will need R2021a as it relies on some functions new in that release, e.g.,

• https://www.mathworks.com/help/textanalytics/ref/textanalytics.unicode.utf32.html
• https://www.mathworks.com/help/textanalytics/ref/textanalytics.unicode.nfd.html

For GPT-2 (which is in the same repository) you should be good with R2020b.

[hongjing lu]

Thanks Jon!

Answer 2

[Steven Lord]

In general we don't discuss future plans. If you believe we should include this word embedding model in our products, please submit that as an enhancement request to Technical Support using the telephone icon in the upper-right corner of this page.

[Saugata Bose]

thanks steven. I would.

[Zalán Kocsis]

In the R2021a updates page

https://www.mathworks.com/products/new_products/latest_features.html

Text Analytics Toolbox

• BERT Transformer Models: Use BERT models for tasks such as text classification, summarization, and sentiment analysis

However, I did not find any related functions or objects in the documentation nor in the Release notes of Text Analytics or Deep Learning.

[Christopher Creutzig]

Zalán, please see Jon Cherrie's reply (currently further up), BERT is available from

• https://github.com/matlab-deep-learning/transformer-models