fastBERTtokens: Tokenizing for BERT in parallel

Version 1.0.0 (1.43 KB) by Ralf Elsas
This function simply divides your text into batches, and tokenizes in parallel. Provides significant speed-up.
16 Downloads
Updated 24 Feb 2023

View License

Function to use Matlab BERT tokenizer in parallel
This function simply divides your text into batches, and tokenizes in parallel. As the Matlab tokenizer is very slow when run on a single processor for large data, this provides a significant speed-up. On an i7-10875H laptop with 8 logical units, tokenizing 76k sentences takes about 100 seconds.
Also note that providing the Matlab BERT model is important, as different BERT models use different encodings for the special BERT tokens like [SEP] etc.

Cite As

Ralf Elsas (2024). fastBERTtokens: Tokenizing for BERT in parallel (https://www.mathworks.com/matlabcentral/fileexchange/125295-fastberttokens-tokenizing-for-bert-in-parallel), MATLAB Central File Exchange. Retrieved .

MATLAB Release Compatibility
Created with R2022b
Compatible with R2021a and later releases
Platform Compatibility
Windows macOS Linux
Acknowledgements

Inspired by: Transformer Models

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!
Version Published Release Notes
1.0.0