arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.The original BERT uses a subword-level tokenization with the vocabulary size of 30K which is learned after input preprocessing and using several heuristics. RoBERTa uses bytes instead of unicode characters as the base for subwords and … Read More