What AI says about SciBERT

SciBERT is a BERT model trained on 1.14 million scientific papers from Semantic Scholar, using a custom vocabulary called scivocab. It achieves state-of-the-art performance on a wide range of scientific natural language processing tasks.