What AI says about LinkBERT

LinkBERT is a knowledge-enabled pretrained language model that extends BERT by pretraining with document links (hyperlinks and citations) to inject cross-document knowledge. It can be used as a drop-in replacement for BERT and improves performance on general language understanding, as well as knowledge-intensive and cross-document tasks like question answering and document retrieval. The project releases pretrained LinkBERT models for general and biomedical domains (LinkBERT-base/large and BioLinkBERT-base) trained on Wikipedia with hyperlinks and PubMed with citation links, in HuggingFace-compatible format.