Microsoft natural language processing

PubMed, the standard repository for biomedical research articles, adds 4, new papers every day and over a million every year. It is impossible to keep track of such rapid progress by manual efforts alone. In the era of big data and precision medicine, the urgency has never been higher to advance natural language processing NLP methods that can help scientists stay versed in the deluge of information.

NLP can help researchers quickly identify and cross-reference important findings in papers that are both directly and tangentially related to their own research at a large scale—instead of researchers having to sift through papers manually for relevant findings or recall them from memory.

In this blog post, we present our recent advances in pretraining neural language models for biomedical NLP. We question the prevailing assumption that pretraining on general-domain text is necessary and useful for specialized domains such as biomedicine. Instead, we show that biomedical text is very different from newswires and web text.

By pretraining solely on biomedical text from scratch, our PubMedBERT model outperforms all prior language models and obtains new state-of-the-art results in a wide range of biomedical applications.

To help accelerate progress in this vitally important area, we have created a comprehensive benchmark and released the first leaderboard for biomedical NLP.

Our findings might also be potentially applicable to other high-value domains, such as finance and law. Pretrained neural language models are the underpinning of state-of-the-art NLP methods. Pretraining works by masking some words from text and training a language model to predict them from the rest. Then, the pre-trained model can be fine-tuned for various downstream tasks using task-specific training data.

As in mainstream NLP, prior work on pretraining is largely concerned about newswires and the Web. For applications in such general domains, the topic is not known a priori , it is thus advantageous to train a broad-coverage model using as much text as one could gather. Read more about grants, fellowships, events and other ways to connect with Microsoft research.

For specialized domains like biomedicine, which has abundant text that is drastically different from general-domain corpora, this rationale no longer applies. Still, the prevailing assumption is that out-domain text, in this case text not related to biomedicine, can be helpful, so prior work typically adopts a mixed-domain approach by starting from a general-domain language model. We challenge this assumption and propose a new paradigm that pretrains entirely on in-domain text from scratch for a specialized domain.

We observe that biomedical text is very different from general-domain text. As shown in the above figure, the standard BERT model pretrained on general-domain text only covers the most frequent biomedical terms. Others will be shattered to non-sensical sub-words. For example, lymphoma is represented as l, ym, ph, or oma.

Acetyltransferase is reduced to ace, ty, lt, ran, sf, eras, or e. For biomedicine, however, such benchmarks and leaderboards are ostensibly absent. Follow us:. Share this page:. Overview Publications Projects Microsoft Research blog The Natural Language Processing group focuses on developing efficient algorithms to process text and to make their information accessible to computer applications.

People People. Chris Brockett Principal Researcher. Natural Language Processing is at the core of what most bots do in interpreting users written or verbal inputs and responding to them in a meaningful way using a language they will understand. While NLP certainly can't work miracles and ensure a bot appropriately responds to every message, it's powerful enough to make-or-break a bot's success.

Don't underestimate this critical and often overlooked aspect of bots. Skip to main content. This browser is no longer supported. Download Microsoft Edge More info. Contents Exit focus mode. Please rate your experience Yes No. Any additional feedback? Submit and view feedback for This product This page.