research topics on nlp

Subscribe to the PwC Newsletter

Join the community, natural language processing, language modelling.

Long-range modeling

Protein language model, sentence pair modeling, representation learning.

Disentanglement

Graph representation learning, sentence embeddings.

Network Embedding

Classification.

Text Classification

Graph Classification

Audio Classification

Medical Image Classification

Text retrieval, deep hashing, table retrieval, question answering.

Open-Ended Question Answering

Open-Domain Question Answering

Conversational question answering.

Knowledge Base Question Answering

Nlp based person retrival, image generation.

Image-to-Image Translation

Text-to-Image Generation

Image Inpainting

Conditional Image Generation

Translation, data augmentation.

Image Augmentation

Text Augmentation

Large language model.

Knowledge Graphs

Machine translation.

Transliteration

Multimodal Machine Translation

Bilingual lexicon induction.

Unsupervised Machine Translation

Knowledge graph completion, triple classification, inductive knowledge graph completion, inductive relation prediction, text generation.

Dialogue Generation

Data-to-Text Generation

Multi-Document Summarization

Story Generation

2d semantic segmentation, image segmentation, text style transfer.

Scene Parsing

Reflection Removal

Visual question answering (vqa).

Visual Question Answering

Machine Reading Comprehension

Chart Question Answering

Chart understanding.

Topic Models

Document Classification

Sentence Classification

Emotion Classification

Data-free knowledge distillation.

Benchmarking

Sentiment analysis.

Aspect-Based Sentiment Analysis (ABSA)

Multimodal Sentiment Analysis

Aspect Sentiment Triplet Extraction

Twitter Sentiment Analysis

Named entity recognition (ner).

Nested Named Entity Recognition

Chinese named entity recognition, few-shot ner, few-shot learning.

One-Shot Learning

Few-Shot Semantic Segmentation

Cross-domain few-shot.

Unsupervised Few-Shot Learning

Optical character recognition (ocr).

Active Learning

Handwriting Recognition

Handwritten digit recognition, irregular text recognition, word embeddings.

Learning Word Embeddings

Multilingual Word Embeddings

Embeddings evaluation, contextualised word representations, continual learning.

Class Incremental Learning

Continual named entity recognition, unsupervised class-incremental learning, information retrieval.

Passage Retrieval

Cross-lingual information retrieval, table search, text summarization.

Abstractive Text Summarization

Document summarization, opinion summarization, relation extraction.

Relation Classification

Document-level relation extraction, joint entity and relation extraction, temporal relation extraction, link prediction.

Inductive Link Prediction

Dynamic link prediction, hyperedge prediction, anchor link prediction, natural language inference.

Answer Generation

Visual Entailment

Cross-lingual natural language inference, reading comprehension.

Intent Recognition

Implicit relations, active object detection, conformal prediction.

Text Simplification

Self-Supervised Image Classification

Music Source Separation

Emotion recognition.

Speech Emotion Recognition

Emotion Recognition in Conversation

Multimodal Emotion Recognition

Emotion-cause pair extraction, natural language understanding, vietnamese social media text processing.

Emotional Dialogue Acts

Image captioning.

3D dense captioning

Controllable image captioning, aesthetic image captioning.

Relational Captioning

Semantic textual similarity.

Paraphrase Identification

Cross-Lingual Semantic Textual Similarity

In-context learning, event extraction, event causality identification, zero-shot event extraction, dialogue state tracking, task-oriented dialogue systems.

Visual Dialog

Dialogue understanding, code generation.

Code Translation

Code documentation generation, class-level code generation, library-oriented code generation, coreference resolution, coreference-resolution, cross document coreference resolution, semantic parsing.

AMR Parsing

Semantic dependency parsing, drs parsing, ucca parsing, semantic similarity.

Sentence Embedding

Sentence compression, joint multilingual sentence representations, sentence embeddings for biomedical texts, specificity, instruction following, visual instruction following, dependency parsing.

Transition-Based Dependency Parsing

Prepositional phrase attachment, unsupervised dependency parsing, cross-lingual zero-shot dependency parsing, information extraction, extractive summarization, temporal information extraction, document-level event extraction, cross-lingual, cross-lingual transfer, cross-lingual document classification.

Cross-Lingual Entity Linking

Cross-language text summarization, common sense reasoning.

Physical Commonsense Reasoning

Riddle sense, memorization, prompt engineering.

Visual Prompting

Response generation, data integration.

Entity Alignment

Entity Resolution

Table annotation, mathematical reasoning.

Math Word Problem Solving

Formal logic, geometry problem solving, abstract algebra, entity linking.

Question Generation

Poll generation.

Topic coverage

Dynamic topic modeling, part-of-speech tagging.

Unsupervised Part-Of-Speech Tagging

Abuse detection, hate speech detection, open information extraction.

Hope Speech Detection

Hate speech normalization, hate speech detection crisishatemm benchmark, data mining.

Argument Mining

Opinion Mining

Subgroup discovery, cognitive diagnosis, sequential pattern mining, bias detection, selection bias, language identification, dialect identification, native language identification, word sense disambiguation.

Word Sense Induction

Fake news detection, few-shot relation classification, implicit discourse relation classification, cause-effect relation classification, intrusion detection.

Network Intrusion Detection

Relational Reasoning

Semantic Role Labeling

Predicate Detection

Semantic role labeling (predicted predicates).

Textual Analogy Parsing

Slot filling.

Zero-shot Slot Filling

Extracting covid-19 events from twitter, grammatical error correction.

Grammatical Error Detection

Text matching, symbolic regression, equation discovery, document text classification.

Learning with noisy labels

Multi-label classification of biomedical texts, political salient issue orientation detection, pos tagging, spoken language understanding, dialogue safety prediction, deep clustering, trajectory clustering, deep nonparametric clustering, nonparametric deep clustering, stance detection, zero-shot stance detection, few-shot stance detection, stance detection (us election 2020 - biden), stance detection (us election 2020 - trump), multi-modal entity alignment, intent detection.

Open Intent Detection

Word similarity, model editing, knowledge editing, cross-modal retrieval, image-text matching, cross-modal retrieval with noisy correspondence, multilingual cross-modal retrieval.

Zero-shot Composed Person Retrieval

Cross-modal retrieval on rsitmd, document ai, document understanding, fact verification, text-to-speech synthesis.

Prosody Prediction

Zero-shot multi-speaker tts, intent classification.

Zero-Shot Cross-Lingual Transfer

Cross-lingual ner, self-learning, language acquisition, grounded language learning, constituency parsing.

Constituency Grammar Induction

Entity typing.

Entity Typing on DH-KGs

Line items extraction, word alignment, ad-hoc information retrieval, document ranking.

Multimodal Deep Learning

Multimodal text and image classification, abstract meaning representation, open-domain dialog, dialogue evaluation, novelty detection.

text-guided-image-editing

Text-based image editing, concept alignment.

Zero-Shot Text-to-Image Generation

Conditional text-to-image synthesis.

Shallow Syntax

Molecular representation, multi-label text classification, explanation generation, discourse parsing, discourse segmentation, connective detection, de-identification, privacy preserving deep learning, morphological analysis.

Text-to-Video Generation

Text-to-video editing, subject-driven video generation, conversational search, sarcasm detection.

Lemmatization

Speech-to-text translation, simultaneous speech-to-text translation.

Aspect Extraction

Aspect category sentiment analysis, extract aspect.

Aspect-Category-Opinion-Sentiment Quadruple Extraction

Aspect-oriented Opinion Extraction

Session search, knowledge distillation.

Self-Knowledge Distillation

Chinese Word Segmentation

Handwritten chinese text recognition, chinese spelling error correction, chinese zero pronoun resolution, offline handwritten chinese character recognition, entity disambiguation, authorship attribution, source code summarization, method name prediction, text clustering.

Short Text Clustering

Open Intent Discovery

Hierarchical text clustering, linguistic acceptability.

Column Type Annotation

Cell entity annotation, columns property annotation, row annotation.

Visual Storytelling

KG-to-Text Generation

Unsupervised KG-to-Text Generation

Abusive language, keyphrase extraction, few-shot text classification, zero-shot out-of-domain detection, safety alignment, key information extraction, key-value pair extraction, multilingual nlp, protein folding, term extraction, text2text generation, keyphrase generation, figurative language visualization, sketch-to-text generation, morphological inflection, phrase grounding, grounded open vocabulary acquisition, deep attention, spam detection, context-specific spam detection, traditional spam detection, word translation, natural language transduction, image-to-text retrieval, summarization, unsupervised extractive summarization, query-focused summarization.

Conversational Response Selection

Cross-lingual word embeddings, knowledge base population, passage ranking, text annotation, authorship verification.

Multimodal Association

Multimodal generation, video generation, image to video generation.

Unconditional Video Generation

Keyword extraction, biomedical information retrieval.

SpO2 estimation

Meme classification, hateful meme classification, news classification, graph-to-sequence, nlg evaluation, automated essay scoring, morphological tagging, key point matching, component classification, argument pair extraction (ape), claim extraction with stance classification (cesc), claim-evidence pair extraction (cepe), temporal processing, timex normalization, document dating, sentence summarization, unsupervised sentence summarization, long-context understanding, weakly supervised classification, weakly supervised data denoising, entity extraction using gan.

Rumour Detection

Semantic retrieval, emotional intelligence, dark humor detection, review generation, semantic composition.

Sentence Ordering

1 image, 2*2 stitchi, comment generation.

Goal-Oriented Dialog

User simulation, lexical simplification, sentence-pair classification, conversational response generation.

Personalized and Emotional Conversation

Token classification, toxic spans detection.

Blackout Poetry Generation

Humor detection.

Passage Re-Ranking

Subjectivity analysis.

Taxonomy Learning

Taxonomy expansion, hypernym discovery, propaganda detection, propaganda span identification, propaganda technique identification, lexical normalization, pronunciation dictionary creation, negation detection, negation scope resolution, question similarity, medical question pair similarity computation, intent discovery, reverse dictionary, lexical analysis, lexical complexity prediction, question rewriting, legal reasoning, punctuation restoration, attribute value extraction.

Hallucination Evaluation

Meeting summarization, table-based fact verification, diachronic word embeddings, pretrained multilingual language models, formality style transfer, semi-supervised formality style transfer, word attribute transfer, aspect category detection, extreme summarization.

Persian Sentiment Analysis

Binary classification, llm-generated text detection, cancer-no cancer per breast classification, cancer-no cancer per image classification, stable mci vs progressive mci, suspicous (birads 4,5)-no suspicous (birads 1,2,3) per image classification, clinical concept extraction.

Clinical Information Retreival

Constrained clustering.

Only Connect Walls Dataset Task 1 (Grouping)

Incremental constrained clustering, recognizing emotion cause in conversations.

Causal Emotion Entailment

trustable and focussed LLM generated content

Game design, dialog act classification, text compression, decipherment, nested mention recognition, probing language models, relationship extraction (distant supervised), semantic entity labeling, handwriting verification, bangla spelling error correction, clickbait detection, code repair, gender bias detection, ccg supertagging, linguistic steganography, toponym resolution, vietnamese datasets.

Timeline Summarization

Multimodal abstractive text summarization, reader-aware summarization, stock prediction, text-based stock prediction, pair trading, event-driven trading, vietnamese visual question answering, explanatory visual question answering, arabic text diacritization, fact selection, thai word segmentation.

Face to Face Translation

Multimodal lexical translation, semantic shift detection, similarity explanation, aggression identification, arabic sentiment analysis, commonsense causal reasoning, complex word identification, sign language production, suggestion mining, temporal relation classification, vietnamese word segmentation, speculation detection, speculation scope resolution, aspect category polarity, cross-lingual bitext mining, morphological disambiguation, multi-agent integration, scientific document summarization, lay summarization, text anonymization, text attribute transfer.

Image-guided Story Ending Generation

Personality generation, personality alignment, abstract argumentation, chinese spell checking, dialogue rewriting, logical reasoning reading comprehension.

Unsupervised Sentence Compression

Stereotypical bias analysis, temporal tagging, anaphora resolution, bridging anaphora resolution.

Abstract Anaphora Resolution

Hope speech detection for english, hope speech detection for malayalam, hope speech detection for tamil, hidden aspect detection, latent aspect detection, attribute mining, cognate prediction, japanese word segmentation, memex question answering, polyphone disambiguation, spelling correction, table-to-text generation.

KB-to-Language Generation

Zero-shot machine translation, zero-shot sentiment classification, conditional text generation, contextualized literature-based discovery, multimedia generative script learning, image-sentence alignment, open-world social event classification, ai and safety, action parsing, author attribution, binary condescension detection, context query reformulation, conversational web navigation, croatian text diacritization, czech text diacritization, definition modelling, document-level re with incomplete labeling, domain labelling, french text diacritization, hungarian text diacritization, irish text diacritization, latvian text diacritization, literature mining, misogynistic aggression identification, morpheme segmentaiton, multi-label condescension detection, news annotation, open relation modeling, personality recognition in conversation.

Reading Order Detection

Record linking, role-filler entity extraction, romanian text diacritization, simultaneous speech-to-speech translation, slovak text diacritization, social media mental health detection, spanish text diacritization, syntax representation, text-to-video search, turkish text diacritization, turning point identification, twitter event detection.

Vietnamese Language Models

Vietnamese scene text, vietnamese text diacritization.

Conversational Sentiment Quadruple Extraction

Attribute extraction, legal outcome extraction, automated writing evaluation, binary text classification, detection of potentially void clauses, chemical indexing, clinical assertion status detection.

Coding Problem Tagging

Collaborative plan acquisition, commonsense reasoning for rl.

Variable Disambiguation

Cross-lingual text-to-image generation, crowdsourced text aggregation.

Description-guided molecule generation

Multi-modal Dialogue Generation

Page stream segmentation.

Email Thread Summarization

Emergent communications on relations, emotion detection and trigger summarization, extractive tags summarization.

Hate Intensity Prediction

Hate span identification, job prediction, joint entity and relation extraction on scientific data, joint ner and classification, math information retrieval, meme captioning, multi-grained named entity recognition, multilingual machine comprehension in english hindi, multimodal text prediction, negation and speculation cue detection, negation and speculation scope resolution, only connect walls dataset task 2 (connections), overlapping mention recognition, paraphrase generation, multilingual paraphrase generation, phrase ranking, phrase tagging, phrase vector embedding, poem meters classification, query wellformedness.

Question-Answer categorization

Readability optimization, reliable intelligence identification, script generation, sentence completion, hurtful sentence completion, speaker attribution in german parliamentary debates (germeval 2023, subtask 1), text effects transfer, text-variation, vietnamese aspect-based sentiment analysis, sentiment dependency learning, vietnamese natural language understanding, vietnamese sentiment analysis, vietnamese multimodal sentiment analysis, web page tagging, workflow discovery, answerability prediction, incongruity detection, multi-word expression embedding, multi-word expression sememe prediction, pcl detection, semeval-2022 task 4-1 (binary pcl detection), semeval-2022 task 4-2 (multi-label pcl detection), automatic writing, complaint comment classification, counterspeech detection, extractive text summarization, face selection, job classification, multi-lingual text-to-image generation, multlingual neural machine translation, optical charater recogntion, bangla text detection, question to declarative sentence, relation mention extraction.

Tweet-Reply Sentiment Analysis

Vietnamese fact checking, vietnamese parsing.

• ODSC EUROPE
• AI+ Training
• Speak at ODSC

• Data Analytics
• Data Engineering
• Data Visualization
• Deep Learning
• Generative AI
• Machine Learning
• NLP and LLMs
• Business & Use Cases
• Career Advice
• Write for us
• ODSC Community Slack Channel
• Upcoming Webinars

Top Recent NLP Research

Featured Post Modeling NLP & LLMs posted by Daniel Gutierrez, ODSC October 1, 2021 Daniel Gutierrez, ODSC

Natural language processing (NLP) including conversational AI is arguably one of the most exciting technology fields today. NLP is important because it works to resolve ambiguity in language and adds useful analytical structure to the data for a plethora of downstream applications such as speech recognition and text analytics. NLP helps computers communicate with humans in their own language and scales other language-centric tasks. For example, NLP makes it possible for computers to read text, listen to speech, interpret conversations, measure sentiment, and determine which segments are important. Even though budgets were hit hard by the pandemic, 53% of technical leaders said their NLP budget was at least 10% higher compared to 2019 . In addition, many NLP breakthroughs are moving from research to production, with much coming from recent NLP research.

The last couple of years have been big for NLP with a number of high-profile research efforts involving: generative pre-training model (GPT), transfer learning, transformers (e.g. BERT, ELMO), multilingual NLP, training models with reinforcement learning, automating customer service with a new era of chatbots, NLP for social media monitoring, fake news detection, and so much more. 

In this article, I’ll help get you up to speed with current NLP research efforts by curating a list of the top recent papers published with a variety of research destinations including: arXiv.org , The International Conference on Learning Representations (ICLR) , The Stanford NLP Group , NeurIPS , and KDD . Enjoy!

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Increasing model size when pretraining natural language representations often result in improved performance on downstream tasks. However, at some point, further model increases become harder due to GPU/TPU memory limitations and longer training times. To address these problems, this paper presents two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows that the proposed methods lead to models that scale much better compared to the original BERT. Also used is a self-supervised loss that focuses on modeling inter-sentence coherence, and shows it consistently helps downstream tasks with multi-sentence inputs. As a result, the best model from this NLP research establishes new state-of-the-art results on the GLUE, RACE, and \squad benchmarks while having fewer parameters compared to BERT-large. The GitHub repo associated with this paper can be found HERE . 

CogLTX: Applying BERT to Long Texts

BERTs are incapable of processing long texts due to quadratically increasing memory and time consumption. The attempts to address this problem, such as slicing the text by a sliding window or simplifying transformers, suffer from insufficient long-range attentions or need customized CUDA kernels. The limited text length of BERT reminds us of the limited capacity (5 ∼ 9 chunks) of the working memory of humans – then how do human beings “Cognize Long TeXts?” Founded on the cognitive theory stemming from Baddeley, the CogLTX framework described in this NLP research paper identifies key sentences by training a judge model, concatenates them for reasoning, and enables multi-step reasoning via rehearsal and decay. Since relevance annotations are usually unavailable, it is proposed to use treatment experiments to create supervision. As a general algorithm, CogLTX outperforms or gets comparable results to SOTA models on NewsQA, HotpotQA, multi-class, and multi-label long-text classification tasks with memory overheads independent of the text length.

ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, this NLP research paper proposes a more sample-efficient pre-training task called replaced token detection . Instead of masking the input, the new approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, the new approach trains a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by this approach substantially outperform the ones learned by BERT given the same model size, data, and compute. 

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pre-trained models with a differentiable access mechanism to explicit non-parametric memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. This NLP research paper explores a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) — models which combine pre-trained parametric and non-parametric memory for language generation. RAG models are introduced where the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. Two RAG formulations are compared, one which conditions on the same retrieved passages across the whole generated sequence, the other can use different passages per token.

ConvBERT: Improving BERT with Span-based Dynamic Convolution

Pre-trained language models like BERT and its variants have recently achieved impressive performance in various natural language understanding tasks. However, BERT heavily relies on the global self-attention block and thus suffers a large memory footprint and computation cost. Although all its attention heads query on the whole input sequence for generating the attention map from a global perspective, some heads only need to learn local dependencies, which means the existence of computation redundancy. This NLP research paper proposes a novel span-based dynamic convolution to replace these self-attention heads to directly model local dependencies. The novel convolution heads, together with the rest self-attention heads, form a new mixed attention block that is more efficient at both global and local context learning. BERT is equipped with this mixed attention design using a ConvBERT model. Experiments have shown that ConvBERT significantly outperforms BERT and its variants in various downstream tasks, with lower training costs and fewer model parameters. 

The Lottery Ticket Hypothesis for Pre-trained BERT Networks

In NLP, enormous pre-trained models like BERT have become the standard starting point for training on a range of downstream tasks, and similar trends are emerging in other areas of deep learning. In parallel, work on the lottery ticket hypothesis has shown that models for NLP and computer vision contain smaller matching subnetworks capable of training in isolation to full accuracy and transferring to other tasks. The work in this paper combines these observations to assess whether such trainable, transferrable subnetworks exist in pre-trained BERT models. For a range of downstream tasks, matching subnetworks at 40% to 90% sparsity is found. These subnetworks are found at (pre-trained) initialization, a deviation from prior NLP research where they emerge only after some amount of training. Subnetworks found on the masked language modeling task (the same task used to pre-train the model) transfer universally; those found on other tasks transfer in a limited fashion if at all. As large-scale pre-training becomes an increasingly central paradigm in deep learning, the results demonstrate that the main lottery ticket observations remain relevant in this context. The GitHub repo associated with this paper can be found HERE .

BERT Loses Patience: Fast and Robust Inference with Early Exit

This NLP research paper proposes Patience-based Early Exit , a straightforward yet effective inference method that can be used as a plug-and-play technique to simultaneously improve the efficiency and robustness of a pretrained language model (PLM). To achieve this, the approach couples an internal-classifier with each layer of a PLM and dynamically stops inference when the intermediate predictions of the internal classifiers do not change for a pre-defined number of steps. The approach improves inference efficiency as it allows the model to predict with fewer layers. Meanwhile, experimental results with an ALBERT model show that the method can improve the accuracy and robustness of the model by preventing it from overthinking and exploiting multiple classifiers for prediction, yielding a better accuracy-speed trade-off compared to existing early exit methods.

The Curious Case of Neural Text Degeneration

Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that even though the use of likelihood as a training objective leads to high-quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive. This NLP research paper reveals surprising distributional differences between human text and machine text. In addition, it’s found that decoding strategies alone can dramatically affect the quality of machine text, even when generated from exactly the same neural language model. The findings motivate Nucleus Sampling , a simple but effective method to draw the best out of neural generation. By sampling text from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution, the resulting text better demonstrates the quality of human text, yielding enhanced diversity without sacrificing fluency and coherence.

Encoding word order in complex embeddings

Sequential word order is important when processing text. Currently, neural networks (NNs) address this by modeling word position using position embeddings. The problem is that position embeddings capture the position of individual words, but not the ordered relationship (e.g., adjacency or precedence) between individual word positions. This NLP research paper presents a novel and principled solution for modeling both the global absolute positions of words and their order relationships. The solution generalizes word embeddings, previously defined as independent vectors, to continuous word functions over a variable (position). The benefit of continuous functions over variable positions is that word representations shift smoothly with increasing positions. Hence, word representations in different positions can correlate with each other in a continuous function. The general solution of these functions is extended to a complex-valued domain due to richer representations. CNN, RNN, and Transformer NNs are extended to complex-valued versions to incorporate complex embedding. 

Stanza: A Python Natural Language Processing Toolkit for Many Human Languages

This paper introduces Stanza , an open-source Python natural language processing toolkit supporting 66 human languages. Compared to existing widely used toolkits, Stanza features a language-agnostic fully neural pipeline for text analysis, including tokenization, multi-word token expansion, lemmatization, part-of-speech and morphological feature tagging, dependency parsing, and named entity recognition. Stanza was trained on a total of 112 datasets, including the Universal Dependencies treebanks and other multilingual corpora, and show that the same neural architecture generalizes well and achieves competitive performance on all languages tested. Additionally, Stanza includes a native Python interface to the widely used Java Stanford CoreNLP software, which further extends its functionality to cover other tasks such as coreference resolution and relation extraction. The GitHub repo associated with this NLP research paper, along with source code, documentation, and pretrained models for 66 languages can be found HERE . 

Mogrifier LSTM

Many advances in NLP have been based upon more expressive models for how inputs interact with the context in which they occur. Recurrent networks, which have enjoyed a modicum of success, still lack the generalization and systematicity ultimately required for modeling language. This NLP research paper proposes an extension to the venerable Long Short-Term Memory (LSTM) in the form of mutual gating of the current input and the previous output. This mechanism affords the modeling of a richer space of interactions between inputs and their context. Equivalently, the model can be viewed as making the transition function given by the LSTM context-dependent. 

DeFINE: Deep Factorized Input Token Embeddings for Neural Sequence Modeling

For sequence models with large vocabularies, a majority of network parameters lie in the input and output layers. This NLP research paper describes a new method, DeFINE, for learning deep token representations efficiently. The architecture uses a hierarchical structure with novel skip-connections which allows for the use of low dimensional input and output layers, reducing total parameters and training time while delivering similar or better performance versus existing methods. DeFINE can be incorporated easily in new or existing sequence models. Compared to state-of-the-art methods including adaptive input representations, this technique results in a 6% to 20% drop in perplexity. 

FreeLB: Enhanced Adversarial Training for Natural Language Understanding

Adversarial training, which minimizes the maximal risk for label-preserving input perturbations, has proved to be effective for improving the generalization of language models. This paper proposes a novel adversarial training algorithm, FreeLB, that promotes higher invariance in the embedding space, by adding adversarial perturbations to word embeddings and minimizing the resultant adversarial risk inside different regions around input samples. To validate the effectiveness of the proposed approach, it is applied to Transformer-based models for natural language understanding and commonsense reasoning tasks. Experiments on the GLUE benchmark show that when applied only to the finetuning stage, it is able to improve the overall test scores of BERT-base model from 78.3 to 79.4, and RoBERTa-large model from 88.5 to 88.8. The GitHub repo associated with this paper can be found HERE . 

Dynabench: Rethinking Benchmarking in NLP

This paper introduces Dynabench, an open-source platform for dynamic dataset creation and model benchmarking. Dynabench runs in a web browser and supports human-and-model-in-the-loop dataset creation: annotators seek to create examples that a target model will misclassify, but that another person will not. It is argued that Dynabench addresses a critical need in the NLP community: contemporary models quickly achieve outstanding performance on benchmark tasks but nonetheless fail on simple challenge examples and falter in real-world scenarios. With Dynabench, dataset creation, model development, and model assessment can directly inform each other, leading to more robust and informative benchmarks. The paper reports on four initial NLP tasks, illustrating these concepts and highlighting the promise of the platform, and address potential objections to dynamic benchmarking as a new standard for the field.

Causal Effects of Linguistic Properties

This paper considers the problem of using observational data to estimate the causal effects of linguistic properties. For example, does writing a complaint politely lead to a faster response time? How much will a positive product review increase sales? This paper addresses two technical challenges related to the problem before developing a practical method. First, formalize the causal quantity of interest as the effect of a writer’s intent, and establish the assumptions necessary to identify this from observational data. Second, in practice, access is only offered to noisy proxies for the linguistic properties of interest—e.g., predictions from classifiers and lexicons. An estimator is proposed for this setting and proof that its bias is bounded when we perform an adjustment for the text. Based on these results, TEXTCAUSE is introduced, an algorithm for estimating causal effects of linguistic properties. The method leverages (1) distant supervision to improve the quality of noisy proxies, and (2) a pre-trained language model (BERT) to adjust for the text. It is shown that the proposed method outperforms related approaches when estimating the effect of Amazon review sentiment on semi-simulated sales figures.

LM-Critic: Language Models for Unsupervised Grammatical Error Correction

Training a model for grammatical error correction (GEC) requires a set of labeled ungrammatical/grammatical sentence pairs, but manually annotating such pairs can be expensive. Recently, the Break-It-Fix-It (BIFI) framework has demonstrated strong results on learning to repair a broken program without any labeled examples, but this relies on a perfect critic (e.g., a compiler) that returns whether an example is valid or not, which does not exist for the GEC task. This paper shows how to leverage a pretrained language model (LM) in defining an LM-Critic, which judges a sentence to be grammatical if the LM assigns it a higher probability than its local perturbations. This LM-Critic and BIFI is applied along with a large set of unlabeled sentences to bootstrap realistic ungrammatical/grammatical pairs for training a corrector. 

Generative Adversarial Transformers

This paper introduces the GANformer, a novel and efficient type of transformer, and explores it for the task of visual generative modeling. The network employs a bipartite structure that enables long-range interactions across the image, while maintaining computation of linear efficiency, that can readily scale to high-resolution synthesis. It iteratively propagates information from a set of latent variables to the evolving visual features and vice versa, to support the refinement of each in light of the other and encourage the emergence of compositional representations of objects and scenes. In contrast to the classic transformer architecture, it utilizes multiplicative integration that allows flexible region-based modulation, and can thus be seen as a generalization of the successful StyleGAN network. The model’s strength and robustness are demonstrated through a careful evaluation over a range of datasets, from simulated multi-object environments to rich real-world indoor and outdoor scenes, showing it achieves state-of-the-art results in terms of image quality and diversity, while enjoying fast learning and better data efficiency. The GitHub repo associated with this paper can be found HERE .

Learn More About NLP and NLP Research at ODSC West 2021

At our upcoming event this November 16th-18th in San Francisco,  ODSC West 2021 will feature a plethora of talks, workshops, and training sessions on NLP and NLP research. You can register now for 30% off all ticket types before the discount drops to 20% in a few weeks. Some highlighted sessions on NLP and NLP research  include:

• Transferable Representation in Natural Language Processing: Kai-Wei Chang, PhD | Director/Assistant Professor | UCLA NLP/UCLA CS
• Build a Question Answering System using DistilBERT in Python: Jayeeta Putatunda | Data Scientist | MediaMath
• Introduction to NLP and Topic Modeling: Zhenya Antić, PhD | NLP Consultant/Founder | Practical Linguistics Inc
• NLP Fundamentals: Leonardo De Marchi | Lead Instructor | ideai.io

Sessions on Deep Learning and Deep Learning Research:

• GANs: Theory and Practice, Image Synthesis With GANs Using TensorFlow: Ajay Baranwal | Center Director | Center for Deep Learning in Electronic Manufacturing, Inc
• Machine Learning With Graphs: Going Beyond Tabular Data: Dr. Clair J. Sullivan | Data Science Advocate | Neo4j
• Deep Dive into Reinforcement Learning with PPO using TF-Agents & TensorFlow 2.0: Oliver Zeigermann | Software Developer | embarc Software Consulting GmbH
• Get Started with Time-Series Forecasting using the Google Cloud AI Platform: Karl Weinmeister | Developer Relations Engineering Manager | Google

Sessions on Machine Learning:

• Towards More Energy-Efficient Neural Networks? Use Your Brain!: Olaf de Leeuw | Data Scientist | Dataworkz
• Practical MLOps: Automation Journey: Evgenii Vinogradov, PhD | Head of DHW Development | YooMoney
• Applications of Modern Survival Modeling with Python: Brian Kent, PhD | Data Scientist | Founder The Crosstab Kite
• Using Change Detection Algorithms for Detecting Anomalous Behavior in Large Systems: Veena Mendiratta, PhD | Adjunct Faculty, Network Reliability, and Analytics Researcher | Northwestern University

Daniel Gutierrez, ODSC

Daniel D. Gutierrez is a practicing data scientist who’s been working with data long before the field came in vogue. As a technology journalist, he enjoys keeping a pulse on this fast-paced industry. Daniel is also an educator having taught data science, machine learning and R classes at the university level. He has authored four computer industry books on database and data science technology, including his most recent title, “Machine Learning and Data Science: An Introduction to Statistical Learning Methods with R.” Daniel holds a BS in Mathematics and Computer Science from UCLA.

How AI Impacts Diversity in College Admissions

Business + Management posted by Zac Amos Aug 27, 2024 Higher education institutions are beginning to use artificial intelligence to review college applications. Where does this...

Scaling AI Initiatives in Retail

West 2024 Conferences posted by ODSC Community Aug 27, 2024 Editor’s note: Fatih is a speaker for ODSC West this October 29th-31st! Be sure to check...

Just Do Something with AI: Bridging the Business Communication Gap for ML Practitioners

West 2024 Career Insights posted by ODSC Community Aug 26, 2024 Editor’s note: Stephanie Kirmer is a speaker for ODSC West this October 29th-31st. Be sure to...

• Skip to main content
• Skip to primary sidebar
• Skip to footer

The Best of Applied Artificial Intelligence, Machine Learning, Automation, Bots, Chatbots

GPT-3 & Beyond: 10 NLP Research Papers You Should Read

November 17, 2020 by Mariya Yao

NLP research advances in 2020 are still dominated by large pre-trained language models, and specifically transformers. There were many interesting updates introduced this year that have made transformer architecture more efficient and applicable to long documents.

Another hot topic relates to the evaluation of NLP models in different applications. We still lack evaluation approaches that clearly show where a model fails and how to fix it.

Also, with the growing capabilities of language models such as GPT-3, conversational AI is enjoying a new wave of interest. Chatbots are improving, with several impressive bots like Meena and Blender introduced this year by top technology companies.

To help you stay up to date with the latest NLP research breakthroughs, we’ve curated and summarized the key research papers in natural language processing from 2020. The papers cover the leading language models, updates to the transformer architecture, novel evaluation approaches, and major advances in conversational AI.

Subscribe to our AI Research mailing list at the bottom of this article to be alerted when we release new summaries.

If you’d like to skip around, here are the papers we featured:

• WinoGrande: An Adversarial Winograd Schema Challenge at Scale
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Reformer: The Efficient Transformer
• Longformer: The Long-Document Transformer
• ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
• Language Models are Few-Shot Learners
• Beyond Accuracy: Behavioral Testing of NLP models with CheckList
• Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics
• Towards a Human-like Open-Domain Chatbot
• Recipes for Building an Open-Domain Chatbot

Best NLP Research Papers 2020

1. winogrande: an adversarial winograd schema challenge at scale , by keisuke sakaguchi, ronan le bras, chandra bhagavatula, yejin choi, original abstract .

The Winograd Schema Challenge (WSC) (Levesque, Davis, and Morgenstern 2011), a benchmark for commonsense reasoning, is a set of 273 expert-crafted pronoun resolution problems originally designed to be unsolvable for statistical models that rely on selectional preferences or word associations. However, recent advances in neural language models have already reached around 90% accuracy on variants of WSC. This raises an important question whether these models have truly acquired robust commonsense capabilities or whether they rely on spurious biases in the datasets that lead to an overestimation of the true capabilities of machine commonsense. 

To investigate this question, we introduce WinoGrande, a large-scale dataset of 44k problems, inspired by the original WSC design, but adjusted to improve both the scale and the hardness of the dataset. The key steps of the dataset construction consist of (1) a carefully designed crowdsourcing procedure, followed by (2) systematic bias reduction using a novel AfLite algorithm that generalizes human-detectable word associations to machine-detectable embedding associations. The best state-of-the-art methods on WinoGrande achieve 59.4-79.1%, which are 15-35% below human performance of 94.0%, depending on the amount of the training data allowed. 

Furthermore, we establish new state-of-the-art results on five related benchmarks – WSC (90.1%), DPR (93.1%), COPA (90.6%), KnowRef (85.6%), and Winogender (97.1%). These results have dual implications: on one hand, they demonstrate the effectiveness of WinoGrande when used as a resource for transfer learning. On the other hand, they raise a concern that we are likely to be overestimating the true capabilities of machine commonsense across all these benchmarks. We emphasize the importance of algorithmic bias reduction in existing and future benchmarks to mitigate such overestimation.

Our Summary 

The research group from the Allen Institute for Artificial Intelligence introduces WinoGrande , a new benchmark for commonsense reasoning. They build on the design of the famous Winograd Schema Challenge (WSC) benchmark but significantly increase the scale of the dataset to 44K problems and reduce systematic bias using a novel AfLite algorithm. The experiments demonstrate that state-of-the-art methods achieve up to 79.1% accuracy on WinoGrande, which is significantly below the human performance of 94%. Furthermore, the researchers show that WinoGrande is an effective resource for transfer learning, by using a RoBERTa model fine-tuned with WinoGrande to achieve new state-of-the-art results on WSC and four other related benchmarks.

What’s the core idea of this paper?

• The authors claim that existing benchmarks for commonsense reasoning suffer from systematic bias and annotation artifacts, leading to overestimation of the true capabilities of machine intelligence on commonsense reasoning.
• Crowdworkers were asked to write twin sentences that meet the WSC requirements and contain certain anchor words. This new requirement is aimed at improving the creativity of crowdworkers.
• Collected problems were validated through a distinct set of three crowdworkers. Out of 77K collected questions, 53K were deemed valid.
• It generalizes human-detectable biases based on word occurrences to machine-detectable biases based on embedding occurrences.
• After applying the AfLite algorithm, the debiased WinoGrande dataset contains 44K samples. 

What’s the key achievement?

• Wino Knowledge Hunting (WKH) and Ensemble LMs only achieve chance-level performance (50%);
• RoBERTa achieves 79.1% test-set accuracy;
• whereas human performance achieves 94% accuracy.
• 90.1% on WSC;
• 93.1% on DPR ; 
• 90.6% on COPA ; 
• 85.6% on KnowRef ; and 
• 97.1% on Winogender .

What does the AI community think?

• The paper received the Outstanding Paper Award at AAAI 2020, one of the key conferences in artificial intelligence.

What are future research areas?

• Exploring new algorithmic approaches for systematic bias reduction.
• Debiasing other NLP benchmarks.

Where can you get implementation code?

• The dataset can be downloaded from the WinoGrande project page .
• The implementation code is available on GitHub .
• And here is the WinoGrande leaderboard .

2. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , by Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.

The Google research team suggests a unified approach to transfer learning in NLP with the goal to set a new state of the art in the field. To this end, they propose treating each NLP problem as a “text-to-text” problem. Such a framework allows using the same model, objective, training procedure, and decoding process for different tasks, including summarization, sentiment analysis, question answering, and machine translation. The researchers call their model a Text-to-Text Transfer Transformer (T5) and train it on the large corpus of web-scraped data to get state-of-the-art results on a number of NLP tasks.

• Providing a comprehensive perspective on where the NLP field stands by exploring and comparing existing techniques.
• The mode understands which tasks should be performed thanks to the task-specific prefix added to the original input sentence (e.g., “translate English to German:”, “summarize:”).
• Presenting and releasing a new dataset consisting of hundreds of gigabytes of clean web-scraped English text, the Colossal Clean Crawled Corpus (C4) .
• Training a large (up to 11B parameters) model, called Text-to-Text Transfer Transformer (T5) on the C4 dataset.
• the GLUE score of 89.7 with substantially improved performance on CoLA, RTE, and WNLI tasks;
• the Exact Match score of 90.06 on SQuAD dataset;
• the SuperGLUE score of 88.9, which is a very significant improvement over the previous state-of-the-art result (84.6) and very close to human performance (89.8);
• the ROUGE-2-F score of 21.55 on CNN/Daily Mail abstractive summarization task.
• Researching the methods to achieve stronger performance with cheaper models.
• Exploring more efficient knowledge extraction techniques.
• Further investigating the language-agnostic models.

What are possible business applications?

• Even though the introduced model has billions of parameters and can be too heavy to be applied in the business setting, the presented ideas can be used to improve the performance on different NLP tasks, including summarization, question answering, and sentiment analysis.
• The pretrained models together with the dataset and code are released on GitHub .

3. Reformer: The Efficient Transformer , by Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya

Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of Transformers. For one, we replace dot-product attention by one that uses locality-sensitive hashing, changing its complexity from O( L 2 ) to O( L log L ), where L is the length of the sequence. Furthermore, we use reversible residual layers instead of the standard residuals, which allows storing activations only once in the training process instead of N times, where N is the number of layers. The resulting model, the Reformer, performs on par with Transformer models while being much more memory-efficient and much faster on long sequences.

The leading Transformer models have become so big that they can be realistically trained only in large research laboratories. To address this problem, the Google Research team introduces several techniques that improve the efficiency of Transformers. In particular, they suggest (1) using reversible layers to allow storing the activations only once instead of for each layer, and (2) using locality-sensitive hashing to avoid costly softmax computation in the case of full dot-product attention. Experiments on several text tasks demonstrate that the introduced Reformer model matches the performance of the full Transformer but runs much faster and with much better memory efficiency.

• The activations of every layer need to be stored for back-propagation.
• The intermediate feed-forward layers account for a large fraction of memory use since their depth is often much larger than the depth of attention activations.
• The complexity of attention on a sequence of length L is O( L 2 ).
• using reversible layers to store only a single copy of activations;
• splitting activations inside the feed-forward layers and processing them in chunks;
• approximating attention computation based on locality-sensitive hashing .
• switching to locality-sensitive hashing attention;
• using reversible layers.
• For example, on the newstest2014 task for machine translation from English to German, the Reformer base model gets a BLEU score of 27.6 compared to Vaswani’s et al. (2017) BLEU score of 27.3. 
• The paper was selected for oral presentation at ICLR 2020, the leading conference in deep learning.
• text generation;
• visual content generation;
• music generation;
• time-series forecasting.
• The official code implementation from Google is publicly available on GitHub .
• The PyTorch implementation of Reformer is also available on GitHub .

4. Longformer: The Long-Document Transformer , by Iz Beltagy, Matthew E. Peters, Arman Cohan

Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer’s attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA.

Self-attention is one of the key factors behind the success of Transformer architecture. However, it also makes transformer-based models hard to apply to long documents. The existing techniques usually divide the long input into a number of chunks and then use complex architectures to combine information across these chunks. The research team from the Allen Institute for Artificial Intelligence introduces a more elegant solution to this problem. The suggested Longformer model employs an attention pattern that combines local windowed attention with task-motivated global attention. This attention mechanism scales linearly with the sequence length and enables processing of documents with thousands of tokens. The experiments demonstrate that Longformer achieves state-of-the-art results on character-level language modeling tasks, and when pre-trained, consistently outperforms RoBERTa on long-document tasks.

• The computational requirements of self-attention grow quadratically with sequence length, making it hard to process on current hardware. 
• allows memory usage to scale linearly, and not quadratically, with the sequence length;
• a windowed local-context self-attention to build contextual representations;
• an end task motivated global attention to encode inductive bias about the task and build full sequence representation.
• Since the implementation of the sliding window attention pattern requires a form of banded matrix multiplication that is not supported in the existing deep learning libraries like PyTorch and Tensorflow, the authors also introduce a custom CUDA kernel for implementing these attention operations.
• BPC of 1.10 on text8 ;
• BPC of 1.00 on enwik8 .
• accuracy of 75.0 vs. 72.4 on WikiHop ;
• F1 score of 75.2 vs. 74.2 on TriviaQA ;
• joint F1 score of 64.4 vs. 63.5 on HotpotQA ;
• average F1 score of 78.6 vs. 78.4 on the OntoNotes coreference resolution task;
• accuracy of 95.7 vs. 95.3 on the IMDB classification task;
• F1 score of 94.0 vs. 87.4 on the Hyperpartisan classification task.
• The performance gains are especially remarkable for the tasks that require a long context (i.e., WikiHop and Hyperpartisan).
• Exploring other attention patterns that are more efficient due to dynamic adaptation to the input. 
• Applying Longformer to other relevant long document tasks such as summarization.
• document classification;
• question answering;
• coreference resolution;
• summarization;
• semantic search.
• The code implementation of Longformer is open-sourced on GitHub .

5. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators , by Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning

Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30× more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.

The pre-training task for popular language models like BERT and XLNet involves masking a small subset of unlabeled input and then training the network to recover this original input. Even though it works quite well, this approach is not particularly data-efficient as it learns from only a small fraction of tokens (typically ~15%). As an alternative, the researchers from Stanford University and Google Brain propose a new pre-training task called replaced token detection . Instead of masking, they suggest replacing some tokens with plausible alternatives generated by a small language model. Then, the pre-trained discriminator is used to predict whether each token is an original or a replacement. As a result, the model learns from all input tokens instead of the small masked fraction, making it much more computationally efficient. The experiments confirm that the introduced approach leads to significantly faster training and higher accuracy on downstream NLP tasks.

• Pre-training methods that are based on masked language modeling are computationally inefficient as they use only a small fraction of tokens for learning.
• some tokens are replaced by samples from a small generator network; 
• a model is pre-trained as a discriminator to distinguish between original and replaced tokens.
• enables the model to learn from all input tokens instead of the small masked-out subset;
• is not adversarial, despite the similarity to GAN, as the generator producing tokens for replacement is trained with maximum likelihood.
• Demonstrating that the discriminative task of distinguishing between real data and challenging negative samples is more efficient than existing generative methods for language representation learning.
• ELECTRA-Small gets a GLUE score of 79.9 and outperforms a comparably small BERT model with a score of 75.1 and a much larger GPT model with a score of 78.8.
• An ELECTRA model that performs comparably to XLNet and RoBERTa uses only 25% of their pre-training compute.
• ELECTRA-Large outscores the alternative state-of-the-art models on the GLUE and SQuAD benchmarks while still requiring less pre-training compute.
• The paper was selected for presentation at ICLR 2020, the leading conference in deep learning.
• Because of its computational efficiency, the ELECTRA approach can make the application of pre-trained text encoders more accessible to business practitioners.
• The original TensorFlow implementation and pre-trained weights are released on GitHub .

6. Language Models are Few-Shot Learners , by Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions – something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10× more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3’s few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

The OpenAI research team draws attention to the fact that the need for a labeled dataset for every new language task limits the applicability of language models. Considering that there is a wide range of possible tasks and it’s often difficult to collect a large labeled training dataset, the researchers suggest an alternative solution, which is scaling up language models to improve task-agnostic few-shot performance. They test their solution by training a 175B-parameter autoregressive language model, called GPT-3 , and evaluating its performance on over two dozen NLP tasks. The evaluation under few-shot learning, one-shot learning, and zero-shot learning demonstrates that GPT-3 achieves promising results and even occasionally outperforms the state of the art achieved by fine-tuned models.

• The GPT-3 model uses the same model and architecture as GPT-2, including the modified initialization, pre-normalization, and reversible tokenization.
• However, in contrast to GPT-2, it uses alternating dense and locally banded sparse attention patterns in the layers of the transformer, as in the Sparse Transformer .
• Few-shot learning , when the model is given a few demonstrations of the task (typically, 10 to 100) at inference time but with no weight updates allowed.
• One-shot learning , when only one demonstration is allowed, together with a natural language description of the task.
• Zero-shot learning , when no demonstrations are allowed and the model has access only to a natural language description of the task.
• On the CoQA benchmark, 81.5 F1 in the zero-shot setting, 84.0 F1 in the one-shot setting, and 85.0 F1 in the few-shot setting, compared to the 90.7 F1 score achieved by fine-tuned SOTA.
• On the TriviaQA benchmark, 64.3% accuracy in the zero-shot setting, 68.0% in the one-shot setting, and 71.2% in the few-shot setting, surpassing the state of the art (68%) by 3.2%.
• On the LAMBADA dataset, 76.2 % accuracy in the zero-shot setting, 72.5% in the one-shot setting, and 86.4% in the few-shot setting, surpassing the state of the art (68%) by 18%.
• The news articles generated by the 175B-parameter GPT-3 model are hard to distinguish from real ones, according to human evaluations (with accuracy barely above the chance level at ~52%).
• “The GPT-3 hype is way too much. It’s impressive (thanks for the nice compliments!) but it still has serious weaknesses and sometimes makes very silly mistakes. AI is going to change the world, but GPT-3 is just a very early glimpse. We have a lot still to figure out.” – Sam Altman, CEO and co-founder of OpenAI .
• “I’m shocked how hard it is to generate text about Muslims from GPT-3 that has nothing to do with violence… or being killed…” – Abubakar Abid, CEO and founder of Gradio .
• “No. GPT-3 fundamentally does not understand the world that it talks about. Increasing corpus further will allow it to generate a more credible pastiche but not fix its fundamental lack of comprehension of the world. Demos of GPT-4 will still require human cherry picking.” – Gary Marcus, CEO and founder of Robust.ai .
• “Extrapolating the spectacular performance of GPT3 into the future suggests that the answer to life, the universe and everything is just 4.398 trillion parameters.” – Geoffrey Hinton, Turing Award winner .
• Improving pre-training sample efficiency.
• Exploring how few-shot learning works.
• Distillation of large models down to a manageable size for real-world applications.
• The model with 175B parameters is hard to apply to real business problems due to its impractical resource requirements, but if the researchers manage to distill this model down to a workable size, it could be applied to a wide range of language tasks, including question answering, dialog agents, and ad copy generation.
• The code itself is not available, but some dataset statistics together with unconditional, unfiltered 2048-token samples from GPT-3 are released on GitHub .

7. Beyond Accuracy: Behavioral Testing of NLP models with CheckList , by Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, Sameer Singh

Although measuring held-out accuracy has been the primary approach to evaluate generalization, it often overestimates the performance of NLP models, while alternative approaches for evaluating models either focus on individual tasks or on specific behaviors. Inspired by principles of behavioral testing in software engineering, we introduce CheckList, a task-agnostic methodology for testing NLP models. CheckList includes a matrix of general linguistic capabilities and test types that facilitate comprehensive test ideation, as well as a software tool to generate a large and diverse number of test cases quickly. We illustrate the utility of CheckList with tests for three tasks, identifying critical failures in both commercial and state-of-art models. In a user study, a team responsible for a commercial sentiment analysis model found new and actionable bugs in an extensively tested model. In another user study, NLP practitioners with CheckList created twice as many tests, and found almost three times as many bugs as users without it.

The authors point out the shortcomings of existing approaches to evaluating performance of NLP models. A single aggregate statistic, like accuracy, makes it difficult to estimate where the model is failing and how to fix it. The alternative evaluation approaches usually focus on individual tasks or specific capabilities. To address the lack of comprehensive evaluation approaches, the researchers introduce CheckList , a new evaluation methodology for testing of NLP models. The approach is inspired by principles of behavioral testing in software engineering. Basically, CheckList is a matrix of linguistic capabilities and test types that facilitates test ideation. Multiple user studies demonstrate that CheckList is very effective at discovering actionable bugs, even in extensively tested NLP models.

• The primary approach to the evaluation of models’ generalization capabilities, which is accuracy on held-out data, may lead to performance overestimation, as the held-out data often contains the same biases as the training data. Moreover, this single aggregate statistic doesn’t help much in figuring out where the NLP model is failing and how to fix these bugs.
• The alternative approaches are usually designed for evaluation of specific behaviors on individual tasks and thus, lack comprehensiveness.
• CheckList provides users with a list of linguistic capabilities to be tested, like vocabulary, named entity recognition, and negation.
• Then, to break down potential capability failures into specific behaviors, CheckList suggests different test types , such as prediction invariance or directional expectation tests in case of certain perturbations.
• Potential tests are structured as a matrix, with capabilities as rows and test types as columns.
• The suggested implementation of CheckList also introduces a variety of abstractions to help users generate large numbers of test cases easily.
• Evaluation of state-of-the-art models with CheckList demonstrated that even though some NLP tasks are considered “solved” based on accuracy results, the behavioral testing highlights many areas for improvement.
• helps to identify and test for capabilities not previously considered;
• results in more thorough and comprehensive testing for previously considered capabilities;
• helps to discover many more actionable bugs.
• The paper received the Best Paper Award at ACL 2020, the leading conference in natural language processing.
• CheckList can be used to create more exhaustive testing for a variety of NLP tasks.
• Such comprehensive testing that helps in identifying many actionable bugs is likely to lead to more robust NLP systems.
• The code for testing NLP models with CheckList is available on GitHub .

8. Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics , by Nitika Mathur, Timothy Baldwin, Trevor Cohn

Automatic metrics are fundamental for the development and evaluation of machine translation systems. Judging whether, and to what extent, automatic metrics concur with the gold standard of human evaluation is not a straightforward problem. We show that current methods for judging metrics are highly sensitive to the translations used for assessment, particularly the presence of outliers, which often leads to falsely confident conclusions about a metric’s efficacy. Finally, we turn to pairwise system ranking, developing a method for thresholding performance improvement under an automatic metric against human judgements, which allows quantification of type I versus type II errors incurred, i.e., insignificant human differences in system quality that are accepted, and significant human differences that are rejected. Together, these findings suggest improvements to the protocols for metric evaluation and system performance evaluation in machine translation.

The most recent Conference on Machine Translation (WMT) has revealed that, based on Pearson’s correlation coefficient, automatic metrics poorly match human evaluations of translation quality when comparing only a few best systems. Even negative correlations were exhibited in some instances. The research team from the University of Melbourne investigates this issue by studying the role of outlier systems, exploring how the correlation coefficient reflects different patterns of errors (type I vs. type II errors), and what magnitude of difference in the metric score corresponds to true improvements in translation quality as judged by humans. Their findings suggest that small BLEU differences (i.e., 1–2 points) have little meaning and other metrics, such as chrF, YiSi-1, and ESIM should be preferred over BLEU. However, only human evaluations can be a reliable basis for drawing important empirical conclusions.

• Automatic metrics are used as a proxy for human translation evaluation, which is considerably more expensive and time-consuming.
• For example, the recent findings show that if the correlation between leading metrics and human evaluations is computed using a large set of translation systems, it is typically very high (i.e., 0.9). However, if only a few best systems are considered, the correlation reduces markedly and can even be negative in some cases.
• The identified problem with Pearson’s correlation is due to the small sample size and not specific to comparing strong MT systems.
• Outlier systems, whose quality is much higher or lower than the rest of the systems, have a disproportionate effect on the computed correlation and should be removed.
• The same correlation coefficient can reflect different patterns of errors. Thus, a better approach for gaining insights into metric reliability is to visualize metric scores against human scores.
• Small BLEU differences of 1-2 points correspond to true improvements in translation quality (as judged by humans) only in 50% of cases.
• Giving preference to such evaluation metrics as chrF, YiSi-1, and ESIM over BLEU and TER.
• Moving away from using small changes in evaluation metrics as the sole basis to draw important empirical conclusions, and always ensuring support from human evaluations before claiming that one MT system significantly outperforms another one.
• The paper received an Honorable Mention at ACL 2020, the leading conference in natural language processing. 
• The implementation code, data, and additional analysis will be released on GitHub .

9. Towards a Human-like Open-Domain Chatbot , by Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, Quoc V. Le

We present Meena, a multi-turn open-domain chatbot trained end-to-end on data mined and filtered from public domain social media conversations. This 2.6B parameter neural network is simply trained to minimize perplexity of the next token. We also propose a human evaluation metric called Sensibleness and Specificity Average (SSA), which captures key elements of a human-like multi-turn conversation. Our experiments show strong correlation between perplexity and SSA. The fact that the best perplexity end-to-end trained Meena scores high on SSA (72% on multi-turn evaluation) suggests that a human-level SSA of 86% is potentially within reach if we can better optimize perplexity. Additionally, the full version of Meena (with a filtering mechanism and tuned decoding) scores 79% SSA, 23% higher in absolute SSA than the existing chatbots we evaluated. 

In contrast to most modern conversational agents, which are highly specialized, the Google research team introduces a chatbot Meena that can chat about virtually anything. It’s built on a large neural network with 2.6B parameters trained on 341 GB of text. The researchers also propose a new human evaluation metric for open-domain chatbots, called Sensibleness and Specificity Average (SSA), which can capture important attributes for human conversation. They demonstrate that this metric correlates highly with perplexity, an automatic metric that is readily available. Thus, the Meena chatbot, which is trained to minimize perplexity, can conduct conversations that are more sensible and specific compared to other chatbots. Particularly, the experiments demonstrate that Meena outperforms existing state-of-the-art chatbots by a large margin in terms of the SSA score (79% vs. 56%) and is closing the gap with human performance (86%).

• Despite recent progress, open-domain chatbots still have significant weaknesses: their responses often do not make sense or are too vague or generic.
• Meena is built on a seq2seq model with Evolved Transformer (ET) that includes 1 ET encoder block and 13 ET decoder blocks.
• The model is trained on multi-turn conversations with the input sequence including all turns of the context (up to 7) and the output sequence being the response.
• making sense,
• being specific.
• The research team discovered that the SSA metric shows high negative correlation (R2 = 0.93) with perplexity, a readily available automatic metric that Meena is trained to minimize.
• Proposing a simple human-evaluation metric for open-domain chatbots.
• The best end-to-end trained Meena model outperforms existing state-of-the-art open-domain chatbots by a large margin, achieving an SSA score of 72% (vs. 56%).
• Furthermore, the full version of Meena, with a filtering mechanism and tuned decoding, further advances the SSA score to 79%, which is not far from the 86% SSA achieved by the average human.
• “Google’s “Meena” chatbot was trained on a full TPUv3 pod (2048 TPU cores) for 30 full days – that’s more than $1,400,000 of compute time to train this chatbot model.” – Elliot Turner, CEO and founder of Hyperia .
• “So I was browsing the results for the new Google chatbot Meena, and they look pretty OK (if boring sometimes). However, every once in a while it enters ‘scary sociopath mode,’ which is, shall we say, sub-optimal” – Graham Neubig, Associate professor at Carnegie Mellon University .

• Lowering the perplexity through improvements in algorithms, architectures, data, and compute.
• Considering other aspects of conversations beyond sensibleness and specificity, such as, for example, personality and factuality.
• Tackling safety and bias in the models.
• further humanizing computer interactions; 
• improving foreign language practice; 
• making interactive movie and videogame characters relatable.
• Considering the challenges related to safety and bias in the models, the authors haven’t released the Meena model yet. However, they are still evaluating the risks and benefits and may decide otherwise in the coming months.

10. Recipes for Building an Open-Domain Chatbot , by Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, Jason Weston

Building open-domain chatbots is a challenging area for machine learning research. While prior work has shown that scaling neural models in the number of parameters and the size of the data they are trained on gives improved results, we show that other ingredients are important for a high-performing chatbot. Good conversation requires a number of skills that an expert conversationalist blends in a seamless way: providing engaging talking points and listening to their partners, and displaying knowledge, empathy and personality appropriately, while maintaining a consistent persona. We show that large scale models can learn these skills when given appropriate training data and choice of generation strategy. We build variants of these recipes with 90M, 2.7B and 9.4B parameter models, and make our models and code publicly available. Human evaluations show our best models are superior to existing approaches in multi-turn dialogue in terms of engagingness and humanness measurements. We then discuss the limitations of this work by analyzing failure cases of our models. 

The Facebook AI Research team shows that with appropriate training data and generation strategy, large-scale models can learn many important conversational skills, such as engagingness, knowledge, empathy, and persona consistency. Thus, to build their state-of-the-art conversational agent, called BlenderBot , they leveraged a model with 9.4B parameters, trained it on a novel task called Blended Skill Talk , and deployed beam search with carefully selected hyperparameters as a generation strategy. Human evaluations demonstrate that BlenderBot outperforms Meena in pairwise comparison 75% to 25% in terms of engagingness and 65% to 35% in terms of humanness.

• Large scale. The largest model has 9.4 billion parameters and was trained on 1.5 billion training examples of extracted conversations.
• Blended skills. The chatbot was trained on the Blended Skill Talk task to learn such skills as engaging use of personality, engaging use of knowledge, and display of empathy.
• Beam search used for decoding. The researchers show that this generation strategy, deployed with carefully selected hyperparameters, gives strong results. In particular, it was demonstrated that the lengths of the agent’s utterances is very important for chatbot performance (i.e, too short responses are often considered dull and too long responses make the chatbot appear to waffle and not listen).
• 75% of the time in terms of engagingness;
• 65% of the time in terms of humanness.
• In an A/B comparison between human-to-human and human-to-BlenderBot conversations, the latter were preferred 49% of the time as more engaging.
• a lack of in-depth knowledge if sufficiently interrogated; 
• a tendency to use simpler language; 
• a tendency to repeat oft-used phrases.
• Further exploring unlikelihood training and retrieve-and-refine mechanisms as potential avenues for fixing these issues.
• Facebook AI open-sourced BlenderBot by releasing code to fine-tune the conversational agent, the model weights, and code to evaluate it.

If you like these research summaries, you might be also interested in the following articles:

• 2020’s Top AI & Machine Learning Research Papers
• Novel Computer Vision Research Papers From 2020
• AAAI 2021: Top Research Papers With Business Applications
• ICLR 2021: Key Research Papers

Enjoy this article? Sign up for more AI research updates.

We’ll let you know when we release more summary articles like this one.

• Email Address *
• Name * First Last
• Natural Language Processing (NLP)
• Chatbots & Conversational AI
• Computer Vision
• Ethics & Safety
• Machine Learning
• Deep Learning
• Reinforcement Learning
• Generative Models
• Other (Please Describe Below)
• What is your biggest challenge with AI research? *
• Email This field is for validation purposes and should be left unchanged.

Reader Interactions

About Mariya Yao

Mariya is the co-author of Applied AI: A Handbook For Business Leaders and former CTO at Metamaven. She "translates" arcane technical concepts into actionable business advice for executives and designs lovable products people actually want to use. Follow her on Twitter at @thinkmariya to raise your AI IQ.

December 9, 2023 at 3:26 am

Thank you so much for sharing such great information with us. Your website is great. We are impressed by the details you have on your website. We have bookmarked this site. keep it up and thanks again.

March 14, 2024 at 8:49 am

quibusdam recusandae sint delectus nobis inventore dolorem nostrum omnis eum voluptas. autem blanditiis voluptatum delectus non asperiores sit. itaque accusamus voluptas repudiandae sed officiis. consectetur aut officia molestiae voluptatem sint quasi illum voluptate.

March 20, 2024 at 11:05 pm

Your article helped me a lot, is there any more related content? Thanks!

March 29, 2024 at 9:49 am

Leave a Reply

You must be logged in to post a comment.

About TOPBOTS

• Expert Contributors
• Terms of Service & Privacy Policy
• Contact TOPBOTS

Subscribe to our biweekly newsletter for the latest in applied AI, cutting-edge research updates, funding deals, and promising AI startups.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts

Collection  20 April 2023

Advances in Natural Language Processing

Natural language processing (NLP) is an interdisciplinary field spanning computational science and artificial intelligence (AI), concerned with the understanding of human language, in both written and verbal forms, by machines. NLP puts an emphasis on employing machine learning and deep learning techniques to complete tasks, like language translation or question answering. In the growing NLP domain, two main methodological branches can be distinguished: natural language understanding (NLU), which aims to improve the machine's reading comprehension, and natural language generation (NLG), focused on enabling machines to produce human language text responses based on a given data input.

In the modern world, the number of NLP applications seems to be following an exponential growth curve: from highly agile chatbots, to sentiment analysis and intent classification, to personalised medicine, the NLP's capacity for improving our lives is ever-growing. At the same time, NLP progress is halted by the limited AI hardware infrastructure which struggles to accommodate more refined NLP models, the sparsity of good-quality NLP-training data, and complex linguistic problems, such as machine's understanding of homonymy or generation of polysemy.

This Collection is dedicated to the latest research on methodology in the vast field of NLP, which addresses and carries the potential to solve at least one of the many struggles the state-of-the-art NLP approaches face. We welcome theoretical-applied and applied research, proposing novel computational and/or hardware solutions.

University of Surrey, UK

Einat Liebenthal

Harvard Medical School, USA

University of Modena and Reggio Emilia, Italy

• Collection content
• How to submit
• About the Guest Editors
• Collection policies

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• Our Promise
• Our Achievements
• Our Mission
• Proposal Writing
• System Development
• Paper Writing
• Paper Publish
• Synopsis Writing
• Thesis Writing
• Assignments
• Survey Paper
• Conference Paper
• Journal Paper
• Empirical Paper
• Journal Support
• Innovative 12+ Natural Language Processing Thesis Topics

Generally, natural language processing is the sub-branch of Artificial Intelligence (AI). Natural language processing is otherwise known as NLP. It is compatible in dealing with multi-linguistic aspects and they convert the text into binary formats in which computers can understand it.  Primarily, the device understands the texts and then translates according to the questions asked. These processes are getting done with the help of several techniques. As this article is concentrated on delivering the natural language processing thesis topics , we are going to reveal each and every aspect that is needed for an effective NLP thesis .

NLP has a wide range of areas to explore in which enormous researches will be conducted. As the matter of fact, they analyses emotions, processes images, summarize texts, answer the questions & translates automatically, and so on.

Thesis writing is one of the important steps in researches. As they can deliver the exact perceptions of the researcher to the opponents hence it is advisable to frame the proper one. Let us begin this article with an overview of the NLP system . Are you ready to sail with us? Come on, guys!!!

“This is the article which is framed to the NLP enthusiasts in order to offer the natural language processing thesis topics”

What is Actually an NLP?

• NLP is the process of retrieving the meaning of the given sentence
• For this they use techniques & algorithms in order to extract the features
• They are also involved with the following,
• Audio capturing
• Text processing
• Conversion of audio into text
• Human-computer interaction

This is a crisp overview of the NLP system. NLP is one of the major technologies that are being used in the day to day life. Without these technologies, we could not even imagine a single scenario . In fact, they minimized the time of human beings by means of spelling checks, grammatical formations and most importantly they are highly capable of handling audio data . In this regard, let us have an idea of how does the NLP works in general. Shall we get into that section? Come let’s move on to that!!!

How does NLP Works?

• Unstructured Data Inputs
• Lingual Knowledge
• Domain Knowledge
• Domain Model
• Corpora Model Training
• Tools & Methods

The above listed are necessary when input is given to the model. The NLP model is in need of the above-itemized aspects to process the unstructured data in order to offer the structured data by means of parsing, stemming and lemmatization, and so on. In fact, NLP is subject to the classifications by their eminent features such as generation & understanding.  Yes my dear students we are going to cover the next sections with the NLP classifications.  

Classifications of NLP

• Natural Language-based Generation
• Natural Language-based Understanding

The above listed are the 2 major classifications of NLP technology . In these classifications let us have further brief explanations of the natural language-based understanding for your better understanding.

• Biometric Domains
• Spam Detection
• Opinion/Data Mining
• Entity Linking
• Named Entity Recognition
• Relationship Extraction

This is how the natural language-based understanding is sub-classified according to its functions. In recent days, NLP is getting boom in which various r esearches and projects are getting investigated and implemented successfully by our technical team. Generally, NLP processes are getting performed in a structural manner. That means they are overlays in several steps in crafting natural language processing thesis topics . Yes dears, we are going to envelop the next section with the steps that are concreted with the natural language processing.

NLP Natural Language Processing Steps

• Segmentation of Sentences
• Tokenization of Words
• PoS Tagging
• Parsing of Syntactic Contexts
• Removing of Stop Words
• Lemmatization & Stemming
• Classification of Texts
• Emotion/Sentiment Analysis

Here POS stands for the Parts of Speech . These are some of the steps involved in natural language processing. NLP performs according to the inputs given. Here you might need examples in these areas. For your better understanding, we are going to illustrate to you about the same with clear bulletin points. Come let us try to understand them.

• Let we take inputs as text & speech
• Text inputs are analyzed by “word tokenization”
• Speech inputs are analyzed by “phonetics”

In addition to that, they both are further processed in the same manner as they are,

• Morphological Analysis
• Syntactic Analysis
• Semantic Understanding
• Speech Processing

The above listed are the steps involved in NLP tasks in general . Word tokenization is one of the major which points out the vocabulary words presented in the word groups . Though, NLP processes are subject to numerous challenges. Our technical team is pointed out to you the challenges involved in the current days for a better understanding. Let’s move on to the current challenges sections.

Before going to the next section, we would like to highlight ourselves here. We are one of the trusted crew of technicians who are dynamically performing the NLP-based projects and researches effectively . As the matter of fact, we are offering so many successful projects all over the world by using the emerging techniques in technology. Now we can have the next section.

Current Challenges in NLP

• Context/Intention Understanding
• Voice Ambiguity/Vagueness
• Data Transformation
• Semantic Context Extracting
• Word Phrase Matching
• Vocabulary/Terminologies Creation
• PoS Tagging & Tokenization

The above listed are the current challenges that get involved in natural language processing. Besides, we can overcome these challenges by improving the NLP model by means of their performance. On the other hand, our technical experts in the concern are usually testing natural language processing approaches to abolish these constraints.

In the following passage, our technical team elaborately explained to you the various natural language processing approaches for the ease of your understanding. In fact, our researchers are always focusing on the students understanding so that they are categorizing each and every edge needed for the NLP-oriented tasks and approaches .  Are you interested to know about that? Now let’s we jump into the section.

Different NLP Approaches

Domain Model-based Approaches

• Loss Centric
• Feature Centric
• Pre-Training
• Pseudo Labeling
• Data Selection
• Model + Data-Centric

Machine Learning-based Approaches

• Association
• K-Means Clustering
• Anomalies Recognition
• Data Parsing
• Regular Emotions/Expressions
• Syntactic Interpretations
• Pattern Matching
• BFS Co-location Data
• BERT & BioBERT
• Decision Trees
• Logistic Regression
• Linear Regression
• Random Forests
• Support Vector Machine
• Gradient-based Networks
• Convolutional Neural Network
• Deep Neural Networks

Text Mining Approaches

• K-nearest Neighbor
• Naïve Bayes
• Predictive Modeling
• Association Rules
• Classification
• Document Indexing
• Term & Inverse Document Frequency
• Document Term Matrix
• Distribution
• Keyword Frequency
• Term Reduction/Compression
• Stemming/lemmatization
• Tokenization
• NLP & Log Parsing
• Text Taxonomies
• Text Classifications
• Text Categorization
• Text Clustering

The above listed are the 3 major approaches that are mainly used for natural languages processing in real-time . However, there are some demerits and merits are presented with the above-listed approaches. It is also important to know about the advantages and disadvantages of the NLP approaches which will help you to focus on the constraints and lead will lead you to the developments. Shall we discuss the pros and cons of NLP approaches? Come on, guys!

Advantages & Disadvantages of NLP Approaches

• Effortless Debugging
• Effective Precisions
• Multi-perspectives
• Short Form Reading
• Ineffective Parsing
• Poor Recalls
• Excessive Skills
• Low Scalability
• Speed Processes
• Resilient Results
• Effective Documentation
• Better Recalls
• High Scalability
• Narrow Understanding
• Poor in Reading Messages
• Huge Annotations
• Complex in Debugging

The foregoing passage conveyed to you the pros and cons of two approaches named machine learning and text mining. The best approach is also having pros and cons. If you do want further explanations or clarifications on that you can feel free to approach our researchers to get benefit from us. Generally, NLP models are trained to perform every task in order to recognize the inputs with latest natural language processing project ideas . Yes, you people guessed right! The next section is all about the training models of the NLP.

Training Models in NLP

• Scratch dataset such as language-specific BERTs & multi-linguistic BERT
• These are the datasets used in model pre-training
• Auxiliary based Pre-Training
• It is the additional data tasks used for labeled adaptive pre-training
• Multi-Phase based Pre-Training
• Domain & broad tasks are the secondary phases of pre-training
• Unlabeled data sources make differences in the multiphase pre-training
• TAPT, DAPT, AdaptaBERT & BioBERT are used datasets

As this article is named as natural language processing thesis topics , here we are going to point out to you the latest thesis topics in NLP for your reference. Commonly, a thesis is the best illustration of the projects or researches done in the determined areas. In fact, they convey the researchers’ perspectives & thoughts to the opponent by the effective structures of the thesis. If you are searching for thesis writing assistance then this is the right platform, you can surely approach our team at any time.

In the following passage, we have itemized some of the latest thesis topics in NLP .  We thought that it would help you a lot. Let’s get into the next section. As this is an important section, you are advised to pay your attention here. Are you really interested in getting into the next section? Come let us also learn them.

Latest Natural Language Processing Thesis Topics

• Cross & Multilingual based NLP Methods
• Multi-modal based NLP Methodologies
• Provocative based NLP Systems
• Graph oriented NLP Techniques
• Data Amplification in NLP
• Reinforcement Learning based NLP
• Dialogue/Voice Assistants
• Market & Customer Behavior Modeling
• Text Classification by Zero-shot/Semi-supervised Learning & Sentiment Analysis
• Text Generation & Summarization
• Relation & Knowledge Extraction for Fine-grained Entity Recognition
• Knowledge & Open-domain based Question & Answering

These are some of the latest thesis topics in NLP . As the matter of fact, we have delivered around 200 to 300 thesis with fruitful outcomes. Actually, they are very innovative and unique by means of their features. Our thesis writing approaches impress the institutes incredibly. At this time, we would like to reveal the future directions of the NLP for the ease of your understanding.

How to select the best thesis topics in NLP?

• See the latest IEEE and other benchmark papers
• Understand the NLP Project ideas recently proposed
• Highlight the problems and gaps
• Get the future scope of each existing work

Come let’s move on to the next section.

Future Research Directions of Natural Language Processing

• Logical Reasoning Chains
• Statistical Integrated Multilingual & Domain Knowledge Processing
• Combination of Interacting Modules

On the whole, NLP requires a better understanding of the texts. In fact, they understand the text’s meaning by relating to the presented word phrases. Conversion of the natural languages in reasoning logic will lead NLP to future directions. By allowing the modules to interact can enhance the NLP pipelines and modules. So far, we have come up with the areas of natural language processing thesis topics and each and every aspect that is needed to do a thesis. If you are in dilemma you could have the valuable opinions of our technical experts.

“Let’s begin to work on your experimental areas and yield the stunning outcomes”

MILESTONE 1: Research Proposal

Finalize journal (indexing).

Before sit down to research proposal writing, we need to decide exact journals. For e.g. SCI, SCI-E, ISI, SCOPUS.

Research Subject Selection

As a doctoral student, subject selection is a big problem. Phdservices.org has the team of world class experts who experience in assisting all subjects. When you decide to work in networking, we assign our experts in your specific area for assistance.

Research Topic Selection

We helping you with right and perfect topic selection, which sound interesting to the other fellows of your committee. For e.g. if your interest in networking, the research topic is VANET / MANET / any other

Literature Survey Writing

To ensure the novelty of research, we find research gaps in 50+ latest benchmark papers (IEEE, Springer, Elsevier, MDPI, Hindawi, etc.)

Case Study Writing

After literature survey, we get the main issue/problem that your research topic will aim to resolve and elegant writing support to identify relevance of the issue.

Problem Statement

Based on the research gaps finding and importance of your research, we conclude the appropriate and specific problem statement.

Writing Research Proposal

Writing a good research proposal has need of lot of time. We only span a few to cover all major aspects (reference papers collection, deficiency finding, drawing system architecture, highlights novelty)

MILESTONE 2: System Development

Fix implementation plan.

We prepare a clear project implementation plan that narrates your proposal in step-by step and it contains Software and OS specification. We recommend you very suitable tools/software that fit for your concept.

Tools/Plan Approval

We get the approval for implementation tool, software, programing language and finally implementation plan to start development process.

Pseudocode Description

Our source code is original since we write the code after pseudocodes, algorithm writing and mathematical equation derivations.

Develop Proposal Idea

We implement our novel idea in step-by-step process that given in implementation plan. We can help scholars in implementation.

Comparison/Experiments

We perform the comparison between proposed and existing schemes in both quantitative and qualitative manner since it is most crucial part of any journal paper.

Graphs, Results, Analysis Table

We evaluate and analyze the project results by plotting graphs, numerical results computation, and broader discussion of quantitative results in table.

Project Deliverables

For every project order, we deliver the following: reference papers, source codes screenshots, project video, installation and running procedures.

MILESTONE 3: Paper Writing

Choosing right format.

We intend to write a paper in customized layout. If you are interesting in any specific journal, we ready to support you. Otherwise we prepare in IEEE transaction level.

Collecting Reliable Resources

Before paper writing, we collect reliable resources such as 50+ journal papers, magazines, news, encyclopedia (books), benchmark datasets, and online resources.

Writing Rough Draft

We create an outline of a paper at first and then writing under each heading and sub-headings. It consists of novel idea and resources

Proofreading & Formatting

We must proofread and formatting a paper to fix typesetting errors, and avoiding misspelled words, misplaced punctuation marks, and so on

Native English Writing

We check the communication of a paper by rewriting with native English writers who accomplish their English literature in University of Oxford.

Scrutinizing Paper Quality

We examine the paper quality by top-experts who can easily fix the issues in journal paper writing and also confirm the level of journal paper (SCI, Scopus or Normal).

Plagiarism Checking

We at phdservices.org is 100% guarantee for original journal paper writing. We never use previously published works.

MILESTONE 4: Paper Publication

Finding apt journal.

We play crucial role in this step since this is very important for scholar’s future. Our experts will help you in choosing high Impact Factor (SJR) journals for publishing.

Lay Paper to Submit

We organize your paper for journal submission, which covers the preparation of Authors Biography, Cover Letter, Highlights of Novelty, and Suggested Reviewers.

Paper Submission

We upload paper with submit all prerequisites that are required in journal. We completely remove frustration in paper publishing.

Paper Status Tracking

We track your paper status and answering the questions raise before review process and also we giving you frequent updates for your paper received from journal.

Revising Paper Precisely

When we receive decision for revising paper, we get ready to prepare the point-point response to address all reviewers query and resubmit it to catch final acceptance.

Get Accept & e-Proofing

We receive final mail for acceptance confirmation letter and editors send e-proofing and licensing to ensure the originality.

Publishing Paper

Paper published in online and we inform you with paper title, authors information, journal name volume, issue number, page number, and DOI link

MILESTONE 5: Thesis Writing

Identifying university format.

We pay special attention for your thesis writing and our 100+ thesis writers are proficient and clear in writing thesis for all university formats.

Gathering Adequate Resources

We collect primary and adequate resources for writing well-structured thesis using published research articles, 150+ reputed reference papers, writing plan, and so on.

Writing Thesis (Preliminary)

We write thesis in chapter-by-chapter without any empirical mistakes and we completely provide plagiarism-free thesis.

Skimming & Reading

Skimming involve reading the thesis and looking abstract, conclusions, sections, & sub-sections, paragraphs, sentences & words and writing thesis chorological order of papers.

Fixing Crosscutting Issues

This step is tricky when write thesis by amateurs. Proofreading and formatting is made by our world class thesis writers who avoid verbose, and brainstorming for significant writing.

Organize Thesis Chapters

We organize thesis chapters by completing the following: elaborate chapter, structuring chapters, flow of writing, citations correction, etc.

Writing Thesis (Final Version)

We attention to details of importance of thesis contribution, well-illustrated literature review, sharp and broad results and discussion and relevant applications study.

How PhDservices.org deal with significant issues ?

1. novel ideas.

Novelty is essential for a PhD degree. Our experts are bringing quality of being novel ideas in the particular research area. It can be only determined by after thorough literature search (state-of-the-art works published in IEEE, Springer, Elsevier, ACM, ScienceDirect, Inderscience, and so on). SCI and SCOPUS journals reviewers and editors will always demand “Novelty” for each publishing work. Our experts have in-depth knowledge in all major and sub-research fields to introduce New Methods and Ideas. MAKING NOVEL IDEAS IS THE ONLY WAY OF WINNING PHD.

2. Plagiarism-Free

To improve the quality and originality of works, we are strictly avoiding plagiarism since plagiarism is not allowed and acceptable for any type journals (SCI, SCI-E, or Scopus) in editorial and reviewer point of view. We have software named as “Anti-Plagiarism Software” that examines the similarity score for documents with good accuracy. We consist of various plagiarism tools like Viper, Turnitin, Students and scholars can get your work in Zero Tolerance to Plagiarism. DONT WORRY ABOUT PHD, WE WILL TAKE CARE OF EVERYTHING.

3. Confidential Info

We intended to keep your personal and technical information in secret and it is a basic worry for all scholars.

• Technical Info: We never share your technical details to any other scholar since we know the importance of time and resources that are giving us by scholars.
• Personal Info: We restricted to access scholars personal details by our experts. Our organization leading team will have your basic and necessary info for scholars.

CONFIDENTIALITY AND PRIVACY OF INFORMATION HELD IS OF VITAL IMPORTANCE AT PHDSERVICES.ORG. WE HONEST FOR ALL CUSTOMERS.

4. Publication

Most of the PhD consultancy services will end their services in Paper Writing, but our PhDservices.org is different from others by giving guarantee for both paper writing and publication in reputed journals. With our 18+ year of experience in delivering PhD services, we meet all requirements of journals (reviewers, editors, and editor-in-chief) for rapid publications. From the beginning of paper writing, we lay our smart works. PUBLICATION IS A ROOT FOR PHD DEGREE. WE LIKE A FRUIT FOR GIVING SWEET FEELING FOR ALL SCHOLARS.

5. No Duplication

After completion of your work, it does not available in our library i.e. we erased after completion of your PhD work so we avoid of giving duplicate contents for scholars. This step makes our experts to bringing new ideas, applications, methodologies and algorithms. Our work is more standard, quality and universal. Everything we make it as a new for all scholars. INNOVATION IS THE ABILITY TO SEE THE ORIGINALITY. EXPLORATION IS OUR ENGINE THAT DRIVES INNOVATION SO LET’S ALL GO EXPLORING.

Client Reviews

I ordered a research proposal in the research area of Wireless Communications and it was as very good as I can catch it.

I had wishes to complete implementation using latest software/tools and I had no idea of where to order it. My friend suggested this place and it delivers what I expect.

It really good platform to get all PhD services and I have used it many times because of reasonable price, best customer services, and high quality.

My colleague recommended this service to me and I’m delighted their services. They guide me a lot and given worthy contents for my research paper.

I’m never disappointed at any kind of service. Till I’m work with professional writers and getting lot of opportunities.

- Christopher

Once I am entered this organization I was just felt relax because lots of my colleagues and family relations were suggested to use this service and I received best thesis writing.

I recommend phdservices.org. They have professional writers for all type of writing (proposal, paper, thesis, assignment) support at affordable price.

You guys did a great job saved more money and time. I will keep working with you and I recommend to others also.

These experts are fast, knowledgeable, and dedicated to work under a short deadline. I had get good conference paper in short span.

Guys! You are the great and real experts for paper writing since it exactly matches with my demand. I will approach again.

I am fully satisfied with thesis writing. Thank you for your faultless service and soon I come back again.

Trusted customer service that you offer for me. I don’t have any cons to say.

I was at the edge of my doctorate graduation since my thesis is totally unconnected chapters. You people did a magic and I get my complete thesis!!!

- Abdul Mohammed

Good family environment with collaboration, and lot of hardworking team who actually share their knowledge by offering PhD Services.

I enjoyed huge when working with PhD services. I was asked several questions about my system development and I had wondered of smooth, dedication and caring.

I had not provided any specific requirements for my proposal work, but you guys are very awesome because I’m received proper proposal. Thank you!

- Bhanuprasad

I was read my entire research proposal and I liked concept suits for my research issues. Thank you so much for your efforts.

- Ghulam Nabi

I am extremely happy with your project development support and source codes are easily understanding and executed.

Hi!!! You guys supported me a lot. Thank you and I am 100% satisfied with publication service.

- Abhimanyu

I had found this as a wonderful platform for scholars so I highly recommend this service to all. I ordered thesis proposal and they covered everything. Thank you so much!!!

Related Pages

Natural Language Processing

Natural Language Processing (NLP) research at Google focuses on algorithms that apply at scale, across languages, and across domains. Our systems are used in numerous ways across Google, impacting user experience in search, mobile, apps, ads, translate and more.

Our work spans the range of traditional NLP tasks, with general-purpose syntax and semantic algorithms underpinning more specialized systems. We are particularly interested in algorithms that scale well and can be run efficiently in a highly distributed environment.

Our syntactic systems predict part-of-speech tags for each word in a given sentence, as well as morphological features such as gender and number. They also label relationships between words, such as subject, object, modification, and others. We focus on efficient algorithms that leverage large amounts of unlabeled data, and recently have incorporated neural net technology.

On the semantic side, we identify entities in free text, label them with types (such as person, location, or organization), cluster mentions of those entities within and across documents (coreference resolution), and resolve the entities to the Knowledge Graph.

Recent work has focused on incorporating multiple sources of knowledge and information to aid with analysis of text, as well as applying frame semantics at the noun phrase, sentence, and document level.

Recent Publications

Some of our teams.

Africa team

Impact-Driven Research, Innovation and Moonshots

We're always looking for more talented, passionate people.

Natural Language Processing

Introduction.

Natural Language Processing (NLP) is one of the hottest areas of artificial intelligence (AI) thanks to applications like text generators that compose coherent essays, chatbots that fool people into thinking they’re sentient, and text-to-image programs that produce photorealistic images of anything you can describe. Recent years have brought a revolution in the ability of computers to understand human languages, programming languages, and even biological and chemical sequences, such as DNA and protein structures, that resemble language. The latest AI models are unlocking these areas to analyze the meanings of input text and generate meaningful, expressive output.

What is Natural Language Processing (NLP)

Natural language processing (NLP) is the discipline of building machines that can manipulate human language — or data that resembles human language — in the way that it is written, spoken, and organized. It evolved from computational linguistics, which uses computer science to understand the principles of language, but rather than developing theoretical frameworks, NLP is an engineering discipline that seeks to build technology to accomplish useful tasks. NLP can be divided into two overlapping subfields: natural language understanding (NLU), which focuses on semantic analysis or determining the intended meaning of text, and natural language generation (NLG), which focuses on text generation by a machine. NLP is separate from — but often used in conjunction with — speech recognition, which seeks to parse spoken language into words, turning sound into text and vice versa.

Why Does Natural Language Processing (NLP) Matter?

NLP is an integral part of everyday life and becoming more so as language technology is applied to diverse fields like retailing (for instance, in customer service chatbots) and medicine (interpreting or summarizing electronic health records). Conversational agents such as Amazon’s Alexa and Apple’s Siri utilize NLP to listen to user queries and find answers. The most sophisticated such agents — such as GPT-3, which was recently opened for commercial applications — can generate sophisticated prose on a wide variety of topics as well as power chatbots that are capable of holding coherent conversations. Google uses NLP to improve its search engine results , and social networks like Facebook use it to detect and filter hate speech . 

NLP is growing increasingly sophisticated, yet much work remains to be done. Current systems are prone to bias and incoherence, and occasionally behave erratically. Despite the challenges, machine learning engineers have many opportunities to apply NLP in ways that are ever more central to a functioning society.

What is Natural Language Processing (NLP) Used For?

NLP is used for a wide variety of language-related tasks, including answering questions, classifying text in a variety of ways, and conversing with users. 

Here are 11 tasks that can be solved by NLP:

• Sentiment analysis is the process of classifying the emotional intent of text. Generally, the input to a sentiment classification model is a piece of text, and the output is the probability that the sentiment expressed is positive, negative, or neutral. Typically, this probability is based on either hand-generated features, word n-grams, TF-IDF features, or using deep learning models to capture sequential long- and short-term dependencies. Sentiment analysis is used to classify customer reviews on various online platforms as well as for niche applications like identifying signs of mental illness in online comments.

• Toxicity classification is a branch of sentiment analysis where the aim is not just to classify hostile intent but also to classify particular categories such as threats, insults, obscenities, and hatred towards certain identities. The input to such a model is text, and the output is generally the probability of each class of toxicity. Toxicity classification models can be used to moderate and improve online conversations by silencing offensive comments , detecting hate speech , or scanning documents for defamation . 
• Machine translation automates translation between different languages. The input to such a model is text in a specified source language, and the output is the text in a specified target language. Google Translate is perhaps the most famous mainstream application. Such models are used to improve communication between people on social-media platforms such as Facebook or Skype. Effective approaches to machine translation can distinguish between words with similar meanings . Some systems also perform language identification; that is, classifying text as being in one language or another. 
• Named entity recognition aims to extract entities in a piece of text into predefined categories such as personal names, organizations, locations, and quantities. The input to such a model is generally text, and the output is the various named entities along with their start and end positions. Named entity recognition is useful in applications such as summarizing news articles and combating disinformation . For example, here is what a named entity recognition model could provide: 

• Spam detection is a prevalent binary classification problem in NLP, where the purpose is to classify emails as either spam or not. Spam detectors take as input an email text along with various other subtexts like title and sender’s name. They aim to output the probability that the mail is spam. Email providers like Gmail use such models to provide a better user experience by detecting unsolicited and unwanted emails and moving them to a designated spam folder. 
• Grammatical error correction models encode grammatical rules to correct the grammar within text. This is viewed mainly as a sequence-to-sequence task, where a model is trained on an ungrammatical sentence as input and a correct sentence as output. Online grammar checkers like Grammarly and word-processing systems like Microsoft Word use such systems to provide a better writing experience to their customers. Schools also use them to grade student essays . 
• Topic modeling is an unsupervised text mining task that takes a corpus of documents and discovers abstract topics within that corpus. The input to a topic model is a collection of documents, and the output is a list of topics that defines words for each topic as well as assignment proportions of each topic in a document. Latent Dirichlet Allocation (LDA), one of the most popular topic modeling techniques, tries to view a document as a collection of topics and a topic as a collection of words. Topic modeling is being used commercially to help lawyers find evidence in legal documents . 
• Autocomplete predicts what word comes next, and autocomplete systems of varying complexity are used in chat applications like WhatsApp. Google uses autocomplete to predict search queries. One of the most famous models for autocomplete is GPT-2, which has been used to write articles , song lyrics , and much more. 
• Database query: We have a database of questions and answers, and we would like a user to query it using natural language. 
• Conversation generation: These chatbots can simulate dialogue with a human partner. Some are capable of engaging in wide-ranging conversations . A high-profile example is Google’s LaMDA, which provided such human-like answers to questions that one of its developers was convinced that it had feelings .
• Information retrieval finds the documents that are most relevant to a query. This is a problem every search and recommendation system faces. The goal is not to answer a particular query but to retrieve, from a collection of documents that may be numbered in the millions, a set that is most relevant to the query. Document retrieval systems mainly execute two processes: indexing and matching. In most modern systems, indexing is done by a vector space model through Two-Tower Networks, while matching is done using similarity or distance scores. Google recently integrated its search function with a multimodal information retrieval model that works with text, image, and video data.

• Extractive summarization focuses on extracting the most important sentences from a long text and combining these to form a summary. Typically, extractive summarization scores each sentence in an input text and then selects several sentences to form the summary.
• Abstractive summarization produces a summary by paraphrasing. This is similar to writing the abstract that includes words and sentences that are not present in the original text. Abstractive summarization is usually modeled as a sequence-to-sequence task, where the input is a long-form text and the output is a summary.
• Multiple choice: The multiple-choice question problem is composed of a question and a set of possible answers. The learning task is to pick the correct answer. 
• Open domain : In open-domain question answering, the model provides answers to questions in natural language without any options provided, often by querying a large number of texts.

How Does Natural Language Processing (NLP) Work?

NLP models work by finding relationships between the constituent parts of language — for example, the letters, words, and sentences found in a text dataset. NLP architectures use various methods for data preprocessing, feature extraction, and modeling. Some of these processes are: 

• Stemming and lemmatization : Stemming is an informal process of converting words to their base forms using heuristic rules. For example, “university,” “universities,” and “university’s” might all be mapped to the base univers . (One limitation in this approach is that “universe” may also be mapped to univers , even though universe and university don’t have a close semantic relationship.) Lemmatization is a more formal way to find roots by analyzing a word’s morphology using vocabulary from a dictionary. Stemming and lemmatization are provided by libraries like spaCy and NLTK. 
• Sentence segmentation breaks a large piece of text into linguistically meaningful sentence units. This is obvious in languages like English, where the end of a sentence is marked by a period, but it is still not trivial. A period can be used to mark an abbreviation as well as to terminate a sentence, and in this case, the period should be part of the abbreviation token itself. The process becomes even more complex in languages, such as ancient Chinese, that don’t have a delimiter that marks the end of a sentence. 
• Stop word removal aims to remove the most commonly occurring words that don’t add much information to the text. For example, “the,” “a,” “an,” and so on.
• Tokenization splits text into individual words and word fragments. The result generally consists of a word index and tokenized text in which words may be represented as numerical tokens for use in various deep learning methods. A method that instructs language models to ignore unimportant tokens can improve efficiency.  

• Bag-of-Words: Bag-of-Words counts the number of times each word or n-gram (combination of n words) appears in a document. For example, below, the Bag-of-Words model creates a numerical representation of the dataset based on how many of each word in the word_index occur in the document. 

• Term Frequency: How important is the word in the document?

TF(word in a document)= Number of occurrences of that word in document / Number of words in document

• Inverse Document Frequency: How important is the term in the whole corpus?

IDF(word in a corpus)=log(number of documents in the corpus / number of documents that include the word)

A word is important if it occurs many times in a document. But that creates a problem. Words like “a” and “the” appear often. And as such, their TF score will always be high. We resolve this issue by using Inverse Document Frequency, which is high if the word is rare and low if the word is common across the corpus. The TF-IDF score of a term is the product of TF and IDF. 

• Word2Vec , introduced in 2013 , uses a vanilla neural network to learn high-dimensional word embeddings from raw text. It comes in two variations: Skip-Gram, in which we try to predict surrounding words given a target word, and Continuous Bag-of-Words (CBOW), which tries to predict the target word from surrounding words. After discarding the final layer after training, these models take a word as input and output a word embedding that can be used as an input to many NLP tasks. Embeddings from Word2Vec capture context. If particular words appear in similar contexts, their embeddings will be similar.
• GLoVE is similar to Word2Vec as it also learns word embeddings, but it does so by using matrix factorization techniques rather than neural learning. The GLoVE model builds a matrix based on the global word-to-word co-occurrence counts. 
• Numerical features extracted by the techniques described above can be fed into various models depending on the task at hand. For example, for classification, the output from the TF-IDF vectorizer could be provided to logistic regression, naive Bayes, decision trees, or gradient boosted trees. Or, for named entity recognition, we can use hidden Markov models along with n-grams. 
• Deep neural networks typically work without using extracted features, although we can still use TF-IDF or Bag-of-Words features as an input. 
• Language Models : In very basic terms, the objective of a language model is to predict the next word when given a stream of input words. Probabilistic models that use Markov assumption are one example:

P(W n )=P(W n |W n−1 )

Deep learning is also used to create such language models. Deep-learning models take as input a word embedding and, at each time state, return the probability distribution of the next word as the probability for every word in the dictionary. Pre-trained language models learn the structure of a particular language by processing a large corpus, such as Wikipedia. They can then be fine-tuned for a particular task. For instance, BERT has been fine-tuned for tasks ranging from fact-checking to writing headlines . 

Top Natural Language Processing (NLP) Techniques

Most of the NLP tasks discussed above can be modeled by a dozen or so general techniques. It’s helpful to think of these techniques in two categories: Traditional machine learning methods and deep learning methods. 

Traditional Machine learning NLP techniques: 

• Logistic regression is a supervised classification algorithm that aims to predict the probability that an event will occur based on some input. In NLP, logistic regression models can be applied to solve problems such as sentiment analysis, spam detection, and toxicity classification.
• Naive Bayes is a supervised classification algorithm that finds the conditional probability distribution P(label | text) using the following Bayes formula:

P(label | text) = P(label) x P(text|label) / P(text) 

and predicts based on which joint distribution has the highest probability. The naive assumption in the Naive Bayes model is that the individual words are independent. Thus: 

P(text|label) = P(word_1|label)*P(word_2|label)*…P(word_n|label)

In NLP, such statistical methods can be applied to solve problems such as spam detection or finding bugs in software code . 

• Decision trees are a class of supervised classification models that split the dataset based on different features to maximize information gain in those splits.

• Latent Dirichlet Allocation (LDA) is used for topic modeling. LDA tries to view a document as a collection of topics and a topic as a collection of words. LDA is a statistical approach. The intuition behind it is that we can describe any topic using only a small set of words from the corpus.
• Hidden Markov models : Markov models are probabilistic models that decide the next state of a system based on the current state. For example, in NLP, we might suggest the next word based on the previous word. We can model this as a Markov model where we might find the transition probabilities of going from word1 to word2, that is, P(word1|word2). Then we can use a product of these transition probabilities to find the probability of a sentence. The hidden Markov model (HMM) is a probabilistic modeling technique that introduces a hidden state to the Markov model. A hidden state is a property of the data that isn’t directly observed. HMMs are used for part-of-speech (POS) tagging where the words of a sentence are the observed states and the POS tags are the hidden states. The HMM adds a concept called emission probability; the probability of an observation given a hidden state. In the prior example, this is the probability of a word, given its POS tag. HMMs assume that this probability can be reversed: Given a sentence, we can calculate the part-of-speech tag from each word based on both how likely a word was to have a certain part-of-speech tag and the probability that a particular part-of-speech tag follows the part-of-speech tag assigned to the previous word. In practice, this is solved using the Viterbi algorithm.

Deep learning NLP Techniques: 

• Convolutional Neural Network (CNN): The idea of using a CNN to classify text was first presented in the paper “ Convolutional Neural Networks for Sentence Classification ” by Yoon Kim. The central intuition is to see a document as an image. However, instead of pixels, the input is sentences or documents represented as a matrix of words.

• Recurrent Neural Network (RNN) : Many techniques for text classification that use deep learning process words in close proximity using n-grams or a window (CNNs). They can see “New York” as a single instance. However, they can’t capture the context provided by a particular text sequence. They don’t learn the sequential structure of the data, where every word is dependent on the previous word or a word in the previous sentence. RNNs remember previous information using hidden states and connect it to the current task. The architectures known as Gated Recurrent Unit (GRU) and long short-term memory (LSTM) are types of RNNs designed to remember information for an extended period. Moreover, the bidirectional LSTM/GRU keeps contextual information in both directions, which is helpful in text classification. RNNs have also been used to generate mathematical proofs and translate human thoughts into words. 

• Autoencoders are deep learning encoder-decoders that approximate a mapping from X to X, i.e., input=output. They first compress the input features into a lower-dimensional representation (sometimes called a latent code, latent vector, or latent representation) and learn to reconstruct the input. The representation vector can be used as input to a separate model, so this technique can be used for dimensionality reduction. Among specialists in many other fields, geneticists have applied autoencoders to spot mutations associated with diseases in amino acid sequences. 

• Encoder-decoder sequence-to-sequence : The encoder-decoder seq2seq architecture is an adaptation to autoencoders specialized for translation, summarization, and similar tasks. The encoder encapsulates the information in a text into an encoded vector. Unlike an autoencoder, instead of reconstructing the input from the encoded vector, the decoder’s task is to generate a different desired output, like a translation or summary. 

• Transformers : The transformer, a model architecture first described in the 2017 paper “ Attention Is All You Need ” (Vaswani, Shazeer, Parmar, et al.), forgoes recurrence and instead relies entirely on a self-attention mechanism to draw global dependencies between input and output. Since this mechanism processes all words at once (instead of one at a time) that decreases training speed and inference cost compared to RNNs, especially since it is parallelizable. The transformer architecture has revolutionized NLP in recent years, leading to models including BLOOM , Jurassic-X , and Turing-NLG . It has also been successfully applied to a variety of different vision tasks , including making 3D images .

Six Important Natural Language Processing (NLP) Models

Over the years, many NLP models have made waves within the AI community, and some have even made headlines in the mainstream news. The most famous of these have been chatbots and language models. Here are some of them:

• Eliza was developed in the mid-1960s to try to solve the Turing Test; that is, to fool people into thinking they’re conversing with another human being rather than a machine. Eliza used pattern matching and a series of rules without encoding the context of the language.
• Tay was a chatbot that Microsoft launched in 2016. It was supposed to tweet like a teen and learn from conversations with real users on Twitter. The bot adopted phrases from users who tweeted sexist and racist comments, and Microsoft deactivated it not long afterward. Tay illustrates some points made by the “Stochastic Parrots” paper, particularly the danger of not debiasing data.
• BERT and his Muppet friends: Many deep learning models for NLP are named after Muppet characters , including ELMo , BERT , Big BIRD , ERNIE , Kermit , Grover , RoBERTa , and Rosita . Most of these models are good at providing contextual embeddings and enhanced knowledge representation.
• Generative Pre-Trained Transformer 3 (GPT-3) is a 175 billion parameter model that can write original prose with human-equivalent fluency in response to an input prompt. The model is based on the transformer architecture. The previous version, GPT-2, is open source. Microsoft acquired an exclusive license to access GPT-3’s underlying model from its developer OpenAI, but other users can interact with it via an application programming interface (API). Several groups including EleutherAI and Meta have released open source interpretations of GPT-3. 
• Language Model for Dialogue Applications (LaMDA) is a conversational chatbot developed by Google. LaMDA is a transformer-based model trained on dialogue rather than the usual web text. The system aims to provide sensible and specific responses to conversations. Google developer Blake Lemoine came to believe that LaMDA is sentient. Lemoine had detailed conversations with AI about his rights and personhood. During one of these conversations, the AI changed Lemoine’s mind about Isaac Asimov’s third law of robotics. Lemoine claimed that LaMDA was sentient, but the idea was disputed by many observers and commentators. Subsequently, Google placed Lemoine on administrative leave for distributing proprietary information and ultimately fired him.
• Mixture of Experts ( MoE): While most deep learning models use the same set of parameters to process every input, MoE models aim to provide different parameters for different inputs based on efficient routing algorithms to achieve higher performance . Switch Transformer is an example of the MoE approach that aims to reduce communication and computational costs.

Programming Languages, Libraries, And Frameworks For Natural Language Processing (NLP)

Many languages and libraries support NLP. Here are a few of the most useful.

• Natural Language Toolkit (NLTK) is one of the first NLP libraries written in Python. It provides easy-to-use interfaces to corpora and lexical resources such as WordNet . It also provides a suite of text-processing libraries for classification, tagging, stemming, parsing, and semantic reasoning.
• spaCy is one of the most versatile open source NLP libraries. It supports more than 66 languages. spaCy also provides pre-trained word vectors and implements many popular models like BERT. spaCy can be used for building production-ready systems for named entity recognition, part-of-speech tagging, dependency parsing, sentence segmentation, text classification, lemmatization, morphological analysis, entity linking, and so on.
• Deep Learning libraries: Popular deep learning libraries include TensorFlow and PyTorch , which make it easier to create models with features like automatic differentiation. These libraries are the most common tools for developing NLP models.
• Hugging Face offers open-source implementations and weights of over 135 state-of-the-art models. The repository enables easy customization and training of the models.
• Gensim provides vector space modeling and topic modeling algorithms.
• R : Many early NLP models were written in R, and R is still widely used by data scientists and statisticians. Libraries in R for NLP include TidyText , Weka , Word2Vec , SpaCyR , TensorFlow , and PyTorch .
• Many other languages including JavaScript, Java, and Julia have libraries that implement NLP methods.

Controversies Surrounding Natural Language Processing (NLP)

NLP has been at the center of a number of controversies. Some are centered directly on the models and their outputs, others on second-order concerns, such as who has access to these systems, and how training them impacts the natural world. 

• Stochastic parrots: A 2021 paper titled “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” by Emily Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell examines how language models may repeat and amplify biases found in their training data. The authors point out that huge, uncurated datasets scraped from the web are bound to include social biases and other undesirable information, and models that are trained on them will absorb these flaws. They advocate greater care in curating and documenting datasets, evaluating a model’s potential impact prior to development, and encouraging research in directions other than designing ever-larger architectures to ingest ever-larger datasets.
• Coherence versus sentience: Recently, a Google engineer tasked with evaluating the LaMDA language model was so impressed by the quality of its chat output that he believed it to be sentient . The fallacy of attributing human-like intelligence to AI dates back to some of the earliest NLP experiments. 
• Environmental impact: Large language models require a lot of energy during both training and inference. One study estimated that training a single large language model can emit five times as much carbon dioxide as a single automobile over its operational lifespan. Another study found that models consume even more energy during inference than training. As for solutions, researchers have proposed using cloud servers located in countries with lots of renewable energy as one way to offset this impact. 
• High cost leaves out non-corporate researchers: The computational requirements needed to train or deploy large language models are too expensive for many small companies . Some experts worry that this could block many capable engineers from contributing to innovation in AI. 
• Black box: When a deep learning model renders an output, it’s difficult or impossible to know why it generated that particular result. While traditional models like logistic regression enable engineers to examine the impact on the output of individual features, neural network methods in natural language processing are essentially black boxes. Such systems are said to be “not explainable,” since we can’t explain how they arrived at their output. An effective approach to achieve explainability is especially important in areas like banking, where regulators want to confirm that a natural language processing system doesn’t discriminate against some groups of people, and law enforcement, where models trained on historical data may perpetuate historical biases against certain groups.

“ Nonsense on stilts ”: Writer Gary Marcus has criticized deep learning-based NLP for generating sophisticated language that misleads users to believe that natural language algorithms understand what they are saying and mistakenly assume they are capable of more sophisticated reasoning than is currently possible.

How To Get Started In Natural Language Processing (NLP)

If you are just starting out, many excellent courses can help.

If you want to learn more about NLP, try reading research papers. Work through the papers that introduced the models and techniques described in this article. Most are easy to find on arxiv.org . You might also take a look at these resources: 

• The Batch : A weekly newsletter that tells you what matters in AI. It’s the best way to keep up with developments in deep learning.
• NLP News : A newsletter from Sebastian Ruder, a research scientist at Google, focused on what’s new in NLP. 
• Papers with Code : A web repository of machine learning research, tasks, benchmarks, and datasets.

We highly recommend learning to implement basic algorithms (linear and logistic regression, Naive Bayes, decision trees, and vanilla neural networks) in Python. The next step is to take an open-source implementation and adapt it to a new dataset or task. 

NLP is one of the fast-growing research domains in AI, with applications that involve tasks including translation, summarization, text generation, and sentiment analysis. Businesses use NLP to power a growing number of applications, both internal — like detecting insurance fraud , determining customer sentiment, and optimizing aircraft maintenance — and customer-facing, like Google Translate. 

Aspiring NLP practitioners can begin by familiarizing themselves with foundational AI skills: performing basic mathematics, coding in Python, and using algorithms like decision trees, Naive Bayes, and logistic regression. Online courses can help you build your foundation. They can also help as you proceed into specialized topics. Specializing in NLP requires a working knowledge of things like neural networks, frameworks like PyTorch and TensorFlow, and various data preprocessing techniques. The transformer architecture, which has revolutionized the field since it was introduced in 2017, is an especially important architecture.

NLP is an exciting and rewarding discipline, and has potential to profoundly impact the world in many positive ways. Unfortunately, NLP is also the focus of several controversies, and understanding them is also part of being a responsible practitioner. For instance, researchers have found that models will parrot biased language found in their training data, whether they’re counterfactual, racist, or hateful. Moreover, sophisticated language models can be used to generate disinformation. A broader concern is that training large models produces substantial greenhouse gas emissions.

This page is only a brief overview of what NLP is all about. If you have an appetite for more, DeepLearning.AI offers courses for everyone in their NLP journey, from AI beginners and those who are ready to specialize . No matter your current level of expertise or aspirations, remember to keep learning!

• SUGGESTED TOPICS
• The Magazine
• Newsletters
• Managing Yourself
• Managing Teams
• Work-life Balance
• The Big Idea
• Data & Visuals
• Reading Lists
• Case Selections
• HBR Learning
• Topic Feeds
• Account Settings
• Email Preferences

The Power of Natural Language Processing

• Ross Gruetzemacher

How companies can use NLP to help with brainstorming, summarizing, and researching.

The conventional wisdom around AI has been that while computers have the edge over humans when it comes to data-driven decision making, it can’t compete on qualitative tasks. That, however, is changing. Natural language processing (NLP) tools have advanced rapidly and can help with writing, coding, and discipline-specific reasoning. Companies that want to make use of this new tech should focus on the following: 1) Identify text data assets and determine how the latest techniques can be leveraged to add value for your firm, 2) understand how you might leverage AI-based language technologies to make better decisions or reorganize your skilled labor, 3) begin incorporating new language-based AI tools for a variety of tasks to better understand their capabilities, and 4) don’t underestimate the transformative potential of AI.

Until recently, the conventional wisdom was that while AI was better than humans at data-driven decision making tasks , it was still inferior to humans for cognitive and creative ones . But in the past two years language-based AI has advanced by leaps and bounds, changing common notions of what this technology can do.

• RG Ross Gruetzemacher is an Assistant Professor of Business Analytics at the W. Frank Barton School of Business at Wichita State University. He is a consultant on AI strategy for organizations in the Bay Area and internationally, and he also works as a Senior Game Master on Intelligence Rising , a strategic role-play game for exploring AI futures.

Partner Center

Cohere and Fujitsu Announce Strategic Partnership To Provide Japanese Enterprise AI Services

Jan 10, 2023

• Hao Wu   ORCID: orcid.org/0009-0007-4813-9297 1 ,
• Shan Li   ORCID: orcid.org/0000-0001-6001-1586 2 ,
• Ying Gao 3 ,
• Jinta Weng 4 &
• Guozhu Ding   ORCID: orcid.org/0000-0002-2043-8320 5  

267 Accesses

Explore all metrics

Natural language processing (NLP) has captivated the attention of educational researchers over the past three decades. In this study, a total of 2,480 studies were retrieved through a comprehensive literature search. We used neural topic modeling and pre-trained language modeling to explore the research topics pertaining to the application of NLP in education. Furthermore, we used linear regression to analyze the changes in the hotness of each topic. The results of topic modeling in different periods were visualized as topic word co-occurrence networks. This study revealed that the application of NLP in education can be broadly categorized into: (1) Technology innovations in language learning, (2) Mining and analysis (Practices, trends, factors, and challenges), (3) Student learning, (4) Interact and evaluation, and (5) Models and algorithms. Moreover, the field followed evolutionary trajectories in language learning, interaction and assessment, educational research and analysis, and learning technologies. Additional examination of the topic word co-occurrence, we found that research topics in this field have transitioned from being primarily technology-oriented to adopting a problem-oriented approach, reflecting a growing trend of diversification.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Text Mining and Automated Scoring

Natural Language Processing and Motivation for Language Learning

Text mining applied to distance higher education: A systematic literature review

Explore related subjects.

• Artificial Intelligence
• Digital Education and Educational Technology

Data availability

The datasets generated and/or analyzed during the current study are available from the corresponding author on reasonable request.

Audrin, C., & Audrin, B. (2022). Key factors in digital literacy in learning and education: A systematic literature review using text mining. Education and Information Technologies, 27 (6), 7395–7419.

Article   Google Scholar  

Bianchi, F., Terragni, S., & Hovy, D. (2020). Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. Annual Meeting of the Association for Computational Linguistics .

Bischof, J., & Airoldi, E.M. (2012). Summarizing topical content with word frequency and exclusivity. ArXiv, abs/1206.4631 .

Blei, D. M. (2012). Probabilistic topic models. Communications of the ACM, 55 (4), 77–84.

Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. Journal of Machine Learning Research, 3 (Jan), 993–1022.

Google Scholar  

Chang, I. C., Yu, T. K., Chang, Y. J., & Yu, T. Y. (2021). Applying text mining, clustering analysis, and latent dirichlet allocation techniques for topic classification of environmental education journals. Sustainability, 13 (19), 10856.

Chen, M. H., Chen, W. F., & Ku, L. W. (2018). Application of sentiment analysis to language learning. Ieee Access : Practical Innovations, Open Solutions, 6 , 24433–24442.

Chen, X., Zou, D., Cheng, G., & Xie, H. (2020). Detecting latent topics and trends in educational technologies over four decades using structural topic modeling: A retrospective of all volumes of computers & education. Computers & Education, 151 , 103855.

Cheng, B., Wang, M., Mørch, A. I., Chen, N. S., & Spector, J. M. (2014). Research on e-learning in the workplace 2000–2012: A bibliometric analysis of the literature. Educational Research Review, 11 , 56–72.

Chowdhary, P.K. (2020). Natural language processing. Fundamentals of artificial intelligence.

Crossley, S., & McNamara, D. (2013). Applications of text analysis tools for spoken response grading. Language Learning & Technology, 17 (2), 171–192.

Daenekindt, S., & Huisman, J. (2020). Mapping the scattered field of research on higher education. A correlated topic model of 17,000 articles, 1991–2018. Higher Education, 80 (3), 571–587.

Devlin, J., Chang, M., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. North American Chapter of the Association for Computational Linguistics .

Diao, L., & Hu, P. (2021). Deep learning and multimodal target recognition of complex and ambiguous words in automated English learning system. Journal of Intelligent & Fuzzy Systems, 40 (4), 7147–7158.

Dieng, A. B., Ruiz, F. J., & Blei, D. M. (2020). Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8 , 439–453.

Ding, R., Nallapati, R., & Xiang, B. (2018). Coherence-aware neural topic modeling. arXiv preprint arXiv:1809.02687 .

Donthu, N., Kumar, S., Mukherjee, D., Pandey, N., & Lim, W. M. (2021). How to conduct a bibliometric analysis: An overview and guidelines. Journal of Business Research, 133 , 285–296.

Engwall, O., & Lopes, J. (2022). Interaction and collaboration in robot-assisted language learning for adults. Computer Assisted Language Learning, 35 (5–6), 1273–1309.

Ferreira-Mello, R., André, M., Pinheiro, A., Costa, E., & Romero, C. (2019). Text mining in education. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 9 (6), e1332.

Foster, R. N. (1986). Working the S-curve: Assessing technological threats. Research Management, 29 (4), 17–20.

Harvey, M. G. (1984). Application of technology life cycles to technology transfers. Journal of Business Strategy, 5 (2), 51–58.

Hassani, H., Beneki, C., Unger, S., Mazinani, M. T., & Yeganegi, M. R. (2020). Text mining in big data analytics. Big Data and Cognitive Computing, 4 (1), 1.

Hoyle, A. M., Goel, P., & Resnik, P. (2020). Improving neural topic models using knowledge distillation. Conference on Empirical Methods in Natural Language Processing.

Hsu, F. Y., Lee, H. M., Chang, T. H., & Sung, Y. T. (2018). Automated estimation of item difficulty for multiple-choice tests: An application of word embedding techniques. Information Processing & Management, 54 (6), 969–984.

Huang, C., Yang, C., Wang, S., Wu, W., Su, J., & Liang, C. (2020). Evolution of topics in education research: A systematic review using bibliometric analysis. Educational Review, 72 (3), 281–297.

Jaén, M. M., & Basanta, C. P. (2009). Developing conversational competence through language awareness and multimodality: The use of DVDs. ReCALL, 21 (3), 283–301.

Kansanen, P. (1999). Teaching as teaching-studying‐learning interaction. Scandinavian Journal of Educational Research, 43 (1), 81–89.

Kherwa, P., & Bansal, P. (2018). Topic modeling: A comprehensive review.  EAI Endorsed Transactions on Scalable Information Systems,  7 (24):e2.

Kingma, D. P., & Welling, M. (2013). Auto-encoding variational bayes. CoRR.  https://arxiv.org/abs/1312.6114

Lau, J. H., Newman, D., & Baldwin, T. (2014). Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics (pp. 530–539).

Lawley, J. (2016). Spelling: Computerised feedback for self-correction. Computer Assisted Language Learning, 29 (5), 868–880.

Lee, J., Kim, H., & Kron, F. (2023). Virtual education strategies in the context of sustainable health care and medical education: A topic modelling analysis of four decades of research. Medical Education, 58 , 47–62.

Li, S. C., & Lai, T. K. (2022). Unfolding knowledge co-construction processes through social annotation and online collaborative writing with text mining techniques. Australasian Journal of Educational Technology, 38 (1), 148–163.

Li, Y. S., Lam, C. S. N., & See, C. (2021). Using a machine learning architecture to create an AI-powered chatbot for anatomy education. Medical Science Educator, 31 , 1729–1730.

Liang, W., Shi, Y., Chi, K. T., Liu, J., Wang, Y., & Cui, X. (2009). Comparison of co-occurrence networks of the Chinese and English languages. Physica A: Statistical Mechanics and Its Applications, 388 (23), 4901–4909.

Liu, M. C., Huang, Y. M., & Chien, Y. C. (2019). Facilitating vocabulary note taking on computers through the deep processing strategy. Journal of Educational Computing Research, 56 (8), 1296–1323.

Morton, H., & Jack, M. (2010). Speech interactive computer-assisted language learning: A cross-cultural evaluation. Computer Assisted Language Learning, 23 (4), 295–319.

Noorbehbahani, F., & Kardan, A. A. (2011). The automatic assessment of free text answers using a modified BLEU algorithm. Computers & Education, 56 (2), 337–345.

Nylander, E., Fejes, A., & Milana, M. (2022). Exploring the themes of the territory: A topic modelling approach to 40 years of publications in International Journal of Lifelong Education (1982–2021). International Journal of Lifelong Education, 41 (1), 27–44.

Odden, T. O. B., Marin, A., & Rudolph, J. L. (2021). How has Science Education changed over the last 100 years? An analysis using natural language processing. Science Education, 105 (4), 653–680.

Organisciak, P., Acar, S., Dumas, D., & Berthiaume, K. (2023). Beyond semantic distance: Automated scoring of divergent thinking greatly improves with large language models. Thinking Skills and Creativity, 49 , 101356.

Pérez-Marín, D., & Pascual-Nieto, I. (2013). An exploratory study on how children interact with pedagogic conversational agents. Behaviour & Information Technology, 32 (9), 955–964.

Pérez-Marín, D., Pascual-Nieto, I., & Rodríguez, P. (2009). Computer-assisted assessment of free-text answers. The Knowledge Engineering Review, 24 (4), 353–374.

Pérez-Paredes, P., Sánchez-Tornel, M., Alcaraz Calero, J. M., & Jiménez, P. A. (2011). Tracking learners’ actual uses of corpora: Guided vs non-guided corpus consultation. Computer Assisted Language Learning, 24 (3), 233–253.

Rahaman, M. A., & Hoque, A. S. M. L. (2022). An effective evaluation system to grade programming assignments automatically. International Journal of Learning Technology, 17 (3), 267–290.

Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using siamese BERT-Networks. Conference on empirical methods in natural language processing .

Rudzewitz, B. (2021). Learning Analytics in Intelligent Computer-Assisted Language Learning (Doctoral dissertation, Universität Tübingen).

Sang, G., Valcke, M., Van Braak, J., Tondeur, J., & Zhu, C. (2011). Predicting ICT integration into classroom teaching in Chinese primary schools: Exploring the complex interplay of teacher-related variables. Journal of Computer Assisted Learning, 27 (2), 160–172.

Srivastava, A., & Sutton, C. (2017). Autoencoding variational inference for topic models. International Conference on Learning Representation s.

Tang, H., Li, M., & Jin, B. (2019). A topic augmented text generation model: Joint learning of semantics and structural features. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) (pp. 5090–5099).

Tono, Y., Satake, Y., & Miura, A. (2014). The effects of using corpora on revision tasks in L2 writing with coded error feedback. ReCALL, 26 (2), 147–162.

Vlugter, P., Knott, A., McDonald, J., & Hall, C. (2009). Dialogue-based CALL: A case study on teaching pronouns. Computer Assisted Language Learning, 22 (2), 115–131.

Wang, P. Y., & Yang, H. C. (2012). Using collaborative filtering to support college students’ use of online forum for English learning. Computers & Education, 59 (2), 628–637.

Woo, H., Kim, J., & Lee, W. (2020). Development of curriculum design support system based on word embedding and terminology extraction. Electronics, 9 (4), 608.

Wu, T. T., & Chen, A. C. (2018). Combining e-books with mind mapping in a reciprocal teaching strategy for a classical Chinese course. Computers & Education, 116 , 64–80.

Wu, W. C. V., Wang, R. J., & Chen, N. S. (2015). Instructional design using an in-house built teaching assistant robot to enhance elementary school English-as-a-foreign-language learning. Interactive Learning Environments, 23 (6), 696–714.

Xue, J., Tang, X., & Zheng, L. (2021). A hierarchical BERT-based transfer learning approach for multi-dimensional essay scoring. IEEE Access, 9 , 125403–125415.

Yeh, H. C., Yang, Y. F., & Wong, W. K. (2010). Interaction chain patterns of online text construction with lexical cohesion. Journal of Educational Technology & Society, 13 (1), 55–68.

Yun, E. (2020). Review of trends in physics education research using topic modeling. Journal of Baltic Science Education, 19 (3), 388–400.

Zhang, H., Daim, T., & Zhang, Y. P. (2021). Integrating patent analysis into technology roadmapping: A latent dirichlet allocation based technology assessment and roadmapping in the field of Blockchain. Technological Forecasting and Social Change, 167 , 120729.

Zhao, H., Phung, D. Q., Huynh, V., Jin, Y., Du, L., & Buntine, W. L. (2021). Topic modelling meets deep neural networks: A Survey. International Joint Conference on Artificial Intelligence .

Zhe, L., Meng, C., Takanori, M., & Juan, L. (2018). Construction and application of korean-english-japanese multilingual teaching aid system based on knowledge map. International Journal of Distance Education Technologies (IJDET), 16 (4), 1–14.

Zhu, G., Xing, W., Costa, S., Scardamalia, M., & Pei, B. (2019). Exploring emotional and cognitive dynamics of knowledge building in grades 1 and 2. User Modeling and User-Adapted Interaction, 29 , 789–820.

Download references

Author information

Authors and affiliations.

School of Computer Science and Cyber Engineering, Guangzhou University, Guangzhou, 510165, China

College of Education, College of Health, Lehigh University, Bethlehem, PA, 18015, USA

School of Information and Intelligent Engineering, Guangzhou Xinhua University, Guangzhou, 523133, China

School of Cyber Security, University of Chinese Academy of Sciences, Beijing, 100000, China

Department of Educational Technology, Guangzhou University, 230 Wai Huan Xi Road, Guangzhou Higher Education Mega Center, Guangzhou, 510165, China

Guozhu Ding

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Guozhu Ding .

Ethics declarations

Disclosure statement.

The authors report that there are no competing interests to declare.

Ethics approval

This study involves no human participants. Ethical approval was exempted from the Research Ethics Board Office at Guangzhou University.

Conflicts of interest

The authors declare that they have no conflict of interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Wu, H., Li, S., Gao, Y. et al. Natural language processing in educational research: The evolution of research topics. Educ Inf Technol (2024). https://doi.org/10.1007/s10639-024-12764-2

Download citation

Received : 21 February 2024

Accepted : 29 April 2024

Published : 23 May 2024

DOI : https://doi.org/10.1007/s10639-024-12764-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Topic modeling
• Evolutionary paths
• Word co-occurrence networks
• Find a journal
• Publish with us
• Track your research

Microsoft Research Lab – Asia

The next 10 years look golden for natural language processing research, share this page.

By Ming Zhou (opens in new tab) , Nan Duan (opens in new tab) , Furu Wei (opens in new tab) , Shujie Liu (opens in new tab) , and  Dongdong Zhang (opens in new tab) , Microsoft Research Asia

Language is the holy grail of Artificial Intelligence. The progress of Natural Language Processing (NLP) technologies will push the entire AI field forward. Here’s a view into what’s next.

Since the inception of Microsoft Research Asia, NLP has been a key research area in the field of Artificial Intelligence (AI). In the past 20 years, Microsoft Research Asia has developed NLP technologies, including those which have been shipped in Windows, Office, Bing, Microsoft Cognitive Services, Xiaoice, and Cortana.

This past work, which includes research in deep learning applied to machine translation, extractive machine reading comprehension, and grammar check, has achieved parity with human performance on related evaluation tasks

So what’s next? We believe that the next 10 years will be a golden era in NLP development, for the following reasons:

• Big data will become more easily collected, processed, and archived.
• NLP technology will extend into new applications for search engines, customer support, business intelligence, translation, education, law, finance, and more.
• Robot and IOT requirements will increasingly include text, speech, and vision capabilities.

These trends will stimulate large-scale investment in NLP and attract more talent to work in NLP research and development.

Areas of focus for the next generation of NLP research will include:

• Integrating knowledge and common sense into data-driven learning approaches.
• More attention to low-resource NLP tasks.
• Contextualized modelling and multi-turn dialogue understanding.
• Semantic analysis, leading to NLP that is knowledge-based, commonsense, and explainable.

Why NLP research is key

Natural Language Understanding (NLU) is a research area that uses the computer to analyze and extract key information from natural language sentences and texts, and then perform information retrieval, question-answering, machine translation, and text-generation activities. It is central to progress in many areas of AI, because the goal of AI overall is to make computers and smart devices listen, speak, and understand language; be able to think and solve problems; and even be able to create new things.

Recent progress in NLP includes:

Neural machine translation

Neural machine translation is a process of simulating how a human brain translates.

The task of translation is to convert a source language sentence into a target language sentence and retain the original meaning. When translating, human brains first try to understand the sentence, then form a semantic representation of the sentence in the mind, and finally transform this semantic representation into a sentence in another language. Neural machine translation simulates this human translation process, through two modular processes—encoding and decoding. The encoder is responsible for compressing source language sentences into vector representations in the semantic space, which are expected to contain the semantic information of source language sentences. The decoder generates semantically equivalent sentences of the target language based on semantic vectors provided by the encoder.

The advantage of the neural machine translation model lies in three aspects: end-to-end training, which reduces error propagation among multiple sub-models; the distributed representation of information, which can automatically learn multi-dimensional translation knowledge; and the use of global context information to complete the translation, rather than just using local context. Recurrent neural machine translation is an important foundational model, over which there have been many improvements on either advanced network structures or novel model training methods.

The translation quality of neural machine translation systems keeps improving, with the goal of reaching human-level performance. In 2018, the Chinese-English machine translation system, developed by Microsoft Research Asia in collaboration with the Microsoft Translator product team, reached a translation quality level comparable to human professional translation on the WMT 2017 news test dataset. This system combines four advanced technologies proposed by Microsoft Research Asia, including joint training and dual-learning techniques that can efficiently utilize large-scale monolingual training data to improve the model training, an agreement regularization technique to address the issue of exposure bias, as well as a deliberation network approach to improving translation quality with two-pass translations that simulate the human translation process.

Human-computer interaction

Human-computer interaction (HCI) aims to build machine intelligence that can communicate with humans by using natural language. Conversation as a Platform (CaaP) is one of the most important concepts for this.

Conversation as a Platform (CaaP) is a brand-new concept proposed by Microsoft CEO Satya Nadella in 2016. Satya thinks that conversation will become the next-generation interface, which will bring progress to both the artificial intelligence and device fields.

The reasons why this concept is important are two-fold. First, conversation-centered apps, such as WeChat and Facebook, have become part of everyone’s life, setting up our expectations for future HCI platforms. Second, a large portion of devices have small screens (such as cell phones) or even no screen (such as some IoT devices). On such devices, natural language presents the most straightforward and natural form of communication. Today, HCI using conversational systems can complete tasks such as buying coffee and booking tickets, and there are several CaaP platforms available for developers around the world to build their own conversation-based HCI systems.

In general, the technologies used for building such HCI systems can be divided into three layers: the chat layer, the search and question/answer (QA) layer, and the task-completion layer. The chat layer, such as Xiaoice, provides chat capability, which can make an HCI system more conversational and increase user engagement. The search and QA layer, such as Bing, provides search and question answering capabilities, which can return accurate search results or provide answers to a user’s questions. The task-completion layer, represented by Cortana, provides task-oriented dialogue capability that can help users complete specific tasks such as restaurant reservations, hotel booking, or weather inquiry; and once it gets to know your personal habits, it can remind you of meetings or suggest somewhere to go. A bot with these three layers can provide a natural and useful user experience.

Machine reading comprehension

Teaching a machine to read and comprehend text is a very important research challenge in natural language understanding.

The task and goal of machine reading comprehension is to design systems that can answer questions after reading a passage or document. There are a wide range of applications for this, including the ability for search engines to provide intelligent and accurate answers for natural-language queries by reading the relevant documents on the Web. In addition, machine reading comprehension can also be used in personal assistants, such as Cortana, so that Cortana can help answer customer support questions after reading documents (such as user manuals or product descriptions). It can be also used in work environments to help users read and process emails or business documents, and then summarize the relevant information. In the education domain, machine reading comprehension can be used to design tests. In legal circles, it can be used to help lawyers or judges by reading and understanding legal questions. In financial applications, machine reading comprehension can be used to extract information for making better financial decisions.

The recent advances in machine reading comprehension have been furthered by the use of large-scale, manually annotated datasets. The Stanford Question Answering Dataset (SQuAD) is the most widely used benchmark dataset for machine reading comprehension research. Stanford released SQuAD in July 2016 and it consists of 100,000 human-labeled question and answer pairs. The passages in SQuAD are from Wikipedia articles and each passage is annotated with no more than five questions, with answers that are exact sub-spans of each passage. Stanford divides the dataset into training, development, and test sets. The training set and development set are publicly available, while the test set is hidden from both researchers and participants. Participants need to submit their systems to the Stanford team to obtain the results on the test set, which will be updated on the SQuAD leaderboard. As of November 2018, there were more than 100 entries from academic and industry research labs.

The leaderboard indicates that there has been great progress in machine reading comprehension research in the last two years. In January 2018, the R-net system from Microsoft Research Asia was the first system to exceed human parity on the SQuAD dataset, in terms of the Exact Match (EM) metrics. In early 2018, systems from Alibaba and iFLYTEK also exceeded the EM test for human parity. In September 2018, the system from Microsoft Research Asia, nLnet, became the first to exceed both EM and F1 human parity on the SQuAD dataset. Google’s BERT then became the leader.

The SQuAD dataset provides a great platform and testing ground for the whole research community to develop, verify, and accumulate techniques to benefit the broader research effort in NLP. The technology stacks behind the recent progress of research on machine reading comprehension include end-to-end neural machine reading comprehension models; pretrained models, such as the ELMo from AI2 and BERT from Google AI, for machine reading comprehension and natural language processing; and system innovations on network structures, automatic data augmentation, and implementation.

AI creation

Infusing AI into creation processes and democratizing creation for ordinary people.

As early as 2005, Microsoft Research Asia successfully developed the Microsoft Couplet system, with the proposal and support of Dr. Harry Shum, who at the time was director of the lab. Given the user’s input of the first line of a couplet, the system can automatically generate the second sentence of a couplet, as well as the streamer description.

After that, we developed two intelligent AI creation systems: Metrical Poetry and Chinese Character Riddles. For example, in Chinese Character Riddles, the system is able to both solve and generate riddles based on Chinese characters.

In 2017, Microsoft Research Asia developed a system for writing modern poetry and composing music (including lyric generation and melody composition). This system of song generation has participated in the CCTV 1’s AI program (Machine vs. Human Intelligence). All of these show that deep learning technology and big data have great potential for mimicking a human’s ability to create, and that they can be used to help artists and others to create.

Taking the capability of lyrics generation as an example, the system will first generate a topic before writing the lyrics. For instance, if you would like to write a song related to “autumn,” “sundown,” and “sigh with feeling,” the user can add keywords such as “autumn wind,” “flowing year,” ” gleaming,” “changing,” and so on. The sequence-to-sequence neural networks are used to generate the sentences in the lyrics line-by-line, under the constraint of the topics.

To compose the melody for lyrics, the system should not only consider the quality of the melody, but also the correspondence between the lyrics and the melody. It requires that each note correspond to each word in the lyrics. Given the lyrics as input, we generate the lyric-conditional melody as well as the exact alignment between the generated melody and the given lyrics, simultaneously. More specifically, we develop the melody composition model based on the sequence-to-sequence framework, which is able to jointly produce musical notes and the corresponding alignment.

Hot NLP topics

We summarize the latest NLP technologies into five hot topics:

Hot topic 1: Pre-trained models (or representations)

How machines learn more general and effective pre-trained models (or representations) will continue to be one of the hottest research topics in the NLP area.

One major difficulty faced by many natural language tasks is the limited amount of training data. Today’s researchers are investigating how to learn general and effective pre-trained representations for language understanding, where words and text are represented as vectors. These are useful when task-specific training data are limited.

A Neural Probabilistic Language Model is a foundational work in neural language modeling. In this work, word embeddings are further fed into a neural sequence encoder to encode contextual information. Following this direction, many works, such Word2vec (opens in new tab) and GloVe (opens in new tab) , emerged to further improve the quality of learned word embeddings. One drawback of word embedding is its lack of context sensitivity: the representation of one word is the same regardless of the context it appears in. Work by Peters et al. with ELMo reveal that such context-sensitive representations have already been built by the neural language model. Instead of only using word embeddings, ELMo also leverages the sequence encoder from the language model; and such context-sensitive representations bring drastic improvements over traditional word embedding methods. More recently, BERT uses a transformer-based encoder and a masked word approach to train a very large bidirectional representation from large amount of text, which again brings astounding gains in a variety of tasks.

In the future, it is worth investigating new network structures, lightweight approaches, as well as incorporating world knowledge and common-sense knowledge to learn general pre-trained representations for language understanding. It is also interesting to see if further scaling up the model size and training on more text can bring further improvements.

Hot topic 2: Transfer learning and multi-task learning

Transfer learning has important and practical significance to NLP tasks that lack enough training data. Multi -task learning uses common knowledge from multiple task supervisions and improves model generalization.

In the era of deep learning, different NLP tasks often share encoders that have a homogeneous network structure, such as RNN, CNN, or transformer. This makes transfer learning more practical and straightforward. Using pre-trained word embeddings such as Word2Vec,ELMo or BERT, we employ a type of transfer learning method where the knowledge (word embeddings) learnt from a large-scale corpus via a language model is transferred to downstream tasks directly, by initializing corresponding network layers of downstream task models. Such methods are important to those tasks with little training data.

Multi-task learning is another paradigm that can use different task supervisions to improve a target task, by learning common knowledge from all involved tasks. In 2008, Collobert and Weston proposed a deep learning-based, multi-task framework, and it was the first work to combine deep learning and multi-task learning for NLP. In 2018, McCann proposed another multi-task learning framework, which treats all involved tasks as question-answering tasks and trains a unified model for ten NLP tasks. Experiments show that all tasks can benefit by using the common knowledge learnt from different task supervisions. Based on such common knowledge, each specific task can be further fine-tuned.

Hot topic 3: Knowledge and common sense

How to utilize knowledge and common sense in natural language understanding has become one of the most important topics in NLP.

With the rapid development of HCI engines (such as chat, QA, and dialogue systems), how to utilize knowledge and common sense in natural language understanding has become one of the most important topics in NLP, as they are essential for conversation engines or other types of HCI engines to understand user queries, manage conversations, and generate responses.

Wikipedia and knowledge graphs (such as Freebase and Satori) are two types of commonly used knowledge bases. Machine Reading Comprehension(MRC) is a typical NLP task based on Wikipedia, where the MRC model aims to extract an answer from the passage based on the input question. Semantic parsing is another typical NLP task based on a knowledge graph, which aims to convert an input question into a machine-readable and executable logical form. Both tasks are hot topics in NLP.

Commonsense knowledge refers to those facts that all humans are expected to know, such as lemons are sour and an elephant is bigger than a butterfly. Many HCI tasks, like QA and dialogue, need common sense to reason and generate responses. However, as most commonsense knowledge is rarely explicitly expressed in textual corpora, NLP models cannot use such knowledge directly. With the rapid development of chat, dialogue, and QA engines, how to build large-scale commonsense knowledgebases and apply them to various NLP tasks have been explored by many researchers in the last two decades.

Hot topic 4: Low-resource NLP tasks

Data augmentation methods are popularly used to enrich the data resource for low-resource NLP tasks, such as introducing domain knowledge (dictionaries and rules) and labeling more useful data with active learning.

For some NLP tasks, such as rare language translation, chatbot and customer service systems in specific domains and in multi-turn tasks, labeled data is hard to acquire and the data sparseness problem becomes serious. These are called low-resource NLP tasks. To enrich the training data, many data augmentation methods can be used. For example, we can introduce domain knowledge (dictionaries and rules) or leverage active learning to maximize the gain of labeling data. Researchers can also employ semi-supervised and unsupervised methods to use the unlabeled data. Labeled data from other tasks and other languages can also be used with multi-task learning and transfer learning.

Taking machine translation as an example, some rare language translation tasks only have a bilingual dictionary for model training, without any bilingual corpus. Based on this small dictionary of only a few thousand entries, cross-lingual word embedding methods can be used to map the source words and the target words into one semantic space, leveraging a large monolingual corpus. In this semantic space, the source word and the corresponding target word have similar word representations. Based on the cross-lingual word embedding, we can compute the semantic similarity of source and target words, which are used to build a word-based translation table. Together with the trained language model, we can build word-based statistical machine translation (SMT) systems, which are used to translate the monolingual corpus into a pseudo-bilingual corpus and turn the unsupervised translation task into a supervised one. Leveraging the pseudo-bilingual corpus, source-to-target and target-to-source neural translation models can be initialized and boosted with each other by using joint training methods and the large, monolingual corpus.

To improve the translation performance of rare languages, we also propose leveraging the large bilingual corpus between rich languages to boost four translation models for rare ones in a joint generalized EM training framework. Given two rich languages, such as X （Chinese） and Y (English), the rare language Z (such as Hebrew) is treated as a hidden state between X and Y. The translation process from X to Y can be redefined as translating X to Z first, and then translating from Z to Y, and similar for the direction from Y to X. Based on this, we can use the large bilingual data between X and Y to jointly train four translation models, which are P(Z|X), P(Y|Z), P(Z|Y), and P(X|Z), with the popularly used generalized EM training in an iterative process.

Hot topic 5: Multi-modal learning

As a typical multi-modal task, visual QA (VQA) receives great interest by researchers from both NLP and computer vision areas.

Before knowing how to speak, infants perceive the world by seeing, listening, and touching. This means language is not the only way to learn and communicate with the world. Therefore, we should substantially consider both language and other modalities for building artificial generic intelligence. This is called multi-modal learning.

As a typical multi-modal task, visual QA (VQA) receives great interest by researchers in the NLP and computer vision areas. Given an image and natural language question, VQA aims to generate the answer to the input question and depends on the deep understanding and sufficient interaction between the input question and image. Recently, researchers from Microsoft Research Asia presented two VQA approaches in this year’s CVPR and KDD, based on question generation and scene graph generation technologies respectively. We achieved state-of-the-art results on VQA benchmark datasets, including COCO and VQA 2.0. Besides VQA, video QA is another popular multi-modal learning task. Different from VQA, video QA returns a short video clip as the answer to the input query, which makes search results more vivid. With the rapid development of short music and video social platforms, live streaming apps, and mixed and artificial reality technology, how to understand, search, create, and utilize videos will become one of the keys to the next generation of HCI engines.

Future prospects

We think an ideal NLP framework could be a general-purpose architecture as described below. Note this would be one of the typical designs, there could be different design choice on using various technologies for a specific task.

As the first step, it works on the natural language sentence and obtains the word sequence, part-of-speech, dependency analysis, entity identification, intent identification, relation identification, and so on.

Then, the encoder will transform the information obtained into a semantic expression. In this procedure, the pre-trained word embedding and pre-trained entity embedding naturally bring in contextual information of a word or an entity. Furthermore, the same sentence will be encoded with other task-specific encoders and information obtained from these encoders are appended into the encoding of the main-task, with appropriate weights via transfer learning. The additional encoding from other task-specific encoding will further enrich the encoding of the input sentence.

Next, based on the semantic expression obtained from the above process, we can use a decoder to generate the expected output. Additionally, multi-task learning can be applied to introduce other NLP tasks as complementary resources to help with the main-task learning. If the task involves multi-turn modeling, we will need to record the output of the previous turn into memory and use it for the decoding and inference of the subsequent turns.

To realize this ideal NLP framework, we will need to implement the following tasks:

• Construct a large-scale commonsense knowledge base and set up effective evaluation tasks to push forward the related research.
• Study more effective expressions of words, phrases, and sentences, and build a more powerful pre-trained network for expressions at different levels.
• Push forward unsupervised learning and semi-supervised learning, by using a limited amount of knowledge to strengthen the learning ability and by building powerful, cross-lingual word-embedding models.
• Leverage the effect of multi-task learning and transfer learning in NLP tasks, and boost the effect of reinforcement learning for typical tasks such as multi-turn dialogue in customer support systems.
• Effectively model discourse and multi-turn conversation and multi-turn semantic analysis.
• Conduct user modeling and apply it to personalized recommendation and output systems.
• Build an expert system for a specific domain that uses the new generation of reasoning systems, task-completion and conversation systems, and integrates both domain knowledge and commonsense knowledge.
• Develop the explainability of NLP systems by using semantic analysis and knowledge systems.

In the next ten years, NLP research will explode. We can expect that there will be big progress in NLP fundamental research, core technologies, and important applications. As Bill Gates said, “We always overestimate the change that will occur in the next two years and underestimate the change that will occur in the next ten.” This is true for NLP as well.

Let’s imagine what changes NLP will bring us in the next ten years.

• In ten years, machine translation systems will be capable of modeling the context of a sentence and handling new terms. People will use a machine system as a spontaneous interpreter at meetings or presentations.
• An electronic personal assistant will understand your natural command and completes orders for food, flowers, and tickets. You will get used to being served by a robot customer support agent.
• When you climb a mountain, you can tell your phone about your thoughts and upload a photo. Then, your phone will pop up a poem with beautiful language and the photo, and that poem can be sent out to your friends.
• Many news articles will be written by a computer.
• A computer teacher corrects your English pronunciation and polishes your sentences through natural conversation.
• A robot will analyze massive documents and provide a data analysis report in a timely manner to help business leaders make decisions.
• News, books, classes, meetings, articles, and goods will be recommended to you by an intelligent recommendation system.
• Robots will help lawyers to find evidence and suggest similar cases. It can also discover the flaws of a contract or write up a legal document.
• And more, limited only by our imaginations.

While some of the above-mentioned scenarios have already emerged, they will become more mature in the next ten years. In the future, NLP and other AI technologies will dramatically change human life. To realize this bright future, we will continue to innovate boldly and solidly advance by balancing research and application. We will create a new generation of technology designed to serve all of human society.

• Follow on X
• Like on Facebook
• Follow on LinkedIn
• Subscribe on Youtube
• Follow on Instagram
• Subscribe to our RSS feed

Share this page:

• Share on Facebook
• Share on LinkedIn
• Share on Reddit

IEEE Account

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

• Frontiers in Artificial Intelligence
• Language and Computation
• Research Topics

Perspectives for Natural Language Processing between AI, Linguistics and Cognitive Science

Total Downloads

Total Views and Downloads

About this Research Topic

Natural Language Processing (NLP) today - like most of Artificial Intelligence (AI) - is much more of an "engineering" discipline than it originally was, when it sought to develop a general theory of human language understanding that not only translates into language technology, but that is also

Natural Language Processing (NLP) today - like most of Artificial Intelligence (AI) - is much more of an "engineering" discipline than it originally was, when it sought to develop a general theory of human language understanding that not only translates into language technology, but that is also linguistically meaningful and cognitively plausible. At first glance, this trend seems to be clearly connected to recent rapid development. Such development, in the last ten years, was driven to a large extent by the adoption of deep learning techniques. However, it can be argued that the move towards deep learning has the potential of bringing NLP back to its roots after all. Some recent activities in this direction include: ● Techniques like multi-task learning have been used to integrate cognitive data as supervision in NLP tasks 1 ; ● Pre-training/fine-tuning regimens are potentially interpretable in terms of cognitive mechanisms like general competencies applied to specific tasks 2 ; ● The ability of modern models for 'few-shot' or even 'zero-shot' performance on novel tasks mirrors human performance 3 ; ● Analysis of complex neural network architectures like transformer models has found evidence of unsupervised structure learning that mirrors classical linguistic structures using so-called 'probing studies' 4,5 . The last generation of neural network architectures has allowed AI and NLP to make unprecedented progress in developing systems endowed with natural language capabilities. Such systems (e.g., GPT) are typically trained with huge computational infrastructures on large amounts of textual data from which they acquire knowledge thanks to their extraordinary ability to record and generalize the statistical patterns found in data. However, the debate about the human-like semantic abilities that such “juggernaut models” really acquire is still wide open. In fact, despite the figures typically reported to show the success of AI on various benchmarks, other research argues that their semantic competence is still very brittle 6,7,8 . Thus, an important limitation of current AI research is the lack of attention to the mechanisms behind human language understanding. The latter does not only consist in a brute-force, data-intensive processing of statistical regularities, but it is also governed by complex inferential mechanisms that integrate linguistic information and contextual knowledge coming from different sources and potentially different modalities ("grounding"). We posit that the possibility for new breakthroughs in the study of human and machine intelligence calls for a new alliance between NLP, AI, linguistic and cognitive research. The current computational paradigms can offer new ways to explore human language learning and processing, while linguistic and cognitive research can contribute by highlighting those aspects of human intelligence that systems need to model or incorporate within their architectures. The current Research Topic aims at fostering this process by discussing perspectives forward for NLP, given the data and learning devices we have at hand and given the conflicting interests of the participating fields. Suitable topics include, but are not limited to: ● What can NLP do for linguistics, and vice versa? ● What can NLP do for cognitive science, and vice versa? ● How does modeling language relate to modeling general intelligence? ● How do we measure short-term long-term success in NLP? ● Is interdisciplinary research the way ahead for NLP? What are hallmarks for successful interdisciplinary research on language? We invite not only empirical work but also theoretical (methodological) considerations and position papers. Information for the authors: - To ensure a quick and high-quality reviewing process, we invite authors to act as reviewers for other submissions to the collection. - We encourage authors to submit an abstract by June 15th to allow the Guest Editors to assess the relevance of the paper to the collection. References: 1. M. Barrett, and A. Søgaard. "Reading behavior predicts syntactic categories." Proceedings of the 19th conference on Computational Natural Language Learning. 2015. 2. T. Flesch, et al. "Comparing continual task learning in minds and machines." Proceedings of the National Academy of Sciences. 2018. 3. A. Lazaridou, et al. "Hubness and pollution: Delving into cross-space mapping for zero-shot learning." Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. 2015. 4. J. Hewitt, and C. D. Manning. "A structural probe for finding syntax in word representations." Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019. 5. I. Tenney, et al. "BERT Rediscovers the Classical NLP Pipeline." Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019. 6. B. M. Lake, and M. Baroni. "Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks”. Proceedings of the 35th International Conference on Machine Learning. 2018. 7. A. Ravichander, et al. "Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?." arXiv:2005.00719. 2020. 8. E. M. Bender, and A. Koller. "Climbing towards NLU: On meaning, form, and understanding in the age of data." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.

Keywords : Natural Language Processing, Linguistics, Cognitive Science, Human Language Understanding, Language Technology, Deep Learning, Multi-task Learning

Important Note : All contributions to this Research Topic must be within the scope of the section and journal to which they are submitted, as defined in their mission statements. Frontiers reserves the right to guide an out-of-scope manuscript to a more suitable section or journal at any stage of peer review.

Topic Editors

Topic coordinators, recent articles, submission deadlines.

Submission closed.

Participating Journals

Total views.

• Demographics

No records found

total views article views downloads topic views

Top countries

Top referring sites, about frontiers research topics.

With their unique mixes of varied contributions from Original Research to Review Articles, Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author.

• Conferences
• Last updated April 20, 2020
• In AI Origins & Evolution

8 Different NLP Scenarios One Can Take Up For A Project

• Published on April 20, 2020
• by Ambika Choudhury

Over the last few years, natural language processing ( NLP ) techniques have witnessed fast growth in quality as well as usability. Today, NLP is one of the most trending topics of research in the field of STEM . Tech giants have been researching NLP, and applying newer deep learning methods to gain a deeper understanding of the consumers.

In this article, we list down – in no particular order – eight different NLP scenarios that one can take up for a project.

1| Question Answering 

Question answering is one of the most prevalent research problems in NLP. Some of its applications are chatbots, information retrieval, dialog systems, among others. It serves as a powerful tool to automatically answer questions asked by humans in natural language, with the help of either a pre-structured database or a collection of natural language documents.

Models: Models like BiDAF, BERT, and XLNet can be used for question-answering projects.

Dataset: Stanford Question Answering Datase t (SQuAD), Conversational Question Answering systems (CoQA), etc.  

2| Text Classification

Text Classification or Text Categorization is the technique of categorizing and analyzing text into some specific groups. This technique supports a comparative evaluation of the impact of linguistic information concerning approaches based on word matching. 

Models: BERT, XLNet, and RoBERTa can be used for text classification.

Dataset: Amazon Reviews dataset , IMDB dataset , SMS Spam Collection , etc. 

3| Text Summarization

Text summarization is one of the most efficient methods to interpret text information. Text summarization methods can be mainly categorized into two parts – extractive summarization and abstractive summarization. In extractive summarization, the process involves selecting sentences of high rank from any document based on word and sentence features and fusing them to generate a summary. On the other hand, an abstractive summarization is mainly used to understand the main concepts in any given document and then express those concepts in any natural language. 

Models: BERTSumExt, BERTSumAbs, and UniLM (s2s-ft) can be used for text summarization.

Dataset: BBC News Summary , Large-Scale Chinese Short Text Summarization Dataset , etc.

4| Sentiment Analysis 

Sentiment Analysis is the technique of understanding human sentiments implied in a text, and helps classify emotions using text analysis methods. This technique has witnessed significant traction due to the growth of social media platforms like Facebook, Instagram, and more. Some of the applications of this technique are market research, brand monitoring, customer service, among others.

Models: Models like Dependency Parser, BERT, and RoBERTa can be used for sentiment analysis.

Dataset: Stanford Sentiment Treebank , Multi-Domain Sentiment Dataset , Sentiment140 , etc.  

5| Sentence Similarity

Sentence similarity portrays an important part in text-related research and applications in areas such as text mining and dialogue systems. This technique has proven to be one of the best to improve retrieval effectiveness, where titles are used to represent documents in the named page finding task. 

Models: BERT, GloVe, etc. can be used for sentence similarity projects.

Dataset: Paraphrase Adversaries from Word Scrambling (PAWS) 

6| Speech Recognition

Speech Recognition is the technique used in identifying spoken words or phrases and translating them into machine language. Speech recognition has gained attention in recent years with the dramatic improvements in acoustic modeling yielded by deep feedforward networks.

Models: BERT, RoBERTa, etc. can be used for speech recognition projects.

Dataset: Google AudioSet , LibriSpeech ASR corpus , etc.

7| Neural Machine Translation

Neural machine translation is one of the most popular approaches in NLP research. The neural machine translation aims at building a single neural network that can be jointly tuned to

maximize translation performance. 

Models: BERT, RNN Encoder-Decoder, etc. 

Dataset: English-Persian parallel corpus , Japanese-English Bilingual Corpus , etc.

8| Document Summarization

Document Summarization is the technique of helping readers catch the main points of a long document with less effort. It also helps as a preprocessing step for some text mining tasks such as document classification. This method can be categorized into two different dimensions – abstract-based and extract-based. An extract-based summary includes sentences that are extracted from the document. In contrast, an abstract-based summary may consist of words and phrases which do not appear in the original document.

Models: Hidden Markov Model can be used for document summarization. 

Dataset: 20 Newsgroups dataset. 

📣 Want to advertise in AIM? Book here

Ambika Choudhury

• machine translation , NLP AI , NLP projects , Question Answering , sentence similarity , sentiment analysis NLP , Speech recognition , text classification , Text summarization

Gemini shines brighter…

Top Editorial Picks

Google Unveils GameNGen: Integrating Generative AI into Gaming  Aditi Suresh

AI4Bharat Invites Contributors to Chitralekha Open-Source Project Siddharth Jindal

NTT DATA Launches Accelerated AI Platform to Simplify Complex Projects in India  Tanisha Bhattacharjee

Dell Technologies to Offer GenAI Foundations Course and Professional Certifications in India Tanisha Bhattacharjee

Cerebras Launches Fastest AI Inference Solution, Claims 20x Speed Advantage Over NVIDIA Siddharth Jindal

Subscribe to The Belamy: Our Weekly Newsletter

Biggest ai stories, delivered to your inbox every week., "> "> flagship events.

Discover how Cypher 2024 expands to the USA, bridging AI innovation gaps and tackling the challenges of enterprise AI adoption

© Analytics India Magazine Pvt Ltd & AIM Media House LLC 2024

• Terms of use
• Privacy Policy

CLL 345: Linguistics: Home

Modern Language Association's index of journals and books on literature, linguistics, language, folklore, and film.

Modern Language Association's index to international journal and books on literature, linguistics, languages, folklore, and film studies, 1926 to present.

Scholarly journal articles, books, and primary sources in a wide variety of disciplines.

Full text of hundreds of scholarly journals in all disciplines dating back to the 1700s.

Scholarly journals, magazines, and newspapers in almost every subject area. A great starting point for any topic.

Comprehensive index to scholarly journals and general periodicals in all subject and interdisciplinary areas, including full text. Coverage prior to 1990 is sparse.

Starting point for education research, from preschool to college. Journal articles, books, curriculum guides, and more. 1966-present.

Indexes and abstracts articles, books, theses, conference proceedings, and unpublished papers in education, 1966 to present.

Index for journal articles about psychology.

Indexes and abstracts the international literature of psychology, 1887 to present.

General Tips for Picking a Topic

• Pick something you're interested in. It sounds obvious, but it will be much more rewarding to work on a research question that you're interested in answering.
• Think back to what you've discussed in class so far. Was there anything that you'd like to research in more depth? Look ahead in the syllabus. Does anything pique your interest there?
• Do some background reading. Check things like Annual Reviews, reference sources, and recent studies to get a sense of what some of the major topics in Linguistics are right now and if there is anything there you want to learn more about.

Yearly summaries of current research in the field for natural and social sciences.

Full text of annual summaries of scholarship in over three dozen academic disciplines up through 2023.

Other Useful Guides

• All Comparative Languages & Linguistics by Julia Muse Last Updated Jul 16, 2024 9 views this year
• All Psychology Resources by Amy Bryant Last Updated Jul 16, 2024 76 views this year

Writing a Research Proposal: Helpful Guides

• Writing a Research Proposal — USC Libraries
• A Guide to Research Proposals — Sacred Heart University Library

• Last Updated: Aug 28, 2024 3:50 PM
• URL: https://library.earlham.edu/linguistics

Earlham College • 801 National Road West • Richmond, Indiana 47374-4095