Real-World Natural Language Processing: Practical applications with deep learning

by Masato Hagiwara

Overview

Real-world Natural Language Processing shows you how to build the practical NLP applications that are transforming the way humans and computers work together.

In Real-world Natural Language Processing you will learn how to:

    Design, develop, and deploy useful NLP applications
    Create named entity taggers
    Build machine translation systems
    Construct language generation systems and chatbots
    Use advanced NLP concepts such as attention and transfer learning

Real-world Natural Language Processing teaches you how to create practical NLP applications without getting bogged down in complex language theory and the mathematics of deep learning. In this engaging book, you’ll explore the core tools and techniques required to build a huge range of powerful NLP apps, including chatbots, language detectors, and text classifiers.

Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications.

About the technology
Training computers to interpret and generate speech and text is a monumental challenge, and the payoff for reducing labor and improving human/computer interaction is huge! Th e field of Natural Language Processing (NLP) is advancing rapidly, with countless new tools and practices. This unique book offers an innovative collection of NLP techniques with applications in machine translation, voice assistants, text generation, and more.

About the book
Real-world Natural Language Processing shows you how to build the practical NLP applications that are transforming the way humans and computers work together. Guided by clear explanations of each core NLP topic, you’ll create many interesting applications including a sentiment analyzer and a chatbot. Along the way, you’ll use Python and open source libraries like AllenNLP and HuggingFace Transformers to speed up your development process.

What's inside

    Design, develop, and deploy useful NLP applications
    Create named entity taggers
    Build machine translation systems
    Construct language generation systems and chatbots

About the reader
For Python programmers. No prior machine learning knowledge assumed.

About the author
Masato Hagiwara received his computer science PhD from Nagoya University in 2009. He has interned at Google and Microsoft Research, and worked at Duolingo as a Senior Machine Learning Engineer. He now runs his own research and consulting company.

Table of Contents
PART 1 BASICS
1 Introduction to natural language processing
2 Your first NLP application
3 Word and document embeddings
4 Sentence classification
5 Sequential labeling and language modeling
PART 2 ADVANCED MODELS
6 Sequence-to-sequence models
7 Convolutional neural networks
8 Attention and Transformer
9 Transfer learning with pretrained language models
PART 3 PUTTING INTO PRODUCTION
10 Best practices in developing NLP applications
11 Deploying and serving NLP applications

Product Details

ISBN-13:	9781638350392
Publisher:	Manning
Publication date:	12/21/2021
Sold by:	SIMON & SCHUSTER
Format:	eBook
Pages:	336
File size:	11 MB
Note:	This product may take a few minutes to download.

About the Author

Masato Hagiwara received his computer science PhD from Nagoya University in 2009, focusing on Natural Language Processing and machine learning. He has interned at Google and Microsoft Research, and worked at Baidu Japan, Duolingo, and Rakuten Institute of Technology. He now runs his own consultancy business advising clients, including startups and research institutions.

Table of Contents

Preface xi

Acknowledgments xiii

About this book xiv

About the author xvii

About the cover illustration xviii

Part 1 Basics 1

1 Introduction to natural language processing 3

1.1 What is natural language processing (NLP)? 4

What is NLP? 4

What is not NLP? 6

AI, ML, DL, and NLP 8

Why NLP? 10

1.2 How NLP is used 12

NLP applications 13

NLP tasks 15

1.3 Building NLP applications 21

Development of NLP applications 21

Structure of NLP applications 24

2 Your first NLP application 26

2.1 Introducing sentiment analysis 27

2.2 Working with NLP datasets 28

What is a dataset? 28

Stanford Sentiment Treebank 29

Train, validation, and test sets 30

Loading SST datasets using AllenNLP 33

2.3 Using word embeddings 34

What are word embeddings? 34

Using word embeddings for sentiment analysis 36

2.4 Neural networks 37

What are neural networks? 37

Recurrent neural networks (RNNs) and linear layers 38

Architecture for sentiment analysis 39

2.5 Loss functions and optimization 41

2.6 Training your own classifier 43

Batching 43

Putting everything together 44

2.7 Evaluating your classifier 45

2.8 Deploying your application 46

Making predictions 46

Serving predictions 46

3 Word and document embeddings 49

3.1 Introducing embeddings 50

What are embeddings? 50

Why are embeddings important? 50

3.2 Building blocks of language: Characters, words, and phrases 52

Characters 52

Words, tokens, morphemes, and phrases 53

N-grams 53

3.3 Tokenization, stemming, and lemmatization 54

Tokenization 54

Stemming 55

Lemmatization 56

3.4 Skip-gram and continuous bag of words (CBOW) 57

Where word embeddings come from 57

Using word associations 58

Linear layers 59

Softmax 61

Implementing Skip-gram on AllenNLP 62

Continuous bag of words (CBOW) model 67

3.5 GloVe 68

How GloVe learns word embeddings 68

Using pretrained GloVe vectors 69

3.6 FastText 72

Making use of subword information 72

Using the fastText toolkit 73

3.7 Document-level embeddings 74

3.8 Visualizing embeddings 76

4 Sentence classification 80

4.1 Recurrent neural networks (RNNs) 81

Handling variable-length input 81

RNN abstraction 82

Simple RNNs and nonlinearity 84

4.2 Long short-term memory units (LSTMs) and gated recurrent units (GRUs) 88

Vanishing gradients problem 88

Long short-term memory (LSTM) 90

Gated recurrent units (GRUs) 92

4.3 Accuracy, precision, recall, and F-measure 93

Accuracy 93

Precision and recall 94

F-measure 96

4.4 Building AllenNLP training pipelines 96

Instances and fields 97

Vocabulary and token indexers 98

Token embedders and RNNs 99

Building your own model 100

Putting it all together 101

4.5 Configuring AllenNLP training pipelines 102

4.6 Case study: Language detection 105

Using characters as input 106

Creating a dataset reader 106

Building the training pipeline 108

Running the detector on unseen instances 110

5 Sequential labeling and language modeling 112

5.1 Introducing sequential labeling 113

What is sequential labeling? 113

Using RNNs to encode sequences 115

Implementing a Seq2Seq encoder in AllenNLP 117

5.2 Building a part-of-speech tagger 118

Reading a dataset 118

Defining the model and the loss 119

Building the training pipeline 121

5.3 Multilayer and bidirectional RNNs 122

Multilayer RNNs 122

Bidirectional RNNs 124

5.4 Named entity recognition 126

What is named entity recognition? 127

Tagging spans 128

Implementing a named entity recognizer 128

5.5 Modeling a language 130

What is a language model? 130

Why are language models useful? 131

Training an RNN language model 132

5.6 Text generation using RNNs 133

Feeding characters to an RNN 134

Evaluating text using a language model 134

Generating text using a language model 136

Part 2 Advanced Models 139

6 Sequence-to-sequence models 141

6.1 Introducing sequence-to-sequence models 142

6.2 Machine translation 101 144

6.3 Building your first translator 147

Preparing the data-sets 148

Training the model 150

Running the translator 153

6.4 How Seq2Seq models work 154

Encoder 154

Decoder 156

Greedy decoding 158

Beam search decoding 161

6.5 Evaluating translation systems 163

Human evaluation 163

Automatic evaluation 163

6.6 Case study: Building a chatbot 165

Introducing dialogue systems 165

Preparing a dataset 166

Training and running a chatbot 167

Next steps 169

7 Convolutional neural networks 171

7.1 Introducing convolutional neural networks (CNNs) 172

RNNs and their shortcomings 172

Pattern matching for sentence classification 173

Convolutional neural networks (CNNs) 174

7.2 Convolutional layers 174

Pattern matching using filters 175

Rectified linear unit (ReLU) 176

Combining scores 178

7.3 Pooling layers 179

7.4 Case study: Text classification 180

Review: Text classification 180

Using CnnEncoder 181

Training and running the classifier 182

8 Attention and Transformer 184

8.1 What is attention? 185

Limitation of vanilla Seq2Seq models 185

Attention mechanism 186

8.2 Sequence-to-sequence with attention 187

Encoder-decoder attention 188

Building a Seq2Seq machine translation with attention 189

8.3 Transformer and self-attention 192

Self-attention 192

Transformer 195

Experiments 197

8.4 Transformer-based language models 200

Transformer as a language model 200

Transformer-XL 203

GPT-2 205

XLM 207

8.5 Case study: Spell-checker 208

Spell correction as machine translation 208

Training a spell-checker 210

Improving a spell-checker 213

9 Transfer learning with pretrained language models 218

9.1 Transfer learning 219

Traditional machine learning 219

Word embeddings 220

What is transfer learning? 220

9.2 BERT 222

Limitations of word embeddings 222

Self-supervised learning 224

Pretraining BERT 225

Adapting BERT 226

9.3 Case study 1: Sentiment analysis with BERT 229

Tokenizing input 230

Building the model 232

Training the model 233

9.4 Other pretrained language models 236

ELMo 236

XLNet 237

RoBERTa 239

DistilBERT 240

ALBERT 241

9.5 Case study 2: Natural language inference with BERT 243

What is natural language inference? 243

Using BERT for sentence-pair classification 244

Using Transformers with AllenNLP 246

Part 3 Putting into Production 253

10 Best practices in developing NLP applications 255

10.1 Batching instances 256

Padding 256

Sorting 257

Masking 259

10.2 Tokenization for neural models 261

Unknown words 261

Character models 262

Subword models 263

10.3 Avoiding overfitting 265

Regularization 265

Early stopping 268

Cross-validation 269

10.4 Dealing with imbalanced datasets 270

Using appropriate evaluation metrics 270

Upsampling and downsampling 271

Weighting losses 272

10.5 Hyperparameter tuning 273

Examples of hyperparameters 274

Grid search vs. random search 275

Hyperparameter tuning with Optuna 276

11 Deploying and serving NLP applications 280

11.1 Architecting your NLP application 281

Before machine learning 282

Choosing the right architecture 282

Project structure 283

Version control 285

11.2 Deploying your NLP model 286

Testing 286

Train-serve skew 288

Monitoring 289

Using CPUs 289

11.3 Case study: Serving and deploying NLP applications 292

Serving models with TorchServe 292

Deploying models with SageMaker 296

11.4 Interpreting and visualizing model predictions 298

11.5 Where to go from here 302

Index 305