Sequence Labelling with a Bi LSTM in Py Torch

H
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Named Entity Recognition with PyTorch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we'll explore how we can use Deep Learning for sequence labelling tasks such as part-of-speech tagging or named entity recognition. We won't focus on getting state-of-the-art accuracy, but rather, implement a first neural network to get the main concepts across."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For our experiments we'll reuse the NER data we've already used for our CRF experiments. The Dutch CoNLL-2002 data has four kinds of named entities (people, locations, organizations and miscellaneous entities) and comes split into a training, development and test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[('De', 'Art', 'O'),\n",
       "  ('tekst', 'N', 'O'),\n",
       "  ('van', 'Prep', 'O'),\n",
       "  ('het', 'Art', 'O'),\n",
       "  ('arrest', 'N', 'O'),\n",
       "  ('is', 'V', 'O'),\n",
       "  ('nog', 'Adv', 'O'),\n",
       "  ('niet', 'Adv', 'O'),\n",
       "  ('schriftelijk', 'Adj', 'O'),\n",
       "  ('beschikbaar', 'Adj', 'O'),\n",
       "  ('maar', 'Conj', 'O'),\n",
       "  ('het', 'Art', 'O'),\n",
       "  ('bericht', 'N', 'O'),\n",
       "  ('werd', 'V', 'O'),\n",
       "  ('alvast', 'Adv', 'O'),\n",
       "  ('bekendgemaakt', 'V', 'O'),\n",
       "  ('door', 'Prep', 'O'),\n",
       "  ('een', 'Art', 'O'),\n",
       "  ('communicatiebureau', 'N', 'O'),\n",
       "  ('dat', 'Conj', 'O'),\n",
       "  ('Floralux', 'N', 'B-ORG'),\n",
       "  ('inhuurde', 'V', 'O'),\n",
       "  ('.', 'Punc', 'O')],\n",
       " [('In', 'Prep', 'O'),\n",
       "  (\"'81\", 'Num', 'O'),\n",
       "  ('regulariseert', 'V', 'O'),\n",
       "  ('de', 'Art', 'O'),\n",
       "  ('toenmalige', 'Adj', 'O'),\n",
       "  ('Vlaamse', 'Adj', 'B-MISC'),\n",
       "  ('regering', 'N', 'O'),\n",
       "  ('de', 'Art', 'O'),\n",
       "  ('toestand', 'N', 'O'),\n",
       "  ('met', 'Prep', 'O'),\n",
       "  ('een', 'Art', 'O'),\n",
       "  ('BPA', 'N', 'B-MISC'),\n",
       "  ('dat', 'Pron', 'O'),\n",
       "  ('het', 'Art', 'O'),\n",
       "  ('bedrijf', 'N', 'O'),\n",
       "  ('op', 'Prep', 'O'),\n",
       "  ('eigen', 'Pron', 'O'),\n",
       "  ('kosten', 'N', 'O'),\n",
       "  ('heeft', 'V', 'O'),\n",
       "  ('laten', 'V', 'O'),\n",
       "  ('opstellen', 'V', 'O'),\n",
       "  ('.', 'Punc', 'O')],\n",
       " [('publicatie', 'N', 'O')]]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "\n",
    "train_sents = list(nltk.corpus.conll2002.iob_sents('ned.train'))\n",
    "dev_sents = list(nltk.corpus.conll2002.iob_sents('ned.testa'))\n",
    "test_sents = list(nltk.corpus.conll2002.iob_sents('ned.testb'))\n",
    "\n",
    "train_sents[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we're going to preprocess the data. For this we use the `torchtext` Python library, which has a number of handy  utilities for preprocessing natural language. We process our data to a Dataset that consists of Examples. Each of these examples has two fields: a text field and a label field. Both contain sequential information (the sequence of tokens, and the sequence of labels). We don't have to tokenize this information anymore, as the CONLL data has already been tokenized for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'labels': <torchtext.data.field.Field object at 0x7fc89bc02e10>, 'text': <torchtext.data.field.Field object at 0x7fc89bc02eb8>}\n",
      "['De', 'tekst', 'van', 'het', 'arrest', 'is', 'nog', 'niet', 'schriftelijk', 'beschikbaar', 'maar', 'het', 'bericht', 'werd', 'alvast', 'bekendgemaakt', 'door', 'een', 'communicatiebureau', 'dat', 'Floralux', 'inhuurde', '.']\n",
      "['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-ORG', 'O', 'O']\n",
      "Train: 15806\n",
      "Dev: 2895\n",
      "Test: 5195\n"
     ]
    }
   ],
   "source": [
    "from torchtext.data import Example\n",
    "from torchtext.data import Field, Dataset\n",
    "\n",
    "text_field = Field(sequential=True, tokenize=lambda x:x, include_lengths=True) # Default behaviour is to tokenize by splitting\n",
    "label_field = Field(sequential=True, tokenize=lambda x:x, is_target=True)\n",
    "\n",
    "def read_data(sentences):\n",
    "    examples = []\n",
    "    fields = {'sentence_labels': ('labels', label_field),\n",
    "              'sentence_tokens': ('text', text_field)}\n",
    "    \n",
    "    for sentence in sentences: \n",
    "        tokens = [t[0] for t in sentence]\n",
    "        labels = [t[2] for t in sentence]\n",
    "        \n",
    "        e = Example.fromdict({\"sentence_labels\": labels, \"sentence_tokens\": tokens},\n",
    "                             fields=fields)\n",
    "        examples.append(e)\n",
    "    \n",
    "    return Dataset(examples, fields=[('labels', label_field), ('text', text_field)])\n",
    "\n",
    "train_data = read_data(train_sents)\n",
    "dev_data = read_data(dev_sents)\n",
    "test_data = read_data(test_sents)\n",
    "\n",
    "print(train_data.fields)\n",
    "print(train_data[0].text)\n",
    "print(train_data[0].labels)\n",
    "\n",
    "print(\"Train:\", len(train_data))\n",
    "print(\"Dev:\", len(dev_data))\n",
    "print(\"Test:\", len(test_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we build a vocabulary for both fields. This vocabulary allows us to map every word and label to their index. One index is kept for unknown words, another one for padding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "VOCAB_SIZE = 20000\n",
    "\n",
    "text_field.build_vocab(train_data, max_size=VOCAB_SIZE)\n",
    "label_field.build_vocab(train_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we're on a machine with a CUDA-enabled GPU, we'd like to use this GPU for training and testing. If not, we'll just use the CPU. The check below allows us to write code that works on both CPU and GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another convenient class in `Torchtext` is the BucketIterator. This iterator creates batches of similar-length examples in the data. It also takes care of mapping the words and labels to the correct indices in their vocabularies, and pads the sentences so that they all have the same length. The Bucketiterator creates batches of similar-length examples to minimize the amount of padding. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchtext.data import BucketIterator\n",
    "\n",
    "BATCH_SIZE = 32\n",
    "train_iter = BucketIterator(dataset=train_data, batch_size=BATCH_SIZE, shuffle=True, \n",
    "                            sort_key=lambda x: len(x.text), sort_within_batch=True)\n",
    "dev_iter = BucketIterator(dataset=dev_data, batch_size=BATCH_SIZE, \n",
    "                          sort_key=lambda x: len(x.text), sort_within_batch=True)\n",
    "test_iter = BucketIterator(dataset=test_data, batch_size=BATCH_SIZE, \n",
    "                           sort_key=lambda x: len(x.text), sort_within_batch=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-trained embeddings\n",
    "\n",
    "Pre-trained embeddings embeddings are generally an easy way of improving the performance of your model, particularly if you have little training data. Thanks to these embeddings, you'll be able to make use of knowledge about the meaning and use of the words in your dataset that was learned from another, typically larger data set. In this way, your model will be able to generalize better between semantically related words. \n",
    "\n",
    "In this example, we make use of the popular FastText embeddings. These are high-quality pre-trained word embeddings that are available for a wide variety of languages. After downloading the `vec` file with the embeddings, we use them to initialize our embedding matrix. We do this by creating a matrix filled with zeros whose number of rows equals the number of words in our vocabulary and whose number of columns equals the number of dimensions in the FastText vectors (300). We have to take care that we insert the FastText embedding for a particular word in the correct row. This is the row whose index corresponds to the index of the word in the vocabulary. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading pre-trained embeddings\n",
      "Initializing embedding matrix\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "import os\n",
    "import numpy as np\n",
    "\n",
    "EMBEDDING_PATH = os.path.join(os.path.expanduser(\"~\"), \"data/embeddings/fasttext/cc.nl.300.vec\")\n",
    "\n",
    "\n",
    "def load_embeddings(path):\n",
    "    \"\"\" Load the FastText embeddings from the embedding file. \"\"\"\n",
    "    print(\"Loading pre-trained embeddings\")\n",
    "    \n",
    "    embeddings = {}\n",
    "    with open(path) as i:\n",
    "        for line in i:\n",
    "            if len(line) > 2: \n",
    "                line = line.strip().split()\n",
    "                word = line[0]\n",
    "                embedding = np.array(line[1:])\n",
    "                embeddings[word] = embedding\n",
    "    \n",
    "    return embeddings\n",
    "    \n",
    "\n",
    "def initialize_embeddings(embeddings, vocabulary):\n",
    "    \"\"\" Use the pre-trained embeddings to initialize an embedding matrix. \"\"\"\n",
    "    print(\"Initializing embedding matrix\")\n",
    "    embedding_size = len(embeddings[\".\"])\n",
    "    embedding_matrix = np.zeros((len(vocabulary), embedding_size), dtype=np.float32)\n",
    "                                \n",
    "    for idx, word in enumerate(vocabulary.itos): \n",
    "        if word in embeddings:\n",
    "            embedding_matrix[idx,:] = embeddings[word]\n",
    "            \n",
    "    return embedding_matrix\n",
    "\n",
    "embeddings = load_embeddings(EMBEDDING_PATH)\n",
    "embedding_matrix = initialize_embeddings(embeddings, text_field.vocab)\n",
    "embedding_matrix = torch.from_numpy(embedding_matrix).to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model\n",
    "\n",
    "Next, we create our BiLSTM model. It consists of four layers:\n",
    "    \n",
    "- An embedding layer that maps one-hot word vectors to dense word embeddings. These embeddings are either pretrained or trained from scratch.\n",
    "- A bidirectional LSTM layer that reads the text both front to back and back to front. For each word, this LSTM produces two output vectors of dimensionality `hidden_dim`, which are concatenated to a vector of `2*hidden_dim`.\n",
    "- A dropout layer that helps us avoid overfitting by dropping a certain percentage of the items in the LSTM output.\n",
    "- A dense layer that projects the LSTM output to an output vector with a dimensionality equal to the number of labels.\n",
    "\n",
    "We initialize these layers in the `__init__` method, and put them together in the `forward` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence\n",
    "\n",
    "class BiLSTMTagger(nn.Module):\n",
    "\n",
    "    def __init__(self, embedding_dim, hidden_dim, vocab_size, output_size, embeddings=None):\n",
    "        super(BiLSTMTagger, self).__init__()\n",
    "        \n",
    "        # 1. Embedding Layer\n",
    "        if embeddings is None:\n",
    "            self.embeddings = nn.Embedding(vocab_size, embedding_dim)\n",
    "        else:\n",
    "            self.embeddings = nn.Embedding.from_pretrained(embeddings)\n",
    "        \n",
    "        # 2. LSTM Layer\n",
    "        self.lstm = nn.LSTM(embedding_dim, hidden_dim, bidirectional=True, num_layers=1)\n",
    "        \n",
    "        # 3. Optional dropout layer\n",
    "        self.dropout_layer = nn.Dropout(p=0.5)\n",
    "\n",
    "        # 4. Dense Layer\n",
    "        self.hidden2tag = nn.Linear(2*hidden_dim, output_size)\n",
    "        \n",
    "    def forward(self, batch_text, batch_lengths):\n",
    "\n",
    "        embeddings = self.embeddings(batch_text)\n",
    "        \n",
    "        packed_seqs = pack_padded_sequence(embeddings, batch_lengths)\n",
    "        lstm_output, _ = self.lstm(packed_seqs)\n",
    "        lstm_output, _ = pad_packed_sequence(lstm_output)\n",
    "        lstm_output = self.dropout_layer(lstm_output)\n",
    "        \n",
    "        logits = self.hidden2tag(lstm_output)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "Then we need to train this model. This involves taking a number of decisions: \n",
    "\n",
    "- We pick a loss function (or `criterion`) to quantify how far away the model predictions are from the correct output. For multiclass tasks such as Named Entity Recognition, a standard loss function is the Cross-Entropy Loss, which here measures the difference between two multinomial probability distributions. PyTorch's `CrossEntropyLoss` does this by first applying a `softmax` to the last layer of the model to transform the output scores to probabilities, and then computing the cross-entropy between the predicted and correct probability distributions. The `ignore_index` parameter allows us to mask the padding items in the training data, so that these do not contribute to the loss. We also remove these masked items from the output afterwards, so they are not taken into account when we evaluate the model output.\n",
    "- Next, we need to choose an optimizer. For many NLP problems, the Adam optimizer is a good first choice. Adam is a variation of Stochastic Gradient Descent with several advantages: it maintains per-parameter learning rates and adapts these learning rates based on how quickly the values of a specific parameter are changing (or, how large its average gradient is).\n",
    "\n",
    "Then the actual training starts. This happens in several epochs. During each epoch, we show all of the training data to the network, in the batches produced by the BucketIterators we created above. Before we show the model a new batch, we set the gradients of the model to zero to avoid accumulating gradients across batches. Then we let the model make its predictions for the batch. We do this by taking the output, and finding out what label received the highest score, using the `torch.max` method. We then compute the loss with respect to the correct labels. `loss.backward()` then computes the gradients for all model parameters; `optimizer.step()` performs an optimization step.\n",
    "\n",
    "When we have shown all the training data in an epoch, we perform the precision, recall and F-score on the training data and development data. Note that we compute the loss for the development data, but we do not optimize the model with it. Whenever the F-score on the development data is better than before, we save the model. If the F-score is lower than the minimum F-score we've seen in the past few epochs (we call this number the patience), we stop training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "from tqdm import tqdm_notebook as tqdm\n",
    "from sklearn.metrics import precision_recall_fscore_support, classification_report\n",
    "\n",
    "\n",
    "def remove_predictions_for_masked_items(predicted_labels, correct_labels): \n",
    "\n",
    "    predicted_labels_without_mask = []\n",
    "    correct_labels_without_mask = []\n",
    "        \n",
    "    for p, c in zip(predicted_labels, correct_labels):\n",
    "        if c > 1:\n",
    "            predicted_labels_without_mask.append(p)\n",
    "            correct_labels_without_mask.append(c)\n",
    "            \n",
    "    return predicted_labels_without_mask, correct_labels_without_mask\n",
    "\n",
    "\n",
    "def train(model, train_iter, dev_iter, batch_size, max_epochs, num_batches, patience, output_path):\n",
    "    criterion = nn.CrossEntropyLoss(ignore_index=1)  # we mask the <pad> labels\n",
    "    optimizer = optim.Adam(model.parameters())\n",
    "\n",
    "    train_f_score_history = []\n",
    "    dev_f_score_history = []\n",
    "    no_improvement = 0\n",
    "    for epoch in range(max_epochs):\n",
    "\n",
    "        total_loss = 0\n",
    "        predictions, correct = [], []\n",
    "        for batch in tqdm(train_iter, total=num_batches, desc=f\"Epoch {epoch}\"):\n",
    "            optimizer.zero_grad()\n",
    "            \n",
    "            text_length, cur_batch_size = batch.text[0].shape\n",
    "            \n",
    "            pred = model(batch.text[0].to(device), batch.text[1].to(device)).view(cur_batch_size*text_length, NUM_CLASSES)\n",
    "            gold = batch.labels.to(device).view(cur_batch_size*text_length)\n",
    "            \n",
    "            loss = criterion(pred, gold)\n",
    "            \n",
    "            total_loss += loss.item()\n",
    "\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "            _, pred_indices = torch.max(pred, 1)\n",
    "            \n",
    "            predicted_labels = list(pred_indices.cpu().numpy())\n",
    "            correct_labels = list(batch.labels.view(cur_batch_size*text_length).numpy())\n",
    "            \n",
    "            predicted_labels, correct_labels = remove_predictions_for_masked_items(predicted_labels, \n",
    "                                                                                   correct_labels)\n",
    "            \n",
    "            predictions += predicted_labels\n",
    "            correct += correct_labels\n",
    "\n",
    "        train_scores = precision_recall_fscore_support(correct, predictions, average=\"micro\")\n",
    "        train_f_score_history.append(train_scores[2])\n",
    "            \n",
    "        print(\"Total training loss:\", total_loss)\n",
    "        print(\"Training performance:\", train_scores)\n",
    "        \n",
    "        total_loss = 0\n",
    "        predictions, correct = [], []\n",
    "        for batch in dev_iter:\n",
    "\n",
    "            text_length, cur_batch_size = batch.text[0].shape\n",
    "\n",
    "            pred = model(batch.text[0].to(device), batch.text[1].to(device)).view(cur_batch_size * text_length, NUM_CLASSES)\n",
    "            gold = batch.labels.to(device).view(cur_batch_size * text_length)\n",
    "            loss = criterion(pred, gold)\n",
    "            total_loss += loss.item()\n",
    "\n",
    "            _, pred_indices = torch.max(pred, 1)\n",
    "            predicted_labels = list(pred_indices.cpu().numpy())\n",
    "            correct_labels = list(batch.labels.view(cur_batch_size*text_length).numpy())\n",
    "            \n",
    "            predicted_labels, correct_labels = remove_predictions_for_masked_items(predicted_labels, \n",
    "                                                                                   correct_labels)\n",
    "            \n",
    "            predictions += predicted_labels\n",
    "            correct += correct_labels\n",
    "\n",
    "        dev_scores = precision_recall_fscore_support(correct, predictions, average=\"micro\")\n",
    "            \n",
    "        print(\"Total development loss:\", total_loss)\n",
    "        print(\"Development performance:\", dev_scores)\n",
    "        \n",
    "        dev_f = dev_scores[2]\n",
    "        if len(dev_f_score_history) > patience and dev_f < max(dev_f_score_history):\n",
    "            no_improvement += 1\n",
    "\n",
    "        elif len(dev_f_score_history) == 0 or dev_f > max(dev_f_score_history):\n",
    "            print(\"Saving model.\")\n",
    "            torch.save(model, output_path)\n",
    "            no_improvement = 0\n",
    "            \n",
    "        if no_improvement > patience:\n",
    "            print(\"Development F-score does not improve anymore. Stop training.\")\n",
    "            dev_f_score_history.append(dev_f)\n",
    "            break\n",
    "            \n",
    "        dev_f_score_history.append(dev_f)\n",
    "        \n",
    "    return train_f_score_history, dev_f_score_history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we test the model, we basically take the same steps as in the evaluation on the development data above: we get the predictions, remove the masked items and print a classification report. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test(model, test_iter, batch_size, labels, target_names): \n",
    "    \n",
    "    total_loss = 0\n",
    "    predictions, correct = [], []\n",
    "    for batch in test_iter:\n",
    "\n",
    "        text_length, cur_batch_size = batch.text[0].shape\n",
    "\n",
    "        pred = model(batch.text[0].to(device), batch.text[1].to(device)).view(cur_batch_size * text_length, NUM_CLASSES)\n",
    "        gold = batch.labels.to(device).view(cur_batch_size * text_length)\n",
    "\n",
    "        _, pred_indices = torch.max(pred, 1)\n",
    "        predicted_labels = list(pred_indices.cpu().numpy())\n",
    "        correct_labels = list(batch.labels.view(cur_batch_size*text_length).numpy())\n",
    "\n",
    "        predicted_labels, correct_labels = remove_predictions_for_masked_items(predicted_labels, \n",
    "                                                                               correct_labels)\n",
    "\n",
    "        predictions += predicted_labels\n",
    "        correct += correct_labels\n",
    "    \n",
    "    print(classification_report(correct, predictions, labels=labels, target_names=target_names))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can start the actual training. We set the embedding dimension to 300 (the dimensionality of the FastText embeddings), and pick a hidden dimensionality for each component of the BiLSTM (which will therefore output 512-dimensional vectors). The number of classes (the length of the vocabulary of the label field) will become the dimensionality of the output layer. Finally, we also compute the number of batches in an epoch, so that we can show a progress bar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ba140815a7334dd28cbf7296dc7cc749",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch 0', max=494, style=ProgressStyle(description_width='ini…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Total training loss: 222.35497660934925\n",
      "Training performance: (0.9244241132231894, 0.9244241132231894, 0.9244241132231894, None)\n",
      "Total development loss: 24.117380052804947\n",
      "Development performance: (0.9296043728606681, 0.9296043728606681, 0.9296043728606681, None)\n",
      "Saving model.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.7/site-packages/torch/serialization.py:251: UserWarning: Couldn't retrieve source code for container of type BiLSTMTagger. It won't be checked for correctness upon loading.\n",
      "  \"type \" + obj.__name__ + \". It won't be checked \"\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e1b4716ec9dd4d3dbe903309888ee6a5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch 1', max=494, style=ProgressStyle(description_width='ini…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Total training loss: 71.356663974002\n",
      "Training performance: (0.9591253627050393, 0.9591253627050393, 0.9591253627050393, None)\n",
      "Total development loss: 18.751499708741903\n",
      "Development performance: (0.9388913949107119, 0.9388913949107119, 0.9388913949107119, None)\n",
      "Saving model.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b88bc0032ec148ffb3e787d5b152b0eb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch 2', max=494, style=ProgressStyle(description_width='ini…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Total training loss: 50.72972834017128\n",
      "Training performance: (0.9680819565346124, 0.9680819565346124, 0.9680819565346124, None)\n",
      "Total development loss: 17.262520626187325\n",
      "Development performance: (0.9425531350333006, 0.9425531350333006, 0.9425531350333006, None)\n",
      "Saving model.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4f170531428d428688eeca047775ebf4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch 3', max=494, style=ProgressStyle(description_width='ini…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Total training loss: 41.36486494448036\n",
      "Training performance: (0.9728291980024082, 0.9728291980024082, 0.9728291980024082, None)\n",
      "Total development loss: 16.504801526665688\n",
      "Development performance: (0.9443309363971661, 0.9443309363971661, 0.9443309363971661, None)\n",
      "Saving model.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1b679486eaf3421bbf20686375c8c41b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch 4', max=494, style=ProgressStyle(description_width='ini…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Total training loss: 35.585738136898726\n",
      "Training performance: (0.9755877302066679, 0.9755877302066679, 0.9755877302066679, None)\n",
      "Total development loss: 16.711221787147224\n",
      "Development performance: (0.9458964629713164, 0.9458964629713164, 0.9458964629713164, None)\n",
      "Saving model.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "041d9f3c52464656bae1af40ac9960b8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch 5', max=494, style=ProgressStyle(description_width='ini…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Total training loss: 30.99800857482478\n",
      "Training performance: (0.9784597619470599, 0.9784597619470599, 0.9784597619470599, None)\n",
      "Total development loss: 16.387646575458348\n",
      "Development performance: (0.9469578369198928, 0.9469578369198928, 0.9469578369198928, None)\n",
      "Saving model.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eb16a30222b04d49a285674d79abe5a0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch 6', max=494, style=ProgressStyle(description_width='ini…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Total training loss: 27.11286850180477\n",
      "Training performance: (0.9801326464144016, 0.9801326464144016, 0.9801326464144016, None)\n",
      "Total development loss: 17.557277165353298\n",
      "Development performance: (0.9460822034123172, 0.9460822034123172, 0.9460822034123172, None)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b8bc31674a8c4d13bc3e54164f85c98e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch 7', max=494, style=ProgressStyle(description_width='ini…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Total training loss: 24.422367892228067\n",
      "Training performance: (0.9820917471032945, 0.9820917471032945, 0.9820917471032945, None)\n",
      "Total development loss: 16.09378209663555\n",
      "Development performance: (0.9491867222119033, 0.9491867222119033, 0.9491867222119033, None)\n",
      "Saving model.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5cc4cc80f7fd453ca8cdc695b90f37bd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch 8', max=494, style=ProgressStyle(description_width='ini…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Total training loss: 22.412145966896787\n",
      "Training performance: (0.9829355914806261, 0.9829355914806261, 0.9829355914806261, None)\n",
      "Total development loss: 19.413369961082935\n",
      "Development performance: (0.9440390585613077, 0.9440390585613077, 0.9440390585613077, None)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "16a0f3a493274d578645e7801e49ea6b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch 9', max=494, style=ProgressStyle(description_width='ini…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Total training loss: 20.09223960270174\n",
      "Training performance: (0.9845344545113598, 0.9845344545113598, 0.9845344545113598, None)\n",
      "Total development loss: 15.523855119943619\n",
      "Development performance: (0.948072279565898, 0.948072279565898, 0.948072279565898, None)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9282699e0d6a41f6a5c479f3e4b324c8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch 10', max=494, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Total training loss: 17.925453061121516\n",
      "Training performance: (0.9860395570557233, 0.9860395570557233, 0.9860395570557233, None)\n",
      "Total development loss: 16.789274506270885\n",
      "Development performance: (0.9485233634940431, 0.9485233634940431, 0.9485233634940432, None)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0dd2a7a78ddb40c5b2bc4f5211866565",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch 11', max=494, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Total training loss: 16.409397734270897\n",
      "Training performance: (0.9867205542725174, 0.9867205542725174, 0.9867205542725174, None)\n",
      "Total development loss: 14.920704647898674\n",
      "Development performance: (0.950195027463051, 0.950195027463051, 0.950195027463051, None)\n",
      "Saving model.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4f8b52d4449940b6bafb77e162872878",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch 12', max=494, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Total training loss: 14.990085303143132\n",
      "Training performance: (0.9879493101202108, 0.9879493101202108, 0.9879493101202108, None)\n",
      "Total development loss: 17.077377565205097\n",
      "Development performance: (0.949876615278478, 0.949876615278478, 0.949876615278478, None)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "446456c24c1443809e9a472e93c8bc21",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch 13', max=494, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Total training loss: 13.492025993036805\n",
      "Training performance: (0.9886845897238506, 0.9886845897238506, 0.9886845897238506, None)\n",
      "Total development loss: 19.90118957636878\n",
      "Development performance: (0.9481253482633268, 0.9481253482633268, 0.9481253482633268, None)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b37c259297374650955ab0b45cf71ed9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch 14', max=494, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Total training loss: 12.34741208187188\n",
      "Training performance: (0.9893507826533231, 0.9893507826533231, 0.9893507826533231, None)\n",
      "Total development loss: 18.731272239238024\n",
      "Development performance: (0.9497970122323347, 0.9497970122323347, 0.9497970122323347, None)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d2796b99700548169493af184b534c8b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch 15', max=494, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Total training loss: 10.979540771790198\n",
      "Training performance: (0.9901650184560116, 0.9901650184560116, 0.9901650184560116, None)\n",
      "Total development loss: 17.838901833223645\n",
      "Development performance: (0.949558203093905, 0.949558203093905, 0.949558203093905, None)\n",
      "Development F-score does not improve anymore. Stop training.\n"
     ]
    }
   ],
   "source": [
    "import math\n",
    "\n",
    "EMBEDDING_DIM = 300\n",
    "HIDDEN_DIM = 256\n",
    "NUM_CLASSES = len(label_field.vocab)\n",
    "MAX_EPOCHS = 50\n",
    "PATIENCE = 3\n",
    "OUTPUT_PATH = \"/tmp/bilstmtagger\"\n",
    "num_batches = math.ceil(len(train_data) / BATCH_SIZE)\n",
    "\n",
    "tagger = BiLSTMTagger(EMBEDDING_DIM, HIDDEN_DIM, VOCAB_SIZE+2, NUM_CLASSES, embeddings=embedding_matrix)  \n",
    "\n",
    "train_f, dev_f = train(tagger.to(device), train_iter, dev_iter, BATCH_SIZE, MAX_EPOCHS, \n",
    "                       num_batches, PATIENCE, OUTPUT_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now plot the evolution of the F-score on our training and development set, to visually evaluate if training went well. If it did, the training F-score should first increase suddenly, then more gradually. The development F-score will increase during the first few epochs, but at some point it will start to decrease again. That's when the model starts overfitting. This is where we abandon training, and why we only save the model when we have reached an optimal F-score on the development data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xd4VGXax/HvnZAAMYGEgKEEkoiIsNIDSpOirsC6Fmzoylp2F11Q0X15XWxbXBWsq6u4roXVfS2o2HDttEgXkCpdAiQBJfQSQtr9/nEmyWQSyACTnMnM/bmuc+XMKTP3BPKbZ55zznNEVTHGGBMeItwuwBhjTO2x0DfGmDBioW+MMWHEQt8YY8KIhb4xxoQRC31jjAkjFvrGGBNGLPSNMSaMWOgbY0wYqed2Ab6aNm2qqampbpdhjDF1ytKlS3eparPqtgu60E9NTWXJkiVul2GMMXWKiGz1Zzvr3jHGmDBioW+MMWHEQt8YY8JI0PXpV6WwsJDs7Gzy8/PdLsV1DRo0IDk5maioKLdLMcbUQXUi9LOzs4mLiyM1NRURcbsc16gqu3fvJjs7m7S0NLfLMcbUQX5174jIEBFZLyKbRGR8FetTRGSGiKwUkdkikuy17jERWe2Zrj2ZIvPz80lMTAzrwAcQERITE+0bjzHmpFUb+iISCUwChgIdgetEpKPPZk8C/1HVzsBDwATPvr8AugNdgXOBcSLS6GQKDffAL2W/B2PMqfCne6cXsElVNwOIyBTgMmCN1zYdgT945mcBH3kt/0ZVi4AiEVkJDAHeDUDtxhhT55SUwE8/wdatlafcXFi4EGqybedP6LcCsrweZ+O02r2tAIYDzwJXAHEikuhZ/mcReQqIAQZR8cPCGGNCSkEBZGdXHerbtjlTQcGx99+7F5o0qbn6AnUgdxzwvIjcBHwD5ADFqvqViPQE5gO5wAKg2HdnERkFjAJo06ZNgEoKrH379vHWW28xevToE9pv2LBhvPXWW8THx5/QfuvWrWPEiBGICFOnTqVt27YntL8xJvCKi53W+I4dsH27E+Degb51q7Nc9fjP07QppKQ4U5s25fMpKRAXV7PvwZ/QzwFaez1O9iwro6rbcVr6iEgscKWq7vOsewR4xLPuLWCD7wuo6kvASwDp6enV/LrcsW/fPl544YVKoV9UVES9esf+NX722Wcn9XofffQRV111FQ888MBJ7W+M8V9hodPlsmNH+bR9e+XHO3c6wX88ERGQnFw5zL1D/rTTaud9VcWf0F8MtBORNJywHwFc772BiDQF9qhqCXAvMNmzPBKIV9XdItIZ6Ax8dSoF11RfV3WfzOPHj+eHH36ga9euREVF0aBBAxISEli3bh0bNmzg8ssvJysri/z8fMaOHcuoUaOA8rGEDh06xNChQ+nXrx/z58+nVatWfPzxxzRs2LDSa3322Wc888wzREZGMmPGDGbNmlUTb9mYkFdc7IR5djbk5FQMcu/53NzqM6BU06bQooUztW5dOdRbtYJgvoym2tBX1SIRuR34EogEJqvq9yLyELBEVacBA4EJIqI43TtjPLtHAXM8Z5wcAG7wHNStcyZOnMjq1atZvnw5s2fP5he/+AWrV68uO19+8uTJNGnShCNHjtCzZ0+uvPJKEhMTKzzHxo0befvtt3n55Ze55ppreP/997nhhhsqvdawYcO47bbbiI2NZdy4cbXy/oypawoKnMDOzq445eSUz2/fXn3LHJzGZFKSE+QtW5aHuu/j5s0hOrrm31tN8qtPX1U/Az7zWfYnr/mpwNQq9svHOYMnYPz9NK5pvXr1qnCB1D/+8Q8+/PBDALKysti4cWOl0E9LS6Nr164A9OjRgy1bttRavcbUJQUFTh+5b6B7h/pPP/mXB82aOd0trVqVB7hvsCclwXF6aUNKmLzNwDvNq1Nu9uzZTJ8+nQULFhATE8PAgQOrvICqfv36ZfORkZEcOXKkVmo1Jpjt3AkrV8KKFc60ciWsWeP0sx9PRIQT3q1aOaFe1dSyJXj92Rks9P0WFxfHwYMHq1y3f/9+EhISiImJYd26dSxcuLCWqzMm+BUWwrp15QFf+vPHH6vevk0bZyptpfsGevPm4dM6DyT7lfkpMTGRvn37cs4559CwYUOSkpLK1g0ZMoQXX3yRDh060L59e8477zwXKzXGfbm5Vbfeqzo/PTYWOneGLl2cqXNn6NTJWW4CTzRYOsk90tPT1ffOWWvXrqVDhw4uVRR87Pdh3KYK+/Y5B0q3b3f62deuLQ/4HTuq3q9t28oBn5rqdNWYUyMiS1U1vbrtrKVvjCmjCgcOlId56WmN3o9Lp6NHj/08pa330oAvbb3X9IVHpnoW+i4bM2YM8+bNq7Bs7Nix3HzzzS5VZEKZKvzwAyxdCllZVYd7Xp5/z9WoUcWzYc48szzg09Ks9R6sLPRdNmnSJLdLMCHs8GFYvBgWLHCmhQud/vbjiYkpP73ReyoN99J563Ovmyz0jQkRqpCZWR7wCxY4fey+Fyc1awbnnef0r1cV7HFxNTvKo3GXhb4xdVReHixZUjHkd+6suE1kJHTrBr17l09nnGGhHs4s9I2pA1SdERy9A375cijyGdSkadOKAd+zp7uDe5ngY6FvTBAqKnK6ZubOhTlzYN68yhcxRUQ4B069Q/7MM60Vb47PQv8k/eUvfwnYgGg2dr45cgQWLXICfu5cmD8fDh2quE2TJpVb8XYKpDlRFvpBwMbODz979jit9zlznGnp0spjzbRtC/36Qf/+0LcvtG9vrXhz6upc6Mtfa+Z/vf65+iuTH3nkEV5//XVOP/10WrduTY8ePfjhhx8YM2YMubm5xMTE8PLLL9OiRQs6d+5MZmYmERERHD58mLPPPpvNmzcT5TPQto2dHx62bStvxc+ZA99/X3G9CHTt6gR8v37O1LKlO7Wa0FbnQt8tS5cuZcqUKSxfvpyioiK6d+9Ojx49GDVqFC+++CLt2rVj0aJFjB49mpkzZ9K1a1cyMjIYNGgQ//3vf7n44osrBT7Y2PmhqKTEGZKgtBU/d64T+t7q14dzzy1vyffuDY0bu1OvCS91LvT9aZHXhDlz5nDFFVcQExMDwKWXXkp+fj7z58/n6quvLtvuqOfa9GuvvZZ33nmHQYMGMWXKlBO+t64JXqrOqZFbthx78h1ZOz7e6aIpbcmnp9uQv8YddS70g0lJSQnx8fEsX7680rpLL72U++67jz179rB06VIGDx7sQoXmZJxMqPtKTi5vxffvDz/7mQ1LYIKDhb6fzj//fG666SbuvfdeioqK+OSTT7j11ltJS0vjvffe4+qrr0ZVWblyJV26dCE2NpaePXsyduxYLrnkEiIjI91+C8aHKqxeDV9/DRs3nlioJyY6o0NWNaWk2Fk1JnhZ6Pupe/fuXHvttXTp0oXTTz+dnj17AvDmm2/y+9//nocffpjCwkJGjBhBly5dAKeL5+qrr2b27NkuVm68HToEM2fCZ585U1ZW1dtZqJtQZePp10H2+zgxGzeWh/zs2RVv5JGUBEOHQo8eFuqmbrPx9E3YOnoUMjLKg37jxvJ1Is5gY8OGwS9+4ZwmaX3tJpz4FfoiMgR4FogEXlHViT7rU4DJQDNgD3CDqmZ71j0O/AKIAL4Gxmqwfb2oJTZ2fs3JyioP+enTK44Jn5AAQ4Y4QT9kiDM+jTHhqtrQF5FIYBJwEZANLBaRaaq6xmuzJ4H/qOrrIjIYmACMFJE+QF+gs2e7ucAAYPaJFqqqSB2/HDEQY+eH6edlJUVFzqBjn37qBP2qVRXXd+3qhPywYc758HYDbWMc/vwp9AI2qepmABGZAlwGeId+R+APnvlZwEeeeQUaANGAAFHATydaZIMGDdi9ezeJiYl1PvhPhaqye/duGjRo4HYptU7V6abJyHBa8l995dyjtVRsLFx0kRPyQ4c6NwExxlTmT+i3ArzPccgGzvXZZgUwHKcL6AogTkQSVXWBiMwCduCE/vOquvZEi0xOTiY7O5vc6m75EwYaNGhAcnKy22XUOFVYv9458JqR4Uy+N9tu397plx82zDkn3i52MqZ6gfrSOw54XkRuAr4BcoBiETkT6ACUptTXItJfVed47ywio4BRAG3atKn05FFRUaSlpQWoVBOMVJ2hC7xD/ief74TNmsHAgTBggNM3b4ORGnPi/An9HKC11+Nkz7Iyqrodp6WPiMQCV6rqPhH5HbBQVQ951n0O9Abm+Oz/EvASOKdsntxbMXVJSYkz6FhGhhP033xT+d6tzZs7AV8a9GefbaNMGnOq/An9xUA7EUnDCfsRwPXeG4hIU2CPqpYA9+KcyQOwDfidiEzA6d4ZADwToNpNHVJS4hxs9Q753bsrbtOyZcWQP+ssC3ljAq3a0FfVIhG5HfgS55TNyar6vYg8BCxR1WnAQGCCiChO984Yz+5TgcHAKpyDul+o6ieBfxsmGB05Am+/DdOmOSG/d2/F9cnJFUPe7vpkTM2rE1fkmrpl61Z44QV45RXnZiGl2rQpD/iBAyEtzULemECxK3JNrVKFWbPgueecln1JibM8PR1GjXJOp0xNdbVEYwwW+uYUHToEb7zhhP0az5UbUVFw3XVwxx3OhVHGmOBhoW9OyqZNMGkS/PvfsH+/s6xFC7jtNqdl37y5u/UZY6pmoW/8VlLiXAn73HPw+edOlw5Anz5Oq374cIiOdrdGY8zxWeibah04AK+95rTsN2xwltWvX96F0727q+UZY06Ahb45pnXr4Pnn4fXXnb57cE6zHD0afvtb5wpZY0zdYqFvKigudkatfO455zaCpQYMcFr1l11mI1YaU5fZn68BnCEQJk+Gf/0LMjOdZQ0bwg03wO23Q+fOx9/fGFM3WOiHMVWYOxf++U94//3y2wimpsKYMXDLLdCkiaslGmMCzEI/DO3fD//3f/Dii86gZ+BcGXvJJc4pl0OGQGSkuzUaY2qGhX4Y+e47J+jfegsOH3aWnX66c1B21CjnhuDGmNBmoR/ijhyBd95xunC+/bZ8+aBBTqv+8svt3HpjwomFfohav95p1b/2WvltBePj4cYbnbA/+2xXyzPGuMRCP4QUFsJHHzmt+lmzypf37Am//z1cey3ExLhXnzHGfRb6IWDbNnjpJXj1VfjxR2dZTAxcf73Tqu/Rw936jDHBw0K/Dlu7FsaPh//+t3wo444dnVb9yJHQuLG79Rljgo+Ffh2k6oyD87//C/n5zlDG117rtOr797cbkxhjjs1Cv47ZscO5aOqLL5zHN90Ejz3mnHppjDHVsdCvQz78EH73O+eG4k2aOEMmXHWV21UZY+qSCLcLMNU7dMi5gGr4cCfwL7oIVq60wDfGnDhr6Qe5hQudQc9++MEZw/6xx5zRLiPs49oYcxL8ig4RGSIi60Vkk4iMr2J9iojMEJGVIjJbRJI9yweJyHKvKV9ELg/0mwhFRUXw179Cv35O4HfuDEuWwNixFvjGmJNXbXyISCQwCRgKdASuE5GOPps9CfxHVTsDDwETAFR1lqp2VdWuwGAgD/gqgPWHpE2bnLD/y1+cUzHHjXOGUDjnHLcrM8bUdf60GXsBm1R1s6oWAFOAy3y26QjM9MzPqmI9wFXA56qad7LFhjpV5wKrrl1h0SLnLlXTp8MTTzhdO8YYc6r8Cf1WQJbX42zPMm8rgOGe+SuAOBFJ9NlmBPD2yRQZDnbtcg7U/va3zgiY117rHKwdPNjtyowxoSRQvcPjgAEisgwYAOQAxaUrRaQF0An4sqqdRWSUiCwRkSW5ubkBKqnu+OIL6NTJGTenUSN44w14+21ISHC7MmNMqPEn9HOA1l6Pkz3LyqjqdlUdrqrdgPs9y/Z5bXIN8KGqFlb1Aqr6kqqmq2p6szC62/aRI86ZOEOHOmPmnH++07r/1a/sqlpjTM3wJ/QXA+1EJE1EonG6aaZ5byAiTUWk9LnuBSb7PMd1WNdOBcuWOQOhPf+8M4zCxIkwc6bdyMQYU7OqDX1VLQJux+maWQu8q6rfi8hDInKpZ7OBwHoR2QAkAY+U7i8iqTjfFDICWnkdVVzsnGt/7rnOgGlnn+2ci//HP9otCo0xNU9U1e0aKkhPT9clS5a4XUaNyMpyRr/M8Hz8jRkDjz9uY9wbY06diCxV1fTqtrMrcmvJwYNw4YWwYQMkJcHkyTBsmNtVGWPCjYV+LVCFW291Ar9TJ5gxA8LoeLUxJojYBf214KWXnFMwY2Phvfcs8I0x7rHQr2HLljnj5YAT/u3bu1uPMSa8WejXoAMH4Jpr4OhRGDUKrrvO7YqMMeHOQr+GqDpDKmzaBF26wDPPuF2RMcZY6NeYF15w+u/j4pyfDRu6XZExxljo14ilS+EPf3DmX3kF2rVztx5jjClloR9g+/c7/fgFBTB6tDNvjDHBwkI/gFThlltg82bo3h2eftrtiowxpiIL/QB67jn44ANneOR337Ubnxhjgo+FfoB8+61zW0Nwhlho29bdeowxpioW+gGwd6/Td19Y6IyPf+WVbldkjDFVs9A/Rapw882wdSv07Oncz9YYY4KVhf4p+vvf4eOPIT4e3nnH+vGNMcHNQv8UlN78BODf/4a0NHfrMcaY6ljon6Tdu51+/KIiuPtuuPxytysyxpjqWeifhJISuPFG505Y557r3N/WGGPqAgv9k/DUU/Dpp5CQ4PTjR0e7XZExxvjHQv8EzZsH997rzL/+OqSkuFuPMcacCAv9E7BrF1x7LRQXOxdi/fKXbldkjDEnxq/QF5EhIrJeRDaJyPgq1qeIyAwRWSkis0Uk2WtdGxH5SkTWisgaEUkNXPm1p6QERo6EnBzo0wcefdTtiowx5sRVG/oiEglMAoYCHYHrRKSjz2ZPAv9R1c7AQ8AEr3X/AZ5Q1Q5AL2BnIAqvbY89Bl98AYmJMGUKREW5XZExxpw4f1r6vYBNqrpZVQuAKcBlPtt0BGZ65meVrvd8ONRT1a8BVPWQquYFpPJa9M038MADzvz//R+0bu1uPcYYc7L8Cf1WQJbX42zPMm8rgOGe+SuAOBFJBM4C9onIByKyTESe8HxzqDN27oQRI5zunfHjYehQtysyxpiTF6gDueOAASKyDBgA5ADFQD2gv2d9T+AM4CbfnUVklIgsEZElubm5ASrp1BUXww03wI4d0K8f/O1vbldkjDGnxp/QzwG8OzSSPcvKqOp2VR2uqt2A+z3L9uF8K1ju6RoqAj4Cuvu+gKq+pKrpqprerFmzk3wrgffoo/D119C0qdOPX6+e2xUZY8yp8Sf0FwPtRCRNRKKBEcA07w1EpKmIlD7XvcBkr33jRaQ0yQcDa0697Jp34EB5y/6NN6CVb4eWMcbUQdWGvqeFfjvwJbAWeFdVvxeRh0TkUs9mA4H1IrIBSAIe8exbjNO1M0NEVgECvBzwd1EDli93xsfv0QMuvtjtaowxJjD86rBQ1c+Az3yW/clrfiow9Rj7fg10PoUaXfHdd87Pbt3crcMYYwLJrsg9htLQ717pCIQxxtRdFvrHYKFvjAlFFvpVyMuDtWshMhI617mOKWOMOTYL/SqsWuVcjNWhAzRs6HY1xhgTOBb6VbCDuMaYUGWhXwXrzzfGhCoL/SosW+b8tNA3xoQaC30fBQVOnz5A167u1mKMMYFmoe9jzRon+Nu1g0aN3K7GGGMCy0Lfhx3ENcaEMgt9H3YQ1xgTyiz0fdhBXGNMKLPQ91Jc7IyuCda9Y4wJTRb6XjZscIZgaNPGuXGKMcaEGgt9L3YQ1xgT6iz0vVh/vjEm1Fnoe7Ezd4wxoc5C30PVQt8YE/os9D0yM2H/fkhKghYt3K7GGGNqhoW+h/dBXBF3azHGmJpioe9hB3GNMeHAr9AXkSEisl5ENonI+CrWp4jIDBFZKSKzRSTZa12xiCz3TNMCWXwgWX++MSYc1KtuAxGJBCYBFwHZwGIRmaaqa7w2exL4j6q+LiKDgQnASM+6I6oa1IMUq8LSpc68hb4xJpRVG/pAL2CTqm4GEJEpwGWAd+h3BP7gmZ8FfBTIImva9u2Qmwvx8ZCa6nY1xtQdhwsOsyB7AXO2ziGvMI/Y6Fi/p+jIaMQOoNU6f0K/FZDl9TgbONdnmxXAcOBZ4AogTkQSVXU30EBElgBFwERVDboPhNL+fDuIa8zxHSo4xLxt88jYmsHsLbNZvH0xRSVFJ/Vc9SLqERcdd8wPhRaxLRiYOpDzU86ncYPGAX4npybnQA4LshdQUFxAw3oNaRjV8Lg/G9RrEDQfcP6Evj/GAc+LyE3AN0AOUOxZl6KqOSJyBjBTRFap6g/eO4vIKGAUQJs2bQJUkv+sP9+Yqh04eoB52+Yxe8tsMrZmsGT7Eoq1uGx9hETQo0UPBqQMoHlscw4VHCqfCsvnDx49WGHdwYKDFJUUsTd/L3vz9x7z9Z9e+DQREkHPlj0ZnDaYwWmD6du6Lw2jGtbG2y+TtT+r7IMuY2sGm/ZsOuHnaFCvQdkHQUxUzDE/JF755Ss1+v78Cf0coLXX42TPsjKquh2npY+IxAJXquo+z7ocz8/NIjIb6Ab84LP/S8BLAOnp6Xoyb+RUWOiHr6z9Wby16i0uPONCerTs4XY5rtufv5852+aQsSWDjK0ZLN2xlBItKVsfKZH0bNmTgakDGZAygH5t+p10K7yguKDih4TPtG7XOmZmzmRRzqKyacLcCURHRtOndR8uSLuAwWmD6dmyJ1GRUYH6FQCwdd/WsoDP2JrB5r2bK6yPjY6lb+u+JDRM4EjhEY4UHTnuz6PFR8kvyie/KP+4H3IAky+dHND34ktUj5+xIlIP2ABcgBP2i4HrVfV7r22aAntUtUREHgGKVfVPIpIA5KnqUc82C4DLfA4CV5Cenq5Lliw55Td2Itq0gawsWLsWzj67Vl/auGTVT6t4Yv4TvL36bYpKioiKiOKJi57gznPvDJqv4bVh75G9ZSE/e+tslv+4vELI14uoR3rLdAakDGBg6kD6tu5LXP24Wq3xUMEh5mydw8zMmczInMHyH5ejlOdWbHQs56ecz+DUwVxwxgV0TupMhPh/NrqqsmXflrKQn71lNlv3b62wTaP6jejfpn/Z76Fbi27Ui/C/o6RES8gvyq/2AyKvMI9fdfrVSf0fFJGlqppe7XbVhb7nyYYBzwCRwGRVfUREHgKWqOo0EbkK54wdxeneGeMJ+j7Av4ASnNNDn1HVV4/3WrUd+rt2QbNmEBMDBw5AZGStvbSpZapKxtYMHp/3OJ9v+hxwuif6tO7D3G1zAbiq41W8eumrNKpft2+QXFRSxL78few5soe9R5wuFO+fOw7tYF7WPFb8uKJCgEZFRNGzVU8GpgxkQOoA+rTuQ2x0rIvvpLLdebvJ2JrBjM0zmLllJut2rauwvknDJgxKHVT2TeCsxLMqhKiq8sPeH8o+6DK2ZJB1IKvCc8Q3iKd/m/5l32i6Nu9KZERwh0NAQ7821Xbof/01/Pzn0KcPzJtXay9ralFxSTEfrP2AJ+Y/weLtiwGIiYrhN91+w93n3U1aQhpT10zllo9v4WDBQdo1acfUa6bSOamzy5U7CooLWPnTSnbn7a4U3nuO7CnrF/defrDgoF/PHRURxbnJ55aFfO/k3pwWfVoNv6PAyjmQw6wts8q+CWzbv63C+lZxrRicNpguSV347sfvyNiSQc7BCj3UJDRIYEDqAAakOFPnpM5BH/K+LPT99NhjMH483H47PPdcrb2sqQVHCo/w2vLXeGrBU/yw1zmM1DSmKXf2upPRPUeTGJNYYfuNuzdy1XtXsfKnlTSo14BJwyZxS7db3Ci9zKcbPmXsF2PL6veXIMQ3iCehYQIJDRLKf3rmExsmkt4ynfOSz6v1g6I1SVXZvHczMzJnMDNzJjMzZ5Kbl1tpu8SGiWUhPzB1IOecfs4JdQkFI39DP1Bn79RZdhA39OzO280Li1/guW+fK/uDPyPhDMb1HseNXW8kJiqmyv3aJbZj4W8WcvtntzN5+WR+M+03zN02l+eHPX/MfWpK5t5M7vryLqatdy5iT2mcQrvEdhWCu6qfTRo2IaFhAo3qN6rzIXYyRIS2TdrStklbRvUYRYmW8P3O75mZOZPVO1fTtXlXBqQOoGOzjmH5+wFr6dOuHWza5Nwbt0uXWntZUwO27NvC0wue5tVlr5JXmAdAest07ulzD8M7DD+hr+uvLX+N0Z+O5kjRETqd3omp10zlrMSzaqr0MkcKj/D4vMeZOG8i+UX5xEXH8ZeBf+GOXncE/AwVE1qse8cP+/c7V+FGR8OhQxBlf1OnpERL2LpvK+t2rSub1u5aS15hHqnxqaTFp5GWkEZafBpnJJxBSnwKDeo1OOXXXbZjGU/Mf4J3v3+37BzyoWcO5Z6+9zAgZcBJn42z6qdVXPnulWzcs5G46DhevfRVrv7Z1adc77F8sv4Txn4xlsx9mQD8qtOveOKiJ2gRZ2N9m+pZ944fVqxwfnbqZIF/Io4UHmHD7g1loV4a8Ot3rye/KL/KfZbuWFrl8pZxLcs+BHw/FFrGtTxm61xVmb55Oo/Pf5zpm6cDzumFIzuNZFyfcQE5CNspqRNLRi3ht9N+y3tr3uOaqddw57Y7eeLnTxAdGX3Kz19q055N3PXFXXy68VPndU/vxPPDnuf8lPMD9hrGlArr0Lf+/GNTVXbl7arQYi+d37JvS4XT/Ly1iG3B2U3PLps6NO1AbHQsW/ZtIXNfJpv3biZzXyaZezPZtn8b2w9uZ/vB7czLqnzqVFREFCnxKc6HQekHQ0Ia+UX5/H3h31n+43LAOU97VPdR3HXeXbRu3LrS85yKRvUb8c5V79D/2/78z1f/wz++/QeLchbx7tXv0qbxqV09nleYx4Q5E3h8/uMUFBfQqH4jHhr4EGN6jTmhc8CNORFh/T/LQt9ResbDvKx5zM+az+qdq1m7ay17juypcvtIieTMJmeWhXppwLdv2p74BvFV7tO7de9Ky4pKisg+kO18EOzNdD4MPB8Im/du5qfDP7Fpz6ZjXvKedFoSY88dy23pt5HQMOHkfwHVEBHuOPcOerXqxTVTr2FRziK6/asbb1zxBkPbDT3h51NVPlr3EXd/eXfZRUC/7vJrHrvwMZrHNg90+cbYqjbvAAAQiklEQVRUENZ9+uecA99/D4sWQa9etfKSQaGwuJDlPy5n7ra5zMuax7ysefx46MdK28VFxznB3qwDZyeWt97bNmkb0O6NY8krzHO+IXg+EEq/JRw4eoARPxvByC4jA3JM4ETsztvNyA9Hll3cdX//+/nrwL/6fZB4w+4N3Pn5nXz5w5cAdEnqwqRhk+jbpm+N1WzCgx3IrUZeHsTFOaNqHjwIDUPnVOVK9ufvZ0H2AuZtcwJ+Uc6isrNbSiU2TKRfm370bd2X7i2606FZB1rEtgirIQn8VaIlTJw7kQdnPUiJljA4bTBvDX+LpNikY+5zuOAwj8x5hKcWPEVBcQGN6zfm4cEPc1v6bdaVYwLCDuRWY9UqKClxWvuhFPiqyrb925wW/LZ5zM2ay6qfVlXqgz8r8Sz6tu5bFvS+l6qbY4uQCO7rfx+9k3tz3fvXMTNzJt3+1Y0pV02pdPBVVXl/7fv84cs/lF3qf3PXm5l44UROP+10N8o3YS5sQz9U+vOLS4pZ+dNK5mXNK+uuyT6QXWGbqIgoerTsQb/W/ejbpi99WvexwAmAQWmDWHbrMka8P4Jvtn7D4NcH8+gFjzKuzzgiJIJ1u9Zxx+d3lJ1d1K15NyYNm1Tl8Q1jaouFfh0LfVVl9c7VzMicwfTN0/lm6zeVxlmJbxBP39Z9y1ry6S3TQ+pS+2DSIq4FM349gwdnPsjEeRP54/Q/MnfbXDo07cDfF/6dwpJCEhok8MjgRxjVY1SdG8/FhB4L/ToQ+lv3bWX65unMyJzBjMwZ7Dy8s8L6MxLOqNBV06FZh7C9xNwN9SLqMeHCCfRt05dff/hrPtnwCZ9s+ARB+G233zLhwgk0jWnqdpnGAGEa+gUFsHq1Mx+MQy/sytvFrMxZZa1538G2Wsa15IK0C7jwjAsZnDaY5EbJLlVqvF1y1iV8d+t33PLxLRSVFPHkz5+kV6swOi3M1AlhGfpr1jjB364dNAqCYdMPFxxm7ra5Za1535tENK7fmEFpg8qCvn1iezvoGqRS41OZeeNMt8sw5pjCMvTd7topLC5k8fbFzNg8g+mZ01mQtYDCksKy9fUj69O3TV8uTLuQC864gO4tuttpfcaYgAjLJHEj9FWVD9Z+wGsrXiNjS0aFg6+C0LNlz7KWfJ/WfezAqzGmRoRl6C9b5vzs1q12Xi9jSwb3TL+Hb3O+LVvWPrF9WcgPTB1Yo8MIGGNMqbAL/eJiZ+x8qPnQX71zNeOnjy8bPTHptCTu738/V3S4wg6+GmNcEXahv2GDMwRDmzbQtIbOoss+kM2fZv2J11e8TomWEBsdyz197uHu3ncH3U2mjTHhJexCvyb78/fl72Pi3Ik8u+hZ8ovyqRdRj9Hpo3lwwIN2BawxJij4FfoiMgR4FogEXlHViT7rU4DJQDNgD3CDqmZ7rW8ErAE+UtXbA1T7SamJ0D9adJRJiyfxyJxHyoYjvuZn1/DI4Ec4s8mZgXshY4w5RdWGvohEApOAi4BsYLGITFPVNV6bPQn8R1VfF5HBwARgpNf6vwHfBK7skxfIg7glWsKbK9/kwVkPlo2LPjB1II9f+Dg9W/U89RcwxpgA86el3wvYpKqbAURkCnAZTsu9VEfgD575WcBHpStEpAeQBHwBVDvsZ01SDUxLX1X56oev+OP0P7LiJ+eei+ecfg6PXfgYQ88cahdOGWOClj+h3wrI8nqcDZzrs80KYDhOF9AVQJyIJAJ7gaeAG4ALT7naU5SZ6dwMPSkJWpzkvaaXbl/KH6f/kRmZMwBIbpTM3wb9jZGdR9pgWsaYoBeoA7njgOdF5CacbpwcoBgYDXymqtnHa/2KyChgFECbNqd239Hj8W7ln2hjfPPezTww8wHeXv024AyNcF//+7ij1x12IZUxps7wJ/RzAO+7TSd7lpVR1e04LX1EJBa4UlX3iUhvoL+IjAZigWgROaSq4332fwl4CZw7Z53sm6lOaX/+iXTt5B7O5eFvHuafS/5JYUkh0ZHR3NHrDu7rfx9NGjapmUKNMaaG+BP6i4F2IpKGE/YjgOu9NxCRpsAeVS0B7sU5kwdV/ZXXNjcB6b6BX5tKW/r+HsR9duGzPDjrQQ4WHEQQRnYeyd8G/Y2U+JSaK9IYY2pQtaGvqkUicjvwJc4pm5NV9XsReQhYoqrTgIHABBFRnO6dMTVY80lRhaVLnXl/WvpTVk/hri/vAuDithfz2IWP0aV5EI7DbIwxJyBsboyekwPJyRAfD3v2HL9PP3NvJl3/1ZUDRw/w7JBnufPcOwNejzHGBJK/N0YPm9sr+XsQt7C4kOvev44DRw9wxdlXcEevO2qnQGOMqQVhE/r+HsT98+w/syhnEcmNknnl0lfsnHtjTEgJm9D35yDujM0zmDh3IhESwZvD37Szc4wxISfsQv9YLf3cw7mM/HAkivLg+Q9yfsr5tVecMcbUkrAI/V27ICsLTjvNuS+uL1Xl5o9vZsehHfRr048Hzn+g9os0xphaEBahX9qf37UrRFYxUsJz3z7Hpxs/Jb5BPG8Of9PuR2uMCVlhEfrH69pZ/uNy/vfr/wXglV++QpvGNTcMhDHGuC2sQt/3IO7hgsOMmDqCguICbu1xK1d2vLL2izPGmFoUVqHv29If+8VY1u9eT8dmHXn64qdrvzBjjKllIR/6+/fDpk0QHQ0dO5Yvf2f1O7y67FXqR9ZnypVTiImKca9IY4ypJSEf+iuce5zQuTNERTnzmXszGfXfUQA8ffHTdErq5FJ1xhhTu0I+9H378wuLC7n+g+s5cPQAl7W/jN+n/9694owxppaFTeiX9uf/NeOvLMxeSKu4Vrx66as2zIIxJqyEVejPypzFo3MeRRDeHP4miTGJ7hZnjDG1LKSvQsrLg7VrnQuyWrTdxXmv31A2zMKA1AFul2eMMbUupFv6q1ZBSQl06KiM+eoWth/cTp/WffjTgD+5XZoxxrgipEO/tGsnZsAkPtnwCY3rN+at4W/ZMAvGmLAV+qGftILvmo4D4OVfvmz3tzXGhLWQDv3FKw7DVSMo4ii/6/47rv7Z1W6XZIwxrgrZ0C8ogFUt74Zm62jfpAPPDHnG7ZKMMcZ1IRv6z379HiXdXkaK6/PO1W/bMAvGGIOfoS8iQ0RkvYhsEpHxVaxPEZEZIrJSRGaLSLLX8u9EZLmIfC8itwX6DVRl676t/Hnp7wDolvskXZp3qY2XNcaYoFdt6ItIJDAJGAp0BK4TkY4+mz0J/EdVOwMPARM8y3cAvVW1K3AuMF5EWgaq+KoUlRRx/QfXc0T3w/pfck3qmJp8OWOMqVP8aen3Ajap6mZVLQCmAJf5bNMRmOmZn1W6XlULVPWoZ3l9P1/vlDyU8RDzs+YTnd8SPp5Mjx42zIIxxpTyJ4RbAVlej7M9y7ytAIZ75q8A4kQkEUBEWovISs9zPKaq20+t5GObvWU2D3/zMIIgH70BeU0r3TjFGGPCWaBa3uOAASKyDBgA5ADFAKqa5en2ORO4UUSSfHcWkVEiskREluTm5p5UAbvzdnPDB84wC6M63MfRdYNISYFEG17HGGPK+BP6OUBrr8fJnmVlVHW7qg5X1W7A/Z5l+3y3AVYD/X1fQFVfUtV0VU1v1qzZCb4FxwuLXyDnYA69k3vTu+DPQOXbIxpjTLjzZzyCxUA7EUnDCfsRwPXeG4hIU2CPqpYA9wKTPcuTgd2qekREEoB+wN8DWH+Z+8+/n9joWK7ocAXP/c25W0pVN0I3xphwVm3oq2qRiNwOfAlEApNV9XsReQhYoqrTgIHABBFR4Bug9JSZDsBTnuUCPKmqq2rgfRAhEdzd+24Ali1zllnoG2NMRaKqbtdQQXp6ui5ZsuSk91eFhATn3rjbt0OLFgEszhhjgpSILFXV9Oq2C7krcjMzncBv3twC3xhjfIVc6PveE9cYY0y5kA196883xpjKQi707SCuMcYcW0iFviosXerMW+gbY0xlIRX627dDbq5z9k6K3SDLGGMqCanQ9z6IKzbOmjHGVBJSoW/9+cYYc3whFfp25o4xxhyfhb4xxoSRkAn93FzIyoLTToN27dyuxhhjglPIhH5pf37XrhARMu/KGGMCK2Ticd8+SEqyrh1jjDkef8bTrxOuucaZCgrcrsQYY4JXyLT0S0VHu12BMcYEr5ALfWOMMcdmoW+MMWHEQt8YY8KIhb4xxoQRC31jjAkjFvrGGBNGLPSNMSaMiKq6XUMFIpILbD2Fp2gK7ApQOTUh2OuD4K8x2OsDqzEQgr0+CK4aU1S1WXUbBV3onyoRWaKq6W7XcSzBXh8Ef43BXh9YjYEQ7PVB3ajRl3XvGGNMGLHQN8aYMBKKof+S2wVUI9jrg+CvMdjrA6sxEIK9PqgbNVYQcn36xhhjji0UW/rGGGOOIWRCX0SGiMh6EdkkIuPdrseXiLQWkVkiskZEvheRsW7XVBURiRSRZSLyX7drqYqIxIvIVBFZJyJrRaS32zV5E5G7Pf++q0XkbRFpEAQ1TRaRnSKy2mtZExH5WkQ2en4mBGGNT3j+nVeKyIciEh9sNXqt+x8RURFp6kZtJyIkQl9EIoFJwFCgI3CdiHR0t6pKioD/UdWOwHnAmCCsEWAssNbtIo7jWeALVT0b6EIQ1SoirYA7gXRVPQeIBEa4WxUArwFDfJaNB2aoajtghuexm16jco1fA+eoamdgA3BvbRfl4zUq14iItAZ+Dmyr7YJORkiEPtAL2KSqm1W1AJgCXOZyTRWo6g5V/c4zfxAnrFq5W1VFIpIM/AJ4xe1aqiIijYHzgVcBVLVAVfe5W1Ul9YCGIlIPiAG2u1wPqvoNsMdn8WXA657514HLa7UoH1XVqKpfqWqR5+FCILnWC6tYT1W/R4C/A/cAdeIAaaiEfisgy+txNkEWqN5EJBXoBixyt5JKnsH5z1vidiHHkAbkAv/2dEG9IiKnuV1UKVXNAZ7EafHtAPar6lfuVnVMSaq6wzP/I5DkZjF+uAX43O0ifInIZUCOqq5wuxZ/hUro1xkiEgu8D9ylqgfcrqeUiFwC7FTVpW7Xchz1gO7AP1W1G3AY97slynj6xS/D+XBqCZwmIje4W1X11DmFL2hbqSJyP0736Jtu1+JNRGKA+4A/uV3LiQiV0M8BWns9TvYsCyoiEoUT+G+q6gdu1+OjL3CpiGzB6R4bLCJvuFtSJdlAtqqWfkOaivMhECwuBDJVNVdVC4EPgD4u13QsP4lICwDPz50u11MlEbkJuAT4lQbf+eVtcT7gV3j+bpKB70SkuatVVSNUQn8x0E5E0kQkGufg2TSXa6pARASnL3qtqj7tdj2+VPVeVU1W1VSc399MVQ2qVqqq/ghkiUh7z6ILgDUuluRrG3CeiMR4/r0vIIgONPuYBtzomb8R+NjFWqokIkNwuhsvVdU8t+vxpaqrVPV0VU31/N1kA909/0+DVkiEvudgz+3Alzh/ZO+q6vfuVlVJX2AkTgt6uWca5nZRddAdwJsishLoCjzqcj1lPN9ApgLfAatw/r5cv2JTRN4GFgDtRSRbRH4DTAQuEpGNON9QJgZhjc8DccDXnr+XF4OwxjrHrsg1xpgwEhItfWOMMf6x0DfGmDBioW+MMWHEQt8YY8KIhb4xxoQRC31jjAkjFvrGGBNGLPSNMSaM/D+6R7bEl3QQyQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib notebook\n",
    "%matplotlib inline\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "# Data\n",
    "df = pd.DataFrame({'epochs': range(0,len(train_f)), \n",
    "                  'train_f': train_f, \n",
    "                   'dev_f': dev_f})\n",
    " \n",
    "# multiple line plot\n",
    "plt.plot('epochs', 'train_f', data=df, color='blue', linewidth=2)\n",
    "plt.plot('epochs', 'dev_f', data=df, color='green', linewidth=2)\n",
    "plt.legend()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we test our model on the test data, we have to run its `eval()` method. This will put the model in eval mode, and deactivate dropout layers and other functionality that is only useful in training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BiLSTMTagger(\n",
       "  (embeddings): Embedding(20002, 300)\n",
       "  (lstm): LSTM(300, 256, bidirectional=True)\n",
       "  (dropout_layer): Dropout(p=0.5)\n",
       "  (hidden2tag): Linear(in_features=512, out_features=11, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tagger = torch.load(OUTPUT_PATH)\n",
    "tagger.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we test the model. You'll notice its performance is significantly lower than that of the CRF we explored in an earlier notebook. Designing a competitive neural network takes considerably more effort than we put in here: you'll need to make the architecture of the network more complex, optimize its hyperparameters, and often also throw considerably more data at your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "      B-LOC       0.83      0.67      0.75       774\n",
      "      I-LOC       0.42      0.45      0.44        49\n",
      "     B-MISC       0.83      0.48      0.60      1187\n",
      "     I-MISC       0.58      0.25      0.35       410\n",
      "      B-ORG       0.72      0.56      0.63       882\n",
      "      I-ORG       0.74      0.57      0.64       551\n",
      "      B-PER       0.82      0.68      0.74      1098\n",
      "      I-PER       0.95      0.71      0.81       807\n",
      "\n",
      "avg / total       0.80      0.58      0.67      5758\n",
      "\n"
     ]
    }
   ],
   "source": [
    "labels = label_field.vocab.itos[3:]\n",
    "labels = sorted(labels, key=lambda x: x.split(\"-\")[-1])\n",
    "label_idxs = [label_field.vocab.stoi[l] for l in labels]\n",
    "\n",
    "test(tagger, test_iter, BATCH_SIZE, labels = label_idxs, target_names = labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "In this notebook we've trained a simple bidirectional LSTM for named entity recognition. Far from achieving state-of-the-art performance, our aim was to understand how neural networks can be implemented and trained in PyTorch. To improve our performance, one of the things that is typically done is to add an additional CRF layer to the neural network. This layer helps us optimize the complete label sequence, and not the labels individually. We leave that for future work. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
About this Algorithm

Named Entity Recognition with PyTorch

In this notebook we'll explore how we can use Deep Learning for sequence labelling tasks such as part-of-speech tagging or named entity recognition. We won't focus on getting state-of-the-art accuracy, but rather, implement a first neural network to get the main concepts across.

Data

For our experiments we'll reuse the NER data we've already used for our CRF experiments. The Dutch CoNLL-2002 data has four kinds of named entities (people, locations, organizations and miscellaneous entities) and comes split into a training, development and test set.

import nltk

train_sents = list(nltk.corpus.conll2002.iob_sents('ned.train'))
dev_sents = list(nltk.corpus.conll2002.iob_sents('ned.testa'))
test_sents = list(nltk.corpus.conll2002.iob_sents('ned.testb'))

train_sents[:3]
[[(&#x27;De&#x27;, &#x27;Art&#x27;, &#x27;O&#x27;),
  (&#x27;tekst&#x27;, &#x27;N&#x27;, &#x27;O&#x27;),
  (&#x27;van&#x27;, &#x27;Prep&#x27;, &#x27;O&#x27;),
  (&#x27;het&#x27;, &#x27;Art&#x27;, &#x27;O&#x27;),
  (&#x27;arrest&#x27;, &#x27;N&#x27;, &#x27;O&#x27;),
  (&#x27;is&#x27;, &#x27;V&#x27;, &#x27;O&#x27;),
  (&#x27;nog&#x27;, &#x27;Adv&#x27;, &#x27;O&#x27;),
  (&#x27;niet&#x27;, &#x27;Adv&#x27;, &#x27;O&#x27;),
  (&#x27;schriftelijk&#x27;, &#x27;Adj&#x27;, &#x27;O&#x27;),
  (&#x27;beschikbaar&#x27;, &#x27;Adj&#x27;, &#x27;O&#x27;),
  (&#x27;maar&#x27;, &#x27;Conj&#x27;, &#x27;O&#x27;),
  (&#x27;het&#x27;, &#x27;Art&#x27;, &#x27;O&#x27;),
  (&#x27;bericht&#x27;, &#x27;N&#x27;, &#x27;O&#x27;),
  (&#x27;werd&#x27;, &#x27;V&#x27;, &#x27;O&#x27;),
  (&#x27;alvast&#x27;, &#x27;Adv&#x27;, &#x27;O&#x27;),
  (&#x27;bekendgemaakt&#x27;, &#x27;V&#x27;, &#x27;O&#x27;),
  (&#x27;door&#x27;, &#x27;Prep&#x27;, &#x27;O&#x27;),
  (&#x27;een&#x27;, &#x27;Art&#x27;, &#x27;O&#x27;),
  (&#x27;communicatiebureau&#x27;, &#x27;N&#x27;, &#x27;O&#x27;),
  (&#x27;dat&#x27;, &#x27;Conj&#x27;, &#x27;O&#x27;),
  (&#x27;Floralux&#x27;, &#x27;N&#x27;, &#x27;B-ORG&#x27;),
  (&#x27;inhuurde&#x27;, &#x27;V&#x27;, &#x27;O&#x27;),
  (&#x27;.&#x27;, &#x27;Punc&#x27;, &#x27;O&#x27;)],
 [(&#x27;In&#x27;, &#x27;Prep&#x27;, &#x27;O&#x27;),
  (&quot;&#x27;81&quot;, &#x27;Num&#x27;, &#x27;O&#x27;),
  (&#x27;regulariseert&#x27;, &#x27;V&#x27;, &#x27;O&#x27;),
  (&#x27;de&#x27;, &#x27;Art&#x27;, &#x27;O&#x27;),
  (&#x27;toenmalige&#x27;, &#x27;Adj&#x27;, &#x27;O&#x27;),
  (&#x27;Vlaamse&#x27;, &#x27;Adj&#x27;, &#x27;B-MISC&#x27;),
  (&#x27;regering&#x27;, &#x27;N&#x27;, &#x27;O&#x27;),
  (&#x27;de&#x27;, &#x27;Art&#x27;, &#x27;O&#x27;),
  (&#x27;toestand&#x27;, &#x27;N&#x27;, &#x27;O&#x27;),
  (&#x27;met&#x27;, &#x27;Prep&#x27;, &#x27;O&#x27;),
  (&#x27;een&#x27;, &#x27;Art&#x27;, &#x27;O&#x27;),
  (&#x27;BPA&#x27;, &#x27;N&#x27;, &#x27;B-MISC&#x27;),
  (&#x27;dat&#x27;, &#x27;Pron&#x27;, &#x27;O&#x27;),
  (&#x27;het&#x27;, &#x27;Art&#x27;, &#x27;O&#x27;),
  (&#x27;bedrijf&#x27;, &#x27;N&#x27;, &#x27;O&#x27;),
  (&#x27;op&#x27;, &#x27;Prep&#x27;, &#x27;O&#x27;),
  (&#x27;eigen&#x27;, &#x27;Pron&#x27;, &#x27;O&#x27;),
  (&#x27;kosten&#x27;, &#x27;N&#x27;, &#x27;O&#x27;),
  (&#x27;heeft&#x27;, &#x27;V&#x27;, &#x27;O&#x27;),
  (&#x27;laten&#x27;, &#x27;V&#x27;, &#x27;O&#x27;),
  (&#x27;opstellen&#x27;, &#x27;V&#x27;, &#x27;O&#x27;),
  (&#x27;.&#x27;, &#x27;Punc&#x27;, &#x27;O&#x27;)],
 [(&#x27;publicatie&#x27;, &#x27;N&#x27;, &#x27;O&#x27;)]]

Next, we're going to preprocess the data. For this we use the torchtext Python library, which has a number of handy utilities for preprocessing natural language. We process our data to a Dataset that consists of Examples. Each of these examples has two fields: a text field and a label field. Both contain sequential information (the sequence of tokens, and the sequence of labels). We don't have to tokenize this information anymore, as the CONLL data has already been tokenized for us.

from torchtext.data import Example
from torchtext.data import Field, Dataset

text_field = Field(sequential=True, tokenize=lambda x:x, include_lengths=True) # Default behaviour is to tokenize by splitting
label_field = Field(sequential=True, tokenize=lambda x:x, is_target=True)

def read_data(sentences):
    examples = []
    fields = {'sentence_labels': ('labels', label_field),
              'sentence_tokens': ('text', text_field)}
    
    for sentence in sentences: 
        tokens = [t[0] for t in sentence]
        labels = [t[2] for t in sentence]
        
        e = Example.fromdict({"sentence_labels": labels, "sentence_tokens": tokens},
                             fields=fields)
        examples.append(e)
    
    return Dataset(examples, fields=[('labels', label_field), ('text', text_field)])

train_data = read_data(train_sents)
dev_data = read_data(dev_sents)
test_data = read_data(test_sents)

print(train_data.fields)
print(train_data[0].text)
print(train_data[0].labels)

print("Train:", len(train_data))
print("Dev:", len(dev_data))
print("Test:", len(test_data))
{&#x27;labels&#x27;: &amp;lt;torchtext.data.field.Field object at 0x7fc89bc02e10&amp;gt;, &#x27;text&#x27;: &amp;lt;torchtext.data.field.Field object at 0x7fc89bc02eb8&amp;gt;}
[&#x27;De&#x27;, &#x27;tekst&#x27;, &#x27;van&#x27;, &#x27;het&#x27;, &#x27;arrest&#x27;, &#x27;is&#x27;, &#x27;nog&#x27;, &#x27;niet&#x27;, &#x27;schriftelijk&#x27;, &#x27;beschikbaar&#x27;, &#x27;maar&#x27;, &#x27;het&#x27;, &#x27;bericht&#x27;, &#x27;werd&#x27;, &#x27;alvast&#x27;, &#x27;bekendgemaakt&#x27;, &#x27;door&#x27;, &#x27;een&#x27;, &#x27;communicatiebureau&#x27;, &#x27;dat&#x27;, &#x27;Floralux&#x27;, &#x27;inhuurde&#x27;, &#x27;.&#x27;]
[&#x27;O&#x27;, &#x27;O&#x27;, &#x27;O&#x27;, &#x27;O&#x27;, &#x27;O&#x27;, &#x27;O&#x27;, &#x27;O&#x27;, &#x27;O&#x27;, &#x27;O&#x27;, &#x27;O&#x27;, &#x27;O&#x27;, &#x27;O&#x27;, &#x27;O&#x27;, &#x27;O&#x27;, &#x27;O&#x27;, &#x27;O&#x27;, &#x27;O&#x27;, &#x27;O&#x27;, &#x27;O&#x27;, &#x27;O&#x27;, &#x27;B-ORG&#x27;, &#x27;O&#x27;, &#x27;O&#x27;]
Train: 15806
Dev: 2895
Test: 5195

Next, we build a vocabulary for both fields. This vocabulary allows us to map every word and label to their index. One index is kept for unknown words, another one for padding.

VOCAB_SIZE = 20000

text_field.build_vocab(train_data, max_size=VOCAB_SIZE)
label_field.build_vocab(train_data)

Training

If we're on a machine with a CUDA-enabled GPU, we'd like to use this GPU for training and testing. If not, we'll just use the CPU. The check below allows us to write code that works on both CPU and GPU.

import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)
cuda

Another convenient class in Torchtext is the BucketIterator. This iterator creates batches of similar-length examples in the data. It also takes care of mapping the words and labels to the correct indices in their vocabularies, and pads the sentences so that they all have the same length. The Bucketiterator creates batches of similar-length examples to minimize the amount of padding.

from torchtext.data import BucketIterator

BATCH_SIZE = 32
train_iter = BucketIterator(dataset=train_data, batch_size=BATCH_SIZE, shuffle=True, 
                            sort_key=lambda x: len(x.text), sort_within_batch=True)
dev_iter = BucketIterator(dataset=dev_data, batch_size=BATCH_SIZE, 
                          sort_key=lambda x: len(x.text), sort_within_batch=True)
test_iter = BucketIterator(dataset=test_data, batch_size=BATCH_SIZE, 
                           sort_key=lambda x: len(x.text), sort_within_batch=True)

Pre-trained embeddings

Pre-trained embeddings embeddings are generally an easy way of improving the performance of your model, particularly if you have little training data. Thanks to these embeddings, you'll be able to make use of knowledge about the meaning and use of the words in your dataset that was learned from another, typically larger data set. In this way, your model will be able to generalize better between semantically related words.

In this example, we make use of the popular FastText embeddings. These are high-quality pre-trained word embeddings that are available for a wide variety of languages. After downloading the vec file with the embeddings, we use them to initialize our embedding matrix. We do this by creating a matrix filled with zeros whose number of rows equals the number of words in our vocabulary and whose number of columns equals the number of dimensions in the FastText vectors (300). We have to take care that we insert the FastText embedding for a particular word in the correct row. This is the row whose index corresponds to the index of the word in the vocabulary.

import random
import os
import numpy as np

EMBEDDING_PATH = os.path.join(os.path.expanduser("~"), "data/embeddings/fasttext/cc.nl.300.vec")


def load_embeddings(path):
    """ Load the FastText embeddings from the embedding file. """
    print("Loading pre-trained embeddings")
    
    embeddings = {}
    with open(path) as i:
        for line in i:
            if len(line) &gt; 2: 
                line = line.strip().split()
                word = line[0]
                embedding = np.array(line[1:])
                embeddings[word] = embedding
    
    return embeddings
    

def initialize_embeddings(embeddings, vocabulary):
    """ Use the pre-trained embeddings to initialize an embedding matrix. """
    print("Initializing embedding matrix")
    embedding_size = len(embeddings["."])
    embedding_matrix = np.zeros((len(vocabulary), embedding_size), dtype=np.float32)
                                
    for idx, word in enumerate(vocabulary.itos): 
        if word in embeddings:
            embedding_matrix[idx,:] = embeddings[word]
            
    return embedding_matrix

embeddings = load_embeddings(EMBEDDING_PATH)
embedding_matrix = initialize_embeddings(embeddings, text_field.vocab)
embedding_matrix = torch.from_numpy(embedding_matrix).to(device)
Loading pre-trained embeddings
Initializing embedding matrix

Model

Next, we create our BiLSTM model. It consists of four layers:

  • An embedding layer that maps one-hot word vectors to dense word embeddings. These embeddings are either pretrained or trained from scratch.
  • A bidirectional LSTM layer that reads the text both front to back and back to front. For each word, this LSTM produces two output vectors of dimensionality hidden_dim, which are concatenated to a vector of 2*hidden_dim.
  • A dropout layer that helps us avoid overfitting by dropping a certain percentage of the items in the LSTM output.
  • A dense layer that projects the LSTM output to an output vector with a dimensionality equal to the number of labels.

We initialize these layers in the __init__ method, and put them together in the forward method.

import torch.nn as nn
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

class BiLSTMTagger(nn.Module):

    def __init__(self, embedding_dim, hidden_dim, vocab_size, output_size, embeddings=None):
        super(BiLSTMTagger, self).__init__()
        
        # 1. Embedding Layer
        if embeddings is None:
            self.embeddings = nn.Embedding(vocab_size, embedding_dim)
        else:
            self.embeddings = nn.Embedding.from_pretrained(embeddings)
        
        # 2. LSTM Layer
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, bidirectional=True, num_layers=1)
        
        # 3. Optional dropout layer
        self.dropout_layer = nn.Dropout(p=0.5)

        # 4. Dense Layer
        self.hidden2tag = nn.Linear(2*hidden_dim, output_size)
        
    def forward(self, batch_text, batch_lengths):

        embeddings = self.embeddings(batch_text)
        
        packed_seqs = pack_padded_sequence(embeddings, batch_lengths)
        lstm_output, _ = self.lstm(packed_seqs)
        lstm_output, _ = pad_packed_sequence(lstm_output)
        lstm_output = self.dropout_layer(lstm_output)
        
        logits = self.hidden2tag(lstm_output)
        return logits

Training

Then we need to train this model. This involves taking a number of decisions:

  • We pick a loss function (or criterion) to quantify how far away the model predictions are from the correct output. For multiclass tasks such as Named Entity Recognition, a standard loss function is the Cross-Entropy Loss, which here measures the difference between two multinomial probability distributions. PyTorch's CrossEntropyLoss does this by first applying a softmax to the last layer of the model to transform the output scores to probabilities, and then computing the cross-entropy between the predicted and correct probability distributions. The ignore_index parameter allows us to mask the padding items in the training data, so that these do not contribute to the loss. We also remove these masked items from the output afterwards, so they are not taken into account when we evaluate the model output.
  • Next, we need to choose an optimizer. For many NLP problems, the Adam optimizer is a good first choice. Adam is a variation of Stochastic Gradient Descent with several advantages: it maintains per-parameter learning rates and adapts these learning rates based on how quickly the values of a specific parameter are changing (or, how large its average gradient is).

Then the actual training starts. This happens in several epochs. During each epoch, we show all of the training data to the network, in the batches produced by the BucketIterators we created above. Before we show the model a new batch, we set the gradients of the model to zero to avoid accumulating gradients across batches. Then we let the model make its predictions for the batch. We do this by taking the output, and finding out what label received the highest score, using the torch.max method. We then compute the loss with respect to the correct labels. loss.backward() then computes the gradients for all model parameters; optimizer.step() performs an optimization step.

When we have shown all the training data in an epoch, we perform the precision, recall and F-score on the training data and development data. Note that we compute the loss for the development data, but we do not optimize the model with it. Whenever the F-score on the development data is better than before, we save the model. If the F-score is lower than the minimum F-score we've seen in the past few epochs (we call this number the patience), we stop training.

import torch.optim as optim
from tqdm import tqdm_notebook as tqdm
from sklearn.metrics import precision_recall_fscore_support, classification_report


def remove_predictions_for_masked_items(predicted_labels, correct_labels): 

    predicted_labels_without_mask = []
    correct_labels_without_mask = []
        
    for p, c in zip(predicted_labels, correct_labels):
        if c &gt; 1:
            predicted_labels_without_mask.append(p)
            correct_labels_without_mask.append(c)
            
    return predicted_labels_without_mask, correct_labels_without_mask


def train(model, train_iter, dev_iter, batch_size, max_epochs, num_batches, patience, output_path):
    criterion = nn.CrossEntropyLoss(ignore_index=1)  # we mask the &lt;pad&gt; labels
    optimizer = optim.Adam(model.parameters())

    train_f_score_history = []
    dev_f_score_history = []
    no_improvement = 0
    for epoch in range(max_epochs):

        total_loss = 0
        predictions, correct = [], []
        for batch in tqdm(train_iter, total=num_batches, desc=f"Epoch {epoch}"):
            optimizer.zero_grad()
            
            text_length, cur_batch_size = batch.text[0].shape
            
            pred = model(batch.text[0].to(device), batch.text[1].to(device)).view(cur_batch_size*text_length, NUM_CLASSES)
            gold = batch.labels.to(device).view(cur_batch_size*text_length)
            
            loss = criterion(pred, gold)
            
            total_loss += loss.item()

            loss.backward()
            optimizer.step()

            _, pred_indices = torch.max(pred, 1)
            
            predicted_labels = list(pred_indices.cpu().numpy())
            correct_labels = list(batch.labels.view(cur_batch_size*text_length).numpy())
            
            predicted_labels, correct_labels = remove_predictions_for_masked_items(predicted_labels, 
                                                                                   correct_labels)
            
            predictions += predicted_labels
            correct += correct_labels

        train_scores = precision_recall_fscore_support(correct, predictions, average="micro")
        train_f_score_history.append(train_scores[2])
            
        print("Total training loss:", total_loss)
        print("Training performance:", train_scores)
        
        total_loss = 0
        predictions, correct = [], []
        for batch in dev_iter:

            text_length, cur_batch_size = batch.text[0].shape

            pred = model(batch.text[0].to(device), batch.text[1].to(device)).view(cur_batch_size * text_length, NUM_CLASSES)
            gold = batch.labels.to(device).view(cur_batch_size * text_length)
            loss = criterion(pred, gold)
            total_loss += loss.item()

            _, pred_indices = torch.max(pred, 1)
            predicted_labels = list(pred_indices.cpu().numpy())
            correct_labels = list(batch.labels.view(cur_batch_size*text_length).numpy())
            
            predicted_labels, correct_labels = remove_predictions_for_masked_items(predicted_labels, 
                                                                                   correct_labels)
            
            predictions += predicted_labels
            correct += correct_labels

        dev_scores = precision_recall_fscore_support(correct, predictions, average="micro")
            
        print("Total development loss:", total_loss)
        print("Development performance:", dev_scores)
        
        dev_f = dev_scores[2]
        if len(dev_f_score_history) &gt; patience and dev_f &lt; max(dev_f_score_history):
            no_improvement += 1

        elif len(dev_f_score_history) == 0 or dev_f &gt; max(dev_f_score_history):
            print("Saving model.")
            torch.save(model, output_path)
            no_improvement = 0
            
        if no_improvement &gt; patience:
            print("Development F-score does not improve anymore. Stop training.")
            dev_f_score_history.append(dev_f)
            break
            
        dev_f_score_history.append(dev_f)
        
    return train_f_score_history, dev_f_score_history

When we test the model, we basically take the same steps as in the evaluation on the development data above: we get the predictions, remove the masked items and print a classification report.

def test(model, test_iter, batch_size, labels, target_names): 
    
    total_loss = 0
    predictions, correct = [], []
    for batch in test_iter:

        text_length, cur_batch_size = batch.text[0].shape

        pred = model(batch.text[0].to(device), batch.text[1].to(device)).view(cur_batch_size * text_length, NUM_CLASSES)
        gold = batch.labels.to(device).view(cur_batch_size * text_length)

        _, pred_indices = torch.max(pred, 1)
        predicted_labels = list(pred_indices.cpu().numpy())
        correct_labels = list(batch.labels.view(cur_batch_size*text_length).numpy())

        predicted_labels, correct_labels = remove_predictions_for_masked_items(predicted_labels, 
                                                                               correct_labels)

        predictions += predicted_labels
        correct += correct_labels
    
    print(classification_report(correct, predictions, labels=labels, target_names=target_names))

Now we can start the actual training. We set the embedding dimension to 300 (the dimensionality of the FastText embeddings), and pick a hidden dimensionality for each component of the BiLSTM (which will therefore output 512-dimensional vectors). The number of classes (the length of the vocabulary of the label field) will become the dimensionality of the output layer. Finally, we also compute the number of batches in an epoch, so that we can show a progress bar.

import math

EMBEDDING_DIM = 300
HIDDEN_DIM = 256
NUM_CLASSES = len(label_field.vocab)
MAX_EPOCHS = 50
PATIENCE = 3
OUTPUT_PATH = "/tmp/bilstmtagger"
num_batches = math.ceil(len(train_data) / BATCH_SIZE)

tagger = BiLSTMTagger(EMBEDDING_DIM, HIDDEN_DIM, VOCAB_SIZE+2, NUM_CLASSES, embeddings=embedding_matrix)  

train_f, dev_f = train(tagger.to(device), train_iter, dev_iter, BATCH_SIZE, MAX_EPOCHS, 
                       num_batches, PATIENCE, OUTPUT_PATH)
HBox(children=(IntProgress(value=0, description=&#x27;Epoch 0&#x27;, max=494, style=ProgressStyle(description_width=&#x27;ini
Total training loss: 222.35497660934925
Training performance: (0.9244241132231894, 0.9244241132231894, 0.9244241132231894, None)
Total development loss: 24.117380052804947
Development performance: (0.9296043728606681, 0.9296043728606681, 0.9296043728606681, None)
Saving model.
/opt/anaconda3/lib/python3.7/site-packages/torch/serialization.py:251: UserWarning: Couldn&#x27;t retrieve source code for container of type BiLSTMTagger. It won&#x27;t be checked for correctness upon loading.
  &quot;type &quot; + obj.__name__ + &quot;. It won&#x27;t be checked &quot;
HBox(children=(IntProgress(value=0, description=&#x27;Epoch 1&#x27;, max=494, style=ProgressStyle(description_width=&#x27;ini
Total training loss: 71.356663974002
Training performance: (0.9591253627050393, 0.9591253627050393, 0.9591253627050393, None)
Total development loss: 18.751499708741903
Development performance: (0.9388913949107119, 0.9388913949107119, 0.9388913949107119, None)
Saving model.
HBox(children=(IntProgress(value=0, description=&#x27;Epoch 2&#x27;, max=494, style=ProgressStyle(description_width=&#x27;ini
Total training loss: 50.72972834017128
Training performance: (0.9680819565346124, 0.9680819565346124, 0.9680819565346124, None)
Total development loss: 17.262520626187325
Development performance: (0.9425531350333006, 0.9425531350333006, 0.9425531350333006, None)
Saving model.
HBox(children=(IntProgress(value=0, description=&#x27;Epoch 3&#x27;, max=494, style=ProgressStyle(description_width=&#x27;ini
Total training loss: 41.36486494448036
Training performance: (0.9728291980024082, 0.9728291980024082, 0.9728291980024082, None)
Total development loss: 16.504801526665688
Development performance: (0.9443309363971661, 0.9443309363971661, 0.9443309363971661, None)
Saving model.
HBox(children=(IntProgress(value=0, description=&#x27;Epoch 4&#x27;, max=494, style=ProgressStyle(description_width=&#x27;ini
Total training loss: 35.585738136898726
Training performance: (0.9755877302066679, 0.9755877302066679, 0.9755877302066679, None)
Total development loss: 16.711221787147224
Development performance: (0.9458964629713164, 0.9458964629713164, 0.9458964629713164, None)
Saving model.
HBox(children=(IntProgress(value=0, description=&#x27;Epoch 5&#x27;, max=494, style=ProgressStyle(description_width=&#x27;ini
Total training loss: 30.99800857482478
Training performance: (0.9784597619470599, 0.9784597619470599, 0.9784597619470599, None)
Total development loss: 16.387646575458348
Development performance: (0.9469578369198928, 0.9469578369198928, 0.9469578369198928, None)
Saving model.
HBox(children=(IntProgress(value=0, description=&#x27;Epoch 6&#x27;, max=494, style=ProgressStyle(description_width=&#x27;ini
Total training loss: 27.11286850180477
Training performance: (0.9801326464144016, 0.9801326464144016, 0.9801326464144016, None)
Total development loss: 17.557277165353298
Development performance: (0.9460822034123172, 0.9460822034123172, 0.9460822034123172, None)
HBox(children=(IntProgress(value=0, description=&#x27;Epoch 7&#x27;, max=494, style=ProgressStyle(description_width=&#x27;ini
Total training loss: 24.422367892228067
Training performance: (0.9820917471032945, 0.9820917471032945, 0.9820917471032945, None)
Total development loss: 16.09378209663555
Development performance: (0.9491867222119033, 0.9491867222119033, 0.9491867222119033, None)
Saving model.
HBox(children=(IntProgress(value=0, description=&#x27;Epoch 8&#x27;, max=494, style=ProgressStyle(description_width=&#x27;ini
Total training loss: 22.412145966896787
Training performance: (0.9829355914806261, 0.9829355914806261, 0.9829355914806261, None)
Total development loss: 19.413369961082935
Development performance: (0.9440390585613077, 0.9440390585613077, 0.9440390585613077, None)
HBox(children=(IntProgress(value=0, description=&#x27;Epoch 9&#x27;, max=494, style=ProgressStyle(description_width=&#x27;ini
Total training loss: 20.09223960270174
Training performance: (0.9845344545113598, 0.9845344545113598, 0.9845344545113598, None)
Total development loss: 15.523855119943619
Development performance: (0.948072279565898, 0.948072279565898, 0.948072279565898, None)
HBox(children=(IntProgress(value=0, description=&#x27;Epoch 10&#x27;, max=494, style=ProgressStyle(description_width=&#x27;in
Total training loss: 17.925453061121516
Training performance: (0.9860395570557233, 0.9860395570557233, 0.9860395570557233, None)
Total development loss: 16.789274506270885
Development performance: (0.9485233634940431, 0.9485233634940431, 0.9485233634940432, None)
HBox(children=(IntProgress(value=0, description=&#x27;Epoch 11&#x27;, max=494, style=ProgressStyle(description_width=&#x27;in
Total training loss: 16.409397734270897
Training performance: (0.9867205542725174, 0.9867205542725174, 0.9867205542725174, None)
Total development loss: 14.920704647898674
Development performance: (0.950195027463051, 0.950195027463051, 0.950195027463051, None)
Saving model.
HBox(children=(IntProgress(value=0, description=&#x27;Epoch 12&#x27;, max=494, style=ProgressStyle(description_width=&#x27;in
Total training loss: 14.990085303143132
Training performance: (0.9879493101202108, 0.9879493101202108, 0.9879493101202108, None)
Total development loss: 17.077377565205097
Development performance: (0.949876615278478, 0.949876615278478, 0.949876615278478, None)
HBox(children=(IntProgress(value=0, description=&#x27;Epoch 13&#x27;, max=494, style=ProgressStyle(description_width=&#x27;in
Total training loss: 13.492025993036805
Training performance: (0.9886845897238506, 0.9886845897238506, 0.9886845897238506, None)
Total development loss: 19.90118957636878
Development performance: (0.9481253482633268, 0.9481253482633268, 0.9481253482633268, None)
HBox(children=(IntProgress(value=0, description=&#x27;Epoch 14&#x27;, max=494, style=ProgressStyle(description_width=&#x27;in
Total training loss: 12.34741208187188
Training performance: (0.9893507826533231, 0.9893507826533231, 0.9893507826533231, None)
Total development loss: 18.731272239238024
Development performance: (0.9497970122323347, 0.9497970122323347, 0.9497970122323347, None)
HBox(children=(IntProgress(value=0, description=&#x27;Epoch 15&#x27;, max=494, style=ProgressStyle(description_width=&#x27;in
Total training loss: 10.979540771790198
Training performance: (0.9901650184560116, 0.9901650184560116, 0.9901650184560116, None)
Total development loss: 17.838901833223645
Development performance: (0.949558203093905, 0.949558203093905, 0.949558203093905, None)
Development F-score does not improve anymore. Stop training.

Let's now plot the evolution of the F-score on our training and development set, to visually evaluate if training went well. If it did, the training F-score should first increase suddenly, then more gradually. The development F-score will increase during the first few epochs, but at some point it will start to decrease again. That's when the model starts overfitting. This is where we abandon training, and why we only save the model when we have reached an optimal F-score on the development data.

%matplotlib notebook
%matplotlib inline

import matplotlib.pyplot as plt
import pandas as pd

# Data
df = pd.DataFrame({'epochs': range(0,len(train_f)), 
                  'train_f': train_f, 
                   'dev_f': dev_f})
 
# multiple line plot
plt.plot('epochs', 'train_f', data=df, color='blue', linewidth=2)
plt.plot('epochs', 'dev_f', data=df, color='green', linewidth=2)
plt.legend()
plt.show()

Before we test our model on the test data, we have to run its eval() method. This will put the model in eval mode, and deactivate dropout layers and other functionality that is only useful in training.

tagger = torch.load(OUTPUT_PATH)
tagger.eval()
BiLSTMTagger(
  (embeddings): Embedding(20002, 300)
  (lstm): LSTM(300, 256, bidirectional=True)
  (dropout_layer): Dropout(p=0.5)
  (hidden2tag): Linear(in_features=512, out_features=11, bias=True)
)

Finally, we test the model. You'll notice its performance is significantly lower than that of the CRF we explored in an earlier notebook. Designing a competitive neural network takes considerably more effort than we put in here: you'll need to make the architecture of the network more complex, optimize its hyperparameters, and often also throw considerably more data at your model.

labels = label_field.vocab.itos[3:]
labels = sorted(labels, key=lambda x: x.split("-")[-1])
label_idxs = [label_field.vocab.stoi[l] for l in labels]

test(tagger, test_iter, BATCH_SIZE, labels = label_idxs, target_names = labels)
             precision    recall  f1-score   support

      B-LOC       0.83      0.67      0.75       774
      I-LOC       0.42      0.45      0.44        49
     B-MISC       0.83      0.48      0.60      1187
     I-MISC       0.58      0.25      0.35       410
      B-ORG       0.72      0.56      0.63       882
      I-ORG       0.74      0.57      0.64       551
      B-PER       0.82      0.68      0.74      1098
      I-PER       0.95      0.71      0.81       807

avg / total       0.80      0.58      0.67      5758

Conclusion

In this notebook we've trained a simple bidirectional LSTM for named entity recognition. Far from achieving state-of-the-art performance, our aim was to understand how neural networks can be implemented and trained in PyTorch. To improve our performance, one of the things that is typically done is to add an additional CRF layer to the neural network. This layer helps us optimize the complete label sequence, and not the labels individually. We leave that for future work.