CS224N-Spring-2024

[course]

Note that this repo is still a work in progress...

In this repository, I share my solutions and additional resources for CS224N in the spring of 2024. While there are already some excellent repos offering solutions for various semesters, I found that none fully cover the latest version of the course. Moreover, many of these solutions seem to be copied from past versions, which may not reflect the latest updates. Hence, I decide to document my own learning process and provide all the solutions. If you spot any mistakes, feel free to file an issue!

Offline Schedule

Just follow your own path.

Wed Sep 4

• Watch Lecture 1 - Intro & Word Vectors and Lecture 2 - Neural Classifiers.

Thu Sep 5

• Watch Lecture 3 - Backprop and Neural Networks.
• Add assignment 1 part1 draft.

Fri Sep 6

• Finish assignment 1.
• Watch Lecture 4 - Syntactic Structure and Dependency Parsing.

Wed Sep 11

• Watch Lecture 5 - Recurrent Neural networks (RNNs) and Lecture 6 - Simple and LSTM RNNs.
• Read cs224n-2019-notes04-dependencyparsing.
• Add assignment 2 coding part and some written.
  • Train a neural dependency parser, which achieves UAS 88.47 on the dev set and 89.18 on the test set.

Thu Sep 12

• Watch Lecture 7 - Translation, Seq2Seq, Attention, Lecture 8 - Self-Attention and Transformers, and Lecture 9 - Pretraining.
• (Optional) Read An Analysis on the Learning Rules of the Skip-Gram Model to section III.
• Add assignment 2 written to Q1 (c).

Fri Sep 13

• Read Dropout: A Simple Way to Prevent Neural Networks from Overfitting.
  • Only focus on the motivation and effect of dropout.
• Finish assignment 2.

Sun Sep 15

• Watch Lecture 10 - Prompting, Reinforcement Learning from Human Feedback.

Mon Sep 16

• Do assignment 3 coding part.
  • Temporarily skip code snippets related to beam search.
  • Train a NMT model, which achieves BLEU 22.33 on the test set.

Tue Sep 17

• Finish assignment 3.

Wed Sep 18

• Read chapters 1 and 2 of Natural Language Processing with Transformers.

Thu Sep 19

• Read chapters 3 to 5 of Natural Language Processing with Transformers.

Fri Sep 20

I switch back to the course schedule in the spring of 2023, because there's no lectures updated for 2024. However, all solutions still follow the 2024 version.

• Watch Lecture11 - Natural Language Generation.
  • I think it's more consistent to take this lecture before Lecture 10 - Prompting, Reinforcement Learning from Human Feedback.
• Read chapter 6 of Natural Language Processing with Transformers.

Sat Sep 21

• Add assignment 4 written to Q1 (c) i and Q2.

Wed Sep 25

• Recently play around with a RAG application.
  • As my custom final project?

Fri Sep 28

• Read chapter 7 of Natural Language Processing with Transformers.
  • Haven't run the corresponding notebook due to the problem of launching Elasticsearch locally.
• Watch Lecture12 - Question Answering.
• Read RoFormer: Enhanced Transformer with Rotary Position Embedding.
  • Have to reread it to grasp the main points.

Sat Sep 29

• Add assignment 4 written and coding parts.
  • Pre-train and fine-tune a simple GPT which achieves accuracy of 29.00 of the dev set in Q3 (g).
  • I will finish Q1 (c) to (e) after reviewing probability theory.

Tue Oct 8

• Watch Lecture 16 - Multimodal Deep Learning, Lecture 17 - Coreference Resolution, Lecture 18 - Model Analysis and Explanation, Lecture 19 - Model Interpretability & Editing.
  • I will dive deeper into lec 16, which looks fascinating. Also, I can't quite get lec 19.
• Add the default final project minBERT with section 3 implemented.

Wed Oct 9

• Finish the first half of minBERT.
  • Fine-tuning the last linear layer for SST: Dev acc 0.931.
  • Fine-tuning the last linear layer for CFIMDB: Dev acc 0.771.
  • Fine-tuning the full model for SST: Dev acc 0.517.
  • Fine-tuning the full model for CFIMDB: Dev acc 0.959.
• Read BERT: Pretraining of Deep Bidirectional Transformers for Language Understanding.