Code

[WIP] Implements Roberta Model#679

Code

Draft

JulienDarve wants to merge 32 commits intostanford-crfm:mainstanford-crfm/levanter:mainfrom JulienDarve:roberta-modelJulienDarve/levanter:roberta-modelCopy head branch name to clipboard

+2,779-23

Commits

Commits on Jul 30, 2024

• [WIP] Implements Roberta Model

  

  JulienDarve

  committed
  d6284ca

  Copy full SHA for d6284ca

• Implements dynamic masking objective

  prady-saligram

  committed
  8f7402e

  Copy full SHA for 8f7402e

• Implements dynamic masked dataset

  prady-saligram

  committed
  670b053

  Copy full SHA for 670b053

• Reintroduced accidentally deleted CausalLMDataset class

  prady-saligram

  committed
  42f5404

  Copy full SHA for 42f5404

Commits on Aug 1, 2024

• Everything works except stuck on the final method,

  Show description for 9ad06af
  

  JulienDarve

  committed
  9ad06af

  Copy full SHA for 9ad06af

Commits on Aug 5, 2024

• [WIP] Re-implements MLM training objective

  prady-saligram

  committed
  53fd8d2

  Copy full SHA for 53fd8d2

Commits on Aug 6, 2024

• Adds error handling and reverts LmExample class to original

  prady-saligram

  committed
  dcd45b2

  Copy full SHA for dcd45b2

Commits on Aug 13, 2024

• Testing Modifications

  

  JulienDarve

  committed
  6f21e0d

  Copy full SHA for 6f21e0d

Commits on Aug 26, 2024

• Merge branch 'stanford-crfm:main' into roberta-model

  prady-saligram

  authored
  730d847

  Copy full SHA for 730d847

• Sets RobertaConfig as model architecture and creates default config file

  prady-saligram

  committed
  027b176

  Copy full SHA for 027b176

Commits on Sep 1, 2024

• Adds compute_loss to roberta and changes positional ids to begin from 0

  prady-saligram

  committed
  399e08c

  Copy full SHA for 399e08c

Commits on Sep 4, 2024

• Investingating precision loss over time within the model using output_hidden_states implementation in jax model

  

  JulienDarve

  committed
  cd4118c

  Copy full SHA for cd4118c

• Merge branch 'roberta-model' of https://github.com/JulienDarve/levanter into roberta-model

  

  JulienDarve

  committed
  96522f1

  Copy full SHA for 96522f1

Commits on Sep 10, 2024

• Model can now successfully import weights from huggingface + made attention mask more robust

  

  JulienDarve

  committed
  8a732e5

  Copy full SHA for 8a732e5

• Merge branch 'roberta-training' into roberta-model-copy-2

  

  JulienDarve

  committed
  5f3d8a2

  Copy full SHA for 5f3d8a2

Commits on Sep 12, 2024

• trial

  

  JulienDarve

  committed
  6c105f5

  Copy full SHA for 6c105f5

• update 1

  

  JulienDarve

  committed
  ab85079

  Copy full SHA for ab85079

• update 2

  

  JulienDarve

  committed
  5b97400

  Copy full SHA for 5b97400

• update 3

  

  JulienDarve

  committed
  bd7d411

  Copy full SHA for bd7d411

• update

  

  JulienDarve

  committed
  b5d8e14

  Copy full SHA for b5d8e14

• update

  

  JulienDarve

  committed
  8717c3f

  Copy full SHA for 8717c3f

• update

  

  JulienDarve

  committed
  10c130c

  Copy full SHA for 10c130c

• update

  

  JulienDarve

  committed
  834d88d

  Copy full SHA for 834d88d

• update

  

  JulienDarve

  committed
  47fe23b

  Copy full SHA for 47fe23b

• update

  

  JulienDarve

  committed
  fb5c55c

  Copy full SHA for fb5c55c

• update

  

  JulienDarve

  committed
  8594e79

  Copy full SHA for 8594e79

• update

  

  JulienDarve

  committed
  3ae80d7

  Copy full SHA for 3ae80d7

• update

  

  JulienDarve

  committed
  de93fc9

  Copy full SHA for de93fc9

• update

  

  JulienDarve

  committed
  896af7d

  Copy full SHA for 896af7d

• update

  

  JulienDarve

  committed
  0be9a83

  Copy full SHA for 0be9a83

• update

  

  JulienDarve

  committed
  0c94a47

  Copy full SHA for 0c94a47

Commits on Sep 13, 2024

• Training works!

  JulienDarve

  committed
  7ae681d

  Copy full SHA for 7ae681d