Code

Add loncxt tasks#2629

Code

Draft

baberabb wants to merge 62 commits intomainEleutherAI/lm-evaluation-harness:mainfrom longcxtEleutherAI/lm-evaluation-harness:longcxtCopy head branch name to clipboard

+2,943-38

Commits

Commits on Dec 18, 2024

• add hotpotqa_e

  baberabb

  committed
  a860161

  Copy full SHA for a860161

• add longbench

  baberabb

  committed
  76e517d

  Copy full SHA for 76e517d

• fix metrics

  baberabb

  committed
  cf8a257

  Copy full SHA for cf8a257

• fix do_sample

  baberabb

  committed
  adbfcce

  Copy full SHA for adbfcce

• fix do_sample

  baberabb

  committed
  d684b9e

  Copy full SHA for d684b9e

Commits on Dec 19, 2024

• add niah

  baberabb

  committed
  c3aaad5

  Copy full SHA for c3aaad5

• add ruler reqs

  baberabb

  committed
  07429c8

  Copy full SHA for 07429c8

• handle nltk punkt_tab

  baberabb

  committed
  0351bb6

  Copy full SHA for 0351bb6

• fix

  baberabb

  committed
  19d5460

  Copy full SHA for 19d5460

• fix aggregation

  baberabb

  committed
  7399dae

  Copy full SHA for 7399dae

• fix tokenizer

  baberabb

  committed
  bd34e85

  Copy full SHA for bd34e85

• nit

  baberabb

  committed
  1f27bd8

  Copy full SHA for 1f27bd8

• nit

  baberabb

  committed
  6c7c413

  Copy full SHA for 6c7c413

• fix metric

  baberabb

  committed
  5c1f194

  Copy full SHA for 5c1f194

• fix template

  baberabb

  committed
  e4a8386

  Copy full SHA for e4a8386

• nit

  baberabb

  committed
  adcacfb

  Copy full SHA for adcacfb

• pass tokenizer to task config

  baberabb

  committed
  105ea3e

  Copy full SHA for 105ea3e

• Merge branch 'main' into longcxt

  baberabb

  committed
  556a79b

  Copy full SHA for 556a79b

Commits on Dec 20, 2024

• add more ruler tasks

  baberabb

  committed
  f604d6c

  Copy full SHA for f604d6c

• nit

  baberabb

  committed
  134b486

  Copy full SHA for 134b486

• add cache

  baberabb

  committed
  c179f7a

  Copy full SHA for c179f7a

• nit

  baberabb

  committed
  bf90aa7

  Copy full SHA for bf90aa7

• nit

  baberabb

  committed
  28abec9

  Copy full SHA for 28abec9

• nit

  baberabb

  committed
  ee4252f

  Copy full SHA for ee4252f

Commits on Jan 17, 2025

• nit

  baberabb

  committed
  37f17c3

  Copy full SHA for 37f17c3

• nit

  baberabb

  committed
  898e0ac

  Copy full SHA for 898e0ac

• nit

  baberabb

  committed
  5c77509

  Copy full SHA for 5c77509

• fixup

  baberabb

  committed
  a61b3ee

  Copy full SHA for a61b3ee

• better api

  baberabb

  committed
  764f6fb

  Copy full SHA for 764f6fb

• add other tasks

  baberabb

  committed
  0b53333

  Copy full SHA for 0b53333

• Merge branch 'main' into longcxt

  baberabb

  committed
  2b56339

  Copy full SHA for 2b56339

• handle None

  baberabb

  committed
  46cb513

  Copy full SHA for 46cb513

• noqa

  baberabb

  committed
  610e7e0

  Copy full SHA for 610e7e0

• Merge branch 'main' into longcxt

  baberabb

  committed
  8e37ce3

  Copy full SHA for 8e37ce3

• fix prompt

  baberabb

  committed
  684a60a

  Copy full SHA for 684a60a

• nit

  baberabb

  committed
  ca34825

  Copy full SHA for ca34825

• nit

  baberabb

  committed
  ebccca1

  Copy full SHA for ebccca1

Commits on Jan 18, 2025

• add gen_prefix

  baberabb

  committed
  7ab8b05

  Copy full SHA for 7ab8b05

• add cwe

  baberabb

  committed
  92953af

  Copy full SHA for 92953af

Commits on Jan 19, 2025

• add other ruler tasks

  baberabb

  committed
  ca9316c

  Copy full SHA for ca9316c

• update pre-commit

  baberabb

  committed
  5e2979d

  Copy full SHA for 5e2979d

• Merge branch 'main' into longcxt

  baberabb

  committed
  7f22572

  Copy full SHA for 7f22572

• pre-commit

  baberabb

  committed
  c8a3f2e

  Copy full SHA for c8a3f2e

• add qa_hotpot

  baberabb

  committed
  2e43c8d

  Copy full SHA for 2e43c8d

• nit

  baberabb

  committed
  537a77f

  Copy full SHA for 537a77f

• nit

  baberabb

  committed
  f283c33

  Copy full SHA for f283c33

• nit

  baberabb

  committed
  be7be18

  Copy full SHA for be7be18

• rename

  baberabb

  committed
  01d89cd

  Copy full SHA for 01d89cd

• pass metadata to groups as well

  baberabb

  committed
  2c8a165

  Copy full SHA for 2c8a165

• cleanup

  baberabb

  committed
  3bbc73d

  Copy full SHA for 3bbc73d

• cleanup

  baberabb

  committed
  bca6918

  Copy full SHA for bca6918

• cleanup

  baberabb

  committed
  f8f0a72

  Copy full SHA for f8f0a72

• cleanup

  baberabb

  committed
  a74d040

  Copy full SHA for a74d040

• allow to pass metadata from main

  baberabb

  committed
  352127a

  Copy full SHA for 352127a

• nit

  baberabb

  committed
  b9614a3

  Copy full SHA for b9614a3

• cleanup; pass metadata

  baberabb

  committed
  2bc9aa6

  Copy full SHA for 2bc9aa6

Commits on Jan 22, 2025

• fix gen_prefix

  baberabb

  committed
  6580aab

  Copy full SHA for 6580aab

• fix cwe prefix

  baberabb

  committed
  b56ec43

  Copy full SHA for b56ec43

• fix max_gen_toks

  baberabb

  committed
  83ca727

  Copy full SHA for 83ca727

Commits on Jan 23, 2025

• fix vt

  baberabb

  committed
  bf64b46

  Copy full SHA for bf64b46

• fixup

  baberabb

  committed
  bb9ed30

  Copy full SHA for bb9ed30

• nit

  baberabb

  committed
  5f2dae7

  Copy full SHA for 5f2dae7