Code

Add tokenizer#64

Code

Open

tomiy-0x62 wants to merge 31 commits intouchan-nos:masteruchan-nos/mikanos:masterfrom tomiy-0x62:add_tokenizertomiy-0x62/mikanos:add_tokenizerCopy head branch name to clipboard

+477-141

Commits

Commits on Aug 20, 2022

• 高機能なトークナイザをターミナルに組み込み

  

  Ryosuke

  committed
  76913c7

  Copy full SHA for 76913c7

• トークナイザのテストを追加

  

  Ryosuke

  committed
  8a7e48d

  Copy full SHA for 8a7e48d

Commits on Aug 24, 2022

• 関数名の変更

  

  Ryosuke

  committed
  e00d4f5

  Copy full SHA for e00d4f5

• 関数名変更の適応し忘れを修正

  

  Ryosuke

  committed
  74a2b91

  Copy full SHA for 74a2b91

Commits on Mar 5, 2023

• fix memory leak

  tomiy-0x62

  committed
  730baa0

  Copy full SHA for 730baa0

• 不要な変数を削除

  tomiy-0x62

  committed
  97ad03f

  Copy full SHA for 97ad03f

• strcmpを==に置換

  tomiy-0x62

  committed
  c196367

  Copy full SHA for c196367

Commits on Mar 9, 2023

• 無駄な行を1行にまとめるように修正

  tomiy-0x62

  committed
  11e14a4

  Copy full SHA for 11e14a4

• 無駄なif文を削除

  tomiy-0x62

  committed
  55b952d

  Copy full SHA for 55b952d

• stringの長さの取得方法を修正

  tomiy-0x62

  committed
  4e2adf6

  Copy full SHA for 4e2adf6

• 引数を改行した箇所の先頭位置を修正

  tomiy-0x62

  committed
  fee03ea

  Copy full SHA for fee03ea

• 列挙子の意味をコメントに追加

  tomiy-0x62

  committed
  41056cf

  Copy full SHA for 41056cf

• メンバ変数の仕様をコメントに追加

  tomiy-0x62

  committed
  998e053

  Copy full SHA for 998e053

• fix typo

  tomiy-0x62

  committed
  a756392

  Copy full SHA for a756392

Commits on Aug 22, 2023

• 関数名に含まれるTokenizerInnerStateの略称TisをTISに変更

  tomiy-0x62

  committed
  bd6bf0f

  Copy full SHA for bd6bf0f

• using c++17

  tomiy-0x62

  committed
  06b5d3e

  Copy full SHA for 06b5d3e

• TokenizerInnerStateにコンストラクタとunipue_prtのエイリアスを追加

  tomiy-0x62

  committed
  31c05bf

  Copy full SHA for 31c05bf

• 構造体のポンタをuniqueポインタに変更

  tomiy-0x62

  committed
  66d2556

  Copy full SHA for 66d2556

• use dep file

  tomiy-0x62

  committed
  701d772

  Copy full SHA for 701d772

• TestTokenizeの戻り値をboolに変更

  tomiy-0x62

  committed
  41f7811

  Copy full SHA for 41f7811

• テストの成功割合を表示

  tomiy-0x62

  committed
  e3de427

  Copy full SHA for e3de427

• 引数のインデントレベルを修正

  tomiy-0x62

  committed
  2f279f6

  Copy full SHA for 2f279f6

• tmp_tokenがstringのコンストラクタを追加

  tomiy-0x62

  committed
  3a4d3cd

  Copy full SHA for 3a4d3cd

• 引数を変更しない実装に変更し、その引数にconstを追加

  tomiy-0x62

  committed
  7522e04

  Copy full SHA for 7522e04

• verboseを追加

  tomiy-0x62

  committed
  630092b

  Copy full SHA for 630092b

• 関数ドキュメントを追加

  tomiy-0x62

  committed
  cf37d70

  Copy full SHA for cf37d70

• 関数ドキュメントを追加

  tomiy-0x62

  committed
  3a45705

  Copy full SHA for 3a45705

• 変更しない引数をconst参照に変更

  tomiy-0x62

  committed
  41e852b

  Copy full SHA for 41e852b

• 削除された領域を指すポインタを生成する問題を修正

  tomiy-0x62

  committed
  5917e7d

  Copy full SHA for 5917e7d

• strchrを削除

  tomiy-0x62

  committed
  b9164c0

  Copy full SHA for b9164c0

• cat の複数のファイル対応を削除

  tomiy-0x62

  committed
  ab09297

  Copy full SHA for ab09297