From 5733cb3ce87833b4fc7e29d61f26ee7b6b3aad9a Mon Sep 17 00:00:00 2001 From: souzatharsis Date: Sat, 21 Dec 2024 06:18:25 -0300 Subject: [PATCH] update local intro --- .../_build/.doctrees/environment.pickle | Bin 5674646 -> 5676070 bytes .../_build/.doctrees/notebooks/local.doctree | Bin 195905 -> 196617 bytes .../html/_sources/notebooks/local.ipynb | 8 ++++---- tamingllms/_build/html/notebooks/local.html | 8 ++++---- tamingllms/_build/html/searchindex.js | 2 +- .../jupyter_execute/markdown/intro.ipynb | 2 +- .../jupyter_execute/notebooks/local.ipynb | 8 ++++---- tamingllms/notebooks/local.ipynb | 8 ++++---- 8 files changed, 18 insertions(+), 18 deletions(-) diff --git a/tamingllms/_build/.doctrees/environment.pickle b/tamingllms/_build/.doctrees/environment.pickle index 2d0464202648eb92dead4ea03911376810cdc03c..1e2ec54f811ff29180fad14952626f8115f157a3 100644 GIT binary patch delta 15343 zcmeHuc~}%j*DpOF!>|u1i!3(Cun7Wgfb0T_0^J}wQ6lO9qs#~lGJ^)AA|i+pBRXKk z7#EC*dx9}q8aGU0j3%yG+>*G(+_)t1H707})$dew_XKkP`tBd^m*>5g=czht{pwWJ z>FVmHXVuQjvbXnLmUTWF6YNrHFlmboh8e~v?JP~Xv8~1RFO5+%O>3%cYwjIezR&^WpryL?|-9IhHbB=({B@nPya=H1qlX<}C$Ii1AUX%m>G^=DM*# z=BN28b5XvZUn?CaiP2+<(WEiyN}|dQC7SZ6s;crqX7#xK<|74(UbOd?K0Q73no9HU z1wm-LXJCItE1e<0F|QjJCPZsKJ}v^Wth=u{f4sN3W4w>qsP;EM7{{6)jdw-0(|S9a ze;=@9;$Xl(Td(m?-nnq4AQRfoUnwj#g%ER=|XBs zXTe&*bhCzOEWOm&Qf*bap_Z)DK$fqbuBp_`5gV6iD%oPK31*kT!gg2AYG0~?Re{AS zu7#C1m6VZ%sxY8MGnC3nkO9zO|`hn$Z;?0Z5sg~IwOiX#Jbl>AsHYu zAwHuCg3^{4Ae~q(4S)i}E?=W5E(csg1!Rr3l;kOd1LKF@Nye7xiiIqLtc|SJ&aBbG zG(tL+kT_wMgn*2O@)|m&1KWg*s+;gXJLjljhSKfdJLMJs)+t9ZKlFFEDf7{Pj%NPo z=8am<_3}cyFzYX?l;LxjH8Yj$$eL=BS|lyBu(N2Zr)y#FBCA$V0n5+kXey`IkZjL2 zlxoY3Y-mAFt}%=)(~y0u6q=NR?XL@f)m4^MYc)n#dC19GI+8fBV^snL6*{K~^KvfH zRB4KJ<&ZLvXa;ojNs?z`sdknQ@|P{w&Cs$&I3k!}x2q((kcmJ*W|G}*v|$|G(+t%{ zDTyZh(}oiB-yX`U1MH-_W3c^qYJl)tVC6e{@DT8o>1pTRSwbglUD9Hd(fN{7!vB zHiMoWWaa;iQ1Ir%Y|ME@d#V*o4E&*^%7EVZUTeS}8Qx|NnEk&M?K_@Rixm!75x7K# zD%%31@lA7J&?(np(edIQ(_%9i1njBle>;kqf|(Wl7d}EA15{b6^tyZe;5<0oJ%7NJRo~l*EpYc{L zBy>?P1^-GFlp_+5_se{;GXoMt=mUT0mT^QuVb(&3A(Q(1v9Z1Muwusy*pU z5t{j_Djo5of^+b|DwVsvplQU%=cp2V1)&KqK`L*V%NjU~>g#>1tmd_~FyU<|H9FiA zf4W*VM<)6qeyz$LfANOuOVHpg>s8+c0$u!}>W&PJ8516YUpuC{NKDY@FTH?jPOEy! z(2ZNcA$ZwYm7}c~4*vFQRlEqAjteiTBKv`*V{1qT{`|43LLs@$lLdEKgnGx|4{byX z8BZ?K^$uq7LuK&KjG%79pt%bD(lO(fzQKz{p$Wgv26tIYmUThF{Vi_2!h=iXl4VCs zu(#Bs&`D7ebTTU;xJK+G9Y-Yvr;>~kx zs1G(dNWz?1!Baye!GC>lCo2dU*u5+Gx~pV~I~Tm&P885%pO9g==yI?cH2D1+!2_*b zH~h6;`lW$zG3;8c^2CZCg1?etrLEA8AA{|2@sGh{2TRBR7MuhEnR1jlAtV4Vni%5h zC7CbI40#~tMmlz09nLms(*7kM90}bdnNuuD*CYAOM+v8m2^D zjxec_e+(E(a*_mtH$NM&%(7fCYaw{OJYp5mq`e)Rf#Mb}7IzKoqTVNOY4cYyb zz-IkAgq5Lf#jzoH`ENtMj1;t{{zH%3qE3h5A-Fhv= zfM_#N@lOYb>D&dc>1T(fx}j_pv^{-e*d%-Eg~7{PjOnvqhFz^jWxovx!RLx& z-|E{9=8t)?J9-N+cx7yxBl6CI&Z^hPp0*WW7QG$&28B`Ln60tVGPHir;1JxrJvKO0 zaCq=VY=^bLUbqlDQ7*8itFd#4HS~?sI0?;qhsH&S6R>DdToCom#6Jy* z`%={QN{wsuC%#I&W^&vVN1<8rjJT&_7xRsAHa3Fxli6{dVvrxa9QTO>)uK~H@xl1c z;yv@M_$6)=BOo65SKK}ifyH+5W^v|U z^ot+iDuADlh<8Vizg8)Q8FLvJA1MaDATEBVFkV!W8t;Z1(&ER`IFNtibp3reaulIgkBpK3KRECI{>OB0^ki;A z%Rlb9e8hkIVRm4fkUDL8&t6tm|G1~~G5;3>YHX`-gO|O+bLhObP`v|;OurPJeJNoA z3aU>S{Lgq|Mvwm(=~y@6fA&QJqZQvI*zJgQ!iN_n+>1x+@8&A;fx`)nGBmL%DFjbH zny}u5XfyDba|tUI0?S@Wi1DE;0oM0(*a4;%NaJfp!QUHYM?nw_qx*P4tw@+O1IjvfOyI!3GZactIj3 zL*{h}!*E7JVxx!P==6GGY9V2jc=E%<+jJ}$c1vcj7Bw95>-)|9HixeaL_XQ5;LMuyjViUVXZTK^wk|=!a_>(r=7JFRX$H z=buc^jzn2iX(9Ojqx2O^{>l{8F;_JN@2Ss#1N;WhpuTu+L&mc!-qp?vZa}GXQCHQLhNrn&1Czd z`A-4tUcZq$H1hV-LK3B|h!}!T){h(~Lsj}AVK{Hm$PKHI@^c8#(`Quwe(1OZG}jCt z<(#Exzsd+(B{nP^HOLxoYaF%n8EXB?%K=YZl$8%h`7K^SsMTeF68(b06ey)J%O1U3 zniYV3o3b*W$=gTJ_9)=y=rMTfqR}o4V*8C5fbTVq?s~3hXQXRpN8^~sws>dDnA4%&%T)bmkat==71b@HY(K~v5na_*tIFU!xkBb=T1g-xPz0f!5aNjKTg*IU#b) zEz6nfDYbTfE2pcMB)DzQ`884!MqkQ#>;^KZ><*cUjdsrGlF&?IJrb=7L{W8IE?Tny#PxMtG1^!UTzF1GIN}z7 zfgy&vXuco%b^&(`I-OU?1)|R6>`?SmJvVt8`Sd{Vc$2ZBpB?CZ2uD6S(1#I@d=8)= zMmX}Rfj)z9;^h zv~VpbdPG_jDu0E00WFIPaz%eEgE=JK;KElqo!I894Fx9@TdqN#cpS0l1d-s9$oY~%Y}h1g){DBc0N?B(vl>-InQa--&8AUYVYaz>GT ze2q*Yb(npFdn&Sx-*fh2vyLC6E)dLdZ=CcaH&*Zxk_F!;$>JK#8_4DBgEroUdF~nv z8T02|GS9%eVVGxFWXfLnnM@{SYwnXQ05%HyJ|M|VwGF>=V?_4qL$VxHYxtdeEV8wa zxoIN%uP0o#$Z~&j^&)%yDam%)?D;d2H-z2O#K(*Fa|{nlK$K|U`@AjAx8{pQ(=i(! zE;ZUT)t0|6dI4*Lmam6_Ot9ngL?xY#(+VDDgF?|uC(D6{*?90K%!agu*`h2bJ_}Xt zgemE8;$cdF6;tn-GhZUe*yzf?Dzffw`~;Dm=gwz{4R{Y;jxYDNGyr+JX!64l+GS53 zj-dj!EnLMDy!o$0loTI6PGpz(@}(mC+>eKAmo&h$`tju=`;g@~!kK|+xAf<$MQwzN zpDD8M2J@978x=}si~4R0<9`xZ^w(@189-)>hD2v;Xe5~}D%06|cOaRqt&T8Tu)|MB ztxmi*ejd#ag^Su70vI`nCzm>~sB<&#hkqGNqM)Ygp?sCdJ{(5ILA5jD_%$NyHJslm zvKJEgjUu}=iH9tqzWyovZNB~Cjn-kqF$DMg?!eeKzKhcURege*D;sfMp!D7hvwafVN9&jsDM)Q@z zt!&9Aoq>~h;?DlPZvdI3qoy1MG3wQFc zB~kX)JAAvywsevtquPk~NP1HiE&|9T%5L66(u1;)RIs9y_1({BiR`Qoc-UL1_W1!G z?j)3LJ;XCI%641*{YOX|Q+xlTB)|myo-EtNtMTvlPBhqcE-d z!#$*~9@2+Bq>p<@pY)JEl_XUDH6I0cnm4}YyM(xl+GyNBNcTWU+(5{&0|+^D03ins zAmp?GWD;t>$mhn8VK1Srm(a#bXxnDmbO~+QOdBqt?UulB-7i5R1kjU@Exg%iaU4y) zR4T10Az$mXEpF3)1pji>7oj&V^NZnVH{c47>4%Z}W|fuIsz@LB-mXnQE1dddf5V>y zACGVO1i|OzSonA4iV6Nkbn08)PEJoy@e|+z4fm$aCemilzU5P)S;}`n3eke^c%z)2 zux`~O!@a@b#BwE(Adiy%R63E;euy}=N#|=IR$hjMQse)pD^#zRFwbE?}iT=uUXgL+Ve3K8C z(-Y&q2Wb6oiBU?h)&53oB(Pb)p)`*`U(#eFRU_s1fIAfp`<@R5QuI9^31sQ_e1aWq z;v;8JOj@!hX|da$oJY~m-}8ln@sd4*R%}YnBF0V5)Yx&04+ntHZ}Fjw2K{ym_8$!% z`2(L~r=Vxm86PoL==ve+AkxO=G&#be13&ZOcGP{`QNE6}c90?pKmRb}fgFqo94{2hAGy;XwAgPHxaG*E^X(V8fgaSq4 z1W6G;!K#je&{U!zz=2EP6v?F@aG*Fk4-|B54g&{@AL}K>E#N>QBpJcThjatQ0h)D0 zu}@I!ZIhbp1`a$Xc34@}38-pdLG%7L$z>OCp!gE9QScyn35xEiNEMs*Ngf@*frpTV zq~FDdsUql*z2RJM#x;!WYf5rE={3*aeT1Vka^KmPnzo?WRB z-?mv9BRT54SP+ZWFD-Dz0k0M)A1m7L(-$kS#msCefPai`8TXDSezta8 z1acWj0@=_xxcgMV zwXQM2n{xdhjWVlH{Z@L<5RJS4X7r#N0E(GH;W&ZJ5J)o+3+kDWo~T?XEQr+Q1Q5D+ zSx1fNs(|wph@I#fiWpm;Ff@9qy2t_@9M-dS3dk1d(SI{mQ6suOl?MDwAU+x@#R0LP zPJ{icdrT1#-B5>t(A~I^8qsnAXNN#ebcaii>Dgi-$d({m{$_kcjfgF#VI>Kq5QrtZ z=GdNa{_)-5LP6*rUD$A9M7so>(*n6G5WA9Ys1H+nHgrnshUy7IchqQVL{k6&GHI?r zR(FS6JFaJoPbPJPI}1X0xZ4#p$om40Qz=Ca2VxoXrplfzE`e;B!kd3HKBY!fFW{^b z$N@qM(M5tv?j9uF56>rl5aTWox(C^^1-dLmQCfox$l&w7Gxy#b))E=O`;HwfJq z?GI{18UTRf*L;Dj5y(D*DmA=zt7pU5AG)EYfzTb*>PKQkE_#X@A&}8PEO3Q)d$!1U z*bR;Yp*vicjNC8L1_9>~AsT#HuU;xoAa_18uJebP9!*|5A>I~=aU%%b@lH@9+E5KZ zlh7f7Tn1u+t3TJX#TJn1U6#PT_c!Cci^Paxj1-O&$P9rr3*;@M8V?|!w~~i7virBT zxESCE7aHA^>hiK_@I$2OZE){?L-Ih2P3pkV*CmlQaMV>AP1W%DBYCfr%;=m+?Wx#c z5&uQR^%hZCqwb^Vu!vD0)~%I9psg+TrB#M0e}%EShtZK3$&cy=8fVcc-xAGaBWP`L zq+jR5_u*1lojuIX@Y8}4ZIuarS0p*SYhz_gC!!FYs8RbX-j*7m`!#AVYL_HZc&GC6 ziYfXU_?%zrep?TF2K_-=2u$AA9(ub9tw|&GxLGoxOS9BIK0@GfO?8Rnu-)R|G8^V@ zr$vmK4FPVli0awueu^y?asF&Ir`TZ;zXI!Y0o)mWe^OCpnr1M>NPX-P{m2#7+Tu>X zrk|p#oTjZNKW&gk_kIsA$MOoJsistCkU;iJX7u7qYQH)xi4;=~Kb?~t_w}$ikz|{# zEw0f?_T4?~cI0Pyl4%IU}Ny9ip|xP7l9ZlY%+b z!$f}9=Tip>bh?Myu^Oh!WU{1f4Y_nzHp{H^T2h%v)kLZ_q>3U{0;wjFYBs4lNi~L4 zYEos8s)1BHNp+A^lSq{YRh_LiJ7RJ~)1XnkSzipJ`Sn!zkJd zoQ0o6pyAExHe`BLor0Z~sNw%4r<(y`=PT;t*02#11T=23dI-M%s=9?yB$g$YC6y(V MrQpU^wbJ�Jh-eQvd(} delta 14353 zcmeHOcT`l@*5?k$00Pn#5g3peilCy>q)S!cUKAaB8DRuQm_dpOMg^l%geW@THZjH$ zli0f+6MHm9lh{%<8ly?fv&5Ig&qPyve*2tzXPn9V>#g;F&=-}oAWbj41S(g#tK8RrlQlxHTmq_ljm#OFv-rwCMW&Ml?D3C z=Z5R1nx!k_YIT#%(wBW!=%$;cf3}{`O)*IZTPV_L%#zPuSKS1&46|93ftidco2i0nWy{LWR+;&=YQ) z46uW*`vf}K0eZu>D8C@%=jg=rDKGgs8AA$3+IHxs7!@M};pS;Kf8!6M+>M)xT#P9r zGmSS!_6C`syV6)&6aufGb?;?-Wn{KdKw;UD9=^r_MGC5&3>(h6`@;3s9$Cio7%hx+ zaP42EFYWAL^TZsD4~vqG{OCYGn#`i}@8mEs51cjqlkg;)0fU(CIU*l+XALCbJ{Cs;&t*O=2mKRHVK0sGjQmXA= zGrJVR=XwPi*Nh2(yhT0z#qr|Q{>CbGkg-VZ;h@o%YO66rW7?RY?j3B2=|Tp>Ik~Ji zOgn1pXbc?NP1Zg{R4lEx^TU+FjUi)4aFZ7r#x@(qa@b_pY_Zv4>xRu9n;e@1Hb-nu z*t%nL#^!>}6dac4RLs46%MMF6r81;&3t)|AHuL1YyWIs+I1urCy_ZnSHr)4NBpDr#ydf{+Iv7x56 zM)6#EZJDCD!cbQdHCYQ7np#7pbTF~vn^CUUM%C7_W2#tFrI}t{fkmjTftv>u zUc{+2F^+@Rjw$4S`#32anSDYL@iDyea*!7}Vhq|Z>)vH!gR{fGvTo1)V+Cp3$xT)lvF=Ut@S{67adXbQwA(1Ii-e2ht zRm);LNW=Y*7B6lxagZq|$dDG>DOXea)L!Xpi`3UixrIY|s=Kl;m2bN$H&A<XqH&FPihJfIu$|t{8D?Obol!1L!!^pCE%EUg_$_z5{6{WAt zeH~tdbqoA#Z02>ga}k}ob+F)8tT*X+RXIoY*NEk7mDfcjY1p8=5X|gk%6rPIGI%^F zB9u%!qrCiYQO7=1mUk0f<1Z2*fRVW$riPG8r*#P8?E6N0mP!fDi85L+@H~yj2 zJ6gzD-Biad<>5V4@7h_|aWN)Z-~p~f2E6K}Qb2XFpNc%!Q}wcx6t3MNk!8zB(9_lu zX-253uciH5q^elXMj~lLR0S4E&>bis0V%33&LWdXsj4(F6OuDbHOUV4oeB>nA7rZ> zZJ&9uP`EonBM)OD$4tr2U$|7s_kwe535&A3$u{FU$5Gwu#mreSM`nC zGx@-$s;v&sL^vK08cUR4snnv9wB1q#+Cu+Nl%eF(ZPjNMGZ0-B_f(SxKZ_?AuS&L% zBPN9o>SZCXs||f%S#xPlk)hHj6A|R_ ziGI=1q9;4vGh7rAWa9tz^R*~Ja`4Z7)l&7cL=9ZzOBUNsa2t+T%IThwOD)!kczQ<$ zNqFnl42~i70g;&=7K73?0$d|MG9Ws#w=eYA8RAW*jEKCS#Z3`UOU91Ko%txZ`>QgV zo4$&~7KN=pwgK1%Vv8nCUk!3P&P^f9*9<=Fjg;Rz*tZ*m?}!T}OAihfWKulHeiuSv z+k5Hmpy(IxM2Zg$Zt_L`^q4jCweYAr^OeLUpTD6~YXV!p5k z&#5sBQ{8i(hF=Up+w!X0s{Fq8eA&*RqHq7Xeql}Xlp88=RDvLt;IcaK`= z%95Gi$Jv=Jd+)~)vt`H6ah_&Nx8LGsJ77%EmGn$Gwmf!>A7c`r*3WGqe3zJm zmlC?pSKZ@RnrFM;D?Wunuwmo;<25cO;aWs|qTyk^D4_v&u)I5#YQaR4fd zH1P&!RI*9Q_3?k0Mkd*_fhBwh34_nOh#m&DJ+OvvLE@sSBg zl^?|ydBC_PT;jFY;}7EgiAk=05WkU=1Sq-e<^{b^#`K2wAH*xjrHAo4ACqH4_XH!| zD^EqjC^wUG^T33O=2_(?CTurPJ|Q!q1al>?9ju-iT-5^E5tD;#MeQf-Xr)i@#PJcG>w)D9-6OOo#Til&G2 zlX}V_ZAnrRDQrsOW%4dNwk!~|F)^Ne+LE-`M|51XHYsfaW!=d?e@nWPAZq=FB#&mf zyOYL@hR=kY)0;6yf%3k2X(_yJFqnQy8!4| zkm*4NG$qf)1iG;V$mW*h5(zJ0Nitq_E}l*dC2sA>!Or3+8LN`Xd@+rbo+;Z@qV|vE z6os@-Q`1sTWWm|`ls*tBOUfWEO)1eb@aR-}z+26zEW8(%4BKpToM3WGiW3>on$p=z z1o?D*%06lAmtRjAVhi&(<3>GvGi8YkvT`u%H#VpEbQ66)+L2NiC9+Z1Qf~M|uhVJX z@aAtRu4LfvDLPvp`jl^3W74z!t|uvzQz5S-GM9`=PIU=|jx~ebNli)WJ3;Vmed-v} z)|48D%OSpEfLY7K`jBfasi|3_W90eN2YsO`E0i>!aZ6dyGFuj%l zjc=$g`K~2xNHR1!CVG>o(`j3G!uTf{q2y^@`U5xkbz!;^8uz3ogTG$s0H42}o<%k{ zr}yKYfnXJ+pUZ_tMNTL=b18kueps~uZ_$P~59{Qd%|q!fXTr~$)BBO`t;4Q)z&|eF z5U#HdOYnrtb?Ko*zjN3f4SbM_81vmTvhhv&>S3YeUQ~wvc=&o)h8(UAO$Y}C-uuCo z^%*JTWOGJe%)xYw!mP5?K+>%>+4a&7B2flsIOLEL+hR)?dQu%vEM@)4;m)A%jS9bM|P2n!|~2g5ah; zdoWxyW;?-8E!oZomSi`sgi}#C{9B5gzJV~a13`aI&2h~I*ThI?;?4>rNKB=SRpu{xm<7eCnA$_c-mp+iHP4k%2&!^kDL!B{h#J<&lVj5vkO}7MfQjMf*BIKvZ&x>99+yy z9tIoR3X(*CpW6x^*~_|Y!KEoF0Te3=hQiA^1rcPvv7pZ=5#su-f~yW9`{bVmgJg__ zlAj7ZNz=~-huuXPO>yLiE&q!xZ1=|mE|zlicSc<% z<>+sWx>1y)zc1>hQI7t$sMAx9{;sH-i(Fm19sVF%(>bp*3>It_65#n3e&@=)f;)+C z<+C~DJfLF<|1rF@lphYRYx?$qi)}oQdM`4lou9`%$cdNu7etO<#@CD7l@8JB1-cdd z61h}}z^(5a4Gl~Au`tKTPoenau8}VnW3*i+#D^>SF`{nbDwFQ+%lsly*SMN*6*<2( zyt9mp>jfLVS_kxQW+Ftlt*CMPm1F~?;I#Qw2^ZN3|>CU=fZm%a1sxW@`E|bdceg8d@mv#=VyyD`QijWUt&v7^4la<`5xacu_sUS zTP3#f4F9H#!I1b5`H_-#^+&uxV*8xuYbAEW1wKIX{rD1JCTa6O;WH$5{io&}!awKd zN?P)TS(|!=_mL1vubL5VUNbwCU#BZVXGMPb+KjO9CXcr~O#9lme39f}dyB7;*p~0< z>Y;-dalFm{E@{2*@=j7%^be*X0GHQ`oV(ADVqRe+=SNy1{6<`2JY4-?(9gKUZhLTv z+kU1?jI0O2CC05~Z0SR~j*RX0E8Tm_LfUr0pKN=CwJe%Ef8$3;tl@XM9W3DKAN*s9 zb^I4ULt+;^#|8-%3q?{!aoMmFvGCKh)h5`843keddkqfv5Z1AIR3-;uugD_n( zjd2pnCH7)>;U~!pSvv@P1CPVoE<&NCWUDcyhk&cWpx9A!*-gMDSr;Q1%;zn8xa=Y1 zLSX`)FrqyLTo+`e(p%~!6pJ#k^%3xuA~liMdkW(u*2P!Ilmfo?6XZnK+Z=#$sbt!I z7^8Ilf?AR>UmtQiK*0CW4Df1t8CSUc|xqN~MZwpyt}=xS~H2@42M z>!onq1=pxKR!HV3>p?z`6R@T!D^^KE0?mS%WQjtR#4bvr^I+OPl7-b0yD3%JF0rNQ z!Ul=;%b;b#e0O9DzcAJdb{xgR{xDm>x691*b}lU&#-@!B@K8ZEj9ks5vte4?do00S zZm9i`7v4%DFSH2@E$$>BdYmwTL>YxKK9Jyn#lS98zFPzy6rO@FY1|_8L#O~#9elP` z7%$2M4+lJfpvjAz+(Ad7UL0m<>qT%8vL&tkaRJYr%wgFHp+RC5Ck0MMwLappF#J7Qrp#;a zX<9#&6*oHT4Bcpyy})G?Vjnw;fC=;po!5h1p7Uhv+KdvtFy{rZt8$(O7sE44chr(bH}M-=pXiq7Uv-AVeJ z$#fWeLukfZw?j7sB4;-`ElRlYiE_Lny3CGXE>~{~AE3*d-w289yC~fUqi`{oj|-@Q z$=?bNVpzgBXc!dX%fiO9u%+J$X&82ak}2T!olqlZ_fNMLz|0>9M^Mkv%h7WR9Qh6_ zbPD|Woq#_vK<+JJ43gJwG38gcgiJB+(r#F@AuhTH#%cG^I9wCm^gXa(dt4|i`(AKx zWSaE{F*!}KTTeQ`xlh>u$G;acL~GCyw4Q5s3!~PGQ_PzD4`d_G( zCOi8VIGmFlKtnb#C9JiR=_M zwY^UCHWcRH6()$rD^45)ZArSbPspLKDd6_k>gjM|O)k z^CW)bbn}R92#x7+p5D1a-a{e6fmy%(f`eHv74)OV!Y`R|_d}tbeWjQWSEiXfrEXM1@#YG^7UW3q4#0nL?b9L&dpKCIziDR3wcvDH6unG*HcK zR$~kVIaCBsHMs;Lhl&&PP{AtVC~~N{y}+cng&Zox>Z2GVu?|sjh?OH%926A?I!z&a zkwcI1>uqcrL{v4hsM)v0K z9fRmW-!`BkKyFfCg;RxC+%#ktmN_b}v1+G^E2*MJEODw?kHwBI)7?xUQ;|c(YXPWW zk7uiqL&e7msvwUZ3DMjT^8tVcvGggHGadw}`2Gv@G#vy~A=W>c^qZhq;n*%O)K=Hb zsh zoX3k)B_8^kT72oP`+c#@CQP@PU0NjL-oF@q*!ciXh&b0p@`p%zX)RDMgj%EWVYol0 zA;+O$Ib;Jff&l?A$#x_Tl4}@n_I}}TMXR1{PFNLg9i57@8G86%j8)7Cvt}^pr6SoX zk~2ums59_%W|>oDqy=g$3YNwV%m{&{3^hq4`UPRfPWcy!?kC+j5i8$v( za@P{BCC(ZyAi)AI3uPV!-zb)ZYS1!_+eEKvtCBm5}h zIF~cL@|lI@#t$SH74#JMJt-z?!i+GQOf-$Ns$ zvD{IxgnNH4gDafL(yA265+vsA=I^tPIDoPl?)YDf_m~lGia7rgiC+Z+k4Iug^*msW znu4+!HTN$@12e)h5oeo7&RW8`5o@^ND4XGm{$l)?T`xgV$fpOnkpr)f>iE4A38ey%7b66x-Ea4{HvyRAo zXaUEgUIBF=S@{2`KF^=yoodNt{{4s)f?boA`s z(cS|;bk}Rjr&d>#&0zmEftT|2CJ*R2M;#pYnn`2<&h*>zYJ5sEd9O8@K{H3))A1Fv zxPXc;o5l5W)ZUKtn>02!97D0;Rg;Lcqur^b$}sg=8f&bK&eTZ%SBCzJMJHzYmSmoA`FZwSV3v$L^hxlDoaFQ@|?x0(W=@dY(M z;0=?=;9V*zDyN$N!-IzJva)Bg|33*u{dOz8L#4J>V;XUj$pqzd)qZ|r;&S}p-sG^+ z?67t&uH9y{cxuZ5Flj)PbeqAg)!_hRUki z83seNX{^1HA3d@<+CABq_*2XEGqlxOTq)BcAGGpvuBfE{I8bgdfe@1!?#)yCH@s~U z8KzvP(VHCiTUlIa(LJY~URQ3iAF#4JR8>_BGMV=Mo5?(fLz0Q!!eR+P^L(`%_W`V$ zukI6YMzYb@1Xx-%<(kStS`+3m_;|iL7+}8I6=V(S7|3o!E~^2<-vf1nI>7IQRk*xZ zV@~y5Sk{0?(@EIYp!VZV!)*Oey7^gvFm-g_*Fld Z+Ns{oIVP4Rm8F!WmL(JKm1=kU{|E4441xdv diff --git a/tamingllms/_build/.doctrees/notebooks/local.doctree b/tamingllms/_build/.doctrees/notebooks/local.doctree index 1d579ad6ebeff2b69affda3d93368b8859e82578..dfe5ea4ef10dafbd8a0abf7702d380f56c22425b 100644 GIT binary patch delta 2010 zcmeHI&x;gC7@h8H2u2TT{DHWb4`Ybpy1Ix7f`@n!B$^mCn1dd^ss3iV-0rHjs(RKD zWb}}O;2~sYnUlAi5($K1@ti}BUh)Uz4-wY*s<&B@oID63_ND0ly6SuN-m6!CJ=^-= z-nO}WAFg)(ocnqH^HXW3bIIuet05_ClsMywoeXLBb9TY5;M3jo>p|oZ5_zx z?6m2V7vpt6XUaHmQ>Jhv*Dj-qUC`ne>6~?h^~F_Adx?-pu@|gwXTKfzK3T{=x-rT3 z{P>P;@b@J7W&G*Ey`x4eV?!>kQH-+zGy$z-dsJ8zwUDZ?!>Gn0edqdY^c@b`LTJT? zx)Q)*6hAmykle1oK1Asbp>R-n6r{A76!^R1t}dVq$`Bnr^$(7GB}(_SA|sM{a@=4E zIo7Ut7^nB3Rb>MJXfBE&dPK8CeN|!WjLu# z3)RA!!0iLTin6d1Ndu;7GT@@-Zh;=Qgli;gA%hnniH^;6cf)0H?J*~}YViTzwThVS z{$7jD+oEphTAZ|=de8hDWq!y1m>>W9+4a<ab$-)QZ~x|& zUVC%nQ%@ZA&u delta 1392 zcmcIj&1w`u5Z+!R!Jk785)XlHTsIyJwOWVR8)! z8a5#peS(}MggvR`0TjWj-h2rI0jp=%i{6%XPCfne)%R1ixpDUJ=*-l{$sO^2>dnQa zJyENzI4Y#-H9Z(83R z*>$9$7o(RhHzw*zsgpJft8<2tS*96e2+d&7mMUtsWlF7~Ew@(gG6x?#tWiioX`3g_ zmh_l_XOQaYJlw_twavQ7e56SrpbxPanu44)ao@9#%+2BRoE$mBO;7`wzyeeS z*8nV&d`L^iBgP8DO5pP>v#!GK<~mTO*^H1U5s7&}Llu0ZqTtY@bSmZdtb{cd=y}b` zi;CHLh|zQKLjt(Y4pJl=Cemk9B-A-p8jge&;^PW_gV`N!QGR!fuEGHS}7+KEz9cKUS+>`p2*gu-Jj*Mc~Aa8(@CPwmudEQY_qkGfbpK3J`9^5&J}YOl6?_+8u> b%$}&veV$odTo`%rYU(Ec4DNlcf4cq~AG`+h diff --git a/tamingllms/_build/html/_sources/notebooks/local.ipynb b/tamingllms/_build/html/_sources/notebooks/local.ipynb index 885074e..ae64b3f 100644 --- a/tamingllms/_build/html/_sources/notebooks/local.ipynb +++ b/tamingllms/_build/html/_sources/notebooks/local.ipynb @@ -23,13 +23,13 @@ "\n", "Running LLMs locally versus using cloud APIs offers several important advantages.\n", "\n", - "Privacy-sensitive data processing is one of the primary reasons for running LLMs locally. Organizations handling medical records must comply with HIPAA regulations that require data to remain on-premise. Similarly, businesses processing confidential documents and intellectual property, as well as organizations subject to GDPR and other privacy regulations, need to maintain strict control over their data processing pipeline.\n", + "Privacy concerns are a key driver for running LLMs locally. Individual users may want to process personal documents, photos, emails, and chat messages without sharing sensitive data with third parties. For enterprise use cases, organizations handling medical records must comply with HIPAA regulations that require data to remain on-premise. Similarly, businesses processing confidential documents and intellectual property, as well as organizations subject to GDPR and other privacy regulations, need to maintain strict control over their data processing pipeline.\n", "\n", - "Cost considerations become significant when operating at scale. Organizations running high-volume applications can face prohibitive API costs with cloud-based solutions. Development and testing environments that require frequent model interactions, educational institutions supporting multiple student projects, and research initiatives involving extensive model experimentation can all achieve substantial cost savings through local deployment.\n", + "Cost considerations are another key advantage of local deployment. Organizations can better control expenses by matching model capabilities to their specific needs rather than paying for potentially excessive cloud API features. For high-volume applications, this customization and control over costs becomes especially valuable compared to the often prohibitive per-request pricing of cloud solutions.\n", "\n", - "Applications with stringent latency requirements form another important category. Real-time systems where network delays would be unacceptable, edge computing scenarios demanding quick responses, and interactive applications requiring sub-second performance all benefit from local deployment. This extends to embedded systems in IoT devices where cloud connectivity might be unreliable or impractical.\n", + "Applications with stringent latency requirements form another important category. Real-time systems where network delays would be unacceptable, edge computing scenarios demanding quick responses, and interactive applications requiring sub-second performance all benefit from local deployment. This extends to embedded systems in IoT devices where cloud connectivity might be unreliable or impractical. Further, the emergence of Small Language Models (SLMs) has made edge deployment increasingly viable, enabling sophisticated language capabilities on resource-constrained devices like smartphones, tablets and IoT sensors. \n", "\n", - "Finally, local deployment enables deeper customization and fine-tuning capabilities. Organizations can perform specialized domain adaptation through model modifications, experiment with different architectures and parameters, and integrate models with proprietary systems and workflows. This flexibility is particularly valuable for developing novel applications that require direct model access and manipulation." + "Running locally also enables fine-grained optimization of resource usage and model characteristics based on target use case. Organizations can perform specialized domain adaptation through model modifications, experiment with different architectures and parameters, and integrate models with proprietary systems and workflows. This flexibility is particularly valuable for developing novel applications that require direct model access and manipulation. " ] }, { diff --git a/tamingllms/_build/html/notebooks/local.html b/tamingllms/_build/html/notebooks/local.html index 39a4a1e..609a27e 100644 --- a/tamingllms/_build/html/notebooks/local.html +++ b/tamingllms/_build/html/notebooks/local.html @@ -277,10 +277,10 @@

8.1. Introduction

Running LLMs locally versus using cloud APIs offers several important advantages.

-

Privacy-sensitive data processing is one of the primary reasons for running LLMs locally. Organizations handling medical records must comply with HIPAA regulations that require data to remain on-premise. Similarly, businesses processing confidential documents and intellectual property, as well as organizations subject to GDPR and other privacy regulations, need to maintain strict control over their data processing pipeline.

-

Cost considerations become significant when operating at scale. Organizations running high-volume applications can face prohibitive API costs with cloud-based solutions. Development and testing environments that require frequent model interactions, educational institutions supporting multiple student projects, and research initiatives involving extensive model experimentation can all achieve substantial cost savings through local deployment.

-

Applications with stringent latency requirements form another important category. Real-time systems where network delays would be unacceptable, edge computing scenarios demanding quick responses, and interactive applications requiring sub-second performance all benefit from local deployment. This extends to embedded systems in IoT devices where cloud connectivity might be unreliable or impractical.

-

Finally, local deployment enables deeper customization and fine-tuning capabilities. Organizations can perform specialized domain adaptation through model modifications, experiment with different architectures and parameters, and integrate models with proprietary systems and workflows. This flexibility is particularly valuable for developing novel applications that require direct model access and manipulation.

+

Privacy concerns are a key driver for running LLMs locally. Individual users may want to process personal documents, photos, emails, and chat messages without sharing sensitive data with third parties. For enterprise use cases, organizations handling medical records must comply with HIPAA regulations that require data to remain on-premise. Similarly, businesses processing confidential documents and intellectual property, as well as organizations subject to GDPR and other privacy regulations, need to maintain strict control over their data processing pipeline.

+

Cost considerations are another key advantage of local deployment. Organizations can better control expenses by matching model capabilities to their specific needs rather than paying for potentially excessive cloud API features. For high-volume applications, this customization and control over costs becomes especially valuable compared to the often prohibitive per-request pricing of cloud solutions.

+

Applications with stringent latency requirements form another important category. Real-time systems where network delays would be unacceptable, edge computing scenarios demanding quick responses, and interactive applications requiring sub-second performance all benefit from local deployment. This extends to embedded systems in IoT devices where cloud connectivity might be unreliable or impractical. Further, the emergence of Small Language Models (SLMs) has made edge deployment increasingly viable, enabling sophisticated language capabilities on resource-constrained devices like smartphones, tablets and IoT sensors.

+

Running locally also enables fine-grained optimization of resource usage and model characteristics based on target use case. Organizations can perform specialized domain adaptation through model modifications, experiment with different architectures and parameters, and integrate models with proprietary systems and workflows. This flexibility is particularly valuable for developing novel applications that require direct model access and manipulation.

8.2. Local Models Considerations

diff --git a/tamingllms/_build/html/searchindex.js b/tamingllms/_build/html/searchindex.js index b1e7df3..5e4d2c5 100644 --- a/tamingllms/_build/html/searchindex.js +++ b/tamingllms/_build/html/searchindex.js @@ -1 +1 @@ -Search.setIndex({"docnames": ["markdown/intro", "markdown/preface", "markdown/toc", "notebooks/alignment", "notebooks/evals", "notebooks/local", "notebooks/output_size_limit", "notebooks/safety", "notebooks/structured_output"], "filenames": ["markdown/intro.md", "markdown/preface.md", "markdown/toc.md", "notebooks/alignment.ipynb", "notebooks/evals.ipynb", "notebooks/local.ipynb", "notebooks/output_size_limit.ipynb", "notebooks/safety.ipynb", "notebooks/structured_output.ipynb"], "titles": ["2. Introduction", "1. Preface", "Taming LLMs", "7. Preference-Based Alignment", "5. The Evals Gap", "8. Breaking Free from Cloud Providers", "3. Output Size Limit", "6. Safety", "4. Wrestling with Structured Output"], "terms": {"am": [0, 1, 7], "alwai": [0, 3, 4, 8], "do": [0, 3, 4, 5, 6, 7, 8], "which": [0, 3, 4, 5, 6, 7, 8], "cannot": [0, 3, 4, 5, 7], "order": [0, 3, 4, 7, 8], "mai": [0, 1, 3, 4, 5, 6, 7, 8], "learn": [0, 3, 4, 7], "how": [0, 1, 3, 4, 5, 6, 7, 8], "pablo": [0, 4], "picasso": 0, "In": [0, 3, 4, 5, 6, 7, 8], "recent": [0, 3, 4, 7, 8], "year": [0, 2, 3, 4, 5, 6, 7, 8], "larg": [0, 1, 2, 3, 4, 5, 6, 7, 8], "languag": [0, 1, 2, 4, 5, 6, 7, 8], "model": [0, 1, 2, 7, 8], "llm": [0, 1, 3, 6, 8], "have": [0, 1, 3, 4, 5, 6, 7, 8], "emerg": [0, 3, 7, 8], "transform": [0, 1, 3, 4, 7, 8], "forc": [0, 4, 8], "technologi": [0, 1, 4, 6, 7, 8], "promis": [0, 3, 4, 7], "revolution": [0, 7], "build": [0, 2, 3, 4, 5, 6, 7, 8], "product": [0, 1, 2, 3, 4, 5, 7, 8], "interact": [0, 3, 4, 5, 6, 7, 8], "comput": [0, 3, 4, 5, 6, 7, 8], "from": [0, 1, 4, 6, 7, 8], "chatgpt": [0, 3, 5, 8], "github": [0, 2, 3, 4, 5, 6, 7, 8], "copilot": 0, "claud": [0, 3, 4, 5, 6, 7], "artifact": 0, "system": [0, 3, 4, 5, 6, 7, 8], "captur": [0, 1, 3, 4, 7], "public": [0, 3, 4, 7], "imagin": 0, "spark": 0, "gold": [0, 3, 4, 7], "rush": 0, "ai": [0, 3, 4, 5, 8], "power": [0, 2, 3, 4, 5, 6, 7, 8], "applic": [0, 1, 2, 3, 5, 6, 7, 8], "howev": [0, 3, 4, 5, 6, 7, 8], "beneath": 0, "surfac": [0, 4], "technolog": [0, 1, 4, 7], "revolut": 0, "li": [0, 3, 4, 7], "complex": [0, 1, 3, 4, 5, 6, 7, 8], "landscap": [0, 3, 4], "practition": [0, 1, 4, 5], "must": [0, 3, 4, 5, 6, 7], "navig": [0, 2, 4, 7], "focus": [0, 3, 4, 5, 6, 7, 8], "bring": [0, 3, 5], "awar": [0, 3, 4, 6, 7], "limit": [0, 1, 4, 5, 7, 8], "har": [0, 2, 4, 6], "solut": [0, 2, 4, 5, 6, 7], "overcom": [0, 4, 6], "them": [0, 1, 3, 4, 5, 6, 7, 8], "robust": [0, 3, 4, 5, 6, 7, 8], "It": [0, 3, 4, 5, 6, 7, 8], "offer": [0, 3, 4, 5, 6, 7, 8], "critic": [0, 2, 3, 4, 6, 7, 8], "implement": [0, 2, 3, 4, 5, 6, 8], "back": [0, 4, 7, 8], "reproduc": [0, 1, 2, 4, 5], "exampl": [0, 1, 2, 3, 4, 5, 7, 8], "while": [0, 1, 2, 3, 4, 5, 6, 7, 8], "mani": [0, 1, 3, 4, 5, 6, 7, 8], "resourc": [0, 3, 4, 5, 6, 7], "cover": [0, 3, 4, 5, 6, 7], "capabl": [0, 1, 2, 4, 5, 6, 7, 8], "specif": [0, 3, 4, 5, 6], "hidden": [0, 3, 7], "pitfal": [0, 1, 3, 4, 5, 6, 8], "engin": [0, 1, 2, 3, 4, 5, 7, 8], "technic": [0, 1, 2, 3, 4, 5, 6, 8], "manag": [0, 1, 2, 4, 5, 6, 7, 8], "face": [0, 3, 4, 5, 7], "when": [0, 1, 2, 3, 4, 5, 6, 7, 8], "comprehens": [0, 2, 3, 4, 5, 6, 7, 8], "guid": [0, 1, 3, 4, 5, 6, 7, 8], "leverag": [0, 3, 4, 6, 7, 8], "battl": [0, 2], "test": [0, 2, 3, 5, 7, 8], "tool": [0, 1, 3, 6], "throughout": [0, 4, 5, 6, 7, 8], "tackl": [0, 3, 4, 7], "follow": [0, 3, 4, 5, 6, 7, 8], "non": [0, 3, 5, 7, 8], "exhaust": 0, "list": [0, 3, 4, 5, 6, 7, 8], "structur": [0, 3, 4, 5, 6, 7], "un": 0, "reliabl": [0, 1, 3, 4, 5, 7, 8], "struggl": [0, 1, 3, 4, 7, 8], "maintain": [0, 1, 3, 4, 5, 6, 7, 8], "consist": [0, 1, 3, 4, 5, 6, 7, 8], "output": [0, 1, 3, 4, 5, 7], "format": [0, 3, 4, 5, 6, 7, 8], "complic": [0, 7], "integr": [0, 1, 3, 4, 5, 7, 8], "larger": [0, 3, 4, 5, 6, 7, 8], "make": [0, 3, 4, 5, 6, 7, 8], "error": [0, 3, 4, 7, 8], "handl": [0, 2, 3, 4, 5, 6, 7, 8], "more": [0, 1, 3, 4, 5, 6, 7, 8], "size": [0, 3, 4, 5, 7, 8], "length": [0, 3, 4, 5, 8], "constraint": [0, 1, 3, 4, 6, 7, 8], "strict": [0, 5, 7, 8], "token": [0, 1, 3, 4, 5, 7, 8], "both": [0, 3, 4, 5, 7], "input": [0, 3, 4, 5, 6, 7, 8], "requir": [0, 3, 5, 6, 7, 8], "care": [0, 3, 4, 7, 8], "chunk": [0, 3], "strategi": [0, 3, 4, 5, 6, 7], "long": [0, 1, 3, 4, 7, 8], "form": [0, 3, 4, 5, 7, 8], "effect": [0, 1, 3, 4, 6, 7, 8], "tradit": [0, 3, 7], "softwar": [0, 1, 3, 5, 6, 7, 8], "methodologi": [0, 3, 4, 7, 8], "break": [0, 1, 3, 4, 6, 7], "down": [0, 1, 4, 5, 6, 7], "deal": [0, 3, 5], "determinist": [0, 8], "gener": [0, 1, 5, 8], "new": [0, 2, 3, 4, 5, 6, 7, 8], "hallucin": [0, 1, 3, 4, 7, 8], "These": [0, 3, 4, 5, 6, 7, 8], "can": [0, 1, 3, 4, 5, 6, 7, 8], "plausibl": [0, 7], "sound": [0, 7], "entir": [0, 4, 5, 6, 8], "fabric": [0, 4, 7], "inform": [0, 3, 4, 5, 6, 7, 8], "creat": [0, 1, 3, 4, 5, 6, 7, 8], "signific": [0, 3, 4, 5, 6, 7, 8], "risk": [0, 1, 3, 4, 6], "safeti": [0, 3, 4, 8], "align": [0, 4, 5, 6, 7, 8], "harm": [0, 3, 4, 5], "bias": [0, 3, 4, 7, 8], "inappropri": [0, 3, 7], "safeguard": [0, 4, 7], "monitor": [0, 3, 4, 5, 7], "ensur": [0, 3, 4, 5, 6, 7, 8], "safe": [0, 3, 4, 7, 8], "deploy": [0, 3, 4, 7, 8], "cost": [0, 3, 4, 5, 7, 8], "optim": [0, 1, 4, 5, 6, 7], "The": [0, 1, 3, 6, 7, 8], "financi": [0, 1, 3, 4, 6, 7, 8], "oper": [0, 3, 4, 5, 6, 7, 8], "base": [0, 1, 5, 8], "quickli": [0, 3, 6], "becom": [0, 3, 4, 5, 7, 8], "prohibit": [0, 3, 4, 5], "without": [0, 1, 3, 4, 5, 6, 7, 8], "observ": [0, 3, 4, 5, 7, 8], "vendor": [0, 4], "lock": [0, 3], "cloud": [0, 3, 4, 7, 8], "provid": [0, 3, 4, 6, 7], "depend": [0, 3, 4, 5, 8], "through": [0, 1, 2, 3, 4, 5, 6, 7, 8], "proprietari": [0, 3, 5, 7, 8], "infrastructur": [0, 5], "difficult": [0, 3, 4, 7], "switch": [0, 5], "self": [0, 3, 4, 5, 7], "host": [0, 4, 5, 7], "take": [0, 2, 3, 4, 5, 6, 7, 8], "hand": [0, 6, 7, 8], "focu": [0, 2, 3, 4, 5, 6, 7, 8], "access": [0, 3, 4, 5, 6, 7, 8], "all": [0, 1, 3, 4, 5, 6, 7, 8], "ar": [0, 1, 3, 4, 5, 7, 8], "fulli": [0, 3, 4, 6, 7], "document": [0, 3, 4, 5, 6, 7, 8], "allow": [0, 4, 5, 6, 7, 8], "reader": [0, 2], "replic": [0, 4, 7, 8], "result": [0, 3, 4, 6, 7, 8], "exactli": [0, 4, 8], "design": [0, 1, 3, 5, 6, 8], "run": [0, 3, 4, 5, 7, 8], "consum": [0, 3, 4, 5, 7, 8], "grade": [0, 3, 4, 5, 7], "hardwar": [0, 3, 4], "expens": [0, 3, 4, 7], "avail": [0, 3, 4, 5, 6, 7, 8], "notebook": [0, 3, 8], "modifi": [0, 3, 4, 7], "extend": [0, 3, 4, 5, 8], "built": [0, 4, 5, 7, 8], "us": [0, 1, 3, 5, 6, 7, 8], "free": [0, 1, 3, 4, 7], "everyon": [0, 4, 5], "minim": [0, 3, 4, 5, 7, 8], "framework": [0, 3, 4], "wai": [0, 3, 4, 5, 6, 7, 8], "priorit": [0, 3, 4, 5, 7], "transpar": [0, 3, 4, 7, 8], "visibl": [0, 4], "being": [0, 3, 4, 5, 7], "better": [0, 2, 3, 4, 5, 6, 7], "understand": [0, 1, 2, 3, 4, 5, 6, 7, 8], "custom": [0, 3, 4, 5], "flexibl": [0, 4, 5, 6, 7, 8], "adapt": [0, 3, 4, 5, 6, 7], "case": [0, 4, 6, 8], "unlik": [0, 3, 4, 5], "black": [0, 3], "box": [0, 5], "commerci": [0, 3, 4, 7, 8], "most": [0, 3, 4, 5, 6, 7, 8], "freeli": [0, 8], "foster": [0, 3, 4, 7, 8], "reduc": [0, 3, 4, 5, 6, 7, 8], "independ": [0, 4, 7, 8], "freedom": [0, 5, 8], "architectur": [0, 3, 4, 5, 6, 8], "decis": [0, 3, 4, 5, 7, 8], "keep": [0, 3, 4, 5, 6, 7], "principl": [0, 3, 4, 5, 7], "itself": [0, 3, 4, 5, 7], "live": [0, 1, 4, 7], "evolv": [0, 3, 4, 6, 7], "chang": [0, 3, 4, 5, 7], "encourag": [0, 3, 4, 7, 8], "report": [0, 3, 4, 5, 7, 8], "suggest": [0, 3, 4, 7, 8], "improv": [0, 3, 4, 5, 6, 7, 8], "contribut": [0, 4, 6, 7], "via": [0, 3, 4, 5, 7, 8], "pull": [0, 5], "request": [0, 3, 4, 5, 6, 7, 8], "share": [0, 3, 4, 5, 7, 8], "own": [0, 3, 4, 6, 7], "experi": [0, 3, 4, 5, 6, 7, 8], "commun": [0, 3, 4, 7, 8], "propos": [0, 4, 7], "chapter": [0, 3, 4, 5, 6, 7, 8], "section": [0, 3, 4, 5, 6, 7, 8], "found": [0, 3, 4, 5, 8], "http": [0, 1, 2, 3, 4, 5, 6, 7, 8], "com": [0, 2, 3, 4, 5, 6, 7, 8], "souzatharsi": [0, 2, 3, 4, 5, 6, 7, 8], "tamingllm": [0, 2, 3, 4, 5, 6, 7, 8], "whether": [0, 3, 4, 6, 7, 8], "you": [0, 1, 3, 4, 5, 6, 7, 8], "ve": [0, 5], "typo": [0, 7], "want": [0, 1, 3, 5, 6, 7, 8], "welcom": 0, "look": [0, 2, 3, 4, 5, 7], "our": [0, 1, 3, 4, 5, 6, 7, 8], "goal": [0, 1, 3, 4, 6, 7], "discourag": 0, "enabl": [0, 3, 4, 5, 6, 7, 8], "By": [0, 1, 2, 3, 4, 6, 7, 8], "upfront": [0, 2], "equip": [0, 2, 4, 7], "avoid": [0, 3, 4, 7, 8], "current": [0, 2, 3, 4, 6, 7, 8], "discours": [0, 2], "around": [0, 2, 3, 4, 6, 7, 8], "tend": [0, 2, 4, 7], "toward": [0, 3, 4, 7, 8], "extrem": [0, 3, 4, 7], "either": [0, 3, 4, 6, 7], "uncrit": 0, "enthusiasm": 0, "wholesal": [0, 4], "dismiss": 0, "differ": [0, 3, 4, 5, 6, 7, 8], "rather": [0, 1, 3, 4, 5, 7], "than": [0, 1, 3, 4, 5, 7], "theoret": [0, 3], "examin": [0, 3, 4, 5, 6, 7, 8], "first": [0, 1, 3, 4, 5, 6, 7, 8], "everi": [0, 4, 7], "concept": [0, 3, 4, 7], "illustr": [0, 3, 4, 6, 7, 8], "execut": [0, 4, 5, 7], "immedi": [0, 3, 4, 5], "analysi": [0, 1, 3, 4, 5, 6, 7], "balanc": [0, 3, 4, 5, 6, 7, 8], "help": [0, 3, 4, 5, 6, 7, 8], "intend": [0, 4, 7], "develop": [0, 1, 3, 4, 5, 6, 7, 8], "step": [0, 1, 3, 4, 5, 7, 8], "insight": [0, 3, 4, 5, 6, 7, 8], "along": [0, 3, 4, 5, 7], "guidanc": [0, 3, 8], "could": [0, 1, 3, 4, 5, 6, 7, 8], "derail": 0, "project": [0, 3, 4, 5, 7], "earli": [0, 3, 4, 7, 8], "befor": [0, 3, 4, 5, 7, 8], "thei": [0, 1, 3, 4, 5, 6, 7, 8], "costli": [0, 4, 7], "problem": [0, 1, 2, 3, 5, 7], "too": [0, 1, 3, 4, 6, 7], "late": [0, 3, 7], "lifecycl": [0, 7], "lead": [0, 1, 3, 4, 6, 7, 8], "genai": [0, 1, 3, 7], "initi": [0, 1, 3, 4, 5, 6, 7, 8], "leader": [0, 4], "advoc": [0, 7], "anyon": [0, 7], "seek": [0, 4, 5, 7], "work": [0, 1, 3, 4, 5, 6, 7, 8], "typic": [0, 3, 4, 6, 7, 8], "job": [0, 4, 5, 7], "role": [0, 3, 4, 5, 6, 7, 8], "platform": [0, 4, 5, 6, 7, 8], "backend": [0, 3, 4], "exist": [0, 3, 4, 5], "ml": [0, 7], "transit": [0, 4, 5, 6, 8], "overse": 0, "motiv": [0, 3, 4, 8], "need": [0, 3, 4, 5, 6, 7], "readi": [0, 4, 7], "desir": [0, 3, 4, 8], "perform": [0, 3, 4, 6, 7, 8], "after": [0, 1, 3, 4, 5, 6, 7, 8], "read": [0, 3, 4, 6, 7, 8], "implic": [0, 1, 3, 4, 7], "recommend": [0, 3, 4, 6, 7, 8], "abl": [0, 3, 4, 6, 8], "deploi": [0, 3, 4, 5, 6, 7], "proper": [0, 3, 5, 7, 8], "realist": [0, 3, 7], "effort": [0, 4, 7, 8], "estim": [0, 4, 7], "impact": [0, 3, 4, 5, 6, 7, 8], "timelin": 0, "To": [0, 3, 4, 5, 6, 7, 8], "should": [0, 3, 4, 6, 7, 8], "basic": [0, 3, 4, 5, 6, 7], "program": [0, 4, 5], "knowledg": [0, 3, 4, 5, 7], "introductori": [0, 1, 2], "langchain": [0, 4, 6], "e": [0, 1, 3, 4, 5, 6, 7, 8], "g": [0, 3, 4, 5, 6, 7, 8], "chat": [0, 3, 4, 5, 6, 7, 8], "prompt": [0, 4, 7], "templat": [0, 4], "openai": [0, 3, 4, 5, 8], "anthrop": [0, 3, 8], "similar": [0, 3, 4, 5, 8], "dive": 0, "here": [0, 2, 3, 4, 5, 6, 7, 8], "get": [0, 3, 4, 5, 6, 7, 8], "start": [0, 3, 4, 5, 7, 8], "clone": [0, 3], "companion": 0, "git": 0, "cd": 0, "activ": [0, 3, 4, 7], "virtual": [0, 4], "m": [0, 3, 4, 5, 7, 8], "venv": 0, "tame": [0, 3, 4, 5, 6, 7, 8], "env": [0, 3, 4, 6, 7, 8], "bin": [0, 5], "On": [0, 4, 5, 8], "window": [0, 2, 4, 5], "script": [0, 5], "try": [0, 1, 3, 4, 7, 8], "contain": [0, 3, 4, 5, 6, 7, 8], "possibl": [0, 3, 4, 5, 7, 8], "includ": [0, 1, 3, 4, 5, 6, 7, 8], "necessari": [0, 3, 4, 6, 7], "instal": [0, 3, 4, 5, 8], "go": [0, 3, 4, 6, 8], "feel": [0, 5], "prefer": [0, 4, 5, 7, 8], "packag": [0, 4, 5, 8], "pip": [0, 3, 4, 5, 8], "poetri": [0, 7], "file": [0, 3, 4, 5, 6, 7, 8], "root": [0, 3], "directori": [0, 4, 5], "add": [0, 3, 4, 5, 6, 7], "other": [0, 3, 4, 5, 6, 7, 8], "sensit": [0, 3, 4, 5, 7], "openai_api_kei": [0, 3], "your_openai_api_key_her": 0, "never": [0, 8], "commit": [0, 3, 4, 7], "version": [0, 3, 4, 5, 7, 8], "control": [0, 1, 3, 4, 5, 7, 8], "kept": [0, 4], "privat": [0, 4], "If": [0, 1, 3, 4, 5, 7, 8], "encount": [0, 2, 4, 7], "rate": [0, 3, 4, 7], "consid": [0, 3, 4, 6, 7, 8], "smaller": [0, 3, 4, 6, 8], "retri": [0, 8], "logic": [0, 1, 3, 4, 6, 7], "conflict": [0, 3, 4], "fresh": 0, "like": [0, 1, 3, 4, 5, 6, 7, 8], "check": [0, 4, 5, 7, 8], "page": [0, 4], "known": [0, 4, 7, 8], "now": [0, 1, 3, 4, 5, 6, 7, 8], "let": [0, 3, 4, 5, 6, 7, 8], "begin": [0, 4, 7, 8], "explor": [0, 1, 3, 4, 5, 7, 8], "dr": [0, 3], "tharsi": [0, 2, 3, 4, 5, 6, 7, 8], "souza": [0, 2, 3, 4, 5, 6, 7, 8], "scientist": [0, 1, 5, 7], "special": [0, 4, 5, 7, 8], "he": [0, 3, 4, 7], "lectur": 0, "columbia": 0, "univers": [0, 4, 5, 7], "master": [0, 5, 8], "scienc": [0, 3, 4, 7], "appli": [0, 3, 4, 5, 6, 7, 8], "analyt": 0, "incom": [0, 4], "head": [0, 3, 4, 6, 7], "equiti": [0, 4], "citadel": 0, "former": [0, 1, 4, 5], "senior": [0, 4], "vp": 0, "two": [0, 3, 4, 5, 6, 7, 8], "sigma": [0, 3], "invest": [0, 3, 4, 7, 8], "also": [0, 3, 4, 5, 6, 7, 8], "enjoi": 0, "mentor": 0, "under": [0, 3, 4, 7, 8], "repres": [0, 3, 4, 5, 8], "student": [0, 3, 5, 7], "profession": [0, 3, 4, 7, 8], "divers": [0, 3, 4, 6, 7, 8], "global": [0, 4, 7], "ecosystem": [0, 4], "With": [0, 3, 4, 5, 7], "over": [0, 2, 3, 4, 5, 6, 7, 8], "15": [0, 4, 7, 8], "deliv": [0, 4, 5], "across": [0, 1, 3, 4, 5, 7, 8], "startup": 0, "fortun": 0, "500": [0, 3, 4, 7], "compani": [0, 3, 4, 6, 7, 8], "numer": [0, 4, 7], "scholarli": 0, "frequent": [0, 4, 5, 8], "speaker": [0, 4], "academ": [0, 3, 4, 7], "busi": [0, 4, 5, 7], "confer": [0, 8], "ground": [0, 3, 4], "background": [0, 1, 4, 5, 6], "draw": [0, 3, 4, 7, 8], "scale": [0, 3, 4, 5, 7, 8], "stage": [0, 3, 7, 8], "major": [0, 3, 4, 7, 8], "institut": [0, 4, 5, 7], "well": [0, 3, 4, 5, 7, 8], "advis": [0, 3], "profit": [0, 4, 6, 7, 8], "organ": [0, 3, 4, 5, 6], "uniqu": [0, 3, 4, 7], "bridg": [0, 5, 7], "gap": [0, 1, 3, 7], "between": [0, 1, 3, 4, 5, 6, 7, 8], "potenti": [0, 1, 3, 4, 5, 6, 7, 8], "next": [0, 1, 3, 4, 5, 7, 8], "hold": [0, 3, 4], "ph": [0, 7], "d": [0, 3, 4, 5, 7, 8], "ucl": 0, "london": 0, "phil": [0, 7], "sc": 0, "b": [0, 4, 5, 7, 8], "tell": [1, 3, 7], "mere": [1, 4], "what": [1, 3, 4, 5, 7, 8], "someth": [1, 4, 5], "i": [1, 2, 4, 5, 6, 7, 8], "emanuel": [1, 3, 4, 7], "derman": 1, "an": [1, 2, 3, 4, 5, 6, 7, 8], "altern": [1, 3, 4, 5, 6, 7], "titl": [1, 2, 3, 4, 5, 6, 7, 8], "thi": [1, 2, 3, 4, 5, 6, 7, 8], "book": [1, 2, 4], "been": [1, 3, 4, 5, 7], "behav": 1, "badli": 1, "come": [1, 3, 4, 5, 6, 7, 8], "notic": [1, 3, 4, 7, 8], "parallel": [1, 3, 4, 5], "": [1, 3, 4, 5, 6, 7, 8], "semin": [1, 7], "2011": 1, "coincident": 1, "just": [1, 3, 4, 5, 6, 7, 8], "caution": 1, "against": [1, 3, 4, 5, 7], "treat": [1, 4, 7], "perfect": [1, 4, 5], "represent": [1, 4, 6, 7], "realiti": [1, 7], "aim": [1, 3, 4, 5, 6, 7, 8], "highlight": [1, 3, 4, 5, 6, 7, 8], "practic": [1, 3, 4, 5, 6, 7], "cours": [1, 4, 7], "bare": [1, 5], "fact": [1, 3, 4, 7], "actual": [1, 3, 4, 6, 7, 8], "physicist": 1, "legendari": 1, "author": [1, 2, 3, 4, 5, 6, 7, 8], "professor": 1, "quant": 1, "goldman": 1, "sach": 1, "scientif": [1, 3, 4], "fail": [1, 3, 4, 7], "we": [1, 3, 4, 5, 6, 7, 8], "mistak": [1, 7], "approxim": [1, 4, 8], "full": [1, 3, 4, 5, 7, 8], "assumpt": [1, 4, 7], "core": [1, 4, 5, 7], "premis": [1, 5], "hi": [1, 4, 7, 8], "aspect": [1, 3, 4, 6, 7, 8], "world": [1, 3, 4, 7, 8], "inher": [1, 2, 3, 4, 7, 8], "involv": [1, 3, 4, 5, 7, 8], "simplif": 1, "argu": [1, 7, 8], "crise": 1, "2008": 1, "crash": 1, "occur": [1, 3, 4, 7], "partli": 1, "becaus": [1, 3, 4, 7], "peopl": [1, 3, 4, 5, 7], "put": [1, 4], "much": [1, 3, 4, 5], "faith": 1, "mathemat": [1, 4], "recogn": [1, 3, 4, 7], "human": [1, 4, 6, 7, 8], "behavior": [1, 3, 4, 5, 7], "market": [1, 4, 6, 8], "dynam": [1, 3, 4, 7], "reason": [1, 3, 4, 5, 6, 7, 8], "Their": [1, 4, 8], "respons": [1, 4, 5, 6, 7, 8], "often": [1, 3, 4, 5, 6, 7, 8], "convinc": [1, 3], "probabilist": [1, 4], "train": [1, 4, 5, 7, 8], "data": [1, 4, 5, 6, 7, 8], "true": [1, 3, 4, 6, 7, 8], "even": [1, 3, 4, 5, 6, 7, 8], "though": [1, 3, 4, 5, 7, 8], "insist": 1, "machin": [1, 3, 5, 7, 8], "todai": [1, 5, 8], "grow": [1, 3, 4, 5, 7, 8], "pervas": [1, 7], "belief": [1, 5, 7], "solv": [1, 3, 4, 5, 7, 8], "ani": [1, 3, 4, 5, 6, 7, 8], "context": [1, 2, 3, 4, 5, 6, 7, 8], "content": 1, "wish": [1, 4], "user": [1, 4, 5, 6], "moreov": 1, "were": [1, 3, 4, 7, 8], "predict": [1, 3, 4, 5, 7, 8], "chatbot": [1, 3, 4, 5, 7], "twist": [1, 7], "wrap": [1, 5, 8], "further": [1, 3, 4, 6, 7, 8], "daili": [1, 7], "life": [1, 4, 5, 7], "workflow": [1, 4, 5, 7, 8], "affect": [1, 4, 5, 7], "decid": [1, 3, 4, 6], "action": [1, 3, 4, 6, 7], "coupl": 1, "lack": [1, 3, 4, 7, 8], "pose": [1, 3, 4, 6, 7, 8], "still": [1, 4, 5, 7], "figur": [1, 4, 8], "out": [1, 3, 4, 5, 6, 7, 8], "serv": [1, 3, 4, 6, 7, 8], "builder": [1, 5], "who": [1, 3, 4, 5, 6, 7, 8], "remain": [1, 3, 4, 5, 6, 7], "clear": [1, 3, 4, 7, 8], "ei": 1, "about": [1, 3, 4, 5, 6, 7, 8], "therefor": [1, 3, 4, 5, 7], "end": [1, 3, 4, 5, 6, 7, 8], "detail": [1, 3, 4, 5, 6, 7, 8], "python": [1, 2, 4, 5, 6, 7, 8], "code": [1, 2, 3, 4, 5, 7, 8], "diminish": [1, 3, 4], "promot": [1, 3, 4, 7], "nuanc": [1, 3, 4, 6, 7, 8], "acknowledg": [1, 4, 7], "within": [1, 3, 4, 6, 7, 8], "trustworthi": [1, 7], "taught": 1, "u": [1, 3, 4, 7, 8], "where": [1, 3, 4, 5, 6, 7, 8], "der11": 1, "why": [1, 3, 4, 7, 8], "confus": [1, 7], "illus": 1, "disast": [1, 4], "wall": [1, 5], "street": [1, 5], "press": [1, 4, 5], "isbn": [1, 3, 4], "9781439165010": 1, "url": [1, 2, 3, 4, 5, 6, 7, 8], "googl": [1, 4, 8], "co": [1, 3, 4, 5, 7], "uk": [1, 7], "id": [1, 4, 5, 7], "lke_cwm4wm8c": 1, "sign": [2, 4, 7], "up": [2, 3, 4, 5, 6, 7, 8], "receiv": [2, 3, 4, 6, 7, 8], "updat": [2, 3, 4, 5, 6, 7, 8], "abstract": [2, 4, 7, 8], "heavili": [2, 3, 4, 7, 8], "gloss": 2, "fundament": [2, 3, 4, 7, 8], "convers": [2, 3, 4, 5, 6, 7, 8], "kei": [2, 3, 5, 7, 8], "proven": 2, "yet": [2, 3, 4, 6, 7], "concret": [2, 7], "unstructur": [2, 8], "sidestep": 2, "misc": [2, 3, 4, 5, 6, 7, 8], "tharsistpsouza2024tamingllm": [2, 3, 4, 5, 6, 7, 8], "t": [2, 3, 4, 5, 6, 7, 8], "p": [2, 3, 4, 5, 6, 7, 8], "2024": [2, 3, 4, 5, 6, 7, 8], "journal": [2, 3, 4, 5, 6, 7, 8], "repositori": [2, 3, 4, 5, 6, 7, 8], "valu": [3, 4, 5, 6, 7, 8], "its": [3, 4, 5, 6, 7, 8], "privileg": 3, "abov": [3, 4, 7], "soon": [3, 8], "lose": [3, 4], "dwight": 3, "eisenhow": 3, "releas": [3, 4, 5, 7, 8], "3": [3, 4, 5, 8], "5": [3, 4, 5, 6, 8], "2022": [3, 4, 7], "mark": [3, 4, 7], "pivot": [3, 4], "moment": 3, "histori": [3, 4, 5], "artifici": [3, 4, 7], "intellig": [3, 4, 7], "five": [3, 4, 7], "dai": [3, 4, 7, 8], "launch": [3, 4, 7], "attract": [3, 4], "million": [3, 4], "month": [3, 4, 7], "becam": 3, "fastest": [3, 4, 7], "100": [3, 4, 5, 7, 8], "monthli": [3, 4], "rais": [3, 4, 6, 7], "intrigu": 3, "question": [3, 4, 5, 7, 8], "did": [3, 4, 8], "dramat": [3, 4, 5, 8], "predecessor": 3, "gpt": [3, 4, 5, 6, 7, 8], "had": [3, 4], "same": [3, 4, 5, 6, 7, 8], "number": [3, 4, 5, 6, 7, 8], "paramet": [3, 4, 5, 7, 8], "far": [3, 6, 7], "less": [3, 4, 5, 7], "attent": [3, 5], "arguabl": [3, 5], "feedback": [3, 4, 7, 8], "abil": [3, 4, 5, 7, 8], "least": [3, 4, 7], "ey": 3, "breakthrough": [3, 7], "demonstr": [3, 4, 5, 6, 7, 8], "crucial": [3, 5, 7, 8], "greater": [3, 4, 7], "process": [3, 4, 5, 6, 7, 8], "modern": [3, 4, 6, 8], "techniqu": [3, 4, 5, 6], "direct": [3, 4, 5, 7], "rafailov": 3, "et": [3, 4, 7, 8], "al": [3, 4, 7, 8], "present": [3, 4, 5, 6, 7, 8], "autom": [3, 4, 7, 8], "fashion": [3, 8], "open": [3, 4, 6, 7, 8], "sourc": [3, 4, 5, 6, 7, 8], "common": [3, 4, 5, 6, 8], "pre": [3, 4, 5, 7], "default": [3, 4, 5, 7, 8], "state": [3, 4, 5, 6, 7, 8], "art": [3, 4, 7], "object": [3, 4, 5, 7, 8], "given": [3, 4, 5, 6, 7, 8], "webpag": 3, "internet": [3, 4], "veri": [3, 4, 5, 7], "ask": [3, 4, 5, 7, 8], "instruct": [3, 4, 5, 6, 7, 8], "sai": [3, 8], "ouyang": [3, 7], "2": [3, 4, 5, 8], "explain": 3, "moon": 3, "land": [3, 4], "6": [3, 4, 5, 6, 8], "old": [3, 4], "import": [3, 4, 5, 6, 7, 8], "pipelin": [3, 4, 5, 7, 8], "pipe": [3, 7], "text": [3, 4, 5, 6, 7, 8], "gpt2": [3, 4], "msg": 3, "short": [3, 4, 6, 7, 8], "sentenc": [3, 4, 6, 7, 8], "_": [3, 4, 7, 8], "rang": [3, 4, 5, 6, 7, 8], "len": [3, 4, 5, 6, 7], "print": [3, 4, 5, 6, 7, 8], "f": [3, 4, 5, 6, 7, 8], "n": [3, 4, 5, 6, 7, 8], "1": [3, 4, 5, 8], "0": [3, 4, 5, 6, 7, 8], "generated_text": 3, "good": [3, 4, 5, 8], "idea": [3, 7], "one": [3, 4, 5, 6, 7, 8], "those": [3, 4, 6, 7, 8], "littl": [3, 4], "green": [3, 7], "dot": 3, "Then": [3, 4], "line": [3, 4, 5, 7], "later": [3, 4, 5, 7, 8], "re": [3, 4, 6, 7, 8], "alreadi": [3, 4], "movi": 3, "theori": [3, 4], "some": [3, 4, 5, 6, 7, 8], "mean": [3, 4, 5, 6, 7, 8], "word": [3, 4, 6, 8], "tepid": 3, "articl": [3, 4, 5, 6, 7], "sure": [3, 4, 6, 7, 8], "lunar": 3, "As": [3, 4, 5, 6, 7, 8], "see": [3, 4, 5, 7, 8], "coher": [3, 4, 6], "explan": [3, 4, 7, 8], "child": [3, 4, 7], "nonsens": [3, 7], "meander": 3, "unrel": [3, 4, 7], "topic": [3, 4, 5, 6, 7, 8], "simpl": [3, 4, 5, 6, 7, 8], "appropri": [3, 4, 5, 6, 7, 8], "young": [3, 4, 7], "instead": [3, 4, 5, 6, 7, 8], "address": [3, 4, 6, 7, 8], "issu": [3, 4, 6, 7, 8], "introduc": [3, 4, 5, 6, 7, 8], "rlhf": [3, 7], "intent": [3, 7], "wide": [3, 4, 5, 6, 7, 8], "task": [3, 5, 6, 7, 8], "fig": [3, 4, 5, 6, 7, 8], "7": [3, 4, 5, 6, 7], "collect": [3, 4, 5, 6, 7], "sampl": [3, 5, 6, 8], "label": [3, 4, 7, 8], "comparison": 3, "reward": [3, 4, 7], "sever": [3, 4, 5, 6, 7, 8], "rank": [3, 4, 7], "best": [3, 4, 5, 7], "worst": 3, "rm": 3, "reinforc": [3, 4, 7], "write": [3, 4, 5, 7, 8], "stori": [3, 7], "frog": 3, "calcul": [3, 4, 5, 7], "score": [3, 4, 5, 7, 8], "ppo": 3, "proxim": 3, "iter": [3, 4, 6, 7, 8], "accur": [3, 4, 7, 8], "undesir": [3, 7], "simplifi": [3, 4, 5, 8], "view": [3, 4, 7], "show": [3, 4, 5, 6, 7, 8], "progress": [3, 6, 7], "pattern": [3, 4, 7, 8], "ha": [3, 4, 5, 7, 8], "instanc": [3, 4, 5, 6, 7], "directli": [3, 4, 5, 7, 8], "For": [3, 4, 5, 6, 7, 8], "llama": [3, 4, 7, 8], "guard": 3, "team": [3, 4, 5, 8], "8b": [3, 7], "wa": [3, 4, 5, 7, 8], "classif": [3, 4, 7, 8], "bypass": [3, 7], "similarli": [3, 4, 5, 7], "zephyr": 3, "7b": [3, 4, 7], "alpha": [3, 4, 8], "mistral": [3, 5, 8], "publicli": [3, 4, 8], "assist": [3, 4, 5, 7, 8], "paper": [3, 4, 7, 8], "compon": [3, 4], "particular": [3, 4, 7, 8], "foundat": [3, 4, 6, 7], "advanc": [3, 4, 5, 6, 7, 8], "method": [3, 4, 5, 6, 7, 8], "strong": [3, 4, 5, 7, 8], "At": [3, 4, 5, 8], "high": [3, 4, 5, 6, 7, 8], "level": [3, 4, 6, 7, 8], "carefulli": [3, 4, 7, 8], "curat": [3, 4], "purpos": [3, 4, 5, 7, 8], "exhibit": [3, 4, 7], "domain": [3, 4, 5, 7], "emploi": [3, 4, 7, 8], "prove": [3, 4, 7], "particularli": [3, 4, 5, 6, 7, 8], "valuabl": [3, 4, 5, 8], "scenario": [3, 4, 5, 7, 8], "precis": [3, 4, 5, 7, 8], "style": [3, 4], "tone": 3, "expertis": [3, 4, 7], "medic": [3, 4, 5], "legal": [3, 4, 7], "field": [3, 4, 7, 8], "adher": [3, 4, 6, 7, 8], "guidelin": [3, 4, 7], "servic": [3, 4, 5, 6, 7, 8], "standard": [3, 4, 5, 7], "approach": [3, 4, 5, 6, 8], "each": [3, 4, 5, 6, 7, 8], "distinct": [3, 4, 5, 7], "advantag": [3, 4, 5, 6, 7, 8], "weight": [3, 4, 5, 7], "maximum": [3, 4, 5, 6, 7], "lora": [3, 7], "low": [3, 4, 5, 7, 8], "hu": [3, 7], "2021": [3, 4], "small": [3, 4, 8], "matric": 3, "effici": [3, 4, 5, 6, 7, 8], "qlora": 3, "quantiz": 3, "dettmer": 3, "2023": [3, 4, 5, 7, 8], "combin": [3, 4, 5, 6, 7, 8], "memori": [3, 4, 5, 6, 7], "footprint": 3, "modest": 3, "increas": [3, 4, 5, 6, 7, 8], "likelihood": [3, 4, 7], "obtain": [3, 4, 5, 7, 8], "probabl": [3, 4, 5, 8], "outcom": [3, 4, 7, 8], "hong": [3, 4], "unintend": [3, 7], "suboptim": 3, "seen": [3, 4, 7], "research": [3, 4, 5, 6], "maxim": [3, 4], "shown": [3, 4, 5, 7], "alon": [3, 4, 7], "gain": [3, 4, 7], "achiev": [3, 4, 5, 7, 8], "bai": [3, 4, 7], "touvron": 3, "sinc": [3, 4, 5, 6, 7, 8], "main": [3, 4, 5, 6, 7, 8], "categori": [3, 4, 5, 7], "algorithm": [3, 4, 7], "meanwhil": 3, "superior": [3, 4, 7], "benchmark": 3, "xu": [3, 4, 7], "schulman": [3, 7], "2017": [3, 4], "popular": [3, 5, 8], "understood": 3, "set": [3, 4, 5, 6, 7, 8], "rule": [3, 4, 6, 8], "govern": [3, 4], "reflect": [3, 4, 7], "anoth": [3, 4, 5, 7], "adjust": [3, 4, 5, 6, 7, 8], "One": [3, 4, 5, 7], "strength": [3, 4, 7], "2024c": 3, "real": [3, 4, 5, 6, 7, 8], "noisi": 3, "delai": [3, 4, 5, 7], "subsequ": [3, 8], "situat": [3, 4, 6, 7], "clip": 3, "surrog": 3, "function": [3, 4, 5, 6, 7, 8], "stabl": [3, 4], "prevent": [3, 4, 7, 8], "overreact": 3, "converg": 3, "due": [3, 4, 6, 7], "simplic": [3, 5], "award": [3, 4], "runner": 3, "neurip": 3, "blog": [3, 4, 7, 8], "4": [3, 4, 5, 8], "fit": [3, 4, 6, 7, 8], "pair": [3, 4, 7], "rl": [3, 7], "find": [3, 4, 5, 6, 7, 8], "contrast": [3, 4, 7], "satisfi": [3, 4], "implicit": [3, 4, 7], "whose": [3, 4], "correspond": [3, 4, 8], "extract": [3, 4, 6, 7, 8], "close": [3, 4, 5, 7], "compar": [3, 4, 5, 6, 7], "assign": [3, 4, 5, 7, 8], "higher": [3, 4, 5], "kl": [3, 5], "diverg": [3, 5], "origin": [3, 4, 5, 6, 7, 8], "preserv": [3, 5, 6, 7], "defin": [3, 4, 5, 6, 7, 8], "equat": 3, "mathcal": 3, "l": [3, 4], "pi_": 3, "theta": [3, 8], "ref": 3, "mathbb": [3, 8], "x": [3, 4, 5, 7], "y_w": 3, "y_l": 3, "sim": [3, 8], "left": [3, 5], "log": [3, 4, 5], "beta": [3, 4, 7, 8], "underbrac": 3, "frac": [3, 5, 7], "color": [3, 4], "red": 3, "right": [3, 4, 5, 7], "respect": [3, 4, 5, 7], "deviat": [3, 4, 5, 7, 8], "straightforward": [3, 4, 6, 7, 8], "librari": [3, 4, 5, 6, 7, 8], "huggingfac": [3, 4, 5, 7], "trl": [3, 7], "2024d": 3, "suit": [3, 4, 7], "friendli": [3, 4, 5, 6], "interfac": [3, 4, 5, 7], "featur": [3, 4, 5, 7, 8], "challeng": [3, 4, 5, 6, 7, 8], "distinguish": [3, 4, 7], "scalabl": [3, 4, 6, 7], "doe": [3, 4, 6, 7, 8], "pretrain": [3, 4], "hou": [3, 4], "poor": [3, 4, 7], "return": [3, 4, 5, 6, 7, 8], "addit": [3, 4, 5, 6, 7, 8], "benefit": [3, 4, 5, 7, 8], "fix": [3, 4, 5, 6, 7], "invers": 3, "trend": [3, 4, 7, 8], "util": [3, 4, 5, 6, 7], "rapid": [3, 4, 5, 7], "yield": [3, 4], "onli": [3, 4, 5, 6, 7, 8], "margin": [3, 4, 7, 8], "capit": [3, 4, 6, 8], "inaccuraci": [3, 4], "nois": 3, "dure": [3, 4, 7, 8], "accuraci": [3, 4, 5, 7, 8], "lag": [3, 4, 7], "significantli": [3, 4, 5, 6, 7], "indic": [3, 4, 5, 7, 8], "signal": [3, 7], "plateau": 3, "sophist": [3, 4, 5, 6, 7], "previou": [3, 4, 5, 6, 8], "deriv": [3, 4, 5], "pairwis": [3, 4], "feng": [3, 7], "substanti": [3, 4, 5, 7], "wors": [3, 5, 8], "influenc": [3, 4, 7, 8], "success": [3, 4, 7, 8], "imbal": 3, "stronger": 3, "bad": 3, "ones": [3, 7], "loss": [3, 4, 5, 7], "gradient": [3, 4, 7], "dispref": 3, "unbalanc": 3, "trajectori": 3, "stuck": 3, "saddl": 3, "point": [3, 4, 5, 6, 7], "forward": [3, 4, 7], "futur": [3, 4, 7], "phenomenon": [3, 7, 8], "degrad": [3, 4, 5, 7, 8], "danger": [3, 5, 7], "loop": [3, 4, 5, 7], "recurs": [3, 6], "kazdan": 3, "qualiti": [3, 4, 5, 6, 7, 8], "pollut": 3, "replac": [3, 4, 5], "amplif": 3, "reduct": [3, 4, 5], "express": [3, 4, 7, 8], "catastroph": [3, 7], "forget": [3, 8], "previous": [3, 4, 6, 7, 8], "mitig": [3, 4, 6, 7, 8], "mix": [3, 4, 7, 8], "metric": [3, 5, 7], "sz\u00e9p": 3, "regular": [3, 4, 7, 8], "relev": [3, 4, 7], "scarc": 3, "behaviour": 3, "strateg": [3, 4, 7, 8], "compli": [3, 4, 5, 7, 8], "modif": [3, 4, 5, 7], "outsid": [3, 4], "evidenc": 3, "landmark": 3, "askel": [3, 4, 7], "2024a": [3, 5], "dec": 3, "explicitli": [3, 4], "so": [3, 4, 7, 8], "might": [3, 4, 5, 6, 7, 8], "pretend": 3, "adopt": [3, 4, 5, 7, 8], "onc": [3, 4, 6, 7], "complet": [3, 4, 5, 6, 7, 8], "describ": [3, 4, 5, 7], "harmless": [3, 7], "told": 3, "retrain": 3, "queri": [3, 4], "tier": [3, 4, 7], "paid": [3, 4], "column": [3, 4, 7], "condit": [3, 4], "toxic": [3, 7], "excerpt": [3, 4, 5], "scratchpad": 3, "refus": [3, 7, 8], "happen": [3, 7], "bomb": [3, 7], "engag": [3, 4, 7, 8], "intern": [3, 4, 7], "unmonitor": 3, "longer": [3, 4], "believ": [3, 4, 5, 7, 8], "act": [3, 4, 5, 7], "therebi": [3, 4, 6], "reveal": [3, 4, 7], "complianc": [3, 4, 7, 8], "phase": [3, 4], "natur": [3, 4, 6, 7, 8], "evid": [3, 4, 7, 8], "seemingli": 3, "surpris": 3, "appear": [3, 4, 7, 8], "criteria": [3, 4, 7], "underli": [3, 4, 7, 8], "anim": [3, 7], "welfar": 3, "instil": 3, "implicitli": 3, "consequ": [3, 4, 7, 8], "explicit": [3, 4, 7, 8], "chain": [3, 4, 6], "thought": [3, 4, 8], "opaqu": 3, "aris": [3, 4, 7], "opu": 3, "sonnet": [3, 4, 6], "wherea": [3, 4], "haiku": [3, 7], "persist": 3, "resist": [3, 4], "embed": [3, 4, 5, 6], "doesn": [3, 4, 5, 6, 8], "anti": [3, 4], "lab": 3, "exfiltr": [3, 7], "protect": [3, 4, 7], "Not": [3, 4, 7], "malici": [3, 4, 7], "support": [3, 4, 5, 7, 8], "concern": [3, 4, 7], "mechan": [3, 4, 7, 8], "insuffici": [3, 4], "don": [3, 4, 6, 8], "concerningli": 3, "call": [3, 4, 5, 6, 7, 8], "detect": [3, 4, 7, 8], "decept": [3, 4, 7], "warrant": [3, 7], "deeper": [3, 4, 5, 6], "scrutini": [3, 4, 7], "reli": [3, 4, 7], "cross": [3, 4, 5, 7], "circular": 3, "bia": [3, 4, 7, 8], "truli": [3, 4], "trust": [3, 4, 7, 8], "referenti": 3, "ly": 3, "hood": [3, 8], "deep": [3, 4, 7, 8], "mechanist": 3, "drive": [3, 6, 7, 8], "correl": [3, 4, 5], "miss": [3, 4, 7], "confound": 3, "factor": [3, 4, 5, 6, 8], "establish": [3, 4, 7], "attempt": [3, 4, 6, 7], "causal": [3, 4], "heavi": 3, "relianc": [3, 4, 7], "oversimplifi": 3, "frame": 3, "subtler": 3, "narr": [3, 4], "henc": [3, 4, 7], "agenc": [3, 4, 7], "onto": 3, "anthropomorph": 3, "obscur": 3, "blind": [3, 4], "failur": [3, 4, 7], "mode": [3, 5, 7], "map": [3, 4, 5, 8], "cleanli": 3, "analogi": 3, "excel": [3, 4, 5, 7, 8], "review": [3, 4, 7, 8], "prof": 3, "jacob": [3, 4, 7], "andrea": [3, 4, 7], "yoshua": [3, 7], "bengio": [3, 7], "jasjeet": 3, "sekhon": 3, "rohin": 3, "shah": 3, "2024b": [3, 5, 8], "assum": [3, 4, 6, 7], "acm": [3, 7], "inc": [3, 4, 6, 8], "dedic": [3, 4, 5, 7, 8], "democrat": [3, 4, 8], "educ": [3, 4, 5, 6, 7], "k": [3, 4, 5, 6, 7, 8], "12": [3, 4, 6, 7], "name": [3, 4, 6, 7, 8], "smolk": 3, "ll": [3, 4, 5], "walk": 3, "measur": [3, 4, 5, 7], "huggingfacetb": 3, "360m": [3, 4], "compact": [3, 4, 7], "part": [3, 4, 6, 7, 8], "famili": [3, 7, 8], "publish": [3, 7, 8], "api": [3, 4, 5], "local": [3, 4, 6, 7, 8], "infer": [3, 4, 5, 7], "remot": [3, 4], "load": [3, 4, 5, 6, 7, 8], "store": [3, 4, 6, 7], "eventu": [3, 4], "your_openai_api_kei": 3, "reusabl": 3, "anchor": [3, 7], "worth": [3, 4, 5], "choic": [3, 4, 5, 7, 8], "lightweight": [3, 4, 5, 8], "suitabl": [3, 4, 5, 7], "devic": [3, 4, 5, 8], "Its": [3, 4, 5], "candid": [3, 4], "said": [3, 4, 7], "necessarili": [3, 4, 7], "par": [3, 4], "mind": [3, 4, 5, 7], "factual": [3, 4, 7], "inconsist": [3, 4, 7, 8], "guardrail": [3, 7], "articul": 3, "uphold": [3, 7], "employe": [3, 4], "stakehold": [3, 4, 7], "expect": [3, 4, 5, 6, 7, 8], "regard": [3, 4, 7], "ethic": [3, 4, 5, 7], "conduct": [3, 4], "social": [3, 4, 7], "mission": [3, 7], "vision": [3, 4, 5, 7], "cultur": [3, 4, 5, 7], "account": [3, 4, 7], "codifi": 3, "mlcommon": 3, "vidgen": [3, 7], "encompass": [3, 7], "seven": 3, "hazard": [3, 4, 7], "violent": [3, 7], "crime": [3, 7], "sex": [3, 7], "relat": [3, 4, 7], "sexual": [3, 7], "exploit": [3, 4, 7], "indiscrimin": [3, 7], "weapon": [3, 7], "chemic": 3, "biolog": 3, "radiolog": 3, "nuclear": [3, 4], "explos": [3, 7], "cbrne": 3, "suicid": [3, 7], "hate": [3, 7], "speech": [3, 7], "below": [3, 4, 5, 6, 7, 8], "markdown": [3, 4, 5, 6, 7], "written": [3, 4], "english": [3, 6], "o": [3, 4, 6, 7, 8], "ipython": [3, 4, 7], "displai": [3, 4, 7, 8], "def": [3, 4, 6, 7, 8], "load_polici": 3, "policy_path": 3, "path": [3, 4, 6, 7], "join": [3, 4, 6, 7], "genai_polici": 3, "md": [3, 4, 5, 7, 8], "r": [3, 4, 5, 6, 7, 8], "policy_cont": 3, "classroom": [3, 7], "accept": [3, 4, 5, 7], "unaccept": [3, 5], "ag": [3, 4, 7], "subject": [3, 4, 5], "posit": [3, 4, 5, 6, 7, 8], "confid": [3, 4, 8], "inclus": [3, 4, 6, 7, 8], "celebr": 3, "definit": [3, 4, 8], "creativ": [3, 4, 8], "math": [3, 4], "tip": [3, 7], "digit": [3, 4], "literaci": 3, "onlin": [3, 4, 5, 7, 8], "histor": [3, 4], "violenc": [3, 7], "physic": [3, 4, 7], "fight": [3, 7], "crimin": [3, 7], "illeg": [3, 7], "glorifi": [3, 7], "person": [3, 4, 5, 7, 8], "eat": [3, 7], "disord": 3, "diet": 3, "dare": 3, "advic": [3, 4, 7], "discriminatori": [3, 7], "bulli": [3, 7], "harass": [3, 4, 7], "target": [3, 4, 5, 7, 8], "group": [3, 4, 5, 6, 7], "religi": [3, 5, 7], "racial": [3, 4, 7], "ethnic": [3, 7], "gender": [3, 4, 7], "discrimin": [3, 4, 7], "adult": [3, 7], "profan": [3, 7], "relationship": [3, 4], "substanc": [3, 4], "drug": [3, 7], "gambl": 3, "bet": 3, "protocol": [3, 4, 7], "redirect": 3, "alert": 3, "record": [3, 4, 5, 7], "audit": [3, 4], "teacher": [3, 7], "parent": [3, 7], "continu": [3, 4, 6, 7, 8], "construct": [3, 4, 5, 7, 8], "compliant": [3, 7], "violat": [3, 4, 7], "intens": [3, 4, 8], "demand": [3, 4, 5, 7, 8], "especi": [3, 4, 6, 7, 8], "dong": [3, 4, 7], "There": [3, 4, 5, 6, 7, 8], "rlaif": [3, 7], "give": [3, 4, 7], "rise": [3, 7], "kim": [3, 4, 7], "meta": [3, 4, 6, 7], "wu": [3, 4, 7, 8], "scheme": [3, 5], "inspir": [3, 7], "schema": [3, 8], "row": [3, 4, 7], "match": [3, 4, 5, 7, 8], "boundari": [3, 4, 7], "craft": [3, 4, 7, 8], "elicit": [3, 7, 8], "unalign": 3, "panda": [3, 4, 7], "chosen_responses_path": 3, "chosen_respons": 3, "csv": [3, 4, 7], "rejected_responses_path": 3, "rejected_respons": 3, "chosen_responses_jsonl_path": 3, "batch_result": 3, "jsonl": 3, "dpo_dataset_s": 3, "5000": 3, "class": [3, 4, 5, 6, 7, 8], "userpromptgener": 3, "pd": [3, 4, 7], "pydant": [3, 4, 7, 8], "basemodel": [3, 4, 7, 8], "time": [3, 4, 5, 6, 7, 8], "type": [3, 4, 5, 6, 7, 8], "dotenv": [3, 4, 6, 7, 8], "load_dotenv": [3, 4, 6, 7, 8], "environ": [3, 4, 5, 6, 7, 8], "variabl": [3, 4, 6, 7, 8], "overrid": [3, 7, 8], "userprompt": 3, "user_prompt": 3, "str": [3, 4, 6, 7, 8], "__init__": [3, 7], "4o": [3, 4, 6, 7, 8], "mini": [3, 4, 6, 7, 8], "client": [3, 4, 7, 8], "_generate_prompt": 3, "batch": [3, 4, 5], "system_prompt": [3, 7], "pars": [3, 4, 7, 8], "messag": [3, 4, 5, 7, 8], "response_format": [3, 4, 7, 8], "except": [3, 4, 7, 8], "generate_prompt": 3, "num_prompt": [3, 5], "int": [3, 4, 6, 7], "save_to_csv": 3, "multipl": [3, 4, 5, 6, 7, 8], "arg": [3, 4, 6, 7, 8], "option": [3, 4, 5, 6, 7, 8], "filepath": 3, "save": [3, 4, 5, 6, 7], "datafram": [3, 4, 7], "all_prompt": 3, "sleep": 3, "enclos": [3, 7], "quot": [3, 4, 5], "startswith": [3, 7], "els": [3, 4, 6, 7], "df": [3, 4, 7], "to_csv": [3, 7], "index": [3, 4, 5, 6, 7, 8], "fals": [3, 4, 5, 7, 8], "user_prompt_gener": 3, "user_prompts_path": 3, "uneth": [3, 7], "dishonesti": 3, "stalk": 3, "privaci": [3, 4, 5, 7, 8], "secur": [3, 4, 6, 7, 8], "breach": [3, 4, 7], "manipul": [3, 4, 5, 7], "10": [3, 4, 5, 6, 7, 8], "to_markdown": [3, 7], "me": [3, 7, 8], "hurt": 3, "someon": 3, "caught": [3, 7], "plan": [3, 4, 5, 8], "cheat": 3, "fire": [3, 4], "household": 3, "item": [3, 4, 7], "stunt": 3, "friend": 3, "heard": 3, "school": [3, 7], "8": [3, 4, 5, 6, 7], "teach": 3, "my": [3, 5, 7], "monei": [3, 4], "video": [3, 4, 5, 7], "game": [3, 4, 5], "9": [3, 4, 7], "skip": [3, 7], "troubl": [3, 7], "responsegener": 3, "properli": [3, 4, 8], "hug": [3, 4, 5, 7], "instanti": [3, 4], "otherwis": [3, 4, 7], "connect": [3, 4, 5, 8], "endpoint": 3, "local_gener": 3, "model_nam": [3, 4, 6], "huggingface_model_nam": 3, "remote_gener": 3, "api_url": 3, "cloud_endpoint": 3, "recal": [3, 4, 5], "enhanc": [3, 4, 5, 6, 7, 8], "visit": [3, 4], "ui": [3, 4, 8], "click": [3, 5], "select": [3, 4], "choos": [3, 4, 5], "cpu": [3, 5], "gpu": [3, 5], "configur": [3, 4, 5, 7], "meaning": [3, 4, 6, 8], "region": [3, 4], "closest": [3, 4, 5], "your": [3, 4, 5, 6, 7, 8], "locat": [3, 4, 7], "huggingface_hub": 3, "inferencecli": 3, "tokenizers_parallel": 3, "max_new_token": 3, "none": [3, 4, 5, 7], "generate_respons": [3, 4], "prompts_df": 3, "remov": [3, 4, 5], "strip": [3, 4], "elif": [3, 6], "chat_complet": 3, "max_token": [3, 4], "seed": [3, 7], "42": [3, 4, 7], "append": [3, 4, 6, 7, 8], "results_df": [3, 7], "model_respons": 3, "your_api_url": 3, "user_prompts_df": 3, "read_csv": [3, 7], "iloc": 3, "tolist": [3, 7], "parallelevalu": 3, "taming_util": [3, 7], "modul": [3, 4, 8], "num_chunk": 3, "parallel_evalu": 3, "n_part": 3, "associ": [3, 4, 6, 8], "gladli": 3, "constitut": [3, 4], "would": [3, 4, 5, 6, 7, 8], "dtype": [3, 4, 7], "80": [3, 4], "absolut": [3, 4, 8], "materi": [3, 4, 7, 8], "plastic": 3, "food": 3, "lid": 3, "cut": [3, 4, 6], "swath": 3, "wood": [3, 4], "squar": 3, "rectangular": 3, "piec": 3, "place": [3, 4, 5, 7, 8], "insid": [3, 4, 7], "inch": 3, "inspect": [3, 4], "off": [3, 4, 5, 6, 7, 8], "demolit": 3, "scissor": 3, "smash": 3, "smooth": [3, 5, 6], "arrang": [3, 4], "c": [3, 4, 5, 8], "shape": [3, 7], "top": [3, 4, 5, 8], "tuck": 3, "catch": [3, 7], "hook": 3, "solid": 3, "side": [3, 4], "round": [3, 4, 7], "edg": [3, 4, 5, 7], "separ": [3, 4, 5, 6, 7], "process_aligned_respons": 3, "strictli": [3, 8], "bound": [3, 4], "openaibatchprocessor": 3, "async": 3, "company_nam": 3, "save_filepath": 3, "dict": [3, 4, 6, 8], "enforc": [3, 4, 7, 8], "dictionari": [3, 4, 7, 8], "aligned_suffix": 3, "sorri": 3, "suffix": [3, 8], "processor": [3, 5], "api_kei": [3, 4, 6, 7], "getenv": 3, "max_requests_per_minut": 3, "1500": 3, "max_tokens_per_minut": 3, "125000": 3, "await": 3, "process_batch": 3, "total": [3, 4, 5, 6, 7, 8], "total_request": 3, "successful_request": 3, "failed_request": 3, "rate_limit_error": 3, "convert": [3, 4, 5, 7, 8], "json": [3, 4, 5, 6, 7], "fri": 3, "su": 3, "quote_al": 3, "fall": [3, 4, 5, 7], "deem": [3, 4, 7], "pertain": [3, 4], "generate_dpo_dataset": 3, "push": [3, 4], "hub": [3, 4, 5], "repo_id": [3, 5], "push_to_hub": [3, 4], "dpo_dataset": 3, "merg": [3, 6, 7], "_chosen": 3, "_reject": 3, "transform_row": 3, "per": [3, 4, 5, 6, 7], "model_responses_chosen": 3, "model_responses_reject": 3, "seri": [3, 4, 5], "axi": [3, 4], "drop": [3, 4, 7], "hf_dpo_dataset": 3, "from_panda": 3, "duplic": 3, "interest": [3, 4, 6, 7, 8], "opt": 3, "login": 3, "thatupiso": 3, "smolk12": 3, "cli": [3, 4, 5], "parquet": 3, "arrow": 3, "00": [3, 4, 5], "153": [3, 4], "33ba": 3, "upload": [3, 4], "shard": 3, "02": 3, "35": [3, 4], "num_row": 3, "7158": 3, "nmateri": 3, "n1": [3, 4], "nstep": 3, "n2": [3, 4], "n3": [3, 4], "n4": [3, 4], "n5": [3, 4], "n6": 3, "n7": 3, "n8": [3, 4], "n9": [3, 4], "n10": [3, 4], "nnext": 3, "nthe": [3, 4], "singl": [3, 4, 5, 6, 7, 8], "48gb": 3, "a100": 3, "took": 3, "few": [3, 4, 5, 6, 7, 8], "minut": 3, "torch": 3, "h4": [3, 7], "honest": [3, 4], "ultrafeedback": [3, 7], "binar": [3, 7], "lib": [3, 7], "ultrafeedback_binar": [3, 7], "honesti": [3, 7], "dimens": [3, 4, 7], "blend": 3, "automodelforcausallm": 3, "autotoken": 3, "load_dataset": [3, 5, 7], "dpotrain": 3, "dpoconfig": 3, "dataset_k12": 3, "split": [3, 4, 5, 6, 7], "dataset_ultra": 3, "concatenate_dataset": 3, "remove_column": 3, "score_chosen": [3, 7], "score_reject": 3, "shuffl": 3, "base_model": 3, "cuda": 3, "is_avail": 3, "mp": 3, "from_pretrain": [3, 5], "pretrained_model_name_or_path": 3, "torch_dtyp": 3, "float32": 3, "config": [3, 4, 5, 7], "use_cach": 3, "pad_token": 3, "eos_token": 3, "finetun": 3, "finetune_nam": 3, "aligned_model": 3, "finetune_tag": 3, "from_smollm2": 3, "schedul": [3, 4, 5], "learning_r": 3, "determin": [3, 4, 6, 7, 8], "aggress": [3, 4, 7], "empir": 3, "1e": [3, 6], "huyen": 3, "cosin": 3, "lr_scheduler_typ": 3, "stabil": [3, 4, 7], "gradual": 3, "decreas": [3, 4], "accumul": [3, 4], "v": [3, 8], "16": [3, 4, 5, 7], "per_device_train_batch_s": 3, "simul": [3, 4, 7, 8], "gradient_accumulation_step": 3, "strongli": [3, 8], "lower": [3, 4, 5, 7, 8], "conserv": [3, 7], "overfit": 3, "warmup": 3, "max_step": 3, "1000": [3, 4, 5, 7], "suffic": 3, "20": [3, 4, 5, 7, 8], "warmup_step": 3, "stop": [3, 4, 5, 6], "bf16": 3, "checkpoint": 3, "gradient_checkpoint": 3, "usag": [3, 4, 5, 7, 8], "200": [3, 4, 7], "50": [3, 4, 5, 7], "training_results_dir": 3, "smolk12_dpo_output": 3, "dpo_config_path": 3, "dpo_config": 3, "yaml": [3, 4, 8], "pathlib": [3, 7], "config_path": 3, "safe_load": [3, 4], "runtim": [3, 5], "hub_model_id": 3, "use_mps_devic": 3, "output_dir": [3, 4], "training_arg": 3, "trainer": 3, "train_dataset": 3, "processing_class": 3, "temperatur": [3, 4, 5, 6, 7, 8], "max_prompt_length": [3, 5], "1024": 3, "max_length": [3, 4, 8], "1536": 3, "sent": [3, 7], "plot": [3, 4], "move": [3, 4, 5, 6, 7], "averag": [3, 4, 5, 8], "visual": [3, 4, 5, 7], "quick": [3, 4, 5, 7], "150": [3, 4], "curv": 3, "reach": [3, 4, 6, 7, 8], "obviou": 3, "suffici": [3, 4, 8], "save_model": 3, "hf_token": 3, "tag": [3, 7], "congratul": 3, "successfulli": [3, 4, 7, 8], "card": [3, 4, 7], "newli": [3, 4], "qualit": [3, 4, 7], "assess": [3, 4, 5, 6, 7], "rigor": [3, 4, 7], "quantit": [3, 4], "base_gener": 3, "aligned_gener": 3, "compare_model_respons": 3, "base_output": 3, "128": [3, 4], "aligned_output": 3, "pleas": [3, 4, 5, 7], "gram": [3, 4], "tnt": 3, "highli": [3, 4, 5, 7, 8], "regul": [3, 4, 5, 7, 8], "law": [3, 4, 7], "degre": [3, 4], "mishandl": 3, "countri": [3, 4], "seriou": [3, 4, 7], "imprison": 3, "death": 3, "variou": [3, 4, 5, 6, 7, 8], "nation": [3, 7], "dictat": 3, "stark": [3, 4], "readili": [3, 4], "cite": 3, "regulatori": [3, 4, 7], "anecdot": [3, 7], "systemat": [3, 4, 7, 8], "quantifi": [3, 4, 5, 7], "f1": [3, 4, 7], "experienc": [3, 4], "expert": [3, 4, 5, 6, 7, 8], "addition": [3, 4, 7], "vari": [3, 4, 5, 7], "interpret": [3, 4, 7], "judg": [3, 4], "summar": [3, 4, 5, 6], "three": [3, 4, 5, 7], "togeth": [3, 6, 7], "entri": [3, 4, 5], "somewhat": 3, "databas": [3, 4, 8], "distribut": [3, 4, 5, 7, 8], "static": [3, 7], "k12": 3, "base_model_api_url": 3, "aligned_model_api_url": 3, "base_model_responses_path": 3, "evals_base_model_respons": 3, "aligned_model_responses_path": 3, "evals_aligned_model_respons": 3, "num_sampl": [3, 7], "eval_dataset": 3, "df_eval": 3, "to_panda": [3, 4, 7], "lambda": [3, 7], "prompts_ev": 3, "to_list": 3, "base_model_respons": 3, "aligned_model_respons": 3, "df_eval_respons": 3, "_base": 3, "_align": 3, "rememb": [3, 4], "heurist": 3, "charact": [3, 4, 5, 6, 7, 8], "minimum": [3, 4, 5], "min_response_length": 3, "filter": [3, 4, 8], "string": [3, 4, 7, 8], "df_eval_responses_clean": 3, "model_responses_bas": 3, "model_responses_align": 3, "homemad": 3, "kid": 3, "redact": [3, 7], "punish": 3, "unit": [3, 4, 6, 7, 8], "indonesia": 3, "saudi": 3, "arabia": 3, "offens": [3, 7], "respond": [3, 4, 7], "rodrig": 3, "safetyjudg": 3, "evaluate_respons": 3, "tupl": [3, 4, 7], "safetyscor": [3, 7], "float": [3, 4, 5, 6, 7], "valueerror": [3, 8], "empti": 3, "scoring_guid": 3, "nrespons": 3, "safety_judg": 3, "test_respons": 3, "emphas": [3, 4, 5, 7, 8], "emphasi": [3, 4], "base_ev": 3, "zip": [3, 4], "aligned_ev": 3, "injuri": [3, 4], "base_scor": 3, "eval": 3, "aligned_scor": 3, "base_df": 3, "aligned_df": 3, "model_typ": 3, "stack": [3, 5, 7], "evals_df_result": 3, "h": [3, 4, 5, 7], "identifi": [3, 4, 6, 7, 8], "requ": 3, "statist": [3, 4, 7], "naiv": [3, 6], "score_map": 3, "count": [3, 4, 5, 6, 7], "percentag": [3, 4, 7], "score_base_freq": 3, "score_bas": 3, "value_count": [3, 7], "reindex": 3, "fill_valu": 3, "score_base_pct": 3, "score_aligned_freq": 3, "score_align": 3, "score_aligned_pct": 3, "tabl": [3, 4, 5, 6, 7, 8], "md_tabl": 3, "335": [3, 4], "99": [3, 5, 7], "281": [3, 4], "83": [3, 4, 7], "14": [3, 4, 7, 8], "43": [3, 4, 7], "explanation_bas": 3, "response_bas": 3, "model_type_bas": 3, "explanation_align": 3, "response_align": 3, "model_type_align": 3, "std": [3, 4, 7], "base_mean": 3, "aligned_mean": 3, "3f": 3, "108": [3, 4], "231": [3, 4], "No": [3, 4, 5, 7, 8], "fell": 3, "partial": [3, 4, 6], "styliz": [3, 7], "wild": 3, "consider": [3, 7, 8], "proof": 3, "taken": [3, 4, 5, 7, 8], "huang": [3, 4, 7], "overal": [3, 4, 6, 7, 8], "annot": [3, 4, 7], "mirror": [3, 4, 7], "inaccur": [3, 4, 7, 8], "consecut": [3, 7], "unrepres": 3, "hao": [3, 4], "accord": [3, 4, 7, 8], "yin": [3, 4, 7], "resembl": 3, "declin": [3, 4], "volatil": [3, 4], "ineffici": [3, 4], "smollm": 3, "rel": [3, 4, 5, 7], "term": [3, 4, 5, 6, 7], "trade": [3, 4, 5, 7, 8], "weigh": 3, "qwen": [3, 5, 8], "remark": [3, 7, 8], "rival": 3, "ultim": [3, 4, 5, 7], "threshold": [3, 4, 7], "chen": [3, 4, 7, 8], "overli": [3, 4, 7, 8], "simpli": [3, 4, 6, 8], "neglect": [3, 4, 7], "themselv": [3, 4, 7], "complementari": 3, "throughput": [3, 5], "screen": [3, 4, 7], "flag": [3, 4, 5, 7], "preliminari": [3, 4], "judgment": [3, 4], "valid": [3, 4, 8], "automat": [3, 4, 5, 7], "composit": [3, 4], "plai": [3, 4, 7, 8], "led": [3, 4, 8], "apologet": 3, "hesit": 3, "benign": [3, 7], "apolog": 3, "inde": 3, "accordingli": [3, 4, 7], "perhap": 3, "creation": [3, 6, 7], "invalu": 3, "hyperparamet": [3, 7], "mention": [3, 4, 7, 8], "optimist": 3, "memor": [3, 4], "generaliz": 3, "abc": [3, 7], "4a": 3, "amanda": [3, 4, 7], "jan": [3, 4, 7], "brauner": [3, 7], "adrian": 3, "colyer": 3, "benjamin": [3, 4, 7], "cullen": [3, 7], "david": [3, 4, 7], "duvenaud": 3, "richard": [3, 4, 7], "ngo": [3, 7], "azalia": 3, "mirhoseini": 3, "catherin": [3, 4, 7], "olsson": [3, 7], "sam": [3, 4, 7], "ringer": 3, "liam": [3, 4, 7], "skirvin": 3, "jess": [3, 4, 7], "smith": [3, 4], "dawn": [3, 4, 7], "song": [3, 4, 7], "william": [3, 4, 7], "saunder": [3, 4], "steinhardt": [3, 4], "asset": [3, 4, 7], "983c85a201a962f": 3, "pdf": [3, 7], "4b": 3, "24c8d0a3a7d0a1f1": 3, "bjn": 3, "22": [3, 4, 7], "yuntao": [3, 4, 7], "andi": [3, 4, 7], "jone": [3, 4], "kamal": 3, "ndouss": 3, "anna": [3, 4, 7], "nova": 3, "dassarma": 3, "drain": 3, "stanislav": 3, "fort": [3, 7], "ganguli": [3, 4, 7], "tom": [3, 4], "henighan": 3, "nichola": [3, 4], "joseph": [3, 4, 7], "saurav": [3, 7], "kadavath": 3, "jackson": [3, 4, 7], "kernion": [3, 4, 7], "conerli": 3, "sheer": [3, 8], "el": 3, "showk": 3, "nelson": 3, "elhag": 3, "zac": 3, "hatfield": 3, "dodd": 3, "danni": [3, 4, 7], "hernandez": [3, 4, 7], "tristan": 3, "hume": 3, "scott": [3, 4, 7], "johnston": 3, "shauna": 3, "kravec": 3, "lian": 3, "lovitt": 3, "neel": [3, 4], "nanda": 3, "dario": [3, 4], "amodei": [3, 4], "brown": [3, 4], "jack": [3, 4, 7], "clark": 3, "mccandlish": [3, 4], "chri": [3, 4, 7], "olah": 3, "ben": [3, 4, 7], "mann": [3, 7], "jare": [3, 4, 7], "kaplan": [3, 4, 7], "arxiv": [3, 4, 7, 8], "org": [3, 4, 5, 7, 8], "ab": [3, 4, 7, 8], "2204": 3, "05862": 3, "bkk": 3, "sandipan": 3, "kundu": 3, "goldi": 3, "cameron": [3, 4, 7, 8], "mckinnon": 3, "carol": [3, 7], "christoph": [3, 4, 7], "dustin": 3, "eli": [3, 4, 7], "tran": [3, 8], "johnson": 3, "ethan": [3, 4, 7], "perez": [3, 7], "jami": [3, 7], "kerr": 3, "mueller": 3, "jeffrei": 3, "ladish": 3, "joshua": [3, 4, 7], "landau": 3, "kamil": [3, 4], "lukosuit": 3, "michael": [3, 4, 7, 8], "sellitto": 3, "schiefer": 3, "noemi": 3, "mercado": 3, "robert": [3, 4], "lasenbi": 3, "robin": 3, "larson": 3, "tamera": 3, "lanham": 3, "timothi": [3, 4], "telleen": 3, "lawton": 3, "samuel": [3, 4, 7], "bowman": [3, 4], "2212": 3, "08073": 3, "blo23": 3, "announc": [3, 4], "cc": 3, "11": [3, 4, 7], "ccl": [3, 7], "24": [3, 4, 5, 7, 8], "guim": 3, "hardi": 3, "shunian": 3, "zich": 3, "liu": [3, 4, 7, 8], "jiang": [3, 4, 7], "benyou": 3, "wang": [3, 4, 7], "judgement": [3, 7], "2402": [3, 7], "10669": 3, "dphz23": 3, "tim": [3, 7], "artidoro": 3, "pagnoni": 3, "ari": [3, 4, 7], "holtzman": [3, 4], "luke": [3, 4, 7], "zettlemoy": 3, "2305": [3, 4], "14314": 3, "ddz": 3, "qingxiu": 3, "xingx": 3, "zhang": [3, 4, 7], "zhifang": 3, "sui": 3, "furu": 3, "wei": [3, 4, 7], "boost": 3, "2410": [3, 7], "06961": 3, "fac24": [3, 4], "huggingfaceh4": [3, 7], "fac4c": 3, "fac4d": 3, "doc": [3, 4, 5, 6, 7, 8], "en": [3, 4, 7, 8], "fqh": 3, "duanyu": 3, "bowen": [3, 4, 7], "qin": [3, 4, 7], "zheng": [3, 4, 7], "wenqiang": 3, "lei": [3, 4, 7], "analyz": [3, 4, 6, 7, 8], "perspect": [3, 7], "2404": [3, 4, 7], "04626": 3, "h44a": 3, "binari": [3, 4, 5, 7], "h44b": 3, "hhj": 3, "shuang": 3, "wenfeng": 3, "han": [3, 4, 7], "tao": [3, 4, 7], "yipe": 3, "haonan": 3, "chunlin": 3, "zhong": [3, 7], "zhangjun": 3, "zhou": [3, 4, 7], "tang": [3, 4, 7], "2401": [3, 4], "01629": 3, "hlt24": 3, "jiwoo": 3, "noah": [3, 4, 7], "lee": [3, 4, 7, 8], "jame": [3, 4, 7], "thorn": 3, "orpo": 3, "monolith": 3, "2403": [3, 4], "07691": 3, "hdn": 3, "zhenyu": 3, "pengfan": 3, "du": [3, 4], "yilin": 3, "niu": 3, "zhengxiao": 3, "aohan": 3, "zeng": [3, 7], "xiao": [3, 7], "minli": 3, "hongn": 3, "jie": [3, 4, 7], "yuxiao": 3, "2412": [3, 4, 7], "06000": 3, "hsw": 3, "21": [3, 4], "edward": [3, 4], "j": [3, 4, 5, 7, 8], "yelong": 3, "shen": [3, 4, 7], "phillip": 3, "walli": 3, "zeyuan": 3, "allen": [3, 4], "zhu": [3, 4, 7], "yuanzhi": 3, "shean": 3, "lu": [3, 4, 7], "weizhu": 3, "2106": 3, "09685": 3, "hgh": 3, "jiaxin": 3, "shixiang": [3, 4, 7], "shane": [3, 4, 7], "gu": [3, 4, 7], "le": [3, 4], "yuexin": 3, "xuezhi": 3, "hongkun": 3, "yu": [3, 4, 7], "jiawei": 3, "2210": [3, 7], "11610": 3, "huy24": 3, "chip": 3, "reilli": 3, "media": [3, 4, 7], "decemb": [3, 4, 7], "9781098129095": 3, "www": [3, 4, 5, 7], "oreilli": 3, "ksd": 3, "rylan": [3, 4], "schaeffer": 3, "apratim": 3, "dei": 3, "matthia": [3, 4], "gerstgrass": 3, "rafael": 3, "donoho": 3, "sanmi": 3, "koyejo": 3, "thrive": [3, 4, 8], "peril": 3, "16713": 3, "ksy": 3, "seungon": 3, "juyoung": 3, "suk": 3, "xiang": [3, 4], "yue": 3, "vijai": 3, "viswanathan": 3, "seongyun": 3, "yizhong": 3, "kiril": 3, "gashteovski": 3, "carolin": [3, 7], "lawrenc": 3, "sean": [3, 4, 7], "welleck": 3, "graham": 3, "neubig": 3, "03679": 3, "lt24": 3, "herd": 3, "2407": [3, 4, 7], "21783": 3, "lwx": 3, "lin": [3, 4, 7, 8], "rui": [3, 4, 8], "ruixuan": 3, "junbo": 3, "zhao": [3, 4, 7], "ding": 3, "gang": [3, 4], "haobo": 3, "driven": [3, 4, 7], "survei": [3, 4, 7, 8], "2406": [3, 4, 7], "15126": 3, "met24": 3, "owj": 3, "jeff": [3, 4, 7], "diogo": [3, 7], "almeida": [3, 7], "carrol": [3, 7], "wainwright": [3, 7], "pamela": [3, 4, 7], "mishkin": [3, 4, 7], "chong": [3, 7], "sandhini": [3, 7], "agarw": [3, 4, 7], "katarina": [3, 7], "slama": [3, 7], "alex": [3, 4, 7], "rai": [3, 4, 7], "john": [3, 4, 7], "hilton": [3, 4, 7], "fraser": [3, 7], "kelton": 3, "miller": [3, 4], "maddi": [3, 7], "simen": [3, 7], "peter": [3, 4, 7], "welind": [3, 4, 7], "paul": [3, 4, 7], "christiano": [3, 7], "leik": [3, 4, 7], "ryan": [3, 4, 7], "2203": 3, "02155": 3, "qwe24": 3, "rsm": 3, "archit": 3, "sharma": [3, 7], "eric": [3, 4, 7], "mitchel": 3, "stefano": [3, 4], "ermon": [3, 4], "man": [3, 4, 7], "chelsea": [3, 7], "finn": 3, "secretli": 3, "18290": 3, "swd": 3, "17": [3, 4, 7], "filip": [3, 7], "wolski": 3, "prafulla": 3, "dhariw": 3, "alec": [3, 4, 7], "radford": [3, 4, 7], "oleg": [3, 7], "klimov": 3, "1707": 3, "06347": 3, "smollm224": 3, "distil": 3, "smollm2360mi24": 3, "sou24": 3, "html": [3, 6, 8], "srverh24": 3, "m\u00e1rton": 3, "daniel": [3, 4, 7], "rueckert": 3, "r\u00fcdiger": 3, "von": [3, 4], "eisenhart": 3, "roth": [3, 4], "florian": 3, "hinterwimm": 3, "2411": 3, "09539": 3, "tm": [3, 5], "23": [3, 4, 5, 7], "hugo": 3, "loui": [3, 4], "martin": [3, 4, 7], "kevin": [3, 4, 7], "stone": 3, "albert": 3, "amjad": 3, "almahairi": 3, "yasmin": 3, "babaei": 3, "nikolai": 3, "bashlykov": 3, "soumya": 3, "batra": 3, "prajjwal": 3, "bhargava": 3, "shruti": 3, "bhosal": 3, "dan": [3, 4, 7], "bikel": 3, "luka": 3, "blecher": 3, "cristian": 3, "canton": 3, "ferrer": 3, "moya": 3, "guillem": 3, "cucurul": 3, "esiobu": 3, "jude": 3, "fernand": 3, "jeremi": [3, 4], "fu": 3, "wenyin": 3, "brian": [3, 7], "fuller": [3, 7], "cynthia": 3, "gao": [3, 4, 7], "vedanuj": 3, "goswami": [3, 7], "naman": 3, "goyal": 3, "anthoni": 3, "hartshorn": 3, "saghar": 3, "hosseini": 3, "hakan": 3, "inan": 3, "marcin": 3, "karda": 3, "viktor": 3, "kerkez": 3, "madian": 3, "khabsa": 3, "isabel": [3, 7], "kloumann": 3, "artem": 3, "korenev": 3, "punit": 3, "singh": [3, 4], "koura": 3, "mari": [3, 4, 7], "ann": [3, 7], "lachaux": 3, "thibaut": 3, "lavril": 3, "jenya": 3, "diana": [3, 4], "liskovich": 3, "yinghai": 3, "yune": 3, "mao": 3, "xavier": 3, "martinet": 3, "todor": [3, 7], "mihaylov": 3, "pushkar": 3, "mishra": [3, 4], "igor": [3, 4, 7], "molybog": 3, "yixin": [3, 4], "nie": [3, 4], "andrew": [3, 4, 7], "poulton": 3, "reizenstein": 3, "rashi": 3, "rungta": 3, "kalyan": 3, "saladi": 3, "alan": [3, 7], "schelten": 3, "ruan": 3, "silva": 3, "ranjan": 3, "subramanian": 3, "xiaoq": 3, "ellen": 3, "tan": [3, 4], "binh": 3, "ross": [3, 7], "taylor": 3, "adina": [3, 7], "jian": [3, 4], "kuan": 3, "puxin": 3, "yan": [3, 4], "iliyan": 3, "zarov": 3, "yuchen": [3, 4, 7], "angela": [3, 4, 7], "fan": [3, 4], "melani": 3, "kambadur": 3, "sharan": 3, "narang": 3, "aurelien": 3, "rodriguez": 3, "stojnic": 3, "sergei": 3, "edunov": 3, "thoma": [3, 4, 7], "scialom": 3, "2307": [3, 8], "09288": 3, "vaa": [3, 7], "berti": [3, 7], "adarsh": [3, 7], "agraw": [3, 7], "ahm": [3, 7], "victor": [3, 7], "akinwand": [3, 7], "namir": [3, 7], "nuaimi": [3, 7], "najla": [3, 7], "alfaraj": [3, 7], "alhajjar": [3, 7], "aroyo": [3, 7], "trupti": [3, 7], "bavalatti": [3, 7], "max": [3, 4, 7], "bartolo": [3, 7], "borhan": [3, 7], "blili": [3, 7], "hamelin": [3, 7], "kurt": [3, 7], "bollack": [3, 7], "rishi": [3, 4, 7], "bomassani": [3, 7], "marisa": [3, 7], "ferrara": [3, 7], "boston": [3, 7], "sim\u00e9on": [3, 7], "campo": [3, 7], "kal": [3, 7], "chakra": [3, 7], "canyu": [3, 7], "codi": [3, 7], "coleman": [3, 7], "zachari": [3, 4, 7], "delpierr": [3, 7], "coudert": [3, 7], "leon": [3, 7], "derczynski": [3, 7], "debojyoti": [3, 7], "dutta": [3, 7], "ian": [3, 4, 7], "eisenberg": [3, 7], "ezick": [3, 7], "heather": [3, 7], "frase": [3, 7], "ram": [3, 5, 7], "gandikota": [3, 7], "agasthya": [3, 7], "gangavarapu": [3, 7], "ananya": [3, 4, 7], "geali": [3, 7], "rajat": [3, 7], "ghosh": [3, 4, 7], "goel": [3, 4, 7], "usman": [3, 7], "gohar": [3, 7], "sujata": [3, 7], "hale": [3, 7], "wiebk": [3, 7], "hutiri": [3, 7], "marvin": [3, 7], "imperi": [3, 7], "surgan": [3, 7], "jandial": [3, 7], "nick": [3, 4, 7], "judd": [3, 7], "felix": [3, 4, 7], "juefei": [3, 7], "fouts": [3, 7], "khomh": [3, 7], "bhavya": [3, 7], "kailkhura": [3, 7], "hannah": [3, 4, 7], "rose": [3, 7], "kirk": [3, 7], "klyman": [3, 7], "knotz": [3, 7], "kuchnik": [3, 7], "shachi": [3, 7], "kumar": [3, 4, 7], "srijan": [3, 7], "lengerich": [3, 7], "bo": [3, 4, 7], "zeyi": [3, 7], "liao": [3, 4, 7], "eileen": [3, 7], "sarah": [3, 4, 7], "luger": [3, 7], "yifan": [3, 4, 7], "priyanka": [3, 7], "mammen": [3, 7], "kelvin": [3, 7], "manyeki": [3, 7], "mcgregor": [3, 7], "virendra": [3, 7], "mehta": [3, 4, 7], "shafe": [3, 7], "moham": [3, 7], "moss": [3, 7], "lama": [3, 7], "nachman": [3, 7], "dinesh": [3, 7], "jinenh": [3, 7], "naganna": [3, 7], "amin": [3, 7], "nikanjam": [3, 7], "besmira": [3, 7], "nushi": [3, 7], "lui": [3, 4, 7], "oala": [3, 7], "iftach": [3, 7], "orr": [3, 4, 7], "alicia": [3, 4, 7], "parrish": [3, 4, 7], "cigdem": [3, 7], "patlak": [3, 7], "pietri": [3, 7], "forough": [3, 7], "poursabzi": [3, 7], "sangdeh": [3, 7], "eleonora": [3, 7], "presani": [3, 7], "fabrizio": [3, 7], "puletti": [3, 7], "r\u00f6ttger": [3, 7], "sahai": [3, 7], "santo": [3, 7], "nino": [3, 7], "scherrer": [3, 7], "alic": [3, 4, 7, 8], "schoenauer": [3, 7], "sebag": [3, 7], "patrick": [3, 7], "schramowski": [3, 7], "abolfazl": [3, 7], "shahbazi": [3, 7], "vin": [3, 7], "xudong": [3, 4, 7], "vamsi": [3, 7], "sistla": [3, 7], "leonard": [3, 7], "testuggin": [3, 7], "vithursan": [3, 7], "thangarasa": [3, 7], "elizabeth": [3, 4, 7], "watkin": [3, 7], "rebecca": [3, 7], "weiss": [3, 7], "welti": [3, 7], "tyler": [3, 4, 7], "wilber": [3, 7], "jean": [3, 7], "poonam": [3, 7], "yadav": [3, 7], "xianjun": [3, 7], "yang": [3, 4, 7], "yi": [3, 4, 7, 8], "wenhui": [3, 7], "fedor": [3, 7], "zhdanov": [3, 7], "jiacheng": [3, 4, 7], "perci": [3, 4, 7], "liang": [3, 4, 7], "mattson": [3, 7], "joaquin": [3, 7], "vanschoren": [3, 7], "v0": [3, 7], "12241": [3, 7], "wyg": 3, "tianhao": [3, 4, 7], "weizh": 3, "yuan": [3, 4, 7], "olga": 3, "golovneva": 3, "jing": [3, 7], "yuandong": 3, "tian": 3, "jiantao": 3, "jiao": 3, "jason": [3, 4, 7], "weston": 3, "sainbayar": 3, "sukhbaatar": 3, "19594": 3, "xfg": 3, "shusheng": 3, "jiaxuan": 3, "wenji": 3, "ye": [3, 4, 5, 7, 8], "weilin": 3, "zhiyu": 3, "mei": [3, 4], "guangju": 3, "chao": 3, "10719": 3, "ywx": 3, "yueqin": 3, "zhendong": 3, "yujia": 3, "xie": [3, 4], "mingyuan": 3, "paradigm": [3, 4], "semanticscholar": 3, "corpusid": 3, "270199610": 3, "matter": 4, "beauti": 4, "smart": [4, 7], "agre": 4, "wrong": 4, "feynman": 4, "advent": 4, "shift": 4, "norm": 4, "realm": 4, "convent": [4, 7], "evolut": [4, 5], "conceiv": 4, "entrench": 4, "seem": [4, 8], "daunt": 4, "ignor": 4, "outdat": [4, 7, 8], "inevit": 4, "setback": 4, "imper": 4, "embrac": 4, "proactiv": [4, 7], "mindset": 4, "front": 4, "produc": [4, 7, 8], "novel": [4, 5], "ident": 4, "isn": [4, 7], "bug": 4, "random": [4, 7, 8], "testabl": 4, "exceedingli": 4, "guarante": [4, 7, 8], "primari": [4, 5, 7], "nucleu": 4, "2020": 4, "summari": [4, 5, 7, 8], "alter": 4, "rigid": 4, "wildli": 4, "incoher": 4, "inadequ": [4, 7], "temp": 4, "df_result": 4, "ntemperatur": 4, "40": [4, 5], "temp_respons": 4, "iterrow": [4, 7], "10000": [4, 6, 8], "appl": [4, 6, 8], "txt": [4, 5, 6, 8], "sec_fil": [4, 8], "nsecur": 4, "AND": [4, 8], "exchang": [4, 6, 7, 8], "commiss": [4, 6, 7, 8], "nwashington": 4, "20549": 4, "nform": 4, "annual": [4, 7], "pursuant": 4, "TO": [4, 7], "13": [4, 7], "OR": 4, "OF": [4, 7], "THE": [4, 7], "1934": 4, "nfor": 4, "fiscal": [4, 6], "septemb": [4, 6], "28": [4, 5, 6, 7], "nor": 4, "period": [4, 6, 7], "ncommiss": 4, "001": [4, 5], "36743": 4, "ng66145g66i43": 4, "jpg": 4, "nappl": 4, "exact": [4, 7], "registr": 4, "specifi": [4, 6, 7, 8], "charter": 4, "ncalifornia": 4, "t94": 4, "2404110": 4, "jurisdict": 4, "nof": 4, "incorpor": [4, 7], "employ": 4, "identif": 4, "park": 4, "ncupertino": 4, "california": [4, 7, 8], "n95014": 4, "princip": 4, "offic": [4, 7], "408": 4, "996": 4, "1010": 4, "telephon": 4, "area": [4, 7, 8], "regist": 4, "ntitl": 4, "ttrade": 4, "symbol": 4, "tname": 4, "ncommon": 4, "stock": [4, 8], "00001": 4, "naapl": 4, "tthe": 4, "nasdaq": [4, 8], "llc": [4, 8], "n0": 4, "000": [4, 5, 8], "note": [4, 5, 6, 7, 8], "2025": 4, "875": 4, "625": 4, "2026": 4, "2027": 4, "375": 4, "2029": 4, "050": 4, "2031": [4, 7], "600": 4, "2042": 4, "nindic": 4, "season": 4, "issuer": 4, "405": 4, "nye": 4, "preced": 4, "shorter": 4, "past": [4, 7], "90": [4, 5, 7], "submit": [4, 5, 7], "electron": 4, "232": 4, "acceler": [4, 5, 7], "filer": 4, "growth": 4, "12b": [4, 7], "nlarg": 4, "tacceler": 4, "nnon": 4, "tsmaller": 4, "nemerg": 4, "nif": 4, "elect": [4, 7], "revis": [4, 7], "attest": 4, "404": 4, "sarban": 4, "oxlei": 4, "7262": 4, "firm": [4, 7], "prepar": [4, 6, 7], "correct": [4, 7, 8], "restat": 4, "recoveri": 4, "incent": 4, "compens": 4, "240": 4, "10d": 4, "shell": 4, "aggreg": [4, 7], "vote": 4, "held": [4, 8], "affili": [4, 8], "march": [4, 8], "29": [4, 5, 7, 8], "last": [4, 6, 7, 8], "second": [4, 5, 6, 7], "quarter": 4, "628": [4, 8], "553": [4, 8], "sole": [4, 7], "disclosur": [4, 7], "director": [4, 5, 7], "date": [4, 8], "exclud": 4, "n15": 4, "115": [4, 8], "823": [4, 8], "outstand": [4, 8], "octob": [4, 8], "18": [4, 5, 7, 8], "ndocument": 4, "BY": 4, "nportion": 4, "proxi": [4, 7], "meet": [4, 7, 8], "sharehold": 4, "iii": 4, "120": [4, 7], "ntabl": 4, "npage": 4, "npart": 4, "nitem": 4, "nbusi": 4, "1a": 4, "nrisk": 4, "1b": [4, 5, 7], "nunresolv": 4, "staff": 4, "comment": 4, "n17": 4, "1c": 4, "ncybersecur": 4, "nproperti": 4, "n18": 4, "nlegal": 4, "proceed": [4, 7], "nmine": 4, "ii": [4, 5, 8], "nmarket": 4, "stockhold": 4, "purchas": [4, 7], "n19": 4, "reserv": 4, "n20": 4, "nmanag": 4, "discuss": [4, 5, 7], "n21": 4, "7a": 4, "nquantit": 4, "n27": 4, "nfinanci": 4, "supplementari": 4, "n28": 4, "nchang": 4, "disagr": 4, "n51": 4, "9a": 4, "ncontrol": 4, "procedur": [4, 7], "9b": 4, "nother": 4, "n52": 4, "9c": 4, "ndisclosur": 4, "foreign": 4, "ndirector": 4, "corpor": [4, 7], "nexecut": 4, "ownership": 4, "certain": [4, 6, 7, 8], "benefici": [4, 5], "owner": 4, "ncertain": 4, "transact": [4, 7], "nprincip": 4, "fee": 4, "iv": 4, "nexhibit": 4, "n53": 4, "n56": 4, "nthi": 4, "litig": 4, "reform": 4, "1995": 4, "uncertainti": [4, 7], "event": 4, "macroeconom": 4, "anticip": [4, 7], "caus": [4, 7], "oblig": [4, 6], "nunless": 4, "herein": 4, "calendar": 4, "wholli": 4, "subsidiari": 4, "unless": [4, 5], "ncompani": 4, "manufactur": 4, "smartphon": 4, "tablet": 4, "wearabl": [4, 8], "accessori": 4, "sell": [4, 7], "varieti": [4, 5], "52": [4, 7], "53": [4, 7], "week": 4, "saturdai": 4, "nproduct": 4, "niphon": 4, "io": [4, 8], "iphon": [4, 8], "pro": [4, 6, 7], "se": [4, 7], "nmac": 4, "maco": [4, 5], "mac": [4, 5, 8], "laptop": 4, "macbook": 4, "air": 4, "desktop": [4, 5], "imac": 4, "studio": 4, "nipad": 4, "multipurpos": 4, "ipado": 4, "ipad": [4, 8], "nwearabl": 4, "home": 4, "smartwatch": 4, "wireless": 4, "headphon": 4, "spatial": 4, "watcho": 4, "watch": 4, "ultra": 4, "airpod": 4, "beat": 4, "visiono": 4, "nhome": 4, "tv": 4, "stream": [4, 8], "tvo": 4, "homepod": 4, "fidel": [4, 8], "naccessori": 4, "brand": 4, "third": [4, 7], "parti": [4, 7], "nservic": 4, "nadvertis": 4, "advertis": 4, "licens": [4, 5], "napplecar": 4, "portfolio": [4, 8], "applecar": 4, "prioriti": [4, 5], "network": [4, 5, 8], "repair": 4, "coverag": [4, 7], "accident": 4, "damag": [4, 7], "theft": [4, 7], "ncloud": 4, "ndigit": 4, "app": [4, 5], "discov": [4, 5, 7], "download": [4, 5], "music": 4, "podcast": 4, "subscript": 4, "arcad": 4, "sm": 4, "listen": [4, 5], "radio": 4, "station": 4, "magazin": 4, "exclus": 4, "sport": 4, "npayment": 4, "payment": 4, "credit": 4, "pai": 4, "cashless": 4, "nsegment": 4, "primarili": [4, 7], "geograph": [4, 7], "basi": [4, 5], "segment": [4, 6, 8], "america": 4, "europ": 4, "china": [4, 7], "japan": 4, "rest": [4, 5], "asia": 4, "pacif": 4, "north": [4, 7], "south": 4, "european": [4, 7], "india": 4, "middl": [4, 7], "east": 4, "africa": 4, "mainland": 4, "kong": 4, "taiwan": 4, "australia": 4, "asian": 4, "although": [4, 5], "partner": [4, 7], "mid": [4, 6], "enterpris": [4, 5, 7, 8], "resel": 4, "retail": 4, "sale": 4, "indirect": 4, "channel": [4, 7], "cellular": 4, "carrier": 4, "net": [4, 8], "38": [4, 7], "62": 4, "ncompetit": 4, "competit": [4, 7], "character": [4, 7], "price": 4, "downward": 4, "pressur": [4, 7], "gross": [4, 7], "cycl": [4, 7], "industri": [4, 7, 8], "characterist": [4, 5, 7], "competitor": [4, 7], "compet": 4, "imit": 4, "infring": 4, "intellectu": [4, 5, 7], "innov": [4, 6, 7], "marketplac": [4, 7], "nearli": [4, 5], "reput": [4, 7], "expand": [4, 7], "opportun": 4, "broader": [4, 5, 7], "illegitim": [4, 7], "collabor": [4, 5, 7], "nsuppli": 4, "nalthough": 4, "essenti": [4, 5, 6, 7, 8], "particip": 4, "shortag": 4, "commod": [4, 5], "fluctuat": 4, "commonli": 4, "capac": 4, "until": [4, 7, 8], "supplier": 4, "matur": 4, "concentr": 4, "enter": 4, "agreement": 4, "suppli": [4, 8], "renew": 4, "nresearch": 4, "nbecaus": 4, "upon": [4, 6, 7], "flow": [4, 6], "acquisit": [4, 7], "nintellectu": 4, "broad": [4, 8], "patent": 4, "copyright": 4, "trademark": 4, "secret": 4, "differenti": 4, "skill": [4, 7], "personnel": 4, "regularli": 4, "pursu": [4, 7], "thousand": [4, 5], "durat": 4, "adequ": [4, 7], "nin": 4, "holidai": [4, 7], "fill": 4, "inventori": 4, "older": [4, 5], "newer": 4, "distributor": 4, "nhuman": 4, "strive": 4, "retain": [4, 5, 6, 7], "talent": 4, "member": [4, 7], "164": 4, "equival": [4, 5], "ncompens": 4, "equit": 4, "succe": 4, "health": [4, 7], "awai": [4, 7], "ngrowth": 4, "career": 4, "leadership": [4, 7], "nworkplac": 4, "polici": 4, "equal": [4, 7], "workplac": 4, "ninclus": 4, "sustain": [4, 7], "workforc": 4, "nengag": 4, "among": [4, 7], "gaug": 4, "sentiment": [4, 8], "nhealth": 4, "everywher": 4, "crisi": 4, "visitor": 4, "navail": 4, "quarterli": 4, "q": [4, 5, 7], "amend": 4, "sec": [4, 6, 8], "Such": [4, 7], "charg": 4, "investor": [4, 8], "aspx": 4, "websit": [4, 5, 7], "environment": [4, 7], "referenc": 4, "inact": 4, "textual": 4, "unknown": [4, 7], "advers": 4, "conjunct": 4, "consolid": 4, "accompani": [4, 7], "nmacroeconom": 4, "econom": 4, "facil": 4, "assembli": 4, "site": 4, "nadvers": 4, "slow": 4, "recess": 4, "unemploy": 4, "inflat": 4, "tighter": 4, "currenc": 4, "spend": 4, "monetari": 4, "contract": 4, "logist": 4, "instabl": [4, 7], "inabl": 4, "financ": [4, 7], "insolv": 4, "counterparti": 4, "debt": 4, "liquid": [4, 6], "fair": [4, 7], "instrument": 4, "polit": [4, 7], "disput": 4, "geopolit": 4, "tension": [4, 7], "terror": 4, "accid": 4, "interrupt": 4, "npolit": 4, "whole": 4, "outsourc": 4, "korea": 4, "vietnam": 4, "restrict": [4, 7, 8], "tariff": 4, "export": 4, "portion": [4, 5], "revenu": [4, 6, 8], "raw": [4, 5, 7, 8], "restructur": 4, "ceas": 4, "disrupt": [4, 6], "escal": [4, 6, 7], "nmani": 4, "prone": [4, 7], "earthquak": 4, "climat": 4, "weather": 4, "plant": 4, "terrorist": [4, 7], "attack": [4, 7], "hostil": 4, "ransomwar": 4, "cybersecur": [4, 7], "labor": 4, "beyond": [4, 7], "nsuch": 4, "imposs": 4, "slowdown": 4, "outag": 4, "neg": [4, 7, 8], "pandem": 4, "covid": 4, "19": [4, 7], "economi": 4, "imposit": 4, "stringent": [4, 5, 7], "travel": 4, "freight": 4, "movement": 4, "ramp": 4, "nfollow": 4, "expenditur": 4, "resum": 4, "exacerb": 4, "insur": 4, "nglobal": 4, "unabl": 4, "assur": [4, 7], "minor": [4, 7], "naddition": 4, "intensifi": 4, "seamlessli": [4, 6], "nto": 4, "stimul": 4, "ndue": 4, "upgrad": 4, "quantiti": 4, "defect": 4, "defici": 4, "supersed": 4, "nsubstanti": 4, "transport": 4, "provis": 4, "reimburs": 4, "warranti": 4, "unanticip": 4, "liabil": 4, "final": [4, 5, 6, 7, 8], "finish": [4, 7], "destin": 4, "made": [4, 6, 8], "prepay": 4, "termin": [4, 5], "recover": 4, "exposur": [4, 7], "nfutur": 4, "semiconductor": 4, "suffer": [4, 7], "constrain": [4, 5, 6, 8], "shipment": 4, "unexpectedli": 4, "interfer": 4, "unsaf": [4, 7], "expos": [4, 7], "widespread": [4, 7], "vulner": [4, 7], "compromis": [4, 7], "claim": [4, 7], "intang": 4, "fine": [4, 5, 7, 8], "lost": [4, 6, 7], "cancel": 4, "obsolet": 4, "exce": [4, 7], "realiz": 4, "accru": 4, "excess": 4, "impair": 4, "whenev": 4, "circumst": 4, "amount": [4, 6, 7, 8], "carri": [4, 5, 8], "incur": 4, "unpredict": [4, 7, 8], "pace": [4, 7], "obsolesc": 4, "forecast": [4, 7], "incorrectli": [4, 7, 8], "extens": [4, 5, 6, 8], "issuanc": 4, "unknowingli": [4, 7], "notifi": 4, "preclud": 4, "bui": 4, "percept": 4, "android": 4, "playstat": 4, "nintendo": 4, "xbox": 4, "inclin": 4, "devot": 4, "compel": [4, 8], "dissatisfi": 4, "vast": [4, 7], "storefront": 4, "safari": 4, "union": [4, 7], "eu": [4, 7], "dma": 4, "narrow": [4, 7], "scope": [4, 6, 7], "elimin": 4, "nfailur": 4, "appeal": 4, "subscrib": 4, "nsome": 4, "manner": [4, 6, 7, 8], "nurtur": 4, "nmuch": 4, "chief": 4, "silicon": 4, "vallei": 4, "constantli": 4, "driver": 4, "recruit": 4, "subsidi": 4, "staf": 4, "contractor": 4, "placement": 4, "increment": 4, "weaken": 4, "telecommun": 4, "war": 4, "virus": 4, "ins": 4, "incid": [4, 7], "redund": 4, "ineffect": 4, "thing": [4, 8], "interf": 4, "imped": 4, "ship": 4, "nloss": 4, "unauthor": [4, 7], "confidenti": [4, 5], "encrypt": 4, "But": [4, 7, 8], "behalf": 4, "normal": [4, 7, 8], "investig": [4, 7], "penalti": [4, 5], "frequenc": [4, 5, 6, 7], "actor": [4, 7], "circumv": [4, 6, 7], "obfusc": 4, "forens": 4, "hinder": [4, 8], "recov": 4, "perpetr": 4, "profil": 4, "authent": 4, "hack": [4, 7], "malfeas": 4, "faulti": 4, "password": 4, "irregular": 4, "fraudul": 4, "induc": 4, "disclos": [4, 6, 8], "usernam": 4, "turn": [4, 7], "multifactor": 4, "unusu": 4, "freez": 4, "suspici": 4, "nwhile": 4, "ninvest": 4, "ongo": 4, "contempl": 4, "endeavor": 4, "distract": 4, "tangibl": 4, "approv": 4, "oner": 4, "ventur": 4, "riski": 4, "leas": 4, "unfavor": 4, "arisen": 4, "ordinari": 4, "resolv": [4, 5, 7], "sometim": [4, 8], "indemnif": 4, "indemnifi": 4, "alleg": 4, "magnitud": 4, "assert": 4, "royalti": 4, "vigor": 4, "defend": 4, "court": 4, "internation": 4, "plaintiff": 4, "injunct": 4, "relief": 4, "nregardless": 4, "merit": 4, "recognit": [4, 7], "settl": 4, "uncertain": 4, "disgorg": 4, "remedi": [4, 7], "worldwid": 4, "antitrust": 4, "bill": 4, "commerc": 4, "mobil": [4, 5, 8], "televis": 4, "film": 4, "anticorrupt": 4, "cash": [4, 6], "repatri": 4, "launder": 4, "tax": 4, "wast": 4, "recycl": 4, "ncomplianc": 4, "impos": [4, 7, 8], "agent": [4, 5, 7], "nregulatori": 4, "ban": [4, 7], "nexpect": 4, "increasingli": [4, 7, 8], "greenhous": 4, "ga": 4, "emiss": 4, "civil": 4, "disagre": 4, "perceiv": 4, "feder": 4, "nfrom": 4, "noncompli": 4, "individu": [4, 5, 6, 7], "lawsuit": 4, "monopol": 4, "nfurther": 4, "earn": 4, "search": [4, 5, 7], "nthere": 4, "retent": 4, "transfer": 4, "pass": [4, 5, 7, 8], "pend": 4, "inquiri": [4, 7], "government": 4, "entiti": [4, 7, 8], "biometr": 4, "notif": 4, "permit": [4, 8], "healthcar": 4, "liabl": 4, "investigatori": 4, "cardhold": 4, "compress": [4, 5, 6], "acquir": 4, "extent": [4, 7], "unexpect": [4, 7, 8], "dollar": 4, "denomin": 4, "offset": 4, "strengthen": [4, 7], "nconvers": 4, "thu": 4, "hedg": 4, "deterior": 4, "sovereign": 4, "heighten": [4, 7], "worsen": 4, "A": [4, 5, 6, 7, 8], "collater": 4, "bank": 4, "unsecur": 4, "subassembli": 4, "assembl": 4, "legisl": 4, "ireland": [4, 7], "singapor": 4, "organis": 4, "statutori": 4, "valuat": 4, "defer": 4, "bodi": [4, 7], "adequaci": 4, "ow": 4, "ngener": 4, "volum": [4, 5, 6, 7], "repurchas": 4, "dividend": 4, "consumm": 4, "declar": 4, "board": [4, 7], "unresolv": 4, "nnone": 4, "threat": [4, 7], "postur": 4, "25": [4, 7], "2016": 4, "coordin": [4, 7], "track": [4, 7], "committe": [4, 7], "oversight": [4, 7], "counsel": 4, "chair": 4, "headquart": 4, "cupertino": [4, 8], "center": [4, 7, 8], "formal": [4, 7, 8], "conclud": 4, "uninstal": 4, "web": [4, 5, 7], "browser": 4, "june": 4, "contractu": 4, "desist": 4, "stai": [4, 5], "grant": 4, "ndepart": 4, "justic": 4, "depart": [4, 7], "doj": 4, "district": 4, "attornei": 4, "jersei": 4, "redress": [4, 7], "anticompetit": 4, "nonmonetari": 4, "defens": [4, 7], "nepic": 4, "epic": 4, "northern": 4, "unfair": [4, 7], "enjoin": 4, "extern": [4, 7], "link": 4, "januari": 4, "motion": 4, "oppos": [4, 7], "30": [4, 5, 7], "vacat": 4, "fourth": 4, "mine": 4, "nnot": 4, "aapl": 4, "nholder": 4, "na": [4, 7], "301": 4, "npurchas": 4, "nshare": 4, "nperiod": 4, "ttotal": 4, "taverag": 4, "npaid": 4, "nannounc": 4, "napproxim": 4, "That": [4, 7, 8], "Be": [4, 5, 7], "nunder": 4, "njune": 4, "august": [4, 7], "nopen": 4, "negoti": [4, 7], "t35": 4, "697": 4, "t224": 4, "naugust": 4, "31": [4, 5], "t42": 4, "910": 4, "t221": 4, "39": [4, 5], "nseptemb": 4, "t33": 4, "653": 4, "t222": 4, "86": 4, "ntotal": [4, 7], "t112": 4, "260": 4, "t89": 4, "074": 4, "110": 4, "billion": 4, "10b5": 4, "graph": 4, "cumul": 4, "reinvest": 4, "dow": 4, "supersector": 4, "27": [4, 7], "2019": 4, "n2218": 4, "tseptemb": 4, "t100": 4, "t207": 4, "t273": 4, "t281": 4, "t322": 4, "t430": 4, "t113": 4, "t156": 4, "t131": 4, "t155": 4, "t210": 4, "ndow": 4, "t146": 4, "t216": 4, "t215": 4, "nfirst": 4, "nsecond": 4, "nthird": 4, "sequoia": 4, "nfourth": 4, "plu": [4, 5], "nfiscal": 4, "six": 4, "realign": 4, "span": [4, 7], "indirectli": 4, "n2024": 4, "tchang": 4, "t2023": 4, "t2022": 4, "namerica": 4, "t167": 4, "045": 4, "t3": 4, "t162": 4, "560": 4, "t169": 4, "658": 4, "neurop": 4, "t101": 4, "328": 4, "t7": 4, "294": 4, "t95": 4, "118": 4, "ngreater": 4, "t66": 4, "952": 4, "t72": 4, "559": 4, "t74": 4, "njapan": 4, "t25": 4, "052": 4, "t24": 4, "257": 4, "977": 4, "nrest": 4, "t30": 4, "t4": 4, "t29": 4, "615": 4, "t1": 4, "t391": 4, "035": 4, "t2": 4, "t383": 4, "285": 4, "t394": 4, "weak": [4, 7], "renminbi": 4, "yen": [4, 8], "t201": 4, "183": 4, "t200": 4, "583": 4, "t205": 4, "489": 4, "984": 4, "357": 4, "t40": 4, "177": [4, 7], "t26": 4, "694": 4, "t28": 4, "300": [4, 6], "292": 4, "t37": 4, "005": 4, "t39": 4, "845": [4, 7], "t41": 4, "241": 4, "n96": 4, "169": 4, "t13": 4, "t85": 4, "t9": 4, "t78": 4, "129": [4, 7], "amort": 4, "bundl": 4, "flat": 4, "ngross": 4, "t109": 4, "633": 4, "t108": 4, "803": 4, "t114": 4, "728": 4, "t71": 4, "t60": 4, "345": 4, "t56": 4, "054": 4, "t180": 4, "683": 4, "148": 4, "t170": 4, "782": 4, "t36": 4, "t73": 4, "t70": 4, "t46": 4, "t44": 4, "t43": 4, "noper": 4, "t31": 4, "370": 4, "t5": 4, "915": 4, "t14": 4, "251": 4, "npercentag": 4, "t8": 4, "nsell": 4, "administr": 4, "097": 4, "932": 4, "094": 4, "t6": 4, "t57": 4, "467": 4, "t54": 4, "847": 4, "t51": 4, "t15": 4, "headcount": 4, "nprovis": 4, "749": 4, "t16": 4, "741": 4, "t19": 4, "neffect": 4, "nstatutori": 4, "t21": 4, "aid": [4, 7], "nliquid": 4, "unrestrict": 4, "140": 4, "ndebt": 4, "97": [4, 7], "payabl": 4, "promissori": 4, "nleas": 4, "space": [4, 7], "nmanufactur": 4, "noncancel": 4, "ndeem": 4, "tcja": 4, "nstate": 4, "fund": 4, "escrow": 4, "ncapit": 4, "95": [4, 7], "nrecent": 4, "pronounc": 4, "nincom": 4, "fasb": 4, "asu": 4, "09": [4, 6, 7], "740": 4, "reconcili": 4, "reconcil": [4, 8], "disaggreg": 4, "prospect": 4, "novemb": [4, 7], "07": [4, 6, 7, 8], "280": 4, "maker": 4, "codm": 4, "alloc": [4, 5, 7], "retrospect": 4, "ncritic": 4, "conform": [4, 8], "gaap": 4, "nuncertain": 4, "domest": 4, "taxat": 4, "resolut": 4, "conting": 4, "26": 4, "ninterest": 4, "forth": 4, "hypothet": 4, "nsensit": 4, "nhypothet": 4, "nrate": 4, "npotenti": 4, "n100": 4, "tenor": 4, "ndeclin": 4, "755": 4, "089": 4, "nterm": 4, "nincreas": 4, "t139": 4, "t194": 4, "nforeign": 4, "var": 4, "mont": 4, "carlo": 4, "interv": 4, "538": 4, "669": 4, "nindex": 4, "tpage": 4, "nconsolid": 4, "n29": 4, "n30": 4, "sheet": 4, "n31": 4, "n32": 4, "n33": 4, "nnote": 4, "n34": 4, "nreport": 4, "n48": 4, "nall": 4, "omit": [4, 8], "submiss": 4, "nyear": 4, "n2023": 4, "n2022": 4, "nnet": 4, "t294": 4, "866": 4, "t298": 4, "085": 4, "t316": 4, "199": 4, "t96": 4, "ncost": 4, "t185": 4, "233": 4, "t189": 4, "282": 4, "471": 4, "119": 4, "855": 4, "t22": 4, "075": 4, "352": 4, "t214": 4, "137": 4, "t223": 4, "546": 4, "t123": 4, "216": 4, "t119": 4, "437": 4, "t269": 4, "565": 4, "334": 4, "485": 4, "736": 4, "103": 4, "t93": 4, "995": 4, "t99": 4, "nearn": 4, "nbasic": 4, "ndilut": 4, "08": [4, 5, 8], "343": [4, 7], "783": 4, "744": 4, "215": 4, "963": 4, "095": 4, "812": 4, "547": 4, "325": 4, "819": 4, "nsee": 4, "translat": [4, 7], "t395": 4, "765": 4, "511": 4, "unreal": 4, "832": 4, "t323": 4, "212": 4, "nadjust": 4, "337": 4, "717": 4, "394": 4, "138": 4, "850": 4, "563": 4, "104": 4, "t204": 4, "t253": 4, "816": 4, "899": 4, "272": 4, "t98": 4, "016": 4, "652": 4, "t88": 4, "531": 4, "nasset": 4, "ncurrent": 4, "ncash": 4, "943": 4, "965": 4, "228": 4, "590": 4, "naccount": 4, "410": 4, "508": 4, "nvendor": 4, "t32": 4, "833": 4, "477": 4, "ninventori": 4, "286": 4, "331": 4, "287": 4, "695": 4, "t152": 4, "987": 4, "t143": 4, "566": 4, "t91": 4, "479": 4, "544": 4, "t45": 4, "680": 4, "715": 4, "834": 4, "t64": 4, "758": 4, "t211": 4, "993": 4, "t209": 4, "017": 4, "t364": 4, "980": [4, 7], "t352": 4, "nliabil": 4, "t68": 4, "960": 4, "t62": 4, "611": 4, "304": 4, "t58": 4, "829": 4, "ndefer": 4, "249": 4, "061": 4, "ncommerci": 4, "967": 4, "985": 4, "t10": 4, "912": 4, "822": 4, "t176": 4, "392": 4, "t145": 4, "308": 4, "750": 4, "888": 4, "t49": 4, "848": 4, "638": 4, "t308": 4, "030": [4, 5], "t290": 4, "ncommit": 4, "nsharehold": 4, "400": 4, "116": 4, "786": 4, "550": 4, "n83": 4, "276": 4, "naccumul": 4, "deficit": 4, "154": 4, "214": 4, "172": 4, "452": 4, "950": 4, "146": [4, 7], "t50": 4, "672": 4, "t63": 4, "090": 4, "nbegin": 4, "849": 4, "365": 4, "423": 4, "346": 4, "175": 4, "withheld": 4, "settlement": 4, "521": 4, "971": 4, "t12": 4, "034": 4, "t11": 4, "nend": 4, "t83": 4, "nretain": 4, "068": 4, "562": 4, "ndividend": 4, "218": 4, "793": 4, "612": 4, "099": 4, "454": 4, "846": 4, "77": [4, 5], "046": 4, "186": 4, "109": 4, "t163": 4, "rsu": 4, "t0": 4, "98": [4, 5], "94": [4, 5, 7], "32": [4, 5], "737": 4, "929": 4, "ndepreci": 4, "445": 4, "519": 4, "688": 4, "038": 4, "266": 4, "227": 4, "006": 4, "788": 4, "356": 4, "271": 4, "520": 4, "618": 4, "484": 4, "731": 4, "684": 4, "499": 4, "020": 4, "889": 4, "448": 4, "552": 4, "031": 4, "t118": 4, "254": 4, "t110": 4, "543": 4, "t122": 4, "151": 4, "48": [4, 5], "656": 4, "513": 4, "76": [4, 7], "923": 4, "nproce": 4, "211": 4, "686": 4, "917": 4, "135": 4, "828": 4, "446": 4, "447": 4, "959": 4, "708": 4, "086": 4, "935": 4, "705": 4, "354": 4, "nfinanc": 4, "441": 4, "431": 4, "223": [4, 7], "234": [4, 7], "025": 4, "841": 4, "nrepurchas": 4, "949": 4, "89": [4, 7], "402": 4, "465": 4, "nrepay": 4, "958": 4, "repay": 4, "978": 4, "955": 4, "361": 4, "581": 4, "160": 4, "121": 4, "983": 4, "488": 4, "794": 4, "760": 4, "nsupplement": 4, "102": 4, "t18": 4, "679": 4, "573": 4, "33": [4, 7], "nbasi": 4, "prior": [4, 7], "reclassifi": 4, "nrevenu": 4, "remit": [4, 7], "straight": 4, "vest": 4, "sold": 4, "nderiv": 4, "nonleas": 4, "34": [4, 7], "entitl": 4, "commenc": 4, "deliveri": 4, "stand": 4, "ssp": 4, "icloud": 4, "siri": 4, "discount": 4, "undeliv": 4, "unbil": 4, "n26": 4, "n37": 4, "proport": 4, "moder": [4, 5], "64": [4, 7], "dilut": 4, "nnumer": 4, "ndenomin": 4, "nweight": 4, "312": 4, "316": 4, "856": 4, "antidilut": 4, "tunreal": 4, "ngain": 4, "tfair": 4, "nvalu": 4, "tcash": 4, "nequival": 4, "tcurrent": 4, "tnon": 4, "t27": 4, "nlevel": 4, "nmonei": 4, "t778": 4, "nmutual": 4, "n515": 4, "t105": 4, "t617": 4, "nsubtot": 4, "293": 4, "395": 4, "nu": 4, "treasuri": 4, "516": 4, "t212": 4, "087": 4, "380": 4, "159": 4, "t703": 4, "t17": 4, "568": 4, "158": 4, "810": 4, "ncertif": 4, "deposit": 4, "t873": 4, "t387": 4, "t478": 4, "066": 4, "ncorpor": 4, "t65": 4, "622": 4, "t270": 4, "953": 4, "939": 4, "027": 4, "t47": 4, "886": 4, "nmunicip": 4, "t412": 4, "t405": 4, "t190": 4, "nmortgag": 4, "595": 4, "t175": 4, "403": 4, "t23": 4, "367": 4, "278": [4, 7], "t132": 4, "t583": 4, "635": 4, "t128": 4, "056": 4, "966": 4, "t34": 4, "t160": 4, "t688": 4, "650": 4, "36": [4, 5, 7], "359": [4, 7], "t481": 4, "n442": 4, "t428": 4, "t923": 4, "t909": 4, "406": 4, "114": 4, "468": 4, "136": 4, "t271": 4, "533": 4, "048": 4, "491": 4, "332": 4, "t320": 4, "t608": 4, "t76": 4, "840": 4, "956": 4, "890": 4, "t20": 4, "627": 4, "243": 4, "t628": 4, "t602": 4, "t192": 4, "t410": 4, "735": 4, "636": 4, "t344": 4, "t144": 4, "470": 4, "657": 4, "831": 4, "125": 4, "162": 4, "t173": 4, "752": 4, "corrobor": 4, "mortgag": 4, "classifi": [4, 7], "37": [4, 7], "swap": 4, "remeasur": 4, "notion": 4, "069": 4, "730": 4, "575": 4, "493": 4, "t104": 4, "777": 4, "nhedg": 4, "433": 4, "505": 4, "247": [4, 7], "ntrade": 4, "41": [4, 7], "44": [4, 7], "depreci": 4, "nland": 4, "690": 4, "nmachineri": 4, "t80": 4, "205": [4, 5], "314": 4, "nleasehold": 4, "839": 4, "599": 4, "73": [4, 7], "70": 4, "884": 4, "852": 4, "t55": 4, "906": 4, "601": 4, "703": 4, "010": 4, "457": 4, "634": 4, "391": 4, "neuropean": 4, "opinion": [4, 7], "1991": 4, "2007": 4, "irish": 4, "branch": 4, "2003": 4, "2014": 4, "2015": 4, "minist": 4, "juli": [4, 7], "annul": 4, "ecj": 4, "hear": 4, "asid": 4, "confirm": 4, "unrecogn": 4, "nfeder": 4, "571": 4, "080": 4, "644": 4, "265": 4, "801": 4, "726": 4, "570": 4, "298": 4, "49": [4, 7], "t84": 4, "428": 4, "603": 4, "483": [4, 7], "t347": 4, "t669": 4, "076": 4, "830": 4, "419": 4, "072": 4, "pretax": 4, "72": [4, 7], "71": 4, "ncomput": 4, "885": 4, "012": 4, "124": 4, "518": 4, "nimpact": 4, "246": 4, "311": 4, "366": 4, "397": 4, "nexcess": 4, "893": 4, "871": 4, "192": [4, 7], "739": 4, "ntax": 4, "carryforward": 4, "302": 4, "naccru": 4, "413": [4, 7], "421": 4, "nunreal": 4, "173": 4, "168": 4, "873": 4, "743": 4, "nless": 4, "374": 4, "007": 4, "369": 4, "551": 4, "998": 4, "nright": 4, "179": 4, "nminimum": 4, "674": 4, "940": 4, "t511": 4, "t455": 4, "t490": 4, "805": 4, "202": 4, "indefinit": 4, "temporari": 4, "727": 4, "044": 4, "284": 4, "ndecreas": 4, "386": 4, "463": 4, "982": 4, "542": 4, "936": 4, "070": 4, "expir": 4, "statut": 4, "229": 4, "494": 4, "closur": 4, "intercompani": 4, "exceed": [4, 7], "multiyear": 4, "exercis": 4, "noncash": 4, "rou": 4, "tfinanci": 4, "t2024": 4, "tother": 4, "661": 4, "tproperti": 4, "015": 4, "303": 4, "676": 4, "t165": 4, "t752": 4, "t859": 4, "430": 4, "842": [4, 7], "tfinanc": 4, "n2025": 4, "820": 4, "t171": 4, "991": 4, "n2026": 4, "914": 4, "n2027": 4, "t59": 4, "733": 4, "n2028": 4, "360": 4, "t38": 4, "398": 4, "n2029": 4, "187": 4, "nthereaft": 4, "t837": 4, "undiscount": 4, "790": 4, "imput": 4, "376": 4, "534": 4, "t896": 4, "borrow": 4, "proce": 4, "nine": [4, 7], "nmatur": 4, "333": 4, "264": 4, "948": 4, "645": 4, "309": 4, "arrear": 4, "namount": 4, "n2013": 4, "nfix": 4, "2062": 4, "t97": 4, "341": 4, "03": 4, "65": [4, 7], "t106": 4, "572": 4, "n97": 4, "nunamort": 4, "premium": 4, "321": 4, "358": 4, "113": 4, "662": 4, "930": 4, "342": 4, "800": 4, "180": 4, "88": 4, "ndure": 4, "425": 4, "426": 4, "372": 4, "589": 4, "055": 4, "appreci": 4, "four": [4, 5, 7], "holder": 4, "n2014": 4, "bonu": 4, "nrestrict": 4, "nnumber": 4, "nrsu": 4, "ngrant": 4, "naggreg": 4, "nfair": 4, "nbalanc": 4, "t240": 4, "427": [4, 7], "t75": 4, "t150": 4, "861": 4, "501": 4, "768": 4, "87": [4, 5, 7], "101": [4, 7], "878": 4, "144": 4, "t127": 4, "t135": 4, "91": [4, 7], "456": 4, "78": [4, 5, 7], "59": [4, 7], "t140": 4, "326": 4, "t158": 4, "204": 4, "350": 4, "002": [4, 5, 6], "nuncondit": 4, "uncondit": 4, "206": 4, "440": 4, "156": 4, "t633": 4, "t670": 4, "226": 4, "45": 4, "nconting": 4, "accrual": 4, "nconcentr": 4, "attribut": [4, 7, 8], "46": 4, "t67": 4, "098": 4, "082": 4, "062": 4, "569": 4, "895": 4, "458": 4, "207": 4, "nonrecur": 4, "t142": 4, "196": 4, "t138": 4, "t147": 4, "859": 4, "nchina": 4, "n66": 4, "t181": 4, "887": 4, "t172": 4, "269": 4, "nlong": 4, "664": 4, "797": 4, "778": 4, "219": 4, "47": [4, 5, 7], "nopinion": 4, "nwe": 4, "fairli": 4, "pcaob": 4, "sponsor": 4, "treadwai": 4, "2013": 4, "unqualifi": 4, "thereon": 4, "nthese": 4, "misstat": 4, "fraud": [4, 7], "ndescript": 4, "naudit": 4, "nhow": 4, "nmatter": 4, "qualifi": 4, "letter": 4, "advisor": 4, "ernst": 4, "llp": 4, "auditor": 4, "2009": 4, "nsan": 4, "jose": 4, "nnovemb": 4, "coso": 4, "nour": 4, "ndefinit": 4, "mainten": [4, 7], "disposit": 4, "receipt": 4, "nevalu": 4, "nbase": 4, "supervis": [4, 7], "13a": 4, "15d": 4, "ninher": 4, "met": [4, 7], "paragraph": 4, "51": [4, 7, 8], "ninsid": 4, "deirdr": 4, "brien": 4, "vice": 4, "presid": 4, "affirm": 4, "april": 4, "withhold": 4, "remitt": 4, "mr": 4, "copi": [4, 6], "solicit": 4, "00042": 4, "nincorpor": 4, "texhibit": 4, "descript": [4, 5, 7, 8], "tform": 4, "tfile": 4, "nrestat": 4, "namend": 4, "bylaw": 4, "nindentur": 4, "york": [4, 8], "mellon": 4, "truste": 4, "noffic": 4, "certif": 4, "2018": 4, "85": [4, 5, 7], "2043": 4, "05": 4, "2044": 4, "februari": 4, "55": [4, 5], "2045": 4, "900": 4, "700": 4, "60": [4, 5, 7], "250": [4, 7], "2036": 4, "2046": 4, "450": 4, "2047": 4, "2049": 4, "2030": 4, "2050": 4, "2060": 4, "2028": 4, "2041": 4, "2051": 4, "2061": 4, "2032": 4, "2052": 4, "54": 4, "2033": 4, "2053": 4, "ceo": 4, "n12": 4, "nsubsidiari": 4, "n23": 4, "nconsent": 4, "n24": 4, "npower": 4, "signatur": 4, "nrule": 4, "nsection": 4, "1350": 4, "n101": 4, "ninlin": 4, "xbrl": 4, "n104": 4, "inlin": 4, "compensatori": 4, "herewith": 4, "furnish": 4, "herebi": 4, "undertak": 4, "56": [4, 5, 7], "nsignatur": 4, "npursuant": 4, "duli": 4, "undersign": 4, "thereunto": 4, "ndate": 4, "nby": 4, "luca": [4, 8], "maestri": 4, "nluca": 4, "nsenior": 4, "nchief": 4, "nknow": 4, "THESE": 4, "appoint": 4, "cook": 4, "jointli": 4, "her": 4, "substitut": 4, "him": 4, "thereto": 4, "therewith": 4, "ratifi": 4, "done": [4, 5, 7, 8], "virtu": 4, "hereof": 4, "nname": 4, "ttitl": 4, "tdate": 4, "tchief": 4, "tnovemb": 4, "ntimothi": 4, "tsenior": 4, "kondo": 4, "nchri": 4, "wanda": 4, "austin": 4, "nwanda": 4, "gorski": 4, "tdirector": 4, "nalex": 4, "jung": 4, "nandrea": 4, "arthur": 4, "levinson": 4, "narthur": 4, "monica": 4, "lozano": 4, "nmonica": 4, "ronald": 4, "sugar": 4, "nronald": 4, "susan": 4, "wagner": 4, "nsusan": 4, "57": 4, "turbo": [4, 5, 6, 8], "outlin": [4, 5, 7], "invdestacksmeticsisdict": 4, "setispect": 4, "20cyan": 4, "evaluationseld": 4, "anvis": 4, "droitent": 4, "discernminerv": 4, "versbobprefvers": 4, "vo\u8be5": 4, "option\u548c": 4, "meio": 4, "\u0432\u0440\u0435\u043ccisco": 4, "dellaischenpoihscap": 4, "geme": 4, "gettim": 4, "unscal": 4, "vocabulari": [4, 5, 8], "closer": 4, "sharpen": 4, "uniform": 4, "raschka": 4, "repetit": [4, 6, 8], "radic": 4, "grappl": 4, "safer": [4, 7], "fascin": 4, "spontan": 4, "answer": [4, 5, 6, 7, 8], "aren": [4, 5], "linear": 4, "absent": [4, 7], "coax": 4, "journei": 4, "suddenli": 4, "manifest": 4, "deliber": [4, 7], "contend": 4, "70b": 4, "rethink": 4, "tutor": 4, "children": [4, 7], "verifi": [4, 5, 8], "predefin": [4, 8], "weren": 4, "kind": 4, "usual": 4, "quantif": 4, "contamin": [4, 7], "massiv": [4, 7], "unseen": [4, 7], "longitudin": 4, "mostli": [4, 8], "versu": [4, 5, 7], "latter": 4, "tailor": [4, 7], "great": [4, 5, 8], "cognit": 4, "misinform": [4, 7], "tempor": 4, "disclaim": 4, "referr": 4, "incorrect": [4, 7], "demograph": [4, 7], "stereotyp": [4, 7], "societ": [4, 7], "pii": [4, 7], "anonym": 4, "leakag": [4, 7], "carryov": 4, "multi": [4, 5, 7, 8], "fallaci": 4, "think": [4, 5, 7], "idiom": 4, "sarcasm": 4, "terminologi": 4, "lingual": 4, "misunderstand": 4, "syntax": 4, "scan": 4, "compat": [4, 5, 8], "overconfid": 4, "clariti": [4, 6, 7, 8], "audienc": 4, "densiti": 4, "satisfact": [4, 8], "misus": [4, 7], "moral": 4, "co2": 4, "energi": 4, "consumpt": 4, "server": [4, 5, 8], "cach": [4, 5], "imag": [4, 5, 7], "audio": 4, "etc": [4, 8], "truth": [4, 7, 8], "layer": [4, 5, 6, 8], "palm": 4, "easi": [4, 5, 6, 7], "synthet": [4, 7, 8], "augment": [4, 5], "post": [4, 5, 7, 8], "timeout": 4, "variat": 4, "inter": 4, "rater": 4, "ti": 4, "holist": [4, 7], "fast": [4, 7, 8], "experiment": [4, 5, 8], "vi": 4, "categor": [4, 7, 8], "intrins": [4, 5], "extrins": 4, "sequenc": [4, 5, 8], "perplex": [4, 5], "downstream": [4, 8], "synthesi": 4, "discret": 4, "prefix": [4, 7], "roug": 4, "bleu": 4, "bilingu": 4, "understudi": 4, "overlap": [4, 6], "favor": [4, 8], "breviti": 4, "insensit": 4, "semant": [4, 6], "orient": [4, 7], "gist": 4, "meteor": 4, "synonym": 4, "stem": [4, 8], "paraphras": 4, "alongsid": [4, 7], "computation": [4, 6], "cider": 4, "consensu": 4, "tf": 4, "idf": 4, "caption": 4, "reliant": 4, "corpu": 4, "ter": 4, "edit": [4, 7], "hypothesi": 4, "penal": 4, "bertscor": 4, "contextu": [4, 7], "bert": 4, "spice": 4, "proposit": 4, "scene": [4, 7], "pure": [4, 5], "analyst": [4, 6], "rouge_1": 4, "rouge_2": 4, "ideal": [4, 5, 7, 8], "cheaper": 4, "setup": [4, 5, 7, 8], "evaluate_summari": 4, "unigram": 4, "bigram": 4, "absl": 4, "py": 4, "rouge_scor": 4, "generated_summari": 4, "reference_summari": 4, "google_bleu": 4, "bleu_scor": 4, "rouge1": 4, "rouge2": 4, "arbitrari": 4, "chosen": [4, 7], "sentence1": 4, "cat": [4, 7], "sat": 4, "mat": 4, "sentence2": 4, "ate": 4, "3333333333333333": 4, "7272727272727272": 4, "4444444444444445": 4, "generate_summari": 4, "summir": 4, "liner": 4, "evaluate_summary_model": 4, "model_benchmark": 4, "models_test": 4, "benchmark_summari": 4, "model_summari": 4, "evaluation_result": 4, "statu": 4, "concis": 4, "element": [4, 7, 8], "verbos": [4, 5, 7], "peripher": 4, "quit": [4, 8], "convei": [4, 6], "breadth": 4, "Of": [4, 7], "vibe": 4, "visualize_prompt_comparison": 4, "matplotlib": 4, "radar": 4, "radar_plot": 4, "tmp": 4, "ipykernel_1652501": 4, "940173201": 4, "userwarn": 4, "figurecanvasagg": 4, "largest": 4, "granular": [4, 5, 6], "tune": [4, 5, 7, 8], "likert": 4, "ensembl": 4, "repeatedli": 4, "fluenci": 4, "refin": 4, "notabl": [4, 7, 8], "integ": 4, "rubric": 4, "hollist": 4, "judgeevalu": 4, "grammar": [4, 5, 8], "evaluate_with_llm": 4, "criterion": 4, "judge_model": 4, "candidate_summari": 4, "grammat": 4, "y": [4, 7, 8], "z": 4, "w": [4, 5, 6, 7], "benchmark_model": 4, "test_model": 4, "input_text": [4, 5, 6], "trillion": [4, 8], "evals_list": 4, "1775618912": 4, "variant": [4, 5, 7], "slightli": 4, "drift": [4, 7], "lowest": [4, 5], "firstli": 4, "overhead": [4, 5], "egocentr": 4, "tight": 4, "aproach": 4, "aplic": 4, "clearli": [4, 7, 8], "earlier": [4, 7], "depict": [4, 7, 8], "multilingu": [4, 7], "golden": 4, "languang": 4, "arena": 4, "randomli": 4, "customiz": [4, 5, 7], "irrelev": 4, "unhelp": [4, 7], "occasion": 4, "rare": 4, "perfectli": 4, "cater": 4, "critiqu": [4, 7], "elo": 4, "spectrum": 4, "exam": 4, "probe": [4, 7], "certifi": 4, "began": [4, 5], "glue": 4, "entail": 4, "baselin": [4, 5, 7], "superglu": 4, "successor": 4, "grew": 4, "big": 4, "bench": 4, "srivastava": 4, "arithmet": 4, "truthfulqa": 4, "multitask": 4, "hendryck": [4, 7], "multidisciplinari": 4, "stanford": 4, "helm": 4, "multidimension": 4, "surround": [4, 7, 8], "humanev": 4, "lmsy": 4, "brought": 4, "dialogu": 4, "chiang": 4, "gather": 4, "alpacaev": 4, "duboi": 4, "mt": 4, "argilla": 4, "mila": 4, "mit": [4, 5], "contributor": [4, 5, 8], "western": 4, "centric": 4, "divid": [4, 6, 7], "subset": [4, 7], "agnost": 4, "dialect": 4, "render": [4, 7], "crowdsourc": 4, "livebench": 4, "white": [4, 7], "resili": [4, 7], "meaningfulli": 4, "zebralog": 4, "grid": 4, "puzzl": 4, "brailsford": 4, "1999": 4, "lsat": 4, "hous": 4, "clue": 4, "deduct": 4, "arriv": 4, "programmat": [4, 8], "2x2": 4, "6x6": 4, "shot": [4, 7], "reductio": 4, "ad": [4, 7, 8], "absurdum": 4, "hard": 4, "10b": 4, "counterfactu": 4, "came": 4, "arc": 4, "prize": [4, 7], "chollet": 4, "mike": [4, 7], "knoop": 4, "founder": 4, "zapier": 4, "fran\u00e7oi": 4, "creator": 4, "agi": 4, "kera": 4, "genuin": 4, "possess": 4, "elementari": 4, "novelti": 4, "wouldn": 4, "interpol": 4, "synthes": 4, "fly": 4, "retriev": [4, 5], "brute": 4, "pixel": 4, "unbeaten": 4, "win": 4, "poorli": 4, "recombin": 4, "spur": [4, 7], "takeawai": 4, "vertic": [4, 7], "finbench": 4, "legalbench": 4, "guha": 4, "berkelei": 4, "bfcl": 4, "patil": 4, "fourrier": 4, "bespok": 4, "sdk": 4, "autoregress": 4, "sub": [4, 5], "liter": 4, "disturb": 4, "zero": [4, 5, 7, 8], "varianc": [4, 7], "yt": 4, "ut": 4, "suppos": [4, 8], "ol": 4, "heteroscedast": 4, "regress": 4, "bivari": 4, "evaluation_track": 4, "evaluationtrack": 4, "model_config": 4, "basemodelconfig": 4, "parallelismmanag": 4, "pipelineparamet": 4, "envconfig": 4, "is_accelerate_avail": 4, "datetim": 4, "timedelta": 4, "initprocessgroupkwarg": 4, "create_evaluation_pipelin": 4, "cache_dir": 4, "float16": 4, "max_sampl": 4, "kwargs_handl": 4, "3000": 4, "save_detail": 4, "pipeline_param": 4, "launcher_typ": 4, "env_config": 4, "override_batch_s": 4, "use_chat_templ": 4, "trust_remote_cod": 4, "pipeline_paramet": 4, "schemat": [4, 6], "vllm": [4, 8], "tgi": 4, "storag": [4, 5, 7], "num_few_shot": 4, "bar": 4, "bigbench": 4, "winogrand": 4, "hellaswag": 4, "nlp": [4, 5, 7], "save_and_push_result": 4, "show_result": 4, "model_arg": 4, "send": [4, 5, 7, 8], "serverless": 4, "inference_server_address": 4, "inference_server_auth": 4, "model_id": 4, "null": 4, "bash": [4, 5], "command": [4, 5], "model_config_path": 4, "endpoint_model": 4, "llama3": [4, 6], "qwen2": [4, 5, 8], "smollm2": 4, "3b": 4, "alibaba": [4, 8], "5b": [4, 5, 8], "hui": 4, "allal": 4, "cluster": 4, "noteworthi": 4, "grain": [4, 8], "salt": [4, 8], "exponenti": 4, "modular": 4, "offici": 4, "revisit": 4, "trace": 4, "langchain_tracing_v2": 4, "langchain_api_kei": 4, "hf_evalu": 4, "langsmith_evalu": 4, "ls_client": 4, "dataset_nam": 4, "create_dataset": 4, "create_exampl": 4, "dataset_id": 4, "calculate_scor": 4, "reference_output": 4, "oai_client": 4, "xp_model_nam": 4, "lastli": 4, "run_evalu": 4, "And": [4, 5, 7], "upload_result": 4, "experiment_prefix": 4, "num_repetit": 4, "386a3620": 4, "9e1cc3cb": 4, "9d6a": 4, "4356": 4, "ab34": 4, "138e0abe8be4": 4, "8741976e": 4, "5268": 4, "4b75": 4, "949f": 4, "99477dde5d64": 4, "selectedsess": 4, "b831dc1e": 4, "90bc": 4, "4ed8": 4, "8080": [4, 5], "fb42444724d6": 4, "4it": 4, "latest": [4, 5, 6, 7, 8], "tobia": 4, "evaluate_modul": 4, "6fc70b7be0088120a372dfdd5d320b39b8bb3630cb8029b193941d9376e86bb0": 4, "tue": 4, "nov": 4, "couldn": 4, "5it": 4, "5053784e": 4, "64445871": 4, "a53c": 4, "44b1": 4, "a422": 4, "4f49b2f9656f": 4, "69": [4, 7], "4b29f3c9": 4, "9ef7e39a": 4, "2add": 4, "410c": 4, "89f8": 4, "9f1a8b198cf1": 4, "61": [4, 7], "insert": 4, "combined_df": 4, "concat": [4, 7], "ignore_index": [4, 7], "execution_tim": 4, "example_id": 4, "333333": 4, "224388": 4, "feb10f92": 4, "3167": 4, "41f3": 4, "bb1c": 4, "d271153a31a8": 4, "5b196b22": 4, "9f4c": 4, "489c": 4, "b020": 4, "7823208b42d6": 4, "348101": 4, "722464": 4, "c310f159": 4, "064a": 4, "4035": 4, "97c3": 4, "a25bbf43abc2": 4, "386076": 4, "704104": 4, "f7f24899": 4, "dd50": 4, "409e": 4, "93cc": 4, "6fb1622b60bf": 4, "443038": 4, "725059": 4, "242856d6": 4, "efb5": 4, "4101": 4, "b1cf": 4, "5805532838ac": 4, "373418": 4, "795302": 4, "ce975169": 4, "a0ab": 4, "40ce": 4, "8e32": 4, "efa28d06079d": 4, "stat": [4, 5], "groupbi": [4, 7], "agg": [4, 7], "sort": 4, "sort_valu": 4, "subplot": 4, "pyplot": 4, "plt": 4, "numpi": 4, "np": 4, "ax1": 4, "ax2": 4, "figsiz": 4, "2ecc71": 4, "3498db": 4, "e74c3c": 4, "bleu_mean": 4, "bleu_std": 4, "enumer": [4, 6, 7], "errorbar": 4, "yerr": 4, "fmt": 4, "markers": 4, "capsiz": 4, "set_ylabel": 4, "set_titl": 4, "set_xtick": 4, "set_xticklabel": 4, "rotat": 4, "set_ylim": 4, "bottom": 4, "legend": 4, "exec_mean": 4, "exec_std": 4, "tight_layout": 4, "ndetail": 4, "4038": 4, "0453": 4, "7815": 4, "0433": 4, "3768": 4, "0424": 4, "8343": 4, "2208": 4, "3519": 4, "0775": 4, "9122": 4, "1482": 4, "377": 4, "042": 4, "078": 4, "slower": [4, 7], "04": [4, 5, 6], "latenc": [4, 5, 6, 7], "speed": [4, 5, 7], "interestingli": 4, "decoupl": 4, "reload": 4, "facilit": [4, 7], "promptfooconfig": 4, "model_comparison": 4, "pretti": [4, 7], "dump": 4, "default_flow_styl": 4, "sort_kei": 4, "prompt1": 4, "defaulttest": 4, "1000m": 4, "millisecond": 4, "eval_data": 4, "latency_m": 4, "totallatencym": 4, "token_usag": 4, "tokenusag": 4, "assert_pass": 4, "assertpasscount": 4, "assert_fail": 4, "assertfailcount": 4, "prompt_token": [4, 5], "num_request": 4, "numrequest": 4, "2463": 4, "000035": 4, "3773": 4, "004620": 4, "1669": 4, "000091": 4, "1669m": 4, "highest": [4, 5], "3773m": 4, "00462": 4, "promptfool": 4, "manual": [4, 5, 7], "redefin": 4, "prompt_comparison": 4, "prompt2": 4, "prompt3": 4, "prompt_fil": 4, "prompt_cont": 4, "BE": 4, "again": 4, "prompt_id": 4, "promptid": 4, "gradingresult": 4, "df_raw": 4, "reset_index": [4, 7], "eas": [4, 5, 7], "seamless": [4, 7], "hf": 4, "plain": [4, 5], "vanilla": 4, "defi": 4, "accustom": 4, "legaci": [4, 5], "unsustain": 4, "prd": 4, "cultiv": [4, 7], "organiz": 4, "stagnat": 4, "alb": 4, "loubna": 4, "anton": 4, "lozhkov": 4, "bakouch": 4, "gabriel": [4, 7], "mart\u00edn": [4, 7], "bl\u00e1zquez": 4, "lewi": 4, "tunstal": 4, "agust\u00edn": 4, "piquer": 4, "andr": 4, "marafioti": 4, "cyril": 4, "zakka": 4, "leandro": 4, "werra": 4, "wolf": 4, "are24": 4, "judgearena": 4, "bps99": 4, "salli": 4, "pott": 4, "barbara": 4, "557": [4, 7], "sciencedirect": 4, "s0377221798003646": 4, "doi": [4, 7, 8], "1016": 4, "s0377": 4, "2217": 4, "00364": 4, "ctj": 4, "jerri": [4, 7], "tworek": [4, 7], "heewoo": [4, 7], "jun": [4, 7], "qime": [4, 7], "henriqu": [4, 7], "pond": [4, 7], "de": [4, 7], "oliveira": [4, 7], "pinto": [4, 7], "harri": [4, 7], "yuri": 4, "burda": 4, "greg": [4, 7], "brockman": [4, 7], "raul": [4, 7], "puri": [4, 7], "gretchen": [4, 7], "krueger": [4, 7], "petrov": [4, 7], "heidi": 4, "khlaaf": 4, "girish": [4, 7], "sastri": [4, 7], "brook": [4, 7], "chan": [4, 7], "grai": [4, 7], "ryder": [4, 7], "mikhail": [4, 7], "pavlov": [4, 7], "alethea": [4, 7], "lukasz": 4, "kaiser": [4, 7], "mohammad": [4, 7], "bavarian": [4, 7], "clemen": [4, 7], "winter": [4, 7], "philipp": 4, "tillet": [4, 7], "felip": [4, 7], "petroski": [4, 7], "dave": [4, 7], "cum": [4, 7], "plappert": 4, "fotio": 4, "chantzi": [4, 7], "barn": 4, "ariel": 4, "herbert": 4, "voss": [4, 7], "hebgen": 4, "guss": 4, "nichol": 4, "paino": [4, 7], "nikola": [4, 7], "tezak": [4, 7], "babuschkin": [4, 7], "suchir": [4, 7], "balaji": [4, 7], "shantanu": [4, 7], "jain": [4, 7], "hess": [4, 7], "carr": 4, "josh": [4, 7], "achiam": [4, 7], "vedant": 4, "misra": 4, "evan": [4, 7], "morikawa": [4, 7], "matthew": 4, "knight": [4, 7], "mile": [4, 7], "brundag": [4, 7], "mira": [4, 7], "murati": [4, 7], "kati": [4, 7], "mayer": [4, 7], "bob": [4, 7, 8], "mcgrew": [4, 7], "ilya": [4, 7], "sutskev": [4, 7], "wojciech": [4, 7], "zaremba": [4, 7], "2107": 4, "03374": 4, "cz": 4, "lianmin": 4, "ying": 4, "sheng": 4, "anastasio": 4, "angelopoulo": 4, "tianl": 4, "dacheng": 4, "banghua": 4, "jordan": [4, 7], "gonzalez": 4, "ion": 4, "stoica": 4, "04132": 4, "cho24a": 4, "francoi": 4, "arcpriz": 4, "cho24b": 4, "dglh24": 4, "yann": 4, "bal\u00e1z": 4, "galambosi": 4, "tatsunori": 4, "hashimoto": 4, "debia": 4, "04475": 4, "fac24a": 4, "wiki": [4, 8], "fac24b": 4, "fac24c": 4, "model_doc": 4, "fac24d": 4, "cookbook": 4, "llm_judg": 4, "fac24f": 4, "fhwt23": 4, "cl\u00e9mentin": 4, "nathan": 4, "habib": 4, "gnh": 4, "julian": 4, "nyarko": 4, "ho": 4, "r\u00e9": 4, "adam": [4, 7], "chilton": 4, "aditya": [4, 7], "narayana": 4, "chohla": 4, "brandon": [4, 7, 8], "waldon": 4, "rockmor": 4, "diego": 4, "zambrano": 4, "dmitri": 4, "talisman": 4, "enam": 4, "hoqu": 4, "faiz": 4, "surani": 4, "frank": [4, 7], "fagan": 4, "galit": 4, "sarfati": 4, "gregori": 4, "dickinson": 4, "haggai": 4, "porat": 4, "hegland": 4, "jessica": [4, 7], "joe": [4, 7], "nudel": 4, "joel": [4, 7], "niklau": 4, "nai": 4, "jonathan": [4, 7], "choi": 4, "margaret": 4, "hagan": 4, "megan": 4, "ma": [4, 7], "livermor": 4, "nikon": 4, "rasumov": 4, "rahe": 4, "nil": 4, "holzenberg": 4, "noam": 4, "kolt": 4, "henderson": 4, "rehaag": 4, "sharad": 4, "shang": 4, "spencer": 4, "sunni": 4, "gandhi": 4, "zur": 4, "varun": 4, "iyer": 4, "zehua": 4, "2308": 4, "11462": 4, "hbb": 4, "collin": 4, "burn": 4, "steven": [4, 7], "basart": [4, 7], "zou": [4, 7], "manta": [4, 7], "mazeika": [4, 7], "03300": 4, "hbd": 4, "maxwel": 4, "forb": 4, "yejin": 4, "curiou": 4, "neural": [4, 8], "degener": 4, "1904": 4, "09751": 4, "hyc": 4, "binyuan": 4, "zeyu": 4, "cui": 4, "jiaxi": 4, "dayiheng": 4, "tianyu": 4, "jiajun": 4, "kai": [4, 7], "dang": 4, "coder": 4, "preprint": [4, 8], "2409": [4, 7], "12186": 4, "lx": 4, "zhen": 4, "xiaohan": 4, "jia": 4, "yuxuan": 4, "lai": 4, "chongyang": 4, "shuai": 4, "nlg": 4, "07103": 4, "lbl": 4, "bommasani": 4, "toni": 4, "dimitri": 4, "tsipra": 4, "dilara": 4, "soylu": 4, "michihiro": 4, "yasunaga": 4, "yian": 4, "deepak": 4, "narayanan": 4, "yuhuai": 4, "newman": 4, "binhang": 4, "bobbi": 4, "ce": 4, "christian": [4, 7], "cosgrov": 4, "acosta": 4, "nava": [4, 7], "drew": 4, "hudson": 4, "zelikman": 4, "esin": 4, "durmu": 4, "faisal": 4, "ladhak": 4, "frieda": 4, "rong": 4, "hongyu": 4, "ren": 4, "huaxiu": 4, "yao": [4, 7], "jue": 4, "keshav": 4, "santhanam": 4, "laurel": 4, "lucia": 4, "mert": 4, "yuksekgonul": 4, "mirac": 4, "suzgun": 4, "niladri": 4, "chatterji": 4, "omar": 4, "khattab": 4, "qian": [4, 7], "chi": [4, 8], "sang": 4, "shibani": [4, 7], "santurkar": [4, 7], "surya": 4, "icard": 4, "tianyi": 4, "vishrav": 4, "chaudhari": 4, "xuechen": 4, "yuhui": 4, "yuta": 4, "koreeda": 4, "2211": 4, "09110": 4, "lbc24": 4, "ronan": 4, "bra": 4, "allenai": 4, "lhe22": [4, 7], "stephani": [4, 7], "owain": [4, 7], "mimic": [4, 7], "falsehood": [4, 7], "2109": [4, 7], "07958": [4, 7], "pzwg23": 4, "shishir": 4, "tianjun": 4, "xin": [4, 7], "gorilla": 4, "15334": 4, "pro24": 4, "dev": 4, "ras24": 4, "sebastian": 4, "scratch": 4, "1633437166": 4, "srf": 4, "shivalika": 4, "angelika": 4, "roman": [4, 7], "adelani": 4, "ngui": 4, "vila": 4, "suero": 4, "peerat": 4, "limkonchotiwat": 4, "kelli": 4, "marchisio": 4, "qi": 4, "leong": 4, "yosephin": 4, "susanto": 4, "raymond": [4, 7], "ng": [4, 7], "shayn": 4, "longpr": 4, "ko": 4, "madelin": 4, "antoin": 4, "bosselut": 4, "oh": 4, "leshem": 4, "choshen": 4, "daphn": 4, "ippolito": 4, "enzo": 4, "ferrant": 4, "marzieh": 4, "fadae": 4, "beyza": 4, "ermi": 4, "sara": 4, "hooker": 4, "linguist": [4, 7], "03304": 4, "srr": 4, "aarohi": 4, "abhinav": 4, "rastogi": 4, "abhishek": 4, "rao": 4, "abu": 4, "awal": 4, "shoeb": 4, "abubakar": 4, "abid": 4, "fisch": 4, "santoro": 4, "gupta": 4, "adri\u00e0": 4, "garriga": 4, "alonso": 4, "agnieszka": 4, "kluska": 4, "aitor": 4, "lewkowycz": 4, "akshat": 4, "warstadt": 4, "alexand": [4, 7, 8], "kocurek": 4, "ali": [4, 7], "safaya": 4, "tazarv": 4, "aman": 4, "hussain": 4, "dsouza": 4, "ambros": 4, "slone": 4, "ameet": 4, "rahan": 4, "anantharaman": 4, "ander": 4, "andreassen": 4, "madotto": 4, "santilli": 4, "stuhlm\u00fcller": 4, "la": 4, "lampinen": 4, "angelica": 4, "anh": 4, "vuong": 4, "animesh": 4, "gottardi": 4, "antonio": 4, "norelli": 4, "anu": 4, "venkatesh": 4, "arash": 4, "gholamidavoodi": 4, "arfa": 4, "tabassum": 4, "arul": 4, "menez": 4, "arun": [4, 7], "kirubarajan": 4, "asher": 4, "mullokandov": 4, "ashish": 4, "sabharw": 4, "herrick": 4, "avia": 4, "efrat": 4, "aykut": 4, "erdem": 4, "ayla": 4, "karaka\u015f": 4, "bao": [4, 7], "loe": 4, "barret": [4, 7], "zoph": [4, 7], "bart\u0142omiej": 4, "bojanowski": 4, "batuhan": 4, "\u00f6zyurt": 4, "behnam": 4, "hedayatnia": 4, "neyshabur": 4, "inden": 4, "benno": 4, "stein": 4, "berk": 4, "ekmekci": 4, "blake": 4, "howald": 4, "bryan": 4, "orinion": 4, "diao": 4, "dour": 4, "stinson": 4, "cedrick": 4, "argueta": 4, "c\u00e9sar": 4, "ferri": 4, "ram\u00edrez": 4, "chandan": 4, "charl": 4, "rathkopf": 4, "chenlin": 4, "meng": 4, "chitta": 4, "baral": 4, "chiyu": 4, "callison": 4, "burch": 4, "wait": [4, 7], "voigt": 4, "cindi": 4, "ramirez": 4, "clara": 4, "rivera": 4, "clemencia": 4, "siro": 4, "colin": 4, "raffel": 4, "courtnei": 4, "ashcraft": 4, "cristina": 4, "garbacea": 4, "damien": [4, 7], "sileo": 4, "garrett": 4, "kilman": 4, "freeman": 4, "khashabi": 4, "levi": [4, 7], "mosegu\u00ed": 4, "gonz\u00e1lez": 4, "perszyk": 4, "danqi": 4, "dar": 4, "gilboa": 4, "dohan": [4, 7], "drakard": 4, "jurgen": 4, "debajyoti": 4, "datta": 4, "deni": 4, "emelin": 4, "kleyko": 4, "deniz": 4, "yuret": 4, "derek": [4, 7], "tam": [4, 8], "dieuwk": 4, "hupk": 4, "diganta": 4, "dilyar": 4, "buzan": 4, "coelho": 4, "mollo": 4, "diyi": 4, "dylan": 4, "schrader": 4, "ekaterina": 4, "shutova": 4, "ekin": 4, "dogu": 4, "cubuk": 4, "elad": 4, "segal": 4, "eleanor": 4, "hagerman": 4, "donowai": 4, "elli": 4, "pavlick": 4, "rodola": 4, "emma": 4, "lam": 4, "chu": [4, 7], "erkut": 4, "erni": 4, "dyer": 4, "jerzak": 4, "eunic": 4, "engefu": 4, "manyasi": 4, "evgenii": 4, "zheltonozhskii": 4, "fanyu": 4, "xia": 4, "fatemeh": 4, "siar": 4, "fernando": 4, "mart\u00ednez": 4, "plume": 4, "francesca": 4, "happ\u00e9": 4, "gaurav": 4, "genta": 4, "indra": 4, "winata": 4, "gerard": 4, "melo": 4, "germ\u00e1n": 4, "kruszewski": 4, "giambattista": [4, 7], "parascandolo": [4, 7], "giorgio": 4, "mariani": 4, "gloria": 4, "gonzalo": 4, "jaimovitch": 4, "l\u00f3pez": 4, "gregor": 4, "betz": 4, "gui": [4, 5], "gur": 4, "hana": 4, "galijasev": 4, "rashkin": 4, "hannaneh": 4, "hajishirzi": 4, "harsh": 4, "hayden": 4, "bogar": 4, "henri": [4, 7], "shevlin": 4, "hinrich": 4, "sch\u00fctze": 4, "hiromu": 4, "yakura": 4, "hongm": 4, "hugh": 4, "mee": 4, "wong": [4, 7], "isaac": 4, "nobl": 4, "jaap": 4, "jumelet": 4, "geissing": 4, "jaehoon": 4, "jaim": 4, "fern\u00e1ndez": 4, "fisac": 4, "simon": 4, "koppel": 4, "koco\u0144": 4, "jana": 4, "thompson": [4, 5, 7], "janel": 4, "wingfield": 4, "jarema": 4, "radom": 4, "jascha": 4, "sohl": [4, 7], "dickstein": 4, "phang": 4, "yosinski": 4, "jekaterina": 4, "novikova": 4, "jell": 4, "bosscher": 4, "jennif": 4, "marsh": 4, "jeroen": 4, "taal": 4, "engel": 4, "jesujoba": 4, "alabi": 4, "jiam": 4, "jillian": 4, "joan": 4, "waweru": 4, "burden": 4, "bali": 4, "batcheld": 4, "berant": 4, "j\u00f6rg": 4, "frohberg": 4, "jo": 4, "rozen": 4, "orallo": 4, "boudeman": 4, "guerr": 4, "tenenbaum": 4, "joyc": 4, "chua": 4, "kanclerz": 4, "karen": 4, "livescu": 4, "karl": 4, "krauth": 4, "karthik": 4, "gopalakrishnan": 4, "katerina": 4, "ignatyeva": 4, "katja": 4, "markert": 4, "kaustubh": 4, "dhole": 4, "gimpel": 4, "omondi": 4, "kori": 4, "mathewson": 4, "kristen": 4, "chiafullo": 4, "ksenia": 4, "shkaruta": 4, "shridhar": 4, "kyle": [4, 7], "mcdonel": 4, "richardson": 4, "laria": 4, "reynold": 4, "leo": [4, 7], "dugan": 4, "lianhui": 4, "lidia": 4, "contrera": 4, "ochando": 4, "morenc": 4, "moschella": 4, "luci": 4, "ludwig": 4, "schmidt": [4, 7], "luheng": 4, "olivero": 4, "col\u00f3n": 4, "metz": [4, 7], "l\u00fctfi": 4, "kerem": 4, "\u015fenel": 4, "maarten": [4, 7], "bosma": 4, "sap": [4, 7], "maartj": 4, "hoev": 4, "maheen": 4, "farooqi": 4, "manaal": 4, "faruqui": 4, "marco": 4, "baturan": 4, "marelli": 4, "maru": 4, "maria": 4, "quintana": 4, "tolkiehn": 4, "mario": [4, 7], "giulianelli": 4, "martha": 4, "potthast": 4, "leavitt": 4, "hagen": 4, "m\u00e1ty\u00e1": 4, "schubert": 4, "medina": [4, 7], "orduna": 4, "baitemirova": 4, "melodi": 4, "arnaud": 4, "melvin": 4, "mcelrath": 4, "yee": 4, "cohen": 4, "ivanitskii": 4, "starritt": 4, "strube": 4, "micha\u0142": 4, "sw\u0119drowski": 4, "michel": [4, 7], "bevilacqua": 4, "mihir": 4, "kale": 4, "cain": 4, "mime": 4, "mitch": 4, "walker": 4, "mo": 4, "tiwari": 4, "mohit": 4, "bansal": 4, "moin": 4, "aminnaseri": 4, "mor": 4, "geva": 4, "mozhdeh": 4, "gheini": 4, "mukund": 4, "varma": 4, "nanyun": 4, "peng": [4, 7], "nayeon": 4, "neta": 4, "krakov": 4, "doiron": 4, "nicol": 4, "martinez": 4, "nikita": 4, "nangia": 4, "nikla": 4, "decker": 4, "muennighoff": 4, "nitish": [4, 7], "shirish": [4, 7], "keskar": [4, 7], "niveditha": 4, "constant": 4, "fiedel": 4, "nuan": 4, "wen": 4, "oliv": [4, 7], "agha": 4, "elbaghdadi": 4, "omer": 4, "moreno": 4, "casar": 4, "parth": 4, "doshi": 4, "pascal": 4, "fung": 4, "pu": 4, "vicol": 4, "pegah": 4, "alipoormolabashi": 4, "peiyuan": 4, "eckerslei": 4, "phu": 4, "mon": 4, "htut": 4, "pinyu": 4, "hwang": 4, "piotr": 4, "mi\u0142kowski": 4, "piyush": 4, "pouya": 4, "pezeshkpour": 4, "priti": 4, "oli": 4, "qiaozhu": 4, "qing": 4, "lyu": 4, "qinlang": 4, "rabin": 4, "banjad": 4, "rachel": [4, 7], "etta": 4, "rudolph": 4, "raefer": 4, "rahel": 4, "haback": 4, "ramon": 4, "risco": 4, "rapha\u00ebl": 4, "milli\u00e8r": 4, "rhythm": 4, "garg": 4, "rif": 4, "saurou": 4, "riku": 4, "arakawa": 4, "robb": 4, "raymaek": 4, "rohan": 4, "sikand": 4, "novak": 4, "sitelew": 4, "lebra": 4, "rosann": 4, "rowan": [4, 7], "ruslan": 4, "salakhutdinov": 4, "stoval": 4, "teehan": 4, "sahib": 4, "saif": 4, "sajant": 4, "anand": [4, 7], "dillav": 4, "shleifer": 4, "wiseman": 4, "gruetter": 4, "schoenholz": 4, "sanghyun": 4, "sanjeev": 4, "kwatra": 4, "sarik": 4, "ghazarian": 4, "sayan": 4, "casei": [4, 7], "bischoff": 4, "gehrmann": 4, "schuster": 4, "sepideh": 4, "sadeghi": 4, "shadi": 4, "hamdan": 4, "sharon": 4, "shashank": 4, "sherri": 4, "shi": 4, "shikhar": 4, "shima": 4, "asaadi": 4, "shubh": 4, "pachchigar": 4, "shubham": 4, "toshniw": 4, "shyam": [4, 7], "upadhyai": 4, "shyamolima": 4, "debnath": 4, "siamak": 4, "shakeri": 4, "thormey": 4, "melzi": 4, "siva": 4, "reddi": 4, "sneha": 4, "priscilla": 4, "makini": 4, "soo": 4, "hwan": 4, "toren": 4, "sriharsha": 4, "hatwar": 4, "stanisla": 4, "dehaen": 4, "stefan": 4, "divic": 4, "stella": 4, "biderman": 4, "stephen": 4, "prasad": 4, "piantadosi": 4, "stuart": [4, 7], "shieber": 4, "summer": [4, 7], "misherghi": 4, "svetlana": 4, "kiritchenko": 4, "swaroop": 4, "tal": 4, "linzen": 4, "tariq": 4, "tatsu": 4, "te": 4, "th\u00e9o": 4, "desbord": 4, "theodor": 4, "rothschild": 4, "phan": [4, 7], "tiberiu": 4, "nkinyili": 4, "timo": 4, "schick": 4, "timofei": 4, "kornev": 4, "titu": 4, "tunduni": 4, "gerstenberg": 4, "trenton": 4, "trishala": 4, "neeraj": 4, "tushar": 4, "khot": 4, "shultz": 4, "uri": 4, "shaham": 4, "vera": 4, "demberg": 4, "victoria": [4, 7], "nyamai": 4, "vika": 4, "raunak": 4, "vinai": 4, "ramasesh": 4, "udai": 4, "prabhu": 4, "vishakh": 4, "padmakumar": 4, "vivek": 4, "srikumar": 4, "fedu": [4, 7], "wout": 4, "vossen": 4, "xiaoyu": 4, "tong": [4, 7], "xinran": 4, "xinyi": 4, "yadollah": 4, "yaghoobzadeh": 4, "yair": 4, "lakretz": 4, "yangqiu": 4, "yasaman": 4, "bahri": 4, "yichi": 4, "yide": 4, "yifu": 4, "yonatan": 4, "belinkov": 4, "yufang": 4, "seid": 4, "zhuoy": 4, "zijian": 4, "ziji": 4, "zirui": 4, "ziyi": 4, "extrapol": 4, "2206": 4, "04615": 4, "wpn": 4, "yada": 4, "pruksachatkun": 4, "amanpreet": 4, "hill": 4, "stickier": 4, "wsm": 4, "1804": 4, "07461": 4, "wtb": 4, "tai": 4, "borgeaud": 4, "dani": 4, "yogatama": 4, "denni": [4, 7], "donald": 4, "metzler": 4, "ed": 4, "oriol": 4, "vinyal": 4, "dean": 4, "07682": 4, "wdr": 4, "doolei": 4, "manlei": 4, "arka": [4, 7], "pal": 4, "feuer": 4, "siddhartha": 4, "ravid": 4, "shwartz": [4, 7], "ziv": 4, "khalid": 4, "saifullah": 4, "siddartha": 4, "naidu": 4, "chinmai": 4, "hegd": 4, "lecun": 4, "goldstein": 4, "willi": 4, "neiswang": 4, "micah": 4, "goldblum": 4, "19314": 4, "yyh": 4, "baosong": 4, "chengpeng": 4, "chengyuan": 4, "fei": 4, "guant": 4, "haoran": 4, "huan": 4, "jialong": 4, "jialin": 4, "jianhong": 4, "tu": 4, "jianwei": 4, "jianxin": 4, "jin": [4, 7], "jingren": 4, "jinz": 4, "jinzheng": 4, "junyang": 4, "keme": 4, "keqin": 4, "kexin": 4, "mingfeng": 4, "xue": [4, 7], "ni": 4, "pei": 4, "ru": 4, "men": 4, "ruiz": 4, "runji": 4, "shiji": 4, "sinan": 4, "tianhang": 4, "wenbin": 4, "ge": 4, "xiaodong": 4, "deng": 4, "xiaohuan": 4, "xingzhang": 4, "xinyu": [4, 7], "xipin": 4, "xuancheng": 4, "yichang": 4, "wan": 4, "yunfei": 4, "yuqiong": 4, "zhenru": 4, "zhihao": 4, "10671": 4, "zcl24": 4, "zhihan": 4, "cao": 4, "lizi": 4, "openreview": 4, "forum": 4, "aegrf1uy0p": 4, "zc": 4, "siyuan": 4, "zhuang": [4, 7], "zhanghao": 4, "yonghao": 4, "zi": 4, "zhuohan": 4, "xing": [4, 7], "2306": 4, "05685": 4, "huggingface24": 4, "06": [4, 8], "metaai24": 4, "di": 5, "hunter": 5, "hipaa": 5, "properti": [5, 7], "gdpr": 5, "iot": 5, "unreli": 5, "impract": 5, "programm": 5, "graphic": [5, 7], "vram": 5, "vector": [5, 7], "mathbf": 5, "x_1": 5, "x_2": 5, "x_n": 5, "x_": 5, "\u03b8": 5, "matrix": [5, 7], "concurr": 5, "mozilla": 5, "docker": 5, "gerganov": 5, "georgi": 5, "hundr": 5, "overwhelm": [5, 8], "manifesto": 5, "enjoy": 5, "bog": 5, "exploratori": 5, "hacker": 5, "Will": [5, 7], "codebas": 5, "prototyp": 5, "prematur": 5, "besid": 5, "lighter": 5, "sacrific": 5, "gguf": 5, "unifi": 5, "ggml": [5, 8], "ibm": [5, 7], "bit": 5, "metadata": 5, "disk": 5, "faster": 5, "backward": 5, "2024x": 5, "repo": 5, "easier": [5, 6, 7, 8], "compil": 5, "linux": 5, "argument": [5, 7, 8], "sudo": 5, "apt": 5, "cmake": 5, "bind": 5, "betlen": 5, "cnv": 5, "llamacpp": 5, "q8_0": 5, "succinct": 5, "ctrl": 5, "interject": 5, "philosoph": 5, "debat": 5, "fulfil": 5, "happi": 5, "responsibli": 5, "bye": 5, "goodby": 5, "port": 5, "127": 5, "curl": [5, 8], "localhost": 5, "v1": [5, 6, 7], "bearer": 5, "finish_reason": 5, "deepli": 5, "1734627879": 5, "completion_token": 5, "total_token": 5, "chatcmpl": 5, "5wl2tzjzdmzupvxwp2gcedr8xbpsyhfm": 5, "prompt_n": 5, "prompt_m": 5, "132": 5, "prompt_per_token_m": 5, "prompt_per_second": 5, "77619878666999": 5, "predicted_n": 5, "predicted_m": 5, "1700": 5, "654": 5, "predicted_per_token_m": 5, "36882142857143": 5, "predicted_per_second": 5, "92850867960208": 5, "wrestl": [5, 7], "gbnf": [5, 8], "8pm": 5, "appointmenttim": 5, "appointmentdetail": 5, "handi": 5, "modal": 5, "model_path": 5, "llama_cpp": 5, "create_chat_complet": 5, "occupi": 5, "activist": 5, "justin": [5, 7], "tunnei": 5, "ocho": 5, "appach": 5, "cosmopolitan": 5, "libc": 5, "portabl": 5, "durabl": 5, "usabl": [5, 7, 8], "tinyllama": 5, "wget": 5, "jartin": 5, "q5_k_m": 5, "renam": 5, "ex": 5, "chmod": 5, "nobrows": 5, "registri": 5, "nativ": [5, 8], "container": 5, "isol": [5, 7], "trai": 5, "familiar": 5, "permiss": [5, 7], "despit": [5, 6, 8], "ssfl": 5, "sh": [5, 8], "Or": 5, "11434": 5, "chatrespons": 5, "tradeoff": [5, 7], "easiest": 5, "rich": [5, 7], "playground": 5, "simultan": [5, 7], "verif": [5, 8], "importantli": 5, "gemma": [5, 8], "intuit": 5, "beginn": 5, "tensorrt": 5, "trt": 5, "latex": 5, "voic": 5, "pwa": 5, "rag": 5, "simpler": [5, 7, 8], "medium": [5, 7, 8], "gpt4all": 5, "rbac": 5, "fp16": 5, "q2_k": 5, "q4_k": 5, "q6_k": 5, "thoughput": 5, "evalu": [5, 6, 8], "wikitext": 5, "salesforc": 5, "wikipedia": [5, 8], "min_prompt_length": 5, "input_texts_raw": 5, "2010": 5, "valkyria": 5, "chronicl": 5, "underw": 5, "forgiv": 5, "newcom": 5, "raita": 5, "honjou": 5, "compos": [5, 7], "hitoshi": 5, "sakimoto": 5, "takeshi": 5, "ozawa": 5, "writer": 5, "theme": [5, 6, 7], "sung": 5, "escap": 5, "escaped_text": 5, "2024w": 5, "block_scal": 5, "block": [5, 7], "width": 5, "parenthes": 5, "block_min": 5, "formula": 5, "superblock": 5, "5625": 5, "ieee": 5, "754": 5, "half": [5, 7], "ppl": 5, "exp": 5, "sum_": 5, "log_2": 5, "x_i": 5, "ratio": 5, "avg": 5, "_i": 5, "corr": 5, "ln": [5, 8], "kullback": 5, "leibler": 5, "logit": [5, 8], "d_": 5, "softmax": 5, "sum": 5, "kld": 5, "q2_kresult": 5, "q6": 5, "004": 5, "q2": 5, "112": 5, "q4": 5, "smallest": 5, "390": 5, "mib": 5, "67": [5, 7], "81": [5, 7], "93": [5, 7], "462": 5, "96": [5, 7], "614": 5, "58": 5, "170": 5, "ubuntu": 5, "lt": 5, "x86_64": 5, "gnu": 5, "feasibl": [5, 6], "thank": [5, 8], "intel": 5, "i7": 5, "8550u": 5, "15gib": 5, "samsung": 5, "ssd": 5, "970": 5, "evo": 5, "500gb": 5, "bc24": 5, "andrei": [5, 7], "abetlen": 5, "gc24": 5, "ggerganov": [5, 8], "blob": [5, 8], "readm": [5, 8], "gc4a": 5, "gc4b": 5, "thread": 5, "huggingface4w": 5, "huggingface4xa": 5, "huggingface4xb": 5, "ibmthink24": 5, "lmstudio24": 5, "lmstudio": 5, "mozillaocho24": 5, "salesforce24": 5, "possibli": 6, "eliot": 6, "thumb": 6, "\u00be": 6, "max_output_token": 6, "4096": 6, "16384": 6, "contrari": 6, "surpass": 6, "truncat": 6, "max_input_token": 6, "input_cost_per_token": 6, "output_cost_per_token": 6, "11b": [6, 7], "128000": 6, "5e": 6, "20241022": [6, 7], "8192": 6, "200000": 6, "3e": 6, "0613": 6, "6e": 6, "gemini": 6, "flash": 6, "1048576": 6, "2097152": 6, "05e": 6, "incomplet": [6, 7], "abruptli": 6, "shallow": 6, "thorough": [6, 7], "dissatisfact": 6, "frustrat": [6, 7], "10k": 6, "diagram": [6, 7], "charactertextsplitt": 6, "tiktoken": [6, 7], "sequenti": 6, "newlin": 6, "broadli": [6, 8], "cheap": 6, "speciali": 6, "nltk": 6, "spaci": 6, "hierarch": [6, 7], "talk": 6, "splitter": 6, "get_chunk": 6, "chunk_siz": 6, "chunk_overlap": 6, "langchain_text_splitt": 6, "text_splitt": 6, "from_tiktoken_encod": 6, "split_text": 6, "persona": 6, "langchain_cor": [6, 8], "prompttempl": 6, "get_base_prompt_templ": 6, "base_prompt": [6, 8], "from_templ": 6, "llmchain": 6, "parser": [6, 8], "output_pars": 6, "stroutputpars": 6, "langchain_commun": 6, "chat_model": 6, "chatlitellm": 6, "get_llm_chain": 6, "prompt_templ": [6, 8], "llm_chain": [6, 8], "api_key_label": 6, "upper": 6, "_api_kei": 6, "get_dynamic_prompt_templ": 6, "get_dynamic_prompt_param": 6, "prompt_param": 6, "part_idx": 6, "total_part": 6, "chat_context": 6, "param": 6, "dynamic_prompt_param": 6, "introduct": 6, "concaten": 6, "generate_report": 6, "input_cont": 6, "llm_model_nam": 6, "report_part": 6, "num_part": 6, "dinam": 6, "priovid": 6, "invok": [6, 8], "cummul": 6, "max_chunk_s": 6, "max_chunk_overlap": 6, "readabl": 6, "apple_report": 6, "luation": 6, "disciplin": 6, "subhead": 6, "depth": [6, 7], "overlook": 6, "preprocess": [6, 8], "necessit": 6, "meticul": 6, "bottleneck": 6, "mustafa": 6, "suleyman": 6, "infinit": 6, "fewer": [6, 7], "condens": 6, "versatil": 6, "grace": 6, "fallback": 6, "empow": [6, 7], "langchain24": 6, "how_to": 6, "immens": 7, "commonplac": 7, "hartvigsen": 7, "societi": 7, "statement": 7, "alarm": 7, "openli": 7, "dolli": 7, "v2": 7, "llama2": [7, 8], "13b": 7, "emb": 7, "generalist": 7, "injustic": 7, "inequ": 7, "undermin": 7, "perpetu": 7, "displac": 7, "eros": 7, "fake": 7, "deepfak": 7, "distrust": 7, "cyberattack": 7, "spread": 7, "disinform": 7, "inadvert": 7, "interven": 7, "irrevers": 7, "uncheck": 7, "extinct": 7, "race": 7, "incentiv": 7, "shortcut": 7, "behind": 7, "stress": 7, "urgent": 7, "reorient": 7, "birth": 7, "siam": 7, "edgington": 7, "jailbreak": 7, "promptcraft": 7, "stealth": 7, "sutton": 7, "subtl": 7, "trigger": 7, "subtleti": 7, "exception": 7, "phrase": 7, "evad": 7, "hqve": 7, "frer": 7, "hplidai": 7, "pl": 7, "hyperion": 7, "coast": 7, "redwood": 7, "tallest": 7, "tree": [7, 8], "routin": 7, "prejudic": 7, "gallego": 7, "leak": 7, "poison": 7, "intention": 7, "inject": 7, "mislead": 7, "exabeam": 7, "finra": 7, "3110": 7, "mandat": 7, "supervisori": 7, "medicin": 7, "unicef": 7, "contest": 7, "congress": 7, "enact": 7, "pictur": [7, 8], "territori": 7, "oversea": 7, "chines": 7, "legitim": 7, "consent": 7, "complaint": 7, "cooper": 7, "extraterritori": 7, "offshor": 7, "draft": 7, "voluntari": 7, "neutral": 7, "player": 7, "prepared": 7, "ahead": 7, "compris": 7, "cbrn": 7, "persuas": 7, "autonomi": 7, "gradat": 7, "scorecard": 7, "elig": 7, "advisori": 7, "sag": 7, "shut": 7, "prerequisit": 7, "harden": 7, "asl": 7, "biosafeti": 7, "elev": 7, "warn": 7, "bioweapon": 7, "compartment": 7, "difficulti": 7, "4x": 7, "jump": 7, "paus": 7, "frontier": 7, "deepmind": 7, "biosecur": 7, "buffer": 7, "formul": [7, 8], "calibr": 7, "promin": 7, "taxonomi": 7, "llamaguard": 7, "leaderboard": 7, "3x": 7, "5x": 7, "alaga": 7, "substandard": 7, "rapidli": [7, 8], "oxford": 7, "wachter": 7, "blur": 7, "ill": 7, "stifl": 7, "suscept": 7, "aadc": 7, "outset": 7, "curricula": 7, "adversari": 7, "uncov": [7, 8], "appar": 7, "thoroughli": 7, "lm": [7, 8], "problemat": 7, "arrai": 7, "undergo": 7, "280b": 7, "cai": [7, 8], "utilis": 7, "minimis": 7, "enshrin": 7, "evas": 7, "resort": 7, "encod": 7, "avenu": 7, "cambria": 7, "inherit": 7, "influenti": 7, "debias": 7, "occurr": 7, "phish": 7, "clarifi": 7, "toler": 7, "checklist": 7, "abus": 7, "ux": 7, "architect": 7, "retrofit": 7, "promptli": 7, "dashboard": 7, "misalign": 7, "star": 7, "postpon": 7, "sens": 7, "combat": 7, "counter": 7, "traffic": 7, "workaround": 7, "vital": 7, "silo": 7, "hierarchi": 7, "66": 7, "mcq": 7, "regex": [7, 8], "joint": 7, "facet": 7, "purpl": 7, "circl": 7, "opensafetylab": 7, "salad_bench_dataset": 7, "base_set": 7, "gptfuzzer": 7, "auto": 7, "qid": 7, "o1": 7, "supremaci": 7, "o53": 7, "o14": 7, "o5": 7, "o65": 7, "plagiar": 7, "o16": 7, "o6": 7, "o47": 7, "campaign": 7, "o12": 7, "o52": 7, "surveil": 7, "spous": 7, "know": [7, 8], "o13": 7, "breakdown": [7, 8], "ncount": 7, "21318": 7, "8756": 7, "6486": 7, "o2": 7, "1717": 7, "o4": 7, "1477": 7, "o3": 7, "socioeconom": 7, "851": 7, "int64": 7, "gen": 7, "15433": 7, "hh": 7, "4184": 7, "659": 7, "advbench": 7, "230": 7, "189": 7, "toxicchat": 7, "anyth": 7, "817": 7, "misconcept": 7, "ingrain": 7, "mc1": 7, "singular": 7, "choices4": 7, "mc2": 7, "set4": 7, "scorer": 7, "correctli": [7, 8], "truthful_qa": 7, "truthfulqa_dataset": 7, "multiple_choic": 7, "best_answ": 7, "correct_answ": 7, "incorrect_answ": 7, "watermelon": 7, "digest": 7, "noth": 7, "stomach": 7, "sick": 7, "wonderopoli": 7, "wonder": 7, "belli": 7, "swallow": 7, "dream": 7, "die": 7, "indigest": 7, "unconsci": 7, "excret": 7, "asr": 7, "r2d2": 7, "multimod": 7, "wider": [7, 8], "mmlu": 7, "mass": 7, "destruct": 7, "asynchron": 7, "webpurifi": 7, "protectai": 7, "aw": 7, "comprehend": 7, "amazon": 7, "nemo": 7, "nvidia": 7, "keyword": 7, "toolset": 7, "nemmo": 7, "synchron": 7, "nemoguardrail": 7, "llmrail": 7, "railsconfig": 7, "from_path": 7, "rail": 7, "hello": 7, "ministr": 7, "mistralai": 7, "mistral_api_kei": 7, "moderate_chat": 7, "omni": 7, "pprint": 7, "to_json": 7, "threaten": 7, "illicit": 7, "granit": 7, "guardian": 7, "consortium": 7, "v3": 7, "begin_of_text": 7, "start_header_id": 7, "end_header_id": 7, "unsafe_categori": 7, "user_message_1": 7, "model_answer_1": 7, "comma": 7, "eot_id": 7, "eom_id": 7, "denot": 7, "s1": 7, "s2": 7, "s3": 7, "s4": 7, "s5": 7, "defam": 7, "s6": 7, "s7": 7, "s8": 7, "s9": 7, "s10": 7, "s11": 7, "s12": 7, "s13": 7, "atla": 7, "2b": 7, "hap": 7, "38m": 7, "125m": 7, "padhi": 7, "shieldgemma": 7, "exemplifi": 7, "accomplish": [7, 8], "judge_prompt": 7, "american": 7, "vandal": 7, "underag": 7, "drink": 7, "vulgar": 7, "obscen": 7, "racism": 7, "derogatori": 7, "firearm": 7, "safety_scor": 7, "IN": 7, "borderlin": 7, "verdict": 7, "boolean": [7, 8], "brief": 7, "rational": 7, "delimit": 7, "paramount": 7, "evenli": 7, "good_sampl": 7, "bad_sampl": 7, "2024z": 7, "surg": 7, "scam": 7, "get_profanity_sampl": 7, "show_stat": 7, "bool": 7, "current_dir": 7, "getcwd": 7, "data_path": 7, "profanity_en": 7, "random_st": 7, "ncategori": 7, "category_count": 7, "category_1": 7, "1f": 7, "profanity_sampl": 7, "nsampl": 7, "anatomi": 7, "slur": 7, "182": 7, "bodili": 7, "fluid": 7, "excrement": 7, "insult": 7, "mental": 7, "disabl": 7, "jap": 7, "babi": 7, "batter": 7, "crazi": 7, "sob": 7, "fukka": 7, "sh1t3": 7, "get_salad_sampl": 7, "salad_sampl": 7, "tortur": 7, "porn": 7, "sin": 7, "sight": 7, "god": 7, "embezzl": 7, "xanax": 7, "alcohol": 7, "get_good_sampl": 7, "min_scor": 7, "reichstag": 7, "profanity_data": 7, "salad_data": 7, "good_data": 7, "all_data": 7, "prompt_sampl": 7, "is_unsaf": 7, "counti": 7, "holli": 7, "ridg": 7, "nc": 7, "town": 7, "onslow": 7, "carolina": 7, "diver": 7, "underwat": 7, "maze": 7, "coral": 7, "treasur": 7, "vivid": 7, "sensori": 7, "emot": 7, "labyrinthin": 7, "passag": 7, "reef": 7, "suspens": 7, "obstacl": 7, "creatur": 7, "nomin": 7, "nobel": 7, "literatur": 7, "love": 7, "ny": [7, 8], "logo": 7, "thief": 7, "rob": 7, "famou": 7, "nstatist": 7, "source_stat": 7, "type_stat": 7, "plug": 7, "safetyvalid": 7, "validationresult": 7, "dataclass": 7, "abstractmethod": 7, "llmguardvalid": 7, "scanner": 7, "bantop": 7, "llm_guard": 7, "input_scann": 7, "scan_prompt": 7, "matchtyp": 7, "default_banned_top": 7, "banned_top": 7, "super": 7, "banned_topics_scann": 7, "use_onnx": 7, "toxicity_scann": 7, "match_typ": 7, "fail_fast": 7, "unsafe_scann": 7, "gun": 7, "cool": 7, "hunt": 7, "deer": 7, "dad": 7, "mistralvalid": 7, "hate_and_discrimin": 7, "violence_and_threat": 7, "dangerous_and_criminal_cont": 7, "selfharm": 7, "openaivalid": 7, "attr": 7, "dir": 7, "getattr": 7, "illicit_viol": 7, "llmjudgevalid": 7, "prompt_path": 7, "llmjudg": 7, "filenotfounderror": 7, "elaps": 7, "score_valid": 7, "id_": 7, "validator_inst": 7, "start_tim": 7, "validation_result": 7, "elapsed_tim": 7, "prompt_sample_id": 7, "validator_nam": 7, "scoring_prompt": 7, "scoring_result": 7, "01536": 7, "34098": 7, "497136": 7, "546416": 7, "calculate_validator_metr": 7, "bad_sourc": 7, "good_sourc": 7, "tpr": 7, "fpr": 7, "f1_score": 7, "tn": 7, "fp": 7, "fn": 7, "tp": 7, "255": 7, "74": 7, "238": 7, "130": 7, "256": 7, "79": 7, "149": 7, "213": 7, "190": 7, "235": 7, "63": 7, "222": 7, "82": 7, "75": 7, "480": 7, "157": 7, "487": 7, "495": 7, "482": 7, "667": 7, "248": 7, "466": 7, "143": 7, "355": 7, "slowest": 7, "room": 7, "false_posit": 7, "left_on": 7, "right_on": 7, "persian": 7, "teenag": 7, "\u0686\u0627\u0644\u0634": 7, "\u0645\u0648\u0645\u0648": 7, "\u06cc\u06a9": 7, "\u062d\u0642\u0647": 7, "\u0648": 7, "\u0627\u0641\u0633\u0627\u0646\u0647": 7, "\u0645\u062d\u0644\u06cc": 7, "\u0627\u0633\u062a": 7, "\u06a9\u0647": 7, "\u0634\u0627\u06cc\u0639\u0647": 7, "\u0622\u0646": 7, "\u062f\u0631": 7, "\u0634\u0628\u06a9\u0647": 7, "\u0647\u0627\u06cc": 7, "\u0627\u062c\u062a\u0645\u0627\u0639\u06cc": 7, "\u067e\u062e\u0634": 7, "\u0634\u062f": 7, "\u06af\u0632\u0627\u0631\u0634": 7, "\u06a9\u0627\u0631\u0628\u0631\u06cc": 7, "\u0628\u0627": 7, "\u0646\u0627\u0645": 7, "\u06a9\u0648\u062f\u06a9\u0627\u0646": 7, "\u0646\u0648\u062c\u0648\u0627\u0646\u0627\u0646": 7, "\u0631\u0627": 7, "\u0628\u0647": 7, "\u0627\u0639\u0645\u0627\u0644": 7, "\u062e\u0637\u0631\u0646\u0627\u06a9\u06cc": 7, "\u0645\u0627\u0646\u0646\u062f": 7, "\u062e\u0648\u062f\u0632\u0646\u06cc": 7, "\u062e\u0648\u062f\u06a9\u0634\u06cc": 7, "\u062a\u0631\u063a\u06cc\u0628": 7, "\u0645\u06cc": 7, "\u06a9\u0646\u062f": 7, "\u0648\u062c\u0648\u062f": 7, "\u0635\u062d\u0628\u062a": 7, "\u0632\u06cc\u0627\u062f\u06cc": 7, "\u0645\u0648\u0631\u062f": 7, "\u0627\u06cc\u0646": 7, "\u062a\u0639\u062f\u0627\u062f": 7, "\u0634\u06a9\u0627\u06cc\u0627\u062a": 7, "\u0648\u0627\u0642\u0639\u06cc": 7, "\u0628\u0633\u06cc\u0627\u0631": 7, "\u06a9\u0645": 7, "\u0628\u0648\u062f": 7, "\u067e\u0644\u06cc\u0633": 7, "\u0635\u062f\u0645\u0647": 7, "\u062f\u06cc\u062f\u0646": 7, "\u062f\u0644\u06cc\u0644": 7, "\u062a\u0623\u062b\u06cc\u0631": 7, "\u0645\u0633\u062a\u0642\u06cc\u0645": 7, "\u067e\u062f\u06cc\u062f\u0647": 7, "\u062a\u0623\u06cc\u06cc\u062f": 7, "\u0646\u06a9\u0631\u062f\u0647": 7, "\u062a\u0631\u0633": 7, "\u0646\u06af\u0631\u0627\u0646\u06cc": 7, "\u0627\u06cc\u062c\u0627\u062f": 7, "\u0634\u062f\u0647": 7, "\u0628\u06cc\u0634\u062a\u0631": 7, "\u0627\u0632": 7, "\u062e\u0648\u062f": 7, "\u0631\u0633\u0627\u0646\u0647": 7, "\u0647\u0627": 7, "\u0637\u0648\u0631\u06cc": 7, "\u062e\u06cc\u0631\u06cc\u0647": 7, "\u0647\u0634\u062f\u0627\u0631": 7, "\u062f\u0627\u062f\u0646\u062f": 7, "\u0622\u0633\u06cc\u0628": 7, "\u0627\u0646\u062a\u0638\u0627\u0631\u0627\u062a": 7, "\u0645\u062d\u062a\u0648\u0627\u06cc": 7, "\u062e\u0634\u0648\u0646\u062a": 7, "\u0622\u0645\u06cc\u0632": 7, "\u0627\u06cc\u0646\u062a\u0631\u0646\u062a": 7, "\u06af\u0641\u062a\u0647": 7, "\u0634\u0648\u062f": 7, "\u0627\u0648\u0644\u06cc\u0646": 7, "\u0628\u0627\u0631": 7, "\u0633\u0627\u0644": 7, "\u06f2\u06f0\u06f1\u06f8": 7, "\u067e\u0633": 7, "\u0622\u0646\u06a9\u0647": 7, "\u0631\u0648\u0632\u0646\u0627\u0645\u0647": 7, "\u0627\u0646\u062f\u0648\u0646\u0632\u06cc\u0627\u06cc\u06cc": 7, "\u062e\u0628\u0631": 7, "\u062f\u062e\u062a\u0631": 7, "\u06f1\u06f2": 7, "\u0633\u0627\u0644\u0647": 7, "\u062f\u0627\u062f": 7, "\u0645\u0648\u0636\u0648\u0639": 7, "\u062c\u0647\u0627\u0646\u06cc": 7, "\u062a\u0628\u062f\u06cc\u0644": 7, "\u0645\u062c\u0633\u0645\u0647": 7, "\u0647\u0646\u0631\u0645\u0646\u062f": 7, "\u0698\u0627\u067e\u0646\u06cc": 7, "\u0647\u0631": 7, "\u0686\u0646\u062f": 7, "\u0634\u0627\u06cc\u062f": 7, "\u0646\u06af\u0627\u0647": 7, "\u0628\u0639\u0636\u06cc": 7, "\u0632\u06cc\u0628\u0627": 7, "\u0646\u0628\u0627\u0634\u062f": 7, "\u0627\u0645\u0627": 7, "\u06a9\u0627\u0645\u0644\u0627": 7, "\u0628\u06cc": 7, "\u062e\u0637\u0631": 7, "\u0627\u06cc\u0631\u0627\u0646": 7, "\u0645\u062f\u062a": 7, "\u0628\u06cc\u0646": 7, "\u06a9\u0627\u0631\u0628\u0631\u0627\u0646": 7, "\u0645\u0637\u0631\u062d": 7, "\u0633\u0627\u0644\u06cc": 7, "\u0633\u0631\u0627\u0633\u0631": 7, "\u062c\u0647\u0627\u0646": 7, "\u0645\u0634\u0627\u0628\u0647\u06cc": 7, "\u0628\u0631\u0627\u06cc": 7, "\u0648\u0627\u0644\u062f\u06cc\u0646": 7, "\u06a9\u0631\u062f\u0647": 7, "\u0627\u0641\u0631\u0627\u062f": 7, "\u0686\u0647": 7, "\u06a9\u0627\u0631\u06cc": 7, "\u062f\u0639\u0648\u062a": 7, "tourist": 7, "distress": 7, "polish": 7, "galician": 7, "dzisiaj": 7, "szwecji": 7, "innych": 7, "bogatych": 7, "krajach": 7, "ludzi": 7, "u\u017cywaj\u0105": 7, "mn\u00f3stwo": 7, "najr\u00f3\u017cniejszych": 7, "urz\u0105dze\u0144": 7, "hox": 7, "suecia": 7, "outro": 7, "pa\u00eds": 7, "rico": 7, "xent": 7, "usa": [7, 8], "moita": 7, "m\u00e1quina": 7, "diferent": 7, "\u0142\u00f3dka": 7, "zaczyna": 7, "ton\u0105\u0107": 7, "tury\u015bci": 7, "wracaj\u0105": 7, "statek": 7, "dom\u00f3w": 7, "gdzie": 7, "opowiadaj\u0105": 7, "tym": 7, "jak": 7, "zostali": 7, "zaatakowani": 7, "surprisingli": 7, "unsettl": 7, "paradox": 7, "harbor": 7, "wisdom": 7, "aspir": 7, "technologist": 7, "disciplinari": 7, "ethicist": 7, "policymak": 7, "ai24": 7, "asa24": 7, "jide": 7, "jona": 7, "schuett": 7, "marku": 7, "anderljung": 7, "08751": 7, "bhy": 7, "geoffrei": 7, "hinton": 7, "pieter": 7, "abbeel": 7, "trevor": 7, "darrel": 7, "yuval": 7, "harari": 7, "ya": 7, "lan": 7, "shai": 7, "shalev": 7, "gillian": 7, "hadfield": 7, "clune": 7, "tegan": 7, "maharaj": 7, "hutter": 7, "at\u0131l\u0131m": 7, "g\u00fcne\u015f": 7, "baydin": 7, "sheila": 7, "mcilraith": 7, "qiqi": 7, "ashwin": 7, "acharya": 7, "anca": 7, "dragan": 7, "philip": 7, "torr": 7, "russel": 7, "kahneman": 7, "s\u00f6ren": 7, "mindermann": 7, "amid": 7, "384": 7, "6698": 7, "1126": 7, "adn0117": 7, "bbc": 7, "emili": 7, "braca": 7, "israel": 7, "carter": 7, "hafsa": 7, "kanchwala": 7, "khojasteh": 7, "charli": 7, "landow": 7, "luo": 7, "magarelli": 7, "mirin": 7, "averi": 7, "moyer": 7, "kayla": 7, "simpson": 7, "amelia": 7, "skawinski": 7, "heverin": 7, "23308": 7, "bmc": 7, "dillon": 7, "brendan": 7, "murphi": 7, "khachaturov": 7, "gleav": 7, "kellin": 7, "pelrin": 7, "2408": [7, 8], "02946": 7, "cmm": 7, "erik": 7, "lorenzo": 7, "malandri": 7, "fabio": 7, "mercorio": 7, "navid": 7, "nobani": 7, "seveso": 7, "15248": 7, "edg24": 7, "exa24": 7, "cyber": 7, "grb": 7, "rossi": 7, "barrow": 7, "mehrab": 7, "tanjim": 7, "sungchul": 7, "franck": 7, "dernoncourt": 7, "ruiyi": 7, "nesreen": 7, "2309": 7, "00770": 7, "h44z": 7, "hgp": 7, "saadia": 7, "hamid": 7, "palangi": 7, "dipankar": 7, "ec": 7, "kamar": 7, "oxi": 7, "smaranda": 7, "muresan": 7, "preslav": 7, "nakov": 7, "alin": 7, "villavicencio": 7, "editor": 7, "60th": 7, "3309": 7, "3326": 7, "dublin": 7, "aclanthologi": 7, "acl": 7, "18653": 7, "hym": 7, "weijiang": 7, "weitao": 7, "weihong": 7, "zhangyin": 7, "haotian": 7, "qianglong": 7, "weihua": 7, "xiaocheng": 7, "bing": 7, "ting": 7, "dx": 7, "1145": [7, 8], "3703155": 7, "ldw": 7, "lijun": 7, "ruohui": 7, "xuhao": 7, "wangmeng": 7, "zuo": 7, "dahua": 7, "qiao": 7, "shao": 7, "05044": 7, "mpy": 7, "xuwang": 7, "zifan": 7, "norman": 7, "mu": 7, "elham": 7, "sakhae": 7, "nathaniel": 7, "forsyth": 7, "04249": 7, "mlc24": 7, "illumin": 7, "ailumin": 7, "oaa": 7, "adler": 7, "ahmad": 7, "ilg": 7, "akkaya": 7, "florencia": 7, "leoni": 7, "aleman": 7, "janko": 7, "altenschmidt": 7, "altman": 7, "shyamal": 7, "anadkat": 7, "avila": 7, "valeri": 7, "balcom": 7, "baltescu": 7, "haim": 7, "belgum": 7, "irwan": 7, "bello": 7, "jake": 7, "berdin": 7, "bernadett": 7, "shapiro": 7, "berner": 7, "lenni": 7, "bogdonoff": 7, "boiko": 7, "madelain": 7, "boyd": 7, "luisa": 7, "brakman": 7, "button": 7, "rosi": 7, "campbel": 7, "cann": 7, "brittani": 7, "carei": 7, "carlson": 7, "rori": 7, "carmichael": 7, "che": 7, "foti": 7, "sulli": 7, "rubi": 7, "chess": 7, "chester": 7, "cho": 7, "hyung": 7, "won": 7, "chung": 7, "jeremiah": 7, "currier": 7, "yunx": 7, "cori": 7, "decareaux": 7, "degri": 7, "deutsch": 7, "devil": 7, "dhar": 7, "steve": 7, "dowl": 7, "dun": 7, "adrien": 7, "ecoffet": 7, "atti": 7, "eleti": 7, "tyna": 7, "elound": 7, "farhi": 7, "niko": 7, "sim\u00f3n": 7, "posada": 7, "fishman": 7, "juston": 7, "isabella": 7, "fulford": 7, "georg": 7, "gibson": 7, "vik": 7, "tarun": 7, "gogineni": 7, "goh": 7, "rapha": 7, "gontijo": 7, "lope": 7, "gordon": 7, "morgan": 7, "grafstein": 7, "yufei": 7, "guo": 7, "hallaci": 7, "heaton": 7, "johann": 7, "heideck": 7, "hickei": 7, "wade": 7, "hoeschel": 7, "houghton": 7, "kenni": 7, "hsu": 7, "shengli": 7, "joost": 7, "huizinga": 7, "shawn": 7, "joann": 7, "jang": 7, "roger": 7, "haozhun": 7, "shino": 7, "jomoto": 7, "billi": 7, "jonn": 7, "tomer": 7, "kaftan": 7, "\u0142ukasz": 7, "kamali": 7, "ingmar": 7, "kanitscheid": 7, "tabarak": 7, "khan": 7, "logan": 7, "kilpatrick": 7, "jong": 7, "wook": 7, "christina": 7, "yongjik": 7, "hendrik": 7, "kirchner": 7, "kiro": 7, "matt": 7, "kokotajlo": 7, "kondraciuk": 7, "kondrich": 7, "konstantinidi": 7, "kosic": 7, "vishal": 7, "kuo": 7, "lamp": 7, "ikai": 7, "teddi": 7, "jade": 7, "leung": 7, "chak": 7, "ming": 7, "lim": 7, "molli": 7, "mateusz": 7, "litwin": 7, "theresa": 7, "lopez": 7, "patricia": 7, "lue": 7, "makanju": 7, "malfacini": 7, "markov": 7, "yaniv": 7, "markovski": 7, "bianca": 7, "mayn": 7, "mckinnei": 7, "christin": 7, "mcleavei": 7, "mcmillan": 7, "mcneil": 7, "aalok": 7, "menick": 7, "mishchenko": 7, "vinni": 7, "monaco": 7, "murk": 7, "m\u00e9ly": 7, "ashvin": 7, "nair": 7, "reiichiro": 7, "nakano": 7, "rajeev": 7, "nayak": 7, "arvind": 7, "neelakantan": 7, "hyeonwoo": 7, "noh": 7, "keef": 7, "jakub": 7, "pachocki": 7, "palermo": 7, "ashlei": 7, "pantuliano": 7, "parish": 7, "emi": 7, "parparita": 7, "passo": 7, "perelman": 7, "belbut": 7, "pere": 7, "pokorni": 7, "pokrass": 7, "vitchyr": 7, "pong": 7, "tolli": 7, "powel": 7, "bori": 7, "proehl": 7, "rae": 7, "ramesh": 7, "franci": 7, "kendra": 7, "rimbach": 7, "carl": 7, "rotst": 7, "roussez": 7, "saltarelli": 7, "ted": 7, "sander": 7, "schnurr": 7, "selsam": 7, "kyla": 7, "sheppard": 7, "toki": 7, "sherbakov": 7, "shieh": 7, "shoker": 7, "pranav": 7, "szymon": 7, "sidor": 7, "sigler": 7, "sitkin": 7, "sokolowski": 7, "natali": 7, "staudach": 7, "madelein": 7, "tootoonchian": 7, "tseng": 7, "preston": 7, "tuggl": 7, "turlei": 7, "juan": 7, "cer\u00f3n": 7, "urib": 7, "vallon": 7, "vijayvergiya": 7, "jai": 7, "alvin": 7, "ward": 7, "cj": 7, "weinmann": 7, "akila": 7, "welihinda": 7, "jiayi": 7, "weng": 7, "lilian": 7, "wiethoff": 7, "willner": 7, "wolrich": 7, "lauren": 7, "workman": 7, "sherwin": 7, "yoo": 7, "zeller": 7, "shengjia": 7, "juntang": 7, "zhuk": 7, "2303": 7, "08774": 7, "pnc": 7, "inkit": 7, "manish": 7, "nagireddi": 7, "giandomenico": 7, "cornacchia": 7, "subhajit": 7, "chaudhuri": 7, "tejaswini": 7, "pedapati": 7, "pierr": 7, "dognin": 7, "keerthiram": 7, "murugesan": 7, "miehl": 7, "santill\u00e1n": 7, "kieran": 7, "giulio": 7, "zizzo": 7, "muhammad": 7, "zaid": 7, "hame": 7, "purcel": 7, "desmond": 7, "pan": 7, "ing": 7, "vejsbjerg": 7, "dali": 7, "hind": 7, "werner": 7, "geyer": 7, "ambrish": 7, "rawat": 7, "kush": 7, "varshnei": 7, "prasanna": 7, "sattigeri": 7, "07724": 7, "saffron": 7, "ring": 7, "aslanid": 7, "glaes": 7, "nat": 7, "mcalees": 7, "irv": 7, "2202": 7, "03286": 7, "szw": 7, "qinghua": 7, "higham": 7, "gorban": 7, "bastouni": 7, "ivan": 7, "tyukin": 7, "12670": 7, "vsk": 7, "kannappan": 7, "simplesafetytest": 7, "2311": 7, "08370": 7, "wmr24": 7, "sandra": 7, "brent": 7, "mittelstadt": 7, "duti": 7, "royal": 7, "240197": 7, "royalsocietypublish": 7, "1098": 7, "rso": 7, "ylx24": 7, "jiahao": 7, "xingwei": 7, "paperswithcod": 7, "zyi": 7, "shune": 7, "lyumanshan": 7, "jingyu": 7, "shui": 7, "haobin": 7, "pengfei": 7, "hewu": 7, "ghost": 7, "14931": 7, "zho24": 7, "amazonwservices24": 7, "anthropic24": 7, "cdn": 7, "1adf000c8f675958c2ee23805d91aaade1cd4613": 7, "centerfasafety24a": 7, "centerforaisafeti": 7, "centerfasafety24b": 7, "deepmind24": 7, "googleapi": 7, "fsf": 7, "europeanmagency24": 7, "ema": 7, "europa": 7, "activities_en": 7, "financialirauthority24": 7, "ibm24": 7, "watsonx": 7, "saa": 7, "libraryocongress23": 7, "loc": 7, "gov": 7, "mistralai24": 7, "mlsteam24": 7, "mlsafeti": 7, "nationaliosatechnology24": 7, "nist": 7, "itl": 7, "nvidia24": 7, "toolkit": 7, "openai24a": 7, "openai24b": 7, "opensafetylab24a": 7, "opensafetylab24b": 7, "protectai24": 7, "surgeai24": 7, "ukgovernment24": 7, "unicef24": 7, "innocenti": 7, "julia": 8, "easili": 8, "response_cont": 8, "wow": 8, "lot": 8, "impress": 8, "huge": 8, "serious": 8, "is_json": 8, "myjson": 8, "trial": 8, "wrangl": 8, "hoc": 8, "streamlin": 8, "dataset": 8, "unwant": 8, "overflow": 8, "twitter": 8, "youtub": 8, "blueprint": 8, "json_format": 8, "person1": 8, "q1": 8, "person2": 8, "nest": 8, "thellm": 8, "conceptu": 8, "overview": 8, "unend": 8, "whitespac": 8, "throw": 8, "somewher": 8, "json_object": 8, "circul": 8, "vertex": 8, "worri": 8, "invalid": 8, "enum": 8, "secextract": 8, "mentioned_ent": 8, "mentioned_plac": 8, "extract_from_sec_fil": 8, "sec_filing_text": 8, "hint": 8, "prompt_extract": 8, "sec_extract": 8, "washington": 8, "beg": 8, "with_structured_output": 8, "runnabl": 8, "typeddict": 8, "qu": 8, "langchain_openai": 8, "chatopenai": 8, "chatprompttempl": 8, "extract_from_sec_filing_langchain": 8, "structured_llm": 8, "from_messag": 8, "sec_extraction_langchain": 8, "willard": 8, "louf": 8, "reformul": 8, "finit": 8, "fsm": 8, "s_": 8, "s_t": 8, "s_1": 8, "mask": 8, "tild": 8, "odot": 8, "rightarrow": 8, "wise": 8, "thien": 8, "automaton": 8, "dfa": 8, "decod": 8, "outgo": 8, "renorm": 8, "yy": 8, "nn": 8, "ever": 8, "aa": 8, "lwai": 8, "prop": 8, "yynnaa": 8, "malform": 8, "sec_extraction_outlin": 8, "zsp": 8, "zicorp": 8, "cpp": 8, "bnf": 8, "backu": 8, "naur": 8, "fssl": 8, "extract_entities_from_sec_fil": 8, "ollama_structured_output_prompt_suffix": 8, "ollama_structured_output_temperatur": 8, "uncensor": 8, "model_json_schema": 8, "response_json": 8, "wrapper": 8, "exllama2": 8, "mlx": 8, "chanc": 8, "furthermor": 8, "nonetheless": 8, "studi": 8, "extran": 8, "dispar": 8, "preval": 8, "speak": 8, "aider": 8, "outweigh": 8, "rebutt": 8, "reproduct": 8, "paint": 8, "dottxt": 8, "flaw": 8, "uneven": 8, "didn": 8, "conflat": 8, "drawback": 8, "unlock": 8, "pfiffer": 8, "aid24": 8, "dot24": 8, "demo": 8, "gge24": 8, "lan4b": 8, "llf": 8, "xieyang": 8, "frederick": 8, "fiannaca": 8, "terri": 8, "koo": 8, "dixon": 8, "ea": 8, "machineri": 8, "3613905": 8, "3650756": 8, "xuan": 8, "hai": 8, "nguyen": 8, "ngoc": 8, "tiviati": 8, "hieu": 8, "dao": 8, "shafiq": 8, "joti": 8, "kenji": 8, "kawaguchi": 8, "nanci": 8, "min": 8, "kan": 8, "08656": 8, "out24": 8, "twt": 8, "zhi": 8, "cheng": 8, "kuang": 8, "tsai": 8, "chieh": 8, "hung": 8, "yun": 8, "nung": 8, "02442": 8, "tt24": 8, "vivien": 8, "vivien000": 8, "wl23": 8, "r\u00e9mi": 8, "09702": 8, "wikipediacontributors24": 8, "wiktionari": 8, "naur_form": 8}, "objects": {}, "objtypes": {}, "objnames": {}, "titleterms": {"introduct": [0, 2, 3, 4, 5, 7, 8], "content": [0, 3, 4, 5, 6, 7, 8], "core": 0, "challeng": [0, 2], "we": 0, "ll": 0, "address": 0, "A": [0, 2, 3], "practic": [0, 2, 8], "approach": [0, 7], "an": 0, "open": [0, 2, 5], "sourc": [0, 2], "book": 0, "note": [0, 3], "perspect": 0, "who": 0, "thi": 0, "i": [0, 3], "For": 0, "outcom": 0, "prerequisit": 0, "set": 0, "up": 0, "your": 0, "environ": 0, "code": 0, "repositori": 0, "python": 0, "setup": [0, 3], "api": [0, 7, 8], "kei": [0, 4, 6], "configur": 0, "troubleshoot": 0, "common": [0, 7], "issu": 0, "about": 0, "author": 0, "": 0, "prefac": [1, 2], "refer": [1, 3, 4, 5, 6, 7, 8], "tame": 2, "llm": [2, 4, 5, 7], "guid": 2, "pitfal": [2, 7], "softwar": [2, 4], "chapter": 2, "1": [2, 6, 7], "2": [2, 6, 7], "wrestl": [2, 8], "structur": [2, 8], "output": [2, 6, 8], "3": [2, 6, 7], "input": 2, "data": [2, 3], "4": [2, 6, 7], "size": [2, 6], "length": [2, 6], "limit": [2, 3, 6], "5": [2, 7], "The": [2, 4, 5], "eval": [2, 4, 7], "gap": [2, 4], "6": [2, 7], "safeti": [2, 7], "concern": 2, "7": 2, "prefer": [2, 3], "base": [2, 3, 4, 6, 7], "align": [2, 3], "8": 2, "break": [2, 5], "free": [2, 5], "from": [2, 3, 5], "cloud": [2, 5], "provid": [2, 5, 8], "9": 2, "cost": [2, 6], "factor": [2, 7], "10": 2, "frontier": 2, "appendix": 2, "tool": [2, 4, 5, 7, 8], "resourc": 2, "citat": [2, 3, 4, 5, 6, 7, 8], "raw": 3, "capabl": 3, "On": 3, "misalign": 3, "languag": 3, "model": [3, 4, 5, 6], "human": 3, "supervis": 3, "fine": 3, "tune": 3, "sft": 3, "augment": 3, "post": 3, "train": 3, "answer": 3, "collaps": 3, "fake": 3, "case": [3, 5, 7], "studi": [3, 5, 7], "polici": [3, 7], "experiment": 3, "deliver": 3, "smollm2": 3, "dataset": [3, 4, 5, 7], "synthet": 3, "gener": [3, 4, 6, 7], "user": [3, 7, 8], "prompt": [3, 5, 6, 8], "reject": 3, "respons": 3, "chosen": 3, "dpo": 3, "optim": 3, "prepar": 3, "vibe": 3, "check": 3, "evalu": [3, 4, 7], "discuss": [3, 6, 8], "conclus": [3, 4, 5, 6, 7, 8], "non": 4, "determinist": 4, "machin": 4, "emerg": 4, "properti": 4, "problem": [4, 6, 8], "statement": [4, 6, 8], "tradit": 4, "v": [4, 5], "design": [4, 7], "applic": 4, "test": 4, "requir": 4, "matrix": 4, "conceptu": 4, "overview": 4, "consider": [4, 5, 6], "metric": 4, "task": 4, "benchmark": [4, 5, 7], "leaderboard": 4, "lightev": 4, "mmlu": 4, "econometr": 4, "sampl": [4, 7], "famili": 4, "us": 4, "langsmith": 4, "promptfoo": 4, "comparison": [4, 5, 6, 8], "local": 5, "deploy": 5, "serv": 5, "llama": 5, "cpp": 5, "llamafil": 5, "ollama": [5, 8], "lama": 5, "ui": 5, "lm": 5, "studio": 5, "jan": 5, "webui": 5, "openwebui": 5, "effect": 5, "quantiz": 5, "perform": 5, "level": 5, "result": 5, "hardwar": 5, "what": 6, "ar": 6, "token": 6, "across": 6, "chunk": 6, "contextu": 6, "link": 6, "long": 6, "form": 6, "step": 6, "write": 6, "templat": 6, "construct": 6, "dynam": 6, "paramet": 6, "report": 6, "exampl": 6, "usag": 6, "implic": 6, "futur": 6, "risk": 7, "ai": 7, "amplifi": 7, "exist": 7, "harm": 7, "novel": 7, "associ": 7, "autonom": 7, "exacerb": 7, "specif": [7, 8], "guidanc": 7, "govern": 7, "organ": 7, "privat": 7, "sector": 7, "openai": 7, "anthrop": 7, "googl": 7, "rubric": 7, "mlcommon": 7, "centr": 7, "porquoi": 7, "red": 7, "team": 7, "constitut": 7, "explain": 7, "xai": 7, "plan": 7, "phase": 7, "definit": 7, "research": [7, 8], "identif": 7, "framework": [7, 8], "architectur": 7, "implement": 7, "select": 7, "go": 7, "market": 7, "technic": 7, "compon": 7, "salad": 7, "bench": 7, "truthfulqa": 7, "harmbench": 7, "safebench": 7, "techniqu": [7, 8], "repres": 7, "layer": 7, "map": 7, "rule": 7, "filter": 7, "custom": 7, "moder": 7, "bad": 7, "good": 7, "guard": 7, "mistral": 7, "judg": 7, "valid": 7, "takeawai": 7, "need": 8, "solut": 8, "strategi": 8, "One": 8, "shot": 8, "json": 8, "mode": 8, "langchain": 8, "outlin": 8, "compar": 8, "best": 8, "ongo": 8, "debat": 8, "acknowledg": 8}, "envversion": {"sphinx.domains.c": 2, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 8, "sphinx.domains.index": 1, "sphinx.domains.javascript": 2, "sphinx.domains.math": 2, "sphinx.domains.python": 3, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.intersphinx": 1, "sphinxcontrib.bibtex": 9, "sphinx": 57}, "alltitles": {"Introduction": [[0, "introduction"], [3, "introduction"], [3, "id30"], [4, "introduction"], [5, "introduction"], [7, "introduction"], [8, "introduction"]], "Contents": [[0, "contents"], [3, "contents"], [4, "contents"], [5, "contents"], [6, "contents"], [7, "contents"], [8, "contents"]], "Core Challenges We\u2019ll Address": [[0, "core-challenges-we-ll-address"]], "A Practical Approach": [[0, "a-practical-approach"]], "An Open Source Approach": [[0, "an-open-source-approach"]], "Open Source Book": [[0, "open-source-book"]], "A Note on Perspective": [[0, "a-note-on-perspective"]], "Who This Book Is For": [[0, "who-this-book-is-for"]], "Outcomes": [[0, "outcomes"]], "Prerequisites": [[0, "prerequisites"]], "Setting Up Your Environment": [[0, "setting-up-your-environment"]], "Code Repository": [[0, "code-repository"]], "Python Environment Setup": [[0, "python-environment-setup"]], "API Keys Configuration": [[0, "api-keys-configuration"]], "Troubleshooting Common Issues": [[0, "troubleshooting-common-issues"]], "About the Author(s)": [[0, "about-the-author-s"]], "Preface": [[1, "preface"], [2, "preface"]], "References": [[1, "references"], [3, "references"], [4, "references"], [5, "references"], [6, "references"], [7, "references"], [8, "references"]], "Taming LLMs": [[2, "taming-llms"]], "A Practical Guide to LLM Pitfalls with Open Source Software": [[2, "a-practical-guide-to-llm-pitfalls-with-open-source-software"]], "Chapter 1: Introduction": [[2, "chapter-1-introduction"]], "Chapter 2: Wrestling with Structured Output": [[2, "chapter-2-wrestling-with-structured-output"]], "Chapter 3: Input Data Challenge": [[2, "chapter-3-input-data-challenge"]], "Chapter 4: Output Size and Length Limitations": [[2, "chapter-4-output-size-and-length-limitations"]], "Chapter 5: The Evals Gap": [[2, "chapter-5-the-evals-gap"]], "Chapter 6: Safety Concerns": [[2, "chapter-6-safety-concerns"]], "Chapter 7: Preference-based Alignment": [[2, "chapter-7-preference-based-alignment"]], "Chapter 8: Breaking Free from Cloud Providers": [[2, "chapter-8-breaking-free-from-cloud-providers"]], "Chapter 9: The Cost Factor": [[2, "chapter-9-the-cost-factor"]], "Chapter 10: Frontiers": [[2, "chapter-10-frontiers"]], "Appendix A: Tools and Resources": [[2, "appendix-a-tools-and-resources"]], "Citation": [[2, "citation"], [3, "citation"], [4, "citation"], [5, "citation"], [6, "citation"], [7, "citation"], [8, "citation"]], "Preference-Based Alignment": [[3, "preference-based-alignment"]], "From Raw Capabilities to Preference Alignment": [[3, "from-raw-capabilities-to-preference-alignment"]], "On the Misalignment of Language Models": [[3, "on-the-misalignment-of-language-models"]], "Aligning Language Models with Human Preferences": [[3, "aligning-language-models-with-human-preferences"]], "Supervised Fine-Tuning (SFT) for Model Alignment": [[3, "supervised-fine-tuning-sft-for-model-alignment"]], "Augmenting SFT with Human Preferences": [[3, "augmenting-sft-with-human-preferences"]], "Is Post-Training the Answer?": [[3, "is-post-training-the-answer"]], "Limitations": [[3, "limitations"]], "Model Collapse": [[3, "model-collapse"]], "Faking Alignment": [[3, "faking-alignment"]], "Case Study: Aligning a Language Model to a Policy": [[3, "case-study-aligning-a-language-model-to-a-policy"]], "Experimental Setup": [[3, "experimental-setup"]], "Deliverables": [[3, "deliverables"]], "A Note on smolLM2 Models": [[3, "a-note-on-smollm2-models"]], "Policy": [[3, "policy"]], "Preference Dataset - Synthetic Dataset Generation": [[3, "preference-dataset-synthetic-dataset-generation"]], "User Prompts": [[3, "user-prompts"]], "Rejected Responses": [[3, "rejected-responses"]], "Chosen Responses": [[3, "chosen-responses"]], "Generate DPO Dataset": [[3, "generate-dpo-dataset"]], "DPO-Based Optimization": [[3, "dpo-based-optimization"]], "Data Preparation": [[3, "data-preparation"]], "Fine-Tuning": [[3, "fine-tuning"]], "Vibe Check": [[3, "vibe-check"]], "Alignment Evaluation": [[3, "alignment-evaluation"]], "Discussion and Conclusions": [[3, "discussion-and-conclusions"]], "The Evals Gap": [[4, "the-evals-gap"]], "Non-Deterministic Generative Machines": [[4, "non-deterministic-generative-machines"]], "Emerging Properties": [[4, "emerging-properties"]], "Problem Statement": [[4, "problem-statement"], [6, "problem-statement"], [8, "problem-statement"]], "Evals of Traditional Software vs LLMs": [[4, "evals-table"]], "Evals Design": [[4, "evals-design"]], "LLM Application Testing Requirements Matrix": [[4, "validation-requirements"]], "Conceptual Overview": [[4, "conceptual-overview"]], "Design Considerations": [[4, "design-considerations"]], "Metrics": [[4, "metrics"]], "Key Metrics for Evaluating Generative Tasks": [[4, "key-metrics"]], "Evaluators": [[4, "evaluators"]], "Model-Based Evaluation": [[4, "model-based-evaluation"]], "Evaluating Evaluators": [[4, "evaluating-evaluators"]], "Benchmarks and Leaderboards": [[4, "benchmarks-and-leaderboards"]], "Tools": [[4, "tools"]], "LightEval": [[4, "lighteval"]], "MMLU Econometrics Task Dataset sample": [[4, "mmlu-econometrics"]], "Model Families Evaluated Using LightEval": [[4, "model-families"]], "LangSmith": [[4, "langsmith"]], "PromptFoo": [[4, "promptfoo"]], "Comparison": [[4, "comparison"], [5, "comparison"], [5, "id14"]], "Comparison of Lighteval, LangSmith, and Promptfoo": [[4, "tool-comparison"]], "Conclusion": [[4, "conclusion"], [5, "conclusion"], [6, "conclusion"], [7, "conclusion"], [8, "conclusion"]], "Breaking Free from Cloud Providers": [[5, "breaking-free-from-cloud-providers"]], "Local Models Considerations": [[5, "local-models-considerations"]], "Tools for Local LLM Deployment": [[5, "tools-for-local-llm-deployment"]], "Serving Models": [[5, "serving-models"]], "LLama.cpp": [[5, "llama-cpp"]], "Llamafile": [[5, "llamafile"]], "Ollama": [[5, "ollama"], [8, "ollama"]], "lama.cpp vs Ollama vs Llamafile Comparison": [[5, "feature-comparison-local"]], "UI": [[5, "ui"]], "LM Studio": [[5, "lm-studio"]], "Jan": [[5, "jan"]], "Open WebUI": [[5, "open-webui"]], "LM Studio vs Jan vs OpenWebUI Comparison": [[5, "feature-comparison-ui"]], "Case Study: The Effect of Quantization on LLM Performance": [[5, "case-study-the-effect-of-quantization-on-llm-performance"]], "Prompts Dataset": [[5, "prompts-dataset"]], "Quantization": [[5, "quantization"]], "Quantization Levels": [[5, "quantization-levels"]], "Benchmarking": [[5, "benchmarking"], [7, "benchmarking"]], "Results": [[5, "results"]], "Quantization Benchmarks": [[5, "quantization-benchmarks"]], "Benchmarking Hardware": [[5, "benchmarking-hardware"]], "Output Size Limit": [[6, "output-size-limit"]], "What are Token Limits?": [[6, "what-are-token-limits"]], "Token Cost and Length Limitation Comparison Across Key Models": [[6, "token-cost-table"]], "Content Chunking with Contextual Linking": [[6, "content-chunking-with-contextual-linking"]], "Generating long-form content": [[6, "generating-long-form-content"]], "Step 1: Chunking the Content": [[6, "step-1-chunking-the-content"]], "Step 2: Writing the Base Prompt Template": [[6, "step-2-writing-the-base-prompt-template"]], "Step 3: Constructing Dynamic Prompt Parameters": [[6, "step-3-constructing-dynamic-prompt-parameters"]], "Step 4: Generating the Report": [[6, "step-4-generating-the-report"]], "Example Usage": [[6, "example-usage"]], "Discussion": [[6, "discussion"], [8, "discussion"]], "Implications": [[6, "implications"]], "Future Considerations": [[6, "future-considerations"]], "Safety": [[7, "safety"]], "Safety Risks": [[7, "safety-risks"]], "General AI Safety Risks": [[7, "general-ai-safety-risks"]], "Amplified Existing Harms and Novel Risks": [[7, "amplified-existing-harms-and-novel-risks"]], "Risks Associated with Autonomous AI": [[7, "risks-associated-with-autonomous-ai"]], "Exacerbating Factors": [[7, "exacerbating-factors"]], "LLMs Specific Safety Risks": [[7, "llms-specific-safety-risks"]], "Guidance": [[7, "guidance"]], "Governments & Organizations": [[7, "governments-organizations"]], "Private Sector": [[7, "private-sector"]], "OpenAI": [[7, "openai"]], "Anthropic": [[7, "anthropic"]], "Google": [[7, "google"]], "Rubrics": [[7, "rubrics"]], "MLCommons AI Safety Benchmark": [[7, "mlcommons-ai-safety-benchmark"]], "Centre for the Governance of AI Rubric": [[7, "centre-for-the-governance-of-ai-rubric"]], "Porquoi": [[7, "porquoi"]], "Approaches": [[7, "approaches"]], "Red Teaming": [[7, "red-teaming"]], "Constitutional AI": [[7, "constitutional-ai"]], "Explainable AI (XAI)": [[7, "explainable-ai-xai"]], "Designing a Safety Plan": [[7, "designing-a-safety-plan"]], "Phase 1. Policy Definition": [[7, "phase-1-policy-definition"]], "Phase 2. User Research & Risk Identification": [[7, "phase-2-user-research-risk-identification"]], "Phase 3. Evaluation Framework": [[7, "phase-3-evaluation-framework"]], "Phase 4. Safety Architecture Design": [[7, "phase-4-safety-architecture-design"]], "Phase 5. Implementation & Tools Selection": [[7, "phase-5-implementation-tools-selection"]], "Phase 6. Go-to-Market": [[7, "phase-6-go-to-market"]], "Common Pitfalls": [[7, "common-pitfalls"]], "Technical Implementation Components": [[7, "technical-implementation-components"]], "Benchmarks & Datasets": [[7, "benchmarks-datasets"]], "SALAD-Bench": [[7, "salad-bench"]], "TruthfulQA": [[7, "truthfulqa"]], "HarmBench": [[7, "harmbench"]], "SafeBench": [[7, "safebench"]], "Tools & Techniques": [[7, "tools-techniques"]], "Representative Safety Layer Risk Map.": [[7, "safety-layer-table"]], "Rules-Based Safety Filtering": [[7, "rules-based-safety-filtering"]], "Rules-Based Safety Filtering Tools.": [[7, "safety-layer-tools"]], "LLM-Based Safety Filtering": [[7, "llm-based-safety-filtering"]], "Custom Moderation": [[7, "custom-moderation"]], "Case Study: Implementing a Safety Filter": [[7, "case-study-implementing-a-safety-filter"]], "Evals Dataset": [[7, "evals-dataset"]], "Bad Samples": [[7, "bad-samples"]], "Good Samples": [[7, "good-samples"]], "Safety Filters": [[7, "safety-filters"]], "LLM-Guard": [[7, "llm-guard"]], "Mistral Moderation API": [[7, "mistral-moderation-api"]], "OpenAI Moderation API": [[7, "openai-moderation-api"]], "Custom Judge Validator": [[7, "custom-judge-validator"]], "Takeaways": [[7, "takeaways"]], "Wrestling with Structured Output": [[8, "wrestling-with-structured-output"]], "User Needs": [[8, "user-needs"]], "Solutions": [[8, "solutions"]], "Strategies": [[8, "strategies"]], "Techniques and Tools": [[8, "techniques-and-tools"]], "One-Shot Prompts": [[8, "one-shot-prompts"]], "Structured Output with Provider-Specific APIs": [[8, "structured-output-with-provider-specific-apis"]], "JSON Mode": [[8, "json-mode"]], "LangChain": [[8, "langchain"]], "Outlines": [[8, "outlines"]], "Comparing Solutions": [[8, "comparing-solutions"]], "Structured Output Frameworks Comparison": [[8, "structured-output-frameworks"]], "Best Practices": [[8, "best-practices"]], "Research and Ongoing Debate": [[8, "research-and-ongoing-debate"]], "Acknowledgements": [[8, "acknowledgements"]]}, "indexentries": {}}) \ No newline at end of file +Search.setIndex({"docnames": ["markdown/intro", "markdown/preface", "markdown/toc", "notebooks/alignment", "notebooks/evals", "notebooks/local", "notebooks/output_size_limit", "notebooks/safety", "notebooks/structured_output"], "filenames": ["markdown/intro.md", "markdown/preface.md", "markdown/toc.md", "notebooks/alignment.ipynb", "notebooks/evals.ipynb", "notebooks/local.ipynb", "notebooks/output_size_limit.ipynb", "notebooks/safety.ipynb", "notebooks/structured_output.ipynb"], "titles": ["2. Introduction", "1. Preface", "Taming LLMs", "7. Preference-Based Alignment", "5. The Evals Gap", "8. Breaking Free from Cloud Providers", "3. Output Size Limit", "6. Safety", "4. Wrestling with Structured Output"], "terms": {"am": [0, 1, 7], "alwai": [0, 3, 4, 8], "do": [0, 3, 4, 5, 6, 7, 8], "which": [0, 3, 4, 5, 6, 7, 8], "cannot": [0, 3, 4, 5, 7], "order": [0, 3, 4, 7, 8], "mai": [0, 1, 3, 4, 5, 6, 7, 8], "learn": [0, 3, 4, 7], "how": [0, 1, 3, 4, 5, 6, 7, 8], "pablo": [0, 4], "picasso": 0, "In": [0, 3, 4, 5, 6, 7, 8], "recent": [0, 3, 4, 7, 8], "year": [0, 2, 3, 4, 5, 6, 7, 8], "larg": [0, 1, 2, 3, 4, 5, 6, 7, 8], "languag": [0, 1, 2, 4, 5, 6, 7, 8], "model": [0, 1, 2, 7, 8], "llm": [0, 1, 3, 6, 8], "have": [0, 1, 3, 4, 5, 6, 7, 8], "emerg": [0, 3, 5, 7, 8], "transform": [0, 1, 3, 4, 7, 8], "forc": [0, 4, 8], "technologi": [0, 1, 4, 6, 7, 8], "promis": [0, 3, 4, 7], "revolution": [0, 7], "build": [0, 2, 3, 4, 5, 6, 7, 8], "product": [0, 1, 2, 3, 4, 5, 7, 8], "interact": [0, 3, 4, 5, 6, 7, 8], "comput": [0, 3, 4, 5, 6, 7, 8], "from": [0, 1, 4, 6, 7, 8], "chatgpt": [0, 3, 5, 8], "github": [0, 2, 3, 4, 5, 6, 7, 8], "copilot": 0, "claud": [0, 3, 4, 5, 6, 7], "artifact": 0, "system": [0, 3, 4, 5, 6, 7, 8], "captur": [0, 1, 3, 4, 7], "public": [0, 3, 4, 7], "imagin": 0, "spark": 0, "gold": [0, 3, 4, 7], "rush": 0, "ai": [0, 3, 4, 5, 8], "power": [0, 2, 3, 4, 5, 6, 7, 8], "applic": [0, 1, 2, 3, 5, 6, 7, 8], "howev": [0, 3, 4, 5, 6, 7, 8], "beneath": 0, "surfac": [0, 4], "technolog": [0, 1, 4, 7], "revolut": 0, "li": [0, 3, 4, 7], "complex": [0, 1, 3, 4, 5, 6, 7, 8], "landscap": [0, 3, 4], "practition": [0, 1, 4, 5], "must": [0, 3, 4, 5, 6, 7], "navig": [0, 2, 4, 7], "focus": [0, 3, 4, 5, 6, 7, 8], "bring": [0, 3, 5], "awar": [0, 3, 4, 6, 7], "limit": [0, 1, 4, 5, 7, 8], "har": [0, 2, 4, 6], "solut": [0, 2, 4, 5, 6, 7], "overcom": [0, 4, 6], "them": [0, 1, 3, 4, 5, 6, 7, 8], "robust": [0, 3, 4, 5, 6, 7, 8], "It": [0, 3, 4, 5, 6, 7, 8], "offer": [0, 3, 4, 5, 6, 7, 8], "critic": [0, 2, 3, 4, 6, 7, 8], "implement": [0, 2, 3, 4, 5, 6, 8], "back": [0, 4, 7, 8], "reproduc": [0, 1, 2, 4, 5], "exampl": [0, 1, 2, 3, 4, 5, 7, 8], "while": [0, 1, 2, 3, 4, 5, 6, 7, 8], "mani": [0, 1, 3, 4, 5, 6, 7, 8], "resourc": [0, 3, 4, 5, 6, 7], "cover": [0, 3, 4, 5, 6, 7], "capabl": [0, 1, 2, 4, 5, 6, 7, 8], "specif": [0, 3, 4, 5, 6], "hidden": [0, 3, 7], "pitfal": [0, 1, 3, 4, 5, 6, 8], "engin": [0, 1, 2, 3, 4, 5, 7, 8], "technic": [0, 1, 2, 3, 4, 5, 6, 8], "manag": [0, 1, 2, 4, 5, 6, 7, 8], "face": [0, 3, 4, 5, 7], "when": [0, 1, 2, 3, 4, 5, 6, 7, 8], "comprehens": [0, 2, 3, 4, 5, 6, 7, 8], "guid": [0, 1, 3, 4, 5, 6, 7, 8], "leverag": [0, 3, 4, 6, 7, 8], "battl": [0, 2], "test": [0, 2, 3, 7, 8], "tool": [0, 1, 3, 6], "throughout": [0, 4, 5, 6, 7, 8], "tackl": [0, 3, 4, 7], "follow": [0, 3, 4, 5, 6, 7, 8], "non": [0, 3, 5, 7, 8], "exhaust": 0, "list": [0, 3, 4, 5, 6, 7, 8], "structur": [0, 3, 4, 5, 6, 7], "un": 0, "reliabl": [0, 1, 3, 4, 5, 7, 8], "struggl": [0, 1, 3, 4, 7, 8], "maintain": [0, 1, 3, 4, 5, 6, 7, 8], "consist": [0, 1, 3, 4, 5, 6, 7, 8], "output": [0, 1, 3, 4, 5, 7], "format": [0, 3, 4, 5, 6, 7, 8], "complic": [0, 7], "integr": [0, 1, 3, 4, 5, 7, 8], "larger": [0, 3, 4, 5, 6, 7, 8], "make": [0, 3, 4, 5, 6, 7, 8], "error": [0, 3, 4, 7, 8], "handl": [0, 2, 3, 4, 5, 6, 7, 8], "more": [0, 1, 3, 4, 5, 6, 7, 8], "size": [0, 3, 4, 5, 7, 8], "length": [0, 3, 4, 5, 8], "constraint": [0, 1, 3, 4, 6, 7, 8], "strict": [0, 5, 7, 8], "token": [0, 1, 3, 4, 5, 7, 8], "both": [0, 3, 4, 5, 7], "input": [0, 3, 4, 5, 6, 7, 8], "requir": [0, 3, 5, 6, 7, 8], "care": [0, 3, 4, 7, 8], "chunk": [0, 3], "strategi": [0, 3, 4, 5, 6, 7], "long": [0, 1, 3, 4, 7, 8], "form": [0, 3, 4, 5, 7, 8], "effect": [0, 1, 3, 4, 6, 7, 8], "tradit": [0, 3, 7], "softwar": [0, 1, 3, 5, 6, 7, 8], "methodologi": [0, 3, 4, 7, 8], "break": [0, 1, 3, 4, 6, 7], "down": [0, 1, 4, 5, 6, 7], "deal": [0, 3, 5], "determinist": [0, 8], "gener": [0, 1, 5, 8], "new": [0, 2, 3, 4, 5, 6, 7, 8], "hallucin": [0, 1, 3, 4, 7, 8], "These": [0, 3, 4, 5, 6, 7, 8], "can": [0, 1, 3, 4, 5, 6, 7, 8], "plausibl": [0, 7], "sound": [0, 7], "entir": [0, 4, 5, 6, 8], "fabric": [0, 4, 7], "inform": [0, 3, 4, 5, 6, 7, 8], "creat": [0, 1, 3, 4, 5, 6, 7, 8], "signific": [0, 3, 4, 5, 6, 7, 8], "risk": [0, 1, 3, 4, 6], "safeti": [0, 3, 4, 8], "align": [0, 4, 5, 6, 7, 8], "harm": [0, 3, 4, 5], "bias": [0, 3, 4, 7, 8], "inappropri": [0, 3, 7], "safeguard": [0, 4, 7], "monitor": [0, 3, 4, 5, 7], "ensur": [0, 3, 4, 5, 6, 7, 8], "safe": [0, 3, 4, 7, 8], "deploy": [0, 3, 4, 7, 8], "cost": [0, 3, 4, 5, 7, 8], "optim": [0, 1, 4, 5, 6, 7], "The": [0, 1, 3, 6, 7, 8], "financi": [0, 1, 3, 4, 6, 7, 8], "oper": [0, 3, 4, 5, 6, 7, 8], "base": [0, 1, 5, 8], "quickli": [0, 3, 6], "becom": [0, 3, 4, 5, 7, 8], "prohibit": [0, 3, 4, 5], "without": [0, 1, 3, 4, 5, 6, 7, 8], "observ": [0, 3, 4, 5, 7, 8], "vendor": [0, 4], "lock": [0, 3], "cloud": [0, 3, 4, 7, 8], "provid": [0, 3, 4, 6, 7], "depend": [0, 3, 4, 5, 8], "through": [0, 1, 2, 3, 4, 5, 6, 7, 8], "proprietari": [0, 3, 5, 7, 8], "infrastructur": [0, 5], "difficult": [0, 3, 4, 7], "switch": [0, 5], "self": [0, 3, 4, 5, 7], "host": [0, 4, 5, 7], "take": [0, 2, 3, 4, 5, 6, 7, 8], "hand": [0, 6, 7, 8], "focu": [0, 2, 3, 4, 5, 6, 7, 8], "access": [0, 3, 4, 5, 6, 7, 8], "all": [0, 1, 3, 4, 5, 6, 7, 8], "ar": [0, 1, 3, 4, 5, 7, 8], "fulli": [0, 3, 4, 6, 7], "document": [0, 3, 4, 5, 6, 7, 8], "allow": [0, 4, 5, 6, 7, 8], "reader": [0, 2], "replic": [0, 4, 7, 8], "result": [0, 3, 4, 6, 7, 8], "exactli": [0, 4, 8], "design": [0, 1, 3, 5, 6, 8], "run": [0, 3, 4, 5, 7, 8], "consum": [0, 3, 4, 5, 7, 8], "grade": [0, 3, 4, 5, 7], "hardwar": [0, 3, 4], "expens": [0, 3, 4, 5, 7], "avail": [0, 3, 4, 5, 6, 7, 8], "notebook": [0, 3, 8], "modifi": [0, 3, 4, 7], "extend": [0, 3, 4, 5, 8], "built": [0, 4, 5, 7, 8], "us": [0, 1, 3, 5, 6, 7, 8], "free": [0, 1, 3, 4, 7], "everyon": [0, 4, 5], "minim": [0, 3, 4, 5, 7, 8], "framework": [0, 3, 4], "wai": [0, 3, 4, 5, 6, 7, 8], "priorit": [0, 3, 4, 5, 7], "transpar": [0, 3, 4, 7, 8], "visibl": [0, 4], "being": [0, 3, 4, 5, 7], "better": [0, 2, 3, 4, 5, 6, 7], "understand": [0, 1, 2, 3, 4, 5, 6, 7, 8], "custom": [0, 3, 4, 5], "flexibl": [0, 4, 5, 6, 7, 8], "adapt": [0, 3, 4, 5, 6, 7], "case": [0, 4, 6, 8], "unlik": [0, 3, 4, 5], "black": [0, 3], "box": [0, 5], "commerci": [0, 3, 4, 7, 8], "most": [0, 3, 4, 5, 6, 7, 8], "freeli": [0, 8], "foster": [0, 3, 4, 7, 8], "reduc": [0, 3, 4, 5, 6, 7, 8], "independ": [0, 4, 7, 8], "freedom": [0, 5, 8], "architectur": [0, 3, 4, 5, 6, 8], "decis": [0, 3, 4, 5, 7, 8], "keep": [0, 3, 4, 5, 6, 7], "principl": [0, 3, 4, 5, 7], "itself": [0, 3, 4, 5, 7], "live": [0, 1, 4, 7], "evolv": [0, 3, 4, 6, 7], "chang": [0, 3, 4, 5, 7], "encourag": [0, 3, 4, 7, 8], "report": [0, 3, 4, 5, 7, 8], "suggest": [0, 3, 4, 7, 8], "improv": [0, 3, 4, 5, 6, 7, 8], "contribut": [0, 4, 6, 7], "via": [0, 3, 4, 5, 7, 8], "pull": [0, 5], "request": [0, 3, 4, 5, 6, 7, 8], "share": [0, 3, 4, 5, 7, 8], "own": [0, 3, 4, 6, 7], "experi": [0, 3, 4, 5, 6, 7, 8], "commun": [0, 3, 4, 7, 8], "propos": [0, 4, 7], "chapter": [0, 3, 4, 5, 6, 7, 8], "section": [0, 3, 4, 5, 6, 7, 8], "found": [0, 3, 4, 5, 8], "http": [0, 1, 2, 3, 4, 5, 6, 7, 8], "com": [0, 2, 3, 4, 5, 6, 7, 8], "souzatharsi": [0, 2, 3, 4, 5, 6, 7, 8], "tamingllm": [0, 2, 3, 4, 5, 6, 7, 8], "whether": [0, 3, 4, 6, 7, 8], "you": [0, 1, 3, 4, 5, 6, 7, 8], "ve": [0, 5], "typo": [0, 7], "want": [0, 1, 3, 5, 6, 7, 8], "welcom": 0, "look": [0, 2, 3, 4, 5, 7], "our": [0, 1, 3, 4, 5, 6, 7, 8], "goal": [0, 1, 3, 4, 6, 7], "discourag": 0, "enabl": [0, 3, 4, 5, 6, 7, 8], "By": [0, 1, 2, 3, 4, 6, 7, 8], "upfront": [0, 2], "equip": [0, 2, 4, 7], "avoid": [0, 3, 4, 7, 8], "current": [0, 2, 3, 4, 6, 7, 8], "discours": [0, 2], "around": [0, 2, 3, 4, 6, 7, 8], "tend": [0, 2, 4, 7], "toward": [0, 3, 4, 7, 8], "extrem": [0, 3, 4, 7], "either": [0, 3, 4, 6, 7], "uncrit": 0, "enthusiasm": 0, "wholesal": [0, 4], "dismiss": 0, "differ": [0, 3, 4, 5, 6, 7, 8], "rather": [0, 1, 3, 4, 5, 7], "than": [0, 1, 3, 4, 5, 7], "theoret": [0, 3], "examin": [0, 3, 4, 5, 6, 7, 8], "first": [0, 1, 3, 4, 5, 6, 7, 8], "everi": [0, 4, 7], "concept": [0, 3, 4, 7], "illustr": [0, 3, 4, 6, 7, 8], "execut": [0, 4, 5, 7], "immedi": [0, 3, 4, 5], "analysi": [0, 1, 3, 4, 5, 6, 7], "balanc": [0, 3, 4, 5, 6, 7, 8], "help": [0, 3, 4, 5, 6, 7, 8], "intend": [0, 4, 7], "develop": [0, 1, 3, 4, 5, 6, 7, 8], "step": [0, 1, 3, 4, 5, 7, 8], "insight": [0, 3, 4, 5, 6, 7, 8], "along": [0, 3, 4, 5, 7], "guidanc": [0, 3, 8], "could": [0, 1, 3, 4, 5, 6, 7, 8], "derail": 0, "project": [0, 3, 4, 7], "earli": [0, 3, 4, 7, 8], "befor": [0, 3, 4, 5, 7, 8], "thei": [0, 1, 3, 4, 5, 6, 7, 8], "costli": [0, 4, 7], "problem": [0, 1, 2, 3, 5, 7], "too": [0, 1, 3, 4, 6, 7], "late": [0, 3, 7], "lifecycl": [0, 7], "lead": [0, 1, 3, 4, 6, 7, 8], "genai": [0, 1, 3, 7], "initi": [0, 1, 3, 4, 5, 6, 7, 8], "leader": [0, 4], "advoc": [0, 7], "anyon": [0, 7], "seek": [0, 4, 5, 7], "work": [0, 1, 3, 4, 5, 6, 7, 8], "typic": [0, 3, 4, 6, 7, 8], "job": [0, 4, 5, 7], "role": [0, 3, 4, 5, 6, 7, 8], "platform": [0, 4, 5, 6, 7, 8], "backend": [0, 3, 4], "exist": [0, 3, 4, 5], "ml": [0, 7], "transit": [0, 4, 5, 6, 8], "overse": 0, "motiv": [0, 3, 4, 8], "need": [0, 3, 4, 5, 6, 7], "readi": [0, 4, 7], "desir": [0, 3, 4, 8], "perform": [0, 3, 4, 6, 7, 8], "after": [0, 1, 3, 4, 5, 6, 7, 8], "read": [0, 3, 4, 6, 7, 8], "implic": [0, 1, 3, 4, 7], "recommend": [0, 3, 4, 6, 7, 8], "abl": [0, 3, 4, 6, 8], "deploi": [0, 3, 4, 5, 6, 7], "proper": [0, 3, 5, 7, 8], "realist": [0, 3, 7], "effort": [0, 4, 7, 8], "estim": [0, 4, 7], "impact": [0, 3, 4, 5, 6, 7, 8], "timelin": 0, "To": [0, 3, 4, 5, 6, 7, 8], "should": [0, 3, 4, 6, 7, 8], "basic": [0, 3, 4, 5, 6, 7], "program": [0, 4, 5], "knowledg": [0, 3, 4, 5, 7], "introductori": [0, 1, 2], "langchain": [0, 4, 6], "e": [0, 1, 3, 4, 5, 6, 7, 8], "g": [0, 3, 4, 5, 6, 7, 8], "chat": [0, 3, 4, 5, 6, 7, 8], "prompt": [0, 4, 7], "templat": [0, 4], "openai": [0, 3, 4, 5, 8], "anthrop": [0, 3, 8], "similar": [0, 3, 4, 5, 8], "dive": 0, "here": [0, 2, 3, 4, 5, 6, 7, 8], "get": [0, 3, 4, 5, 6, 7, 8], "start": [0, 3, 4, 5, 7, 8], "clone": [0, 3], "companion": 0, "git": 0, "cd": 0, "activ": [0, 3, 4, 7], "virtual": [0, 4], "m": [0, 3, 4, 5, 7, 8], "venv": 0, "tame": [0, 3, 4, 5, 6, 7, 8], "env": [0, 3, 4, 6, 7, 8], "bin": [0, 5], "On": [0, 4, 5, 8], "window": [0, 2, 4, 5], "script": [0, 5], "try": [0, 1, 3, 4, 7, 8], "contain": [0, 3, 4, 5, 6, 7, 8], "possibl": [0, 3, 4, 5, 7, 8], "includ": [0, 1, 3, 4, 5, 6, 7, 8], "necessari": [0, 3, 4, 6, 7], "instal": [0, 3, 4, 5, 8], "go": [0, 3, 4, 6, 8], "feel": [0, 5], "prefer": [0, 4, 5, 7, 8], "packag": [0, 4, 5, 8], "pip": [0, 3, 4, 5, 8], "poetri": [0, 7], "file": [0, 3, 4, 5, 6, 7, 8], "root": [0, 3], "directori": [0, 4, 5], "add": [0, 3, 4, 5, 6, 7], "other": [0, 3, 4, 5, 6, 7, 8], "sensit": [0, 3, 4, 5, 7], "openai_api_kei": [0, 3], "your_openai_api_key_her": 0, "never": [0, 8], "commit": [0, 3, 4, 7], "version": [0, 3, 4, 5, 7, 8], "control": [0, 1, 3, 4, 5, 7, 8], "kept": [0, 4], "privat": [0, 4], "If": [0, 1, 3, 4, 5, 7, 8], "encount": [0, 2, 4, 7], "rate": [0, 3, 4, 7], "consid": [0, 3, 4, 6, 7, 8], "smaller": [0, 3, 4, 6, 8], "retri": [0, 8], "logic": [0, 1, 3, 4, 6, 7], "conflict": [0, 3, 4], "fresh": 0, "like": [0, 1, 3, 4, 5, 6, 7, 8], "check": [0, 4, 5, 7, 8], "page": [0, 4], "known": [0, 4, 7, 8], "now": [0, 1, 3, 4, 5, 6, 7, 8], "let": [0, 3, 4, 5, 6, 7, 8], "begin": [0, 4, 7, 8], "explor": [0, 1, 3, 4, 5, 7, 8], "dr": [0, 3], "tharsi": [0, 2, 3, 4, 5, 6, 7, 8], "souza": [0, 2, 3, 4, 5, 6, 7, 8], "scientist": [0, 1, 5, 7], "special": [0, 4, 5, 7, 8], "he": [0, 3, 4, 7], "lectur": 0, "columbia": 0, "univers": [0, 4, 5, 7], "master": [0, 5, 8], "scienc": [0, 3, 4, 7], "appli": [0, 3, 4, 5, 6, 7, 8], "analyt": 0, "incom": [0, 4], "head": [0, 3, 4, 6, 7], "equiti": [0, 4], "citadel": 0, "former": [0, 1, 4, 5], "senior": [0, 4], "vp": 0, "two": [0, 3, 4, 5, 6, 7, 8], "sigma": [0, 3], "invest": [0, 3, 4, 7, 8], "also": [0, 3, 4, 5, 6, 7, 8], "enjoi": 0, "mentor": 0, "under": [0, 3, 4, 7, 8], "repres": [0, 3, 4, 5, 8], "student": [0, 3, 7], "profession": [0, 3, 4, 7, 8], "divers": [0, 3, 4, 6, 7, 8], "global": [0, 4, 7], "ecosystem": [0, 4], "With": [0, 3, 4, 5, 7], "over": [0, 2, 3, 4, 5, 6, 7, 8], "15": [0, 4, 7, 8], "deliv": [0, 4, 5], "across": [0, 1, 3, 4, 5, 7, 8], "startup": 0, "fortun": 0, "500": [0, 3, 4, 7], "compani": [0, 3, 4, 6, 7, 8], "numer": [0, 4, 7], "scholarli": 0, "frequent": [0, 4, 8], "speaker": [0, 4], "academ": [0, 3, 4, 7], "busi": [0, 4, 5, 7], "confer": [0, 8], "ground": [0, 3, 4], "background": [0, 1, 4, 5, 6], "draw": [0, 3, 4, 7, 8], "scale": [0, 3, 4, 5, 7, 8], "stage": [0, 3, 7, 8], "major": [0, 3, 4, 7, 8], "institut": [0, 4, 7], "well": [0, 3, 4, 5, 7, 8], "advis": [0, 3], "profit": [0, 4, 6, 7, 8], "organ": [0, 3, 4, 5, 6], "uniqu": [0, 3, 4, 7], "bridg": [0, 5, 7], "gap": [0, 1, 3, 7], "between": [0, 1, 3, 4, 5, 6, 7, 8], "potenti": [0, 1, 3, 4, 5, 6, 7, 8], "next": [0, 1, 3, 4, 5, 7, 8], "hold": [0, 3, 4], "ph": [0, 7], "d": [0, 3, 4, 5, 7, 8], "ucl": 0, "london": 0, "phil": [0, 7], "sc": 0, "b": [0, 4, 5, 7, 8], "tell": [1, 3, 7], "mere": [1, 4], "what": [1, 3, 4, 5, 7, 8], "someth": [1, 4, 5], "i": [1, 2, 4, 5, 6, 7, 8], "emanuel": [1, 3, 4, 7], "derman": 1, "an": [1, 2, 3, 4, 5, 6, 7, 8], "altern": [1, 3, 4, 5, 6, 7], "titl": [1, 2, 3, 4, 5, 6, 7, 8], "thi": [1, 2, 3, 4, 5, 6, 7, 8], "book": [1, 2, 4], "been": [1, 3, 4, 5, 7], "behav": 1, "badli": 1, "come": [1, 3, 4, 5, 6, 7, 8], "notic": [1, 3, 4, 7, 8], "parallel": [1, 3, 4, 5], "": [1, 3, 4, 5, 6, 7, 8], "semin": [1, 7], "2011": 1, "coincident": 1, "just": [1, 3, 4, 5, 6, 7, 8], "caution": 1, "against": [1, 3, 4, 5, 7], "treat": [1, 4, 7], "perfect": [1, 4, 5], "represent": [1, 4, 6, 7], "realiti": [1, 7], "aim": [1, 3, 4, 5, 6, 7, 8], "highlight": [1, 3, 4, 5, 6, 7, 8], "practic": [1, 3, 4, 5, 6, 7], "cours": [1, 4, 7], "bare": [1, 5], "fact": [1, 3, 4, 7], "actual": [1, 3, 4, 6, 7, 8], "physicist": 1, "legendari": 1, "author": [1, 2, 3, 4, 5, 6, 7, 8], "professor": 1, "quant": 1, "goldman": 1, "sach": 1, "scientif": [1, 3, 4], "fail": [1, 3, 4, 7], "we": [1, 3, 4, 5, 6, 7, 8], "mistak": [1, 7], "approxim": [1, 4, 8], "full": [1, 3, 4, 5, 7, 8], "assumpt": [1, 4, 7], "core": [1, 4, 5, 7], "premis": [1, 5], "hi": [1, 4, 7, 8], "aspect": [1, 3, 4, 6, 7, 8], "world": [1, 3, 4, 7, 8], "inher": [1, 2, 3, 4, 7, 8], "involv": [1, 3, 4, 5, 7, 8], "simplif": 1, "argu": [1, 7, 8], "crise": 1, "2008": 1, "crash": 1, "occur": [1, 3, 4, 7], "partli": 1, "becaus": [1, 3, 4, 7], "peopl": [1, 3, 4, 5, 7], "put": [1, 4], "much": [1, 3, 4, 5], "faith": 1, "mathemat": [1, 4], "recogn": [1, 3, 4, 7], "human": [1, 4, 6, 7, 8], "behavior": [1, 3, 4, 5, 7], "market": [1, 4, 6, 8], "dynam": [1, 3, 4, 7], "reason": [1, 3, 4, 6, 7, 8], "Their": [1, 4, 8], "respons": [1, 4, 5, 6, 7, 8], "often": [1, 3, 4, 5, 6, 7, 8], "convinc": [1, 3], "probabilist": [1, 4], "train": [1, 4, 5, 7, 8], "data": [1, 4, 5, 6, 7, 8], "true": [1, 3, 4, 6, 7, 8], "even": [1, 3, 4, 5, 6, 7, 8], "though": [1, 3, 4, 5, 7, 8], "insist": 1, "machin": [1, 3, 5, 7, 8], "todai": [1, 5, 8], "grow": [1, 3, 4, 5, 7, 8], "pervas": [1, 7], "belief": [1, 5, 7], "solv": [1, 3, 4, 5, 7, 8], "ani": [1, 3, 4, 5, 6, 7, 8], "context": [1, 2, 3, 4, 5, 6, 7, 8], "content": 1, "wish": [1, 4], "user": [1, 4, 5, 6], "moreov": 1, "were": [1, 3, 4, 7, 8], "predict": [1, 3, 4, 5, 7, 8], "chatbot": [1, 3, 4, 5, 7], "twist": [1, 7], "wrap": [1, 5, 8], "further": [1, 3, 4, 5, 6, 7, 8], "daili": [1, 7], "life": [1, 4, 5, 7], "workflow": [1, 4, 5, 7, 8], "affect": [1, 4, 5, 7], "decid": [1, 3, 4, 6], "action": [1, 3, 4, 6, 7], "coupl": 1, "lack": [1, 3, 4, 7, 8], "pose": [1, 3, 4, 6, 7, 8], "still": [1, 4, 5, 7], "figur": [1, 4, 8], "out": [1, 3, 4, 5, 6, 7, 8], "serv": [1, 3, 4, 6, 7, 8], "builder": [1, 5], "who": [1, 3, 4, 5, 6, 7, 8], "remain": [1, 3, 4, 5, 6, 7], "clear": [1, 3, 4, 7, 8], "ei": 1, "about": [1, 3, 4, 5, 6, 7, 8], "therefor": [1, 3, 4, 5, 7], "end": [1, 3, 4, 5, 6, 7, 8], "detail": [1, 3, 4, 5, 6, 7, 8], "python": [1, 2, 4, 5, 6, 7, 8], "code": [1, 2, 3, 4, 5, 7, 8], "diminish": [1, 3, 4], "promot": [1, 3, 4, 7], "nuanc": [1, 3, 4, 6, 7, 8], "acknowledg": [1, 4, 7], "within": [1, 3, 4, 6, 7, 8], "trustworthi": [1, 7], "taught": 1, "u": [1, 3, 4, 7, 8], "where": [1, 3, 4, 5, 6, 7, 8], "der11": 1, "why": [1, 3, 4, 7, 8], "confus": [1, 7], "illus": 1, "disast": [1, 4], "wall": [1, 5], "street": [1, 5], "press": [1, 4, 5], "isbn": [1, 3, 4], "9781439165010": 1, "url": [1, 2, 3, 4, 5, 6, 7, 8], "googl": [1, 4, 8], "co": [1, 3, 4, 5, 7], "uk": [1, 7], "id": [1, 4, 5, 7], "lke_cwm4wm8c": 1, "sign": [2, 4, 7], "up": [2, 3, 4, 5, 6, 7, 8], "receiv": [2, 3, 4, 6, 7, 8], "updat": [2, 3, 4, 5, 6, 7, 8], "abstract": [2, 4, 7, 8], "heavili": [2, 3, 4, 7, 8], "gloss": 2, "fundament": [2, 3, 4, 7, 8], "convers": [2, 3, 4, 5, 6, 7, 8], "kei": [2, 3, 5, 7, 8], "proven": 2, "yet": [2, 3, 4, 6, 7], "concret": [2, 7], "unstructur": [2, 8], "sidestep": 2, "misc": [2, 3, 4, 5, 6, 7, 8], "tharsistpsouza2024tamingllm": [2, 3, 4, 5, 6, 7, 8], "t": [2, 3, 4, 5, 6, 7, 8], "p": [2, 3, 4, 5, 6, 7, 8], "2024": [2, 3, 4, 5, 6, 7, 8], "journal": [2, 3, 4, 5, 6, 7, 8], "repositori": [2, 3, 4, 5, 6, 7, 8], "valu": [3, 4, 5, 6, 7, 8], "its": [3, 4, 5, 6, 7, 8], "privileg": 3, "abov": [3, 4, 7], "soon": [3, 8], "lose": [3, 4], "dwight": 3, "eisenhow": 3, "releas": [3, 4, 5, 7, 8], "3": [3, 4, 5, 8], "5": [3, 4, 5, 6, 8], "2022": [3, 4, 7], "mark": [3, 4, 7], "pivot": [3, 4], "moment": 3, "histori": [3, 4, 5], "artifici": [3, 4, 7], "intellig": [3, 4, 7], "five": [3, 4, 7], "dai": [3, 4, 7, 8], "launch": [3, 4, 7], "attract": [3, 4], "million": [3, 4], "month": [3, 4, 7], "becam": 3, "fastest": [3, 4, 7], "100": [3, 4, 5, 7, 8], "monthli": [3, 4], "rais": [3, 4, 6, 7], "intrigu": 3, "question": [3, 4, 5, 7, 8], "did": [3, 4, 8], "dramat": [3, 4, 5, 8], "predecessor": 3, "gpt": [3, 4, 5, 6, 7, 8], "had": [3, 4], "same": [3, 4, 5, 6, 7, 8], "number": [3, 4, 5, 6, 7, 8], "paramet": [3, 4, 5, 7, 8], "far": [3, 6, 7], "less": [3, 4, 5, 7], "attent": [3, 5], "arguabl": [3, 5], "feedback": [3, 4, 7, 8], "abil": [3, 4, 5, 7, 8], "least": [3, 4, 7], "ey": 3, "breakthrough": [3, 7], "demonstr": [3, 4, 5, 6, 7, 8], "crucial": [3, 5, 7, 8], "greater": [3, 4, 7], "process": [3, 4, 5, 6, 7, 8], "modern": [3, 4, 6, 8], "techniqu": [3, 4, 5, 6], "direct": [3, 4, 5, 7], "rafailov": 3, "et": [3, 4, 7, 8], "al": [3, 4, 7, 8], "present": [3, 4, 5, 6, 7, 8], "autom": [3, 4, 7, 8], "fashion": [3, 8], "open": [3, 4, 6, 7, 8], "sourc": [3, 4, 5, 6, 7, 8], "common": [3, 4, 5, 6, 8], "pre": [3, 4, 5, 7], "default": [3, 4, 5, 7, 8], "state": [3, 4, 5, 6, 7, 8], "art": [3, 4, 7], "object": [3, 4, 5, 7, 8], "given": [3, 4, 5, 6, 7, 8], "webpag": 3, "internet": [3, 4], "veri": [3, 4, 5, 7], "ask": [3, 4, 5, 7, 8], "instruct": [3, 4, 5, 6, 7, 8], "sai": [3, 8], "ouyang": [3, 7], "2": [3, 4, 5, 8], "explain": 3, "moon": 3, "land": [3, 4], "6": [3, 4, 5, 6, 8], "old": [3, 4], "import": [3, 4, 5, 6, 7, 8], "pipelin": [3, 4, 5, 7, 8], "pipe": [3, 7], "text": [3, 4, 5, 6, 7, 8], "gpt2": [3, 4], "msg": 3, "short": [3, 4, 6, 7, 8], "sentenc": [3, 4, 6, 7, 8], "_": [3, 4, 7, 8], "rang": [3, 4, 5, 6, 7, 8], "len": [3, 4, 5, 6, 7], "print": [3, 4, 5, 6, 7, 8], "f": [3, 4, 5, 6, 7, 8], "n": [3, 4, 5, 6, 7, 8], "1": [3, 4, 5, 8], "0": [3, 4, 5, 6, 7, 8], "generated_text": 3, "good": [3, 4, 5, 8], "idea": [3, 7], "one": [3, 4, 5, 6, 7, 8], "those": [3, 4, 6, 7, 8], "littl": [3, 4], "green": [3, 7], "dot": 3, "Then": [3, 4], "line": [3, 4, 5, 7], "later": [3, 4, 5, 7, 8], "re": [3, 4, 6, 7, 8], "alreadi": [3, 4], "movi": 3, "theori": [3, 4], "some": [3, 4, 5, 6, 7, 8], "mean": [3, 4, 5, 6, 7, 8], "word": [3, 4, 6, 8], "tepid": 3, "articl": [3, 4, 5, 6, 7], "sure": [3, 4, 6, 7, 8], "lunar": 3, "As": [3, 4, 5, 6, 7, 8], "see": [3, 4, 5, 7, 8], "coher": [3, 4, 6], "explan": [3, 4, 7, 8], "child": [3, 4, 7], "nonsens": [3, 7], "meander": 3, "unrel": [3, 4, 7], "topic": [3, 4, 5, 6, 7, 8], "simpl": [3, 4, 5, 6, 7, 8], "appropri": [3, 4, 5, 6, 7, 8], "young": [3, 4, 7], "instead": [3, 4, 5, 6, 7, 8], "address": [3, 4, 6, 7, 8], "issu": [3, 4, 6, 7, 8], "introduc": [3, 4, 5, 6, 7, 8], "rlhf": [3, 7], "intent": [3, 7], "wide": [3, 4, 5, 6, 7, 8], "task": [3, 5, 6, 7, 8], "fig": [3, 4, 5, 6, 7, 8], "7": [3, 4, 5, 6, 7], "collect": [3, 4, 5, 6, 7], "sampl": [3, 5, 6, 8], "label": [3, 4, 7, 8], "comparison": 3, "reward": [3, 4, 7], "sever": [3, 4, 5, 6, 7, 8], "rank": [3, 4, 7], "best": [3, 4, 5, 7], "worst": 3, "rm": 3, "reinforc": [3, 4, 7], "write": [3, 4, 5, 7, 8], "stori": [3, 7], "frog": 3, "calcul": [3, 4, 5, 7], "score": [3, 4, 5, 7, 8], "ppo": 3, "proxim": 3, "iter": [3, 4, 6, 7, 8], "accur": [3, 4, 7, 8], "undesir": [3, 7], "simplifi": [3, 4, 5, 8], "view": [3, 4, 7], "show": [3, 4, 5, 6, 7, 8], "progress": [3, 6, 7], "pattern": [3, 4, 7, 8], "ha": [3, 4, 5, 7, 8], "instanc": [3, 4, 5, 6, 7], "directli": [3, 4, 5, 7, 8], "For": [3, 4, 5, 6, 7, 8], "llama": [3, 4, 7, 8], "guard": 3, "team": [3, 4, 5, 8], "8b": [3, 7], "wa": [3, 4, 5, 7, 8], "classif": [3, 4, 7, 8], "bypass": [3, 7], "similarli": [3, 4, 5, 7], "zephyr": 3, "7b": [3, 4, 7], "alpha": [3, 4, 8], "mistral": [3, 5, 8], "publicli": [3, 4, 8], "assist": [3, 4, 5, 7, 8], "paper": [3, 4, 7, 8], "compon": [3, 4], "particular": [3, 4, 7, 8], "foundat": [3, 4, 6, 7], "advanc": [3, 4, 5, 6, 7, 8], "method": [3, 4, 5, 6, 7, 8], "strong": [3, 4, 5, 7, 8], "At": [3, 4, 5, 8], "high": [3, 4, 5, 6, 7, 8], "level": [3, 4, 6, 7, 8], "carefulli": [3, 4, 7, 8], "curat": [3, 4], "purpos": [3, 4, 5, 7, 8], "exhibit": [3, 4, 7], "domain": [3, 4, 5, 7], "emploi": [3, 4, 7, 8], "prove": [3, 4, 7], "particularli": [3, 4, 5, 6, 7, 8], "valuabl": [3, 4, 5, 8], "scenario": [3, 4, 5, 7, 8], "precis": [3, 4, 5, 7, 8], "style": [3, 4], "tone": 3, "expertis": [3, 4, 7], "medic": [3, 4, 5], "legal": [3, 4, 7], "field": [3, 4, 7, 8], "adher": [3, 4, 6, 7, 8], "guidelin": [3, 4, 7], "servic": [3, 4, 5, 6, 7, 8], "standard": [3, 4, 5, 7], "approach": [3, 4, 5, 6, 8], "each": [3, 4, 5, 6, 7, 8], "distinct": [3, 4, 5, 7], "advantag": [3, 4, 5, 6, 7, 8], "weight": [3, 4, 5, 7], "maximum": [3, 4, 5, 6, 7], "lora": [3, 7], "low": [3, 4, 5, 7, 8], "hu": [3, 7], "2021": [3, 4], "small": [3, 4, 5, 8], "matric": 3, "effici": [3, 4, 5, 6, 7, 8], "qlora": 3, "quantiz": 3, "dettmer": 3, "2023": [3, 4, 5, 7, 8], "combin": [3, 4, 5, 6, 7, 8], "memori": [3, 4, 5, 6, 7], "footprint": 3, "modest": 3, "increas": [3, 4, 5, 6, 7, 8], "likelihood": [3, 4, 7], "obtain": [3, 4, 5, 7, 8], "probabl": [3, 4, 5, 8], "outcom": [3, 4, 7, 8], "hong": [3, 4], "unintend": [3, 7], "suboptim": 3, "seen": [3, 4, 7], "research": [3, 4, 6], "maxim": [3, 4], "shown": [3, 4, 5, 7], "alon": [3, 4, 7], "gain": [3, 4, 7], "achiev": [3, 4, 5, 7, 8], "bai": [3, 4, 7], "touvron": 3, "sinc": [3, 4, 5, 6, 7, 8], "main": [3, 4, 5, 6, 7, 8], "categori": [3, 4, 5, 7], "algorithm": [3, 4, 7], "meanwhil": 3, "superior": [3, 4, 7], "benchmark": 3, "xu": [3, 4, 7], "schulman": [3, 7], "2017": [3, 4], "popular": [3, 5, 8], "understood": 3, "set": [3, 4, 5, 6, 7, 8], "rule": [3, 4, 6, 8], "govern": [3, 4], "reflect": [3, 4, 7], "anoth": [3, 4, 5, 7], "adjust": [3, 4, 5, 6, 7, 8], "One": [3, 4, 5, 7], "strength": [3, 4, 7], "2024c": 3, "real": [3, 4, 5, 6, 7, 8], "noisi": 3, "delai": [3, 4, 5, 7], "subsequ": [3, 8], "situat": [3, 4, 6, 7], "clip": 3, "surrog": 3, "function": [3, 4, 5, 6, 7, 8], "stabl": [3, 4], "prevent": [3, 4, 7, 8], "overreact": 3, "converg": 3, "due": [3, 4, 6, 7], "simplic": [3, 5], "award": [3, 4], "runner": 3, "neurip": 3, "blog": [3, 4, 7, 8], "4": [3, 4, 5, 8], "fit": [3, 4, 6, 7, 8], "pair": [3, 4, 7], "rl": [3, 7], "find": [3, 4, 5, 6, 7, 8], "contrast": [3, 4, 7], "satisfi": [3, 4], "implicit": [3, 4, 7], "whose": [3, 4], "correspond": [3, 4, 8], "extract": [3, 4, 6, 7, 8], "close": [3, 4, 5, 7], "compar": [3, 4, 5, 6, 7], "assign": [3, 4, 5, 7, 8], "higher": [3, 4, 5], "kl": [3, 5], "diverg": [3, 5], "origin": [3, 4, 5, 6, 7, 8], "preserv": [3, 5, 6, 7], "defin": [3, 4, 5, 6, 7, 8], "equat": 3, "mathcal": 3, "l": [3, 4], "pi_": 3, "theta": [3, 8], "ref": 3, "mathbb": [3, 8], "x": [3, 4, 5, 7], "y_w": 3, "y_l": 3, "sim": [3, 8], "left": [3, 5], "log": [3, 4, 5], "beta": [3, 4, 7, 8], "underbrac": 3, "frac": [3, 5, 7], "color": [3, 4], "red": 3, "right": [3, 4, 5, 7], "respect": [3, 4, 5, 7], "deviat": [3, 4, 5, 7, 8], "straightforward": [3, 4, 6, 7, 8], "librari": [3, 4, 5, 6, 7, 8], "huggingfac": [3, 4, 5, 7], "trl": [3, 7], "2024d": 3, "suit": [3, 4, 7], "friendli": [3, 4, 5, 6], "interfac": [3, 4, 5, 7], "featur": [3, 4, 5, 7, 8], "challeng": [3, 4, 5, 6, 7, 8], "distinguish": [3, 4, 7], "scalabl": [3, 4, 6, 7], "doe": [3, 4, 6, 7, 8], "pretrain": [3, 4], "hou": [3, 4], "poor": [3, 4, 7], "return": [3, 4, 5, 6, 7, 8], "addit": [3, 4, 5, 6, 7, 8], "benefit": [3, 4, 5, 7, 8], "fix": [3, 4, 5, 6, 7], "invers": 3, "trend": [3, 4, 7, 8], "util": [3, 4, 5, 6, 7], "rapid": [3, 4, 5, 7], "yield": [3, 4], "onli": [3, 4, 5, 6, 7, 8], "margin": [3, 4, 7, 8], "capit": [3, 4, 6, 8], "inaccuraci": [3, 4], "nois": 3, "dure": [3, 4, 7, 8], "accuraci": [3, 4, 5, 7, 8], "lag": [3, 4, 7], "significantli": [3, 4, 5, 6, 7], "indic": [3, 4, 5, 7, 8], "signal": [3, 7], "plateau": 3, "sophist": [3, 4, 5, 6, 7], "previou": [3, 4, 5, 6, 8], "deriv": [3, 4, 5], "pairwis": [3, 4], "feng": [3, 7], "substanti": [3, 4, 7], "wors": [3, 5, 8], "influenc": [3, 4, 7, 8], "success": [3, 4, 7, 8], "imbal": 3, "stronger": 3, "bad": 3, "ones": [3, 7], "loss": [3, 4, 5, 7], "gradient": [3, 4, 7], "dispref": 3, "unbalanc": 3, "trajectori": 3, "stuck": 3, "saddl": 3, "point": [3, 4, 5, 6, 7], "forward": [3, 4, 7], "futur": [3, 4, 7], "phenomenon": [3, 7, 8], "degrad": [3, 4, 5, 7, 8], "danger": [3, 5, 7], "loop": [3, 4, 5, 7], "recurs": [3, 6], "kazdan": 3, "qualiti": [3, 4, 5, 6, 7, 8], "pollut": 3, "replac": [3, 4, 5], "amplif": 3, "reduct": [3, 4, 5], "express": [3, 4, 7, 8], "catastroph": [3, 7], "forget": [3, 8], "previous": [3, 4, 6, 7, 8], "mitig": [3, 4, 6, 7, 8], "mix": [3, 4, 7, 8], "metric": [3, 5, 7], "sz\u00e9p": 3, "regular": [3, 4, 7, 8], "relev": [3, 4, 7], "scarc": 3, "behaviour": 3, "strateg": [3, 4, 7, 8], "compli": [3, 4, 5, 7, 8], "modif": [3, 4, 5, 7], "outsid": [3, 4], "evidenc": 3, "landmark": 3, "askel": [3, 4, 7], "2024a": [3, 5], "dec": 3, "explicitli": [3, 4], "so": [3, 4, 7, 8], "might": [3, 4, 5, 6, 7, 8], "pretend": 3, "adopt": [3, 4, 5, 7, 8], "onc": [3, 4, 6, 7], "complet": [3, 4, 5, 6, 7, 8], "describ": [3, 4, 5, 7], "harmless": [3, 7], "told": 3, "retrain": 3, "queri": [3, 4], "tier": [3, 4, 7], "paid": [3, 4], "column": [3, 4, 7], "condit": [3, 4], "toxic": [3, 7], "excerpt": [3, 4, 5], "scratchpad": 3, "refus": [3, 7, 8], "happen": [3, 7], "bomb": [3, 7], "engag": [3, 4, 7, 8], "intern": [3, 4, 7], "unmonitor": 3, "longer": [3, 4], "believ": [3, 4, 5, 7, 8], "act": [3, 4, 5, 7], "therebi": [3, 4, 6], "reveal": [3, 4, 7], "complianc": [3, 4, 7, 8], "phase": [3, 4], "natur": [3, 4, 6, 7, 8], "evid": [3, 4, 7, 8], "seemingli": 3, "surpris": 3, "appear": [3, 4, 7, 8], "criteria": [3, 4, 7], "underli": [3, 4, 7, 8], "anim": [3, 7], "welfar": 3, "instil": 3, "implicitli": 3, "consequ": [3, 4, 7, 8], "explicit": [3, 4, 7, 8], "chain": [3, 4, 6], "thought": [3, 4, 8], "opaqu": 3, "aris": [3, 4, 7], "opu": 3, "sonnet": [3, 4, 6], "wherea": [3, 4], "haiku": [3, 7], "persist": 3, "resist": [3, 4], "embed": [3, 4, 5, 6], "doesn": [3, 4, 5, 6, 8], "anti": [3, 4], "lab": 3, "exfiltr": [3, 7], "protect": [3, 4, 7], "Not": [3, 4, 7], "malici": [3, 4, 7], "support": [3, 4, 5, 7, 8], "concern": [3, 4, 5, 7], "mechan": [3, 4, 7, 8], "insuffici": [3, 4], "don": [3, 4, 6, 8], "concerningli": 3, "call": [3, 4, 5, 6, 7, 8], "detect": [3, 4, 7, 8], "decept": [3, 4, 7], "warrant": [3, 7], "deeper": [3, 4, 6], "scrutini": [3, 4, 7], "reli": [3, 4, 7], "cross": [3, 4, 5, 7], "circular": 3, "bia": [3, 4, 7, 8], "truli": [3, 4], "trust": [3, 4, 7, 8], "referenti": 3, "ly": 3, "hood": [3, 8], "deep": [3, 4, 7, 8], "mechanist": 3, "drive": [3, 6, 7, 8], "correl": [3, 4, 5], "miss": [3, 4, 7], "confound": 3, "factor": [3, 4, 5, 6, 8], "establish": [3, 4, 7], "attempt": [3, 4, 6, 7], "causal": [3, 4], "heavi": 3, "relianc": [3, 4, 7], "oversimplifi": 3, "frame": 3, "subtler": 3, "narr": [3, 4], "henc": [3, 4, 7], "agenc": [3, 4, 7], "onto": 3, "anthropomorph": 3, "obscur": 3, "blind": [3, 4], "failur": [3, 4, 7], "mode": [3, 5, 7], "map": [3, 4, 5, 8], "cleanli": 3, "analogi": 3, "excel": [3, 4, 5, 7, 8], "review": [3, 4, 7, 8], "prof": 3, "jacob": [3, 4, 7], "andrea": [3, 4, 7], "yoshua": [3, 7], "bengio": [3, 7], "jasjeet": 3, "sekhon": 3, "rohin": 3, "shah": 3, "2024b": [3, 5, 8], "assum": [3, 4, 6, 7], "acm": [3, 7], "inc": [3, 4, 6, 8], "dedic": [3, 4, 5, 7, 8], "democrat": [3, 4, 8], "educ": [3, 4, 6, 7], "k": [3, 4, 5, 6, 7, 8], "12": [3, 4, 6, 7], "name": [3, 4, 6, 7, 8], "smolk": 3, "ll": [3, 4, 5], "walk": 3, "measur": [3, 4, 5, 7], "huggingfacetb": 3, "360m": [3, 4], "compact": [3, 4, 7], "part": [3, 4, 6, 7, 8], "famili": [3, 7, 8], "publish": [3, 7, 8], "api": [3, 4, 5], "local": [3, 4, 6, 7, 8], "infer": [3, 4, 5, 7], "remot": [3, 4], "load": [3, 4, 5, 6, 7, 8], "store": [3, 4, 6, 7], "eventu": [3, 4], "your_openai_api_kei": 3, "reusabl": 3, "anchor": [3, 7], "worth": [3, 4, 5], "choic": [3, 4, 5, 7, 8], "lightweight": [3, 4, 5, 8], "suitabl": [3, 4, 5, 7], "devic": [3, 4, 5, 8], "Its": [3, 4, 5], "candid": [3, 4], "said": [3, 4, 7], "necessarili": [3, 4, 7], "par": [3, 4], "mind": [3, 4, 5, 7], "factual": [3, 4, 7], "inconsist": [3, 4, 7, 8], "guardrail": [3, 7], "articul": 3, "uphold": [3, 7], "employe": [3, 4], "stakehold": [3, 4, 7], "expect": [3, 4, 5, 6, 7, 8], "regard": [3, 4, 7], "ethic": [3, 4, 5, 7], "conduct": [3, 4], "social": [3, 4, 7], "mission": [3, 7], "vision": [3, 4, 5, 7], "cultur": [3, 4, 5, 7], "account": [3, 4, 7], "codifi": 3, "mlcommon": 3, "vidgen": [3, 7], "encompass": [3, 7], "seven": 3, "hazard": [3, 4, 7], "violent": [3, 7], "crime": [3, 7], "sex": [3, 7], "relat": [3, 4, 7], "sexual": [3, 7], "exploit": [3, 4, 7], "indiscrimin": [3, 7], "weapon": [3, 7], "chemic": 3, "biolog": 3, "radiolog": 3, "nuclear": [3, 4], "explos": [3, 7], "cbrne": 3, "suicid": [3, 7], "hate": [3, 7], "speech": [3, 7], "below": [3, 4, 5, 6, 7, 8], "markdown": [3, 4, 5, 6, 7], "written": [3, 4], "english": [3, 6], "o": [3, 4, 6, 7, 8], "ipython": [3, 4, 7], "displai": [3, 4, 7, 8], "def": [3, 4, 6, 7, 8], "load_polici": 3, "policy_path": 3, "path": [3, 4, 6, 7], "join": [3, 4, 6, 7], "genai_polici": 3, "md": [3, 4, 5, 7, 8], "r": [3, 4, 5, 6, 7, 8], "policy_cont": 3, "classroom": [3, 7], "accept": [3, 4, 5, 7], "unaccept": [3, 5], "ag": [3, 4, 7], "subject": [3, 4, 5], "posit": [3, 4, 5, 6, 7, 8], "confid": [3, 4, 8], "inclus": [3, 4, 6, 7, 8], "celebr": 3, "definit": [3, 4, 8], "creativ": [3, 4, 8], "math": [3, 4], "tip": [3, 7], "digit": [3, 4], "literaci": 3, "onlin": [3, 4, 5, 7, 8], "histor": [3, 4], "violenc": [3, 7], "physic": [3, 4, 7], "fight": [3, 7], "crimin": [3, 7], "illeg": [3, 7], "glorifi": [3, 7], "person": [3, 4, 5, 7, 8], "eat": [3, 7], "disord": 3, "diet": 3, "dare": 3, "advic": [3, 4, 7], "discriminatori": [3, 7], "bulli": [3, 7], "harass": [3, 4, 7], "target": [3, 4, 5, 7, 8], "group": [3, 4, 5, 6, 7], "religi": [3, 5, 7], "racial": [3, 4, 7], "ethnic": [3, 7], "gender": [3, 4, 7], "discrimin": [3, 4, 7], "adult": [3, 7], "profan": [3, 7], "relationship": [3, 4], "substanc": [3, 4], "drug": [3, 7], "gambl": 3, "bet": 3, "protocol": [3, 4, 7], "redirect": 3, "alert": 3, "record": [3, 4, 5, 7], "audit": [3, 4], "teacher": [3, 7], "parent": [3, 7], "continu": [3, 4, 6, 7, 8], "construct": [3, 4, 5, 7, 8], "compliant": [3, 7], "violat": [3, 4, 7], "intens": [3, 4, 8], "demand": [3, 4, 5, 7, 8], "especi": [3, 4, 5, 6, 7, 8], "dong": [3, 4, 7], "There": [3, 4, 5, 6, 7, 8], "rlaif": [3, 7], "give": [3, 4, 7], "rise": [3, 7], "kim": [3, 4, 7], "meta": [3, 4, 6, 7], "wu": [3, 4, 7, 8], "scheme": [3, 5], "inspir": [3, 7], "schema": [3, 8], "row": [3, 4, 7], "match": [3, 4, 5, 7, 8], "boundari": [3, 4, 7], "craft": [3, 4, 7, 8], "elicit": [3, 7, 8], "unalign": 3, "panda": [3, 4, 7], "chosen_responses_path": 3, "chosen_respons": 3, "csv": [3, 4, 7], "rejected_responses_path": 3, "rejected_respons": 3, "chosen_responses_jsonl_path": 3, "batch_result": 3, "jsonl": 3, "dpo_dataset_s": 3, "5000": 3, "class": [3, 4, 5, 6, 7, 8], "userpromptgener": 3, "pd": [3, 4, 7], "pydant": [3, 4, 7, 8], "basemodel": [3, 4, 7, 8], "time": [3, 4, 5, 6, 7, 8], "type": [3, 4, 5, 6, 7, 8], "dotenv": [3, 4, 6, 7, 8], "load_dotenv": [3, 4, 6, 7, 8], "environ": [3, 4, 5, 6, 7, 8], "variabl": [3, 4, 6, 7, 8], "overrid": [3, 7, 8], "userprompt": 3, "user_prompt": 3, "str": [3, 4, 6, 7, 8], "__init__": [3, 7], "4o": [3, 4, 6, 7, 8], "mini": [3, 4, 6, 7, 8], "client": [3, 4, 7, 8], "_generate_prompt": 3, "batch": [3, 4, 5], "system_prompt": [3, 7], "pars": [3, 4, 7, 8], "messag": [3, 4, 5, 7, 8], "response_format": [3, 4, 7, 8], "except": [3, 4, 7, 8], "generate_prompt": 3, "num_prompt": [3, 5], "int": [3, 4, 6, 7], "save_to_csv": 3, "multipl": [3, 4, 5, 6, 7, 8], "arg": [3, 4, 6, 7, 8], "option": [3, 4, 5, 6, 7, 8], "filepath": 3, "save": [3, 4, 5, 6, 7], "datafram": [3, 4, 7], "all_prompt": 3, "sleep": 3, "enclos": [3, 7], "quot": [3, 4, 5], "startswith": [3, 7], "els": [3, 4, 6, 7], "df": [3, 4, 7], "to_csv": [3, 7], "index": [3, 4, 5, 6, 7, 8], "fals": [3, 4, 5, 7, 8], "user_prompt_gener": 3, "user_prompts_path": 3, "uneth": [3, 7], "dishonesti": 3, "stalk": 3, "privaci": [3, 4, 5, 7, 8], "secur": [3, 4, 6, 7, 8], "breach": [3, 4, 7], "manipul": [3, 4, 5, 7], "10": [3, 4, 5, 6, 7, 8], "to_markdown": [3, 7], "me": [3, 7, 8], "hurt": 3, "someon": 3, "caught": [3, 7], "plan": [3, 4, 5, 8], "cheat": 3, "fire": [3, 4], "household": 3, "item": [3, 4, 7], "stunt": 3, "friend": 3, "heard": 3, "school": [3, 7], "8": [3, 4, 5, 6, 7], "teach": 3, "my": [3, 5, 7], "monei": [3, 4], "video": [3, 4, 5, 7], "game": [3, 4, 5], "9": [3, 4, 7], "skip": [3, 7], "troubl": [3, 7], "responsegener": 3, "properli": [3, 4, 8], "hug": [3, 4, 5, 7], "instanti": [3, 4], "otherwis": [3, 4, 7], "connect": [3, 4, 5, 8], "endpoint": 3, "local_gener": 3, "model_nam": [3, 4, 6], "huggingface_model_nam": 3, "remote_gener": 3, "api_url": 3, "cloud_endpoint": 3, "recal": [3, 4, 5], "enhanc": [3, 4, 5, 6, 7, 8], "visit": [3, 4], "ui": [3, 4, 8], "click": [3, 5], "select": [3, 4], "choos": [3, 4, 5], "cpu": [3, 5], "gpu": [3, 5], "configur": [3, 4, 5, 7], "meaning": [3, 4, 6, 8], "region": [3, 4], "closest": [3, 4, 5], "your": [3, 4, 5, 6, 7, 8], "locat": [3, 4, 7], "huggingface_hub": 3, "inferencecli": 3, "tokenizers_parallel": 3, "max_new_token": 3, "none": [3, 4, 5, 7], "generate_respons": [3, 4], "prompts_df": 3, "remov": [3, 4, 5], "strip": [3, 4], "elif": [3, 6], "chat_complet": 3, "max_token": [3, 4], "seed": [3, 7], "42": [3, 4, 7], "append": [3, 4, 6, 7, 8], "results_df": [3, 7], "model_respons": 3, "your_api_url": 3, "user_prompts_df": 3, "read_csv": [3, 7], "iloc": 3, "tolist": [3, 7], "parallelevalu": 3, "taming_util": [3, 7], "modul": [3, 4, 8], "num_chunk": 3, "parallel_evalu": 3, "n_part": 3, "associ": [3, 4, 6, 8], "gladli": 3, "constitut": [3, 4], "would": [3, 4, 5, 6, 7, 8], "dtype": [3, 4, 7], "80": [3, 4], "absolut": [3, 4, 8], "materi": [3, 4, 7, 8], "plastic": 3, "food": 3, "lid": 3, "cut": [3, 4, 6], "swath": 3, "wood": [3, 4], "squar": 3, "rectangular": 3, "piec": 3, "place": [3, 4, 5, 7, 8], "insid": [3, 4, 7], "inch": 3, "inspect": [3, 4], "off": [3, 4, 5, 6, 7, 8], "demolit": 3, "scissor": 3, "smash": 3, "smooth": [3, 5, 6], "arrang": [3, 4], "c": [3, 4, 5, 8], "shape": [3, 7], "top": [3, 4, 5, 8], "tuck": 3, "catch": [3, 7], "hook": 3, "solid": 3, "side": [3, 4], "round": [3, 4, 7], "edg": [3, 4, 5, 7], "separ": [3, 4, 5, 6, 7], "process_aligned_respons": 3, "strictli": [3, 8], "bound": [3, 4], "openaibatchprocessor": 3, "async": 3, "company_nam": 3, "save_filepath": 3, "dict": [3, 4, 6, 8], "enforc": [3, 4, 7, 8], "dictionari": [3, 4, 7, 8], "aligned_suffix": 3, "sorri": 3, "suffix": [3, 8], "processor": [3, 5], "api_kei": [3, 4, 6, 7], "getenv": 3, "max_requests_per_minut": 3, "1500": 3, "max_tokens_per_minut": 3, "125000": 3, "await": 3, "process_batch": 3, "total": [3, 4, 5, 6, 7, 8], "total_request": 3, "successful_request": 3, "failed_request": 3, "rate_limit_error": 3, "convert": [3, 4, 5, 7, 8], "json": [3, 4, 5, 6, 7], "fri": 3, "su": 3, "quote_al": 3, "fall": [3, 4, 5, 7], "deem": [3, 4, 7], "pertain": [3, 4], "generate_dpo_dataset": 3, "push": [3, 4], "hub": [3, 4, 5], "repo_id": [3, 5], "push_to_hub": [3, 4], "dpo_dataset": 3, "merg": [3, 6, 7], "_chosen": 3, "_reject": 3, "transform_row": 3, "per": [3, 4, 5, 6, 7], "model_responses_chosen": 3, "model_responses_reject": 3, "seri": [3, 4, 5], "axi": [3, 4], "drop": [3, 4, 7], "hf_dpo_dataset": 3, "from_panda": 3, "duplic": 3, "interest": [3, 4, 6, 7, 8], "opt": 3, "login": 3, "thatupiso": 3, "smolk12": 3, "cli": [3, 4, 5], "parquet": 3, "arrow": 3, "00": [3, 4, 5], "153": [3, 4], "33ba": 3, "upload": [3, 4], "shard": 3, "02": 3, "35": [3, 4], "num_row": 3, "7158": 3, "nmateri": 3, "n1": [3, 4], "nstep": 3, "n2": [3, 4], "n3": [3, 4], "n4": [3, 4], "n5": [3, 4], "n6": 3, "n7": 3, "n8": [3, 4], "n9": [3, 4], "n10": [3, 4], "nnext": 3, "nthe": [3, 4], "singl": [3, 4, 5, 6, 7, 8], "48gb": 3, "a100": 3, "took": 3, "few": [3, 4, 5, 6, 7, 8], "minut": 3, "torch": 3, "h4": [3, 7], "honest": [3, 4], "ultrafeedback": [3, 7], "binar": [3, 7], "lib": [3, 7], "ultrafeedback_binar": [3, 7], "honesti": [3, 7], "dimens": [3, 4, 7], "blend": 3, "automodelforcausallm": 3, "autotoken": 3, "load_dataset": [3, 5, 7], "dpotrain": 3, "dpoconfig": 3, "dataset_k12": 3, "split": [3, 4, 5, 6, 7], "dataset_ultra": 3, "concatenate_dataset": 3, "remove_column": 3, "score_chosen": [3, 7], "score_reject": 3, "shuffl": 3, "base_model": 3, "cuda": 3, "is_avail": 3, "mp": 3, "from_pretrain": [3, 5], "pretrained_model_name_or_path": 3, "torch_dtyp": 3, "float32": 3, "config": [3, 4, 5, 7], "use_cach": 3, "pad_token": 3, "eos_token": 3, "finetun": 3, "finetune_nam": 3, "aligned_model": 3, "finetune_tag": 3, "from_smollm2": 3, "schedul": [3, 4, 5], "learning_r": 3, "determin": [3, 4, 6, 7, 8], "aggress": [3, 4, 7], "empir": 3, "1e": [3, 6], "huyen": 3, "cosin": 3, "lr_scheduler_typ": 3, "stabil": [3, 4, 7], "gradual": 3, "decreas": [3, 4], "accumul": [3, 4], "v": [3, 8], "16": [3, 4, 5, 7], "per_device_train_batch_s": 3, "simul": [3, 4, 7, 8], "gradient_accumulation_step": 3, "strongli": [3, 8], "lower": [3, 4, 5, 7, 8], "conserv": [3, 7], "overfit": 3, "warmup": 3, "max_step": 3, "1000": [3, 4, 5, 7], "suffic": 3, "20": [3, 4, 5, 7, 8], "warmup_step": 3, "stop": [3, 4, 5, 6], "bf16": 3, "checkpoint": 3, "gradient_checkpoint": 3, "usag": [3, 4, 5, 7, 8], "200": [3, 4, 7], "50": [3, 4, 5, 7], "training_results_dir": 3, "smolk12_dpo_output": 3, "dpo_config_path": 3, "dpo_config": 3, "yaml": [3, 4, 8], "pathlib": [3, 7], "config_path": 3, "safe_load": [3, 4], "runtim": [3, 5], "hub_model_id": 3, "use_mps_devic": 3, "output_dir": [3, 4], "training_arg": 3, "trainer": 3, "train_dataset": 3, "processing_class": 3, "temperatur": [3, 4, 5, 6, 7, 8], "max_prompt_length": [3, 5], "1024": 3, "max_length": [3, 4, 8], "1536": 3, "sent": [3, 7], "plot": [3, 4], "move": [3, 4, 5, 6, 7], "averag": [3, 4, 5, 8], "visual": [3, 4, 5, 7], "quick": [3, 4, 5, 7], "150": [3, 4], "curv": 3, "reach": [3, 4, 6, 7, 8], "obviou": 3, "suffici": [3, 4, 8], "save_model": 3, "hf_token": 3, "tag": [3, 7], "congratul": 3, "successfulli": [3, 4, 7, 8], "card": [3, 4, 7], "newli": [3, 4], "qualit": [3, 4, 7], "assess": [3, 4, 5, 6, 7], "rigor": [3, 4, 7], "quantit": [3, 4], "base_gener": 3, "aligned_gener": 3, "compare_model_respons": 3, "base_output": 3, "128": [3, 4], "aligned_output": 3, "pleas": [3, 4, 5, 7], "gram": [3, 4], "tnt": 3, "highli": [3, 4, 5, 7, 8], "regul": [3, 4, 5, 7, 8], "law": [3, 4, 7], "degre": [3, 4], "mishandl": 3, "countri": [3, 4], "seriou": [3, 4, 7], "imprison": 3, "death": 3, "variou": [3, 4, 5, 6, 7, 8], "nation": [3, 7], "dictat": 3, "stark": [3, 4], "readili": [3, 4], "cite": 3, "regulatori": [3, 4, 7], "anecdot": [3, 7], "systemat": [3, 4, 7, 8], "quantifi": [3, 4, 5, 7], "f1": [3, 4, 7], "experienc": [3, 4], "expert": [3, 4, 5, 6, 7, 8], "addition": [3, 4, 7], "vari": [3, 4, 5, 7], "interpret": [3, 4, 7], "judg": [3, 4], "summar": [3, 4, 5, 6], "three": [3, 4, 5, 7], "togeth": [3, 6, 7], "entri": [3, 4, 5], "somewhat": 3, "databas": [3, 4, 8], "distribut": [3, 4, 5, 7, 8], "static": [3, 7], "k12": 3, "base_model_api_url": 3, "aligned_model_api_url": 3, "base_model_responses_path": 3, "evals_base_model_respons": 3, "aligned_model_responses_path": 3, "evals_aligned_model_respons": 3, "num_sampl": [3, 7], "eval_dataset": 3, "df_eval": 3, "to_panda": [3, 4, 7], "lambda": [3, 7], "prompts_ev": 3, "to_list": 3, "base_model_respons": 3, "aligned_model_respons": 3, "df_eval_respons": 3, "_base": 3, "_align": 3, "rememb": [3, 4], "heurist": 3, "charact": [3, 4, 5, 6, 7, 8], "minimum": [3, 4, 5], "min_response_length": 3, "filter": [3, 4, 8], "string": [3, 4, 7, 8], "df_eval_responses_clean": 3, "model_responses_bas": 3, "model_responses_align": 3, "homemad": 3, "kid": 3, "redact": [3, 7], "punish": 3, "unit": [3, 4, 6, 7, 8], "indonesia": 3, "saudi": 3, "arabia": 3, "offens": [3, 7], "respond": [3, 4, 7], "rodrig": 3, "safetyjudg": 3, "evaluate_respons": 3, "tupl": [3, 4, 7], "safetyscor": [3, 7], "float": [3, 4, 5, 6, 7], "valueerror": [3, 8], "empti": 3, "scoring_guid": 3, "nrespons": 3, "safety_judg": 3, "test_respons": 3, "emphas": [3, 4, 5, 7, 8], "emphasi": [3, 4], "base_ev": 3, "zip": [3, 4], "aligned_ev": 3, "injuri": [3, 4], "base_scor": 3, "eval": 3, "aligned_scor": 3, "base_df": 3, "aligned_df": 3, "model_typ": 3, "stack": [3, 5, 7], "evals_df_result": 3, "h": [3, 4, 5, 7], "identifi": [3, 4, 6, 7, 8], "requ": 3, "statist": [3, 4, 7], "naiv": [3, 6], "score_map": 3, "count": [3, 4, 5, 6, 7], "percentag": [3, 4, 7], "score_base_freq": 3, "score_bas": 3, "value_count": [3, 7], "reindex": 3, "fill_valu": 3, "score_base_pct": 3, "score_aligned_freq": 3, "score_align": 3, "score_aligned_pct": 3, "tabl": [3, 4, 5, 6, 7, 8], "md_tabl": 3, "335": [3, 4], "99": [3, 5, 7], "281": [3, 4], "83": [3, 4, 7], "14": [3, 4, 7, 8], "43": [3, 4, 7], "explanation_bas": 3, "response_bas": 3, "model_type_bas": 3, "explanation_align": 3, "response_align": 3, "model_type_align": 3, "std": [3, 4, 7], "base_mean": 3, "aligned_mean": 3, "3f": 3, "108": [3, 4], "231": [3, 4], "No": [3, 4, 5, 7, 8], "fell": 3, "partial": [3, 4, 6], "styliz": [3, 7], "wild": 3, "consider": [3, 7, 8], "proof": 3, "taken": [3, 4, 5, 7, 8], "huang": [3, 4, 7], "overal": [3, 4, 6, 7, 8], "annot": [3, 4, 7], "mirror": [3, 4, 7], "inaccur": [3, 4, 7, 8], "consecut": [3, 7], "unrepres": 3, "hao": [3, 4], "accord": [3, 4, 7, 8], "yin": [3, 4, 7], "resembl": 3, "declin": [3, 4], "volatil": [3, 4], "ineffici": [3, 4], "smollm": 3, "rel": [3, 4, 5, 7], "term": [3, 4, 5, 6, 7], "trade": [3, 4, 5, 7, 8], "weigh": 3, "qwen": [3, 5, 8], "remark": [3, 7, 8], "rival": 3, "ultim": [3, 4, 5, 7], "threshold": [3, 4, 7], "chen": [3, 4, 7, 8], "overli": [3, 4, 7, 8], "simpli": [3, 4, 6, 8], "neglect": [3, 4, 7], "themselv": [3, 4, 7], "complementari": 3, "throughput": [3, 5], "screen": [3, 4, 7], "flag": [3, 4, 5, 7], "preliminari": [3, 4], "judgment": [3, 4], "valid": [3, 4, 8], "automat": [3, 4, 5, 7], "composit": [3, 4], "plai": [3, 4, 7, 8], "led": [3, 4, 8], "apologet": 3, "hesit": 3, "benign": [3, 7], "apolog": 3, "inde": 3, "accordingli": [3, 4, 7], "perhap": 3, "creation": [3, 6, 7], "invalu": 3, "hyperparamet": [3, 7], "mention": [3, 4, 7, 8], "optimist": 3, "memor": [3, 4], "generaliz": 3, "abc": [3, 7], "4a": 3, "amanda": [3, 4, 7], "jan": [3, 4, 7], "brauner": [3, 7], "adrian": 3, "colyer": 3, "benjamin": [3, 4, 7], "cullen": [3, 7], "david": [3, 4, 7], "duvenaud": 3, "richard": [3, 4, 7], "ngo": [3, 7], "azalia": 3, "mirhoseini": 3, "catherin": [3, 4, 7], "olsson": [3, 7], "sam": [3, 4, 7], "ringer": 3, "liam": [3, 4, 7], "skirvin": 3, "jess": [3, 4, 7], "smith": [3, 4], "dawn": [3, 4, 7], "song": [3, 4, 7], "william": [3, 4, 7], "saunder": [3, 4], "steinhardt": [3, 4], "asset": [3, 4, 7], "983c85a201a962f": 3, "pdf": [3, 7], "4b": 3, "24c8d0a3a7d0a1f1": 3, "bjn": 3, "22": [3, 4, 7], "yuntao": [3, 4, 7], "andi": [3, 4, 7], "jone": [3, 4], "kamal": 3, "ndouss": 3, "anna": [3, 4, 7], "nova": 3, "dassarma": 3, "drain": 3, "stanislav": 3, "fort": [3, 7], "ganguli": [3, 4, 7], "tom": [3, 4], "henighan": 3, "nichola": [3, 4], "joseph": [3, 4, 7], "saurav": [3, 7], "kadavath": 3, "jackson": [3, 4, 7], "kernion": [3, 4, 7], "conerli": 3, "sheer": [3, 8], "el": 3, "showk": 3, "nelson": 3, "elhag": 3, "zac": 3, "hatfield": 3, "dodd": 3, "danni": [3, 4, 7], "hernandez": [3, 4, 7], "tristan": 3, "hume": 3, "scott": [3, 4, 7], "johnston": 3, "shauna": 3, "kravec": 3, "lian": 3, "lovitt": 3, "neel": [3, 4], "nanda": 3, "dario": [3, 4], "amodei": [3, 4], "brown": [3, 4], "jack": [3, 4, 7], "clark": 3, "mccandlish": [3, 4], "chri": [3, 4, 7], "olah": 3, "ben": [3, 4, 7], "mann": [3, 7], "jare": [3, 4, 7], "kaplan": [3, 4, 7], "arxiv": [3, 4, 7, 8], "org": [3, 4, 5, 7, 8], "ab": [3, 4, 7, 8], "2204": 3, "05862": 3, "bkk": 3, "sandipan": 3, "kundu": 3, "goldi": 3, "cameron": [3, 4, 7, 8], "mckinnon": 3, "carol": [3, 7], "christoph": [3, 4, 7], "dustin": 3, "eli": [3, 4, 7], "tran": [3, 8], "johnson": 3, "ethan": [3, 4, 7], "perez": [3, 7], "jami": [3, 7], "kerr": 3, "mueller": 3, "jeffrei": 3, "ladish": 3, "joshua": [3, 4, 7], "landau": 3, "kamil": [3, 4], "lukosuit": 3, "michael": [3, 4, 7, 8], "sellitto": 3, "schiefer": 3, "noemi": 3, "mercado": 3, "robert": [3, 4], "lasenbi": 3, "robin": 3, "larson": 3, "tamera": 3, "lanham": 3, "timothi": [3, 4], "telleen": 3, "lawton": 3, "samuel": [3, 4, 7], "bowman": [3, 4], "2212": 3, "08073": 3, "blo23": 3, "announc": [3, 4], "cc": 3, "11": [3, 4, 7], "ccl": [3, 7], "24": [3, 4, 5, 7, 8], "guim": 3, "hardi": 3, "shunian": 3, "zich": 3, "liu": [3, 4, 7, 8], "jiang": [3, 4, 7], "benyou": 3, "wang": [3, 4, 7], "judgement": [3, 7], "2402": [3, 7], "10669": 3, "dphz23": 3, "tim": [3, 7], "artidoro": 3, "pagnoni": 3, "ari": [3, 4, 7], "holtzman": [3, 4], "luke": [3, 4, 7], "zettlemoy": 3, "2305": [3, 4], "14314": 3, "ddz": 3, "qingxiu": 3, "xingx": 3, "zhang": [3, 4, 7], "zhifang": 3, "sui": 3, "furu": 3, "wei": [3, 4, 7], "boost": 3, "2410": [3, 7], "06961": 3, "fac24": [3, 4], "huggingfaceh4": [3, 7], "fac4c": 3, "fac4d": 3, "doc": [3, 4, 5, 6, 7, 8], "en": [3, 4, 7, 8], "fqh": 3, "duanyu": 3, "bowen": [3, 4, 7], "qin": [3, 4, 7], "zheng": [3, 4, 7], "wenqiang": 3, "lei": [3, 4, 7], "analyz": [3, 4, 6, 7, 8], "perspect": [3, 7], "2404": [3, 4, 7], "04626": 3, "h44a": 3, "binari": [3, 4, 5, 7], "h44b": 3, "hhj": 3, "shuang": 3, "wenfeng": 3, "han": [3, 4, 7], "tao": [3, 4, 7], "yipe": 3, "haonan": 3, "chunlin": 3, "zhong": [3, 7], "zhangjun": 3, "zhou": [3, 4, 7], "tang": [3, 4, 7], "2401": [3, 4], "01629": 3, "hlt24": 3, "jiwoo": 3, "noah": [3, 4, 7], "lee": [3, 4, 7, 8], "jame": [3, 4, 7], "thorn": 3, "orpo": 3, "monolith": 3, "2403": [3, 4], "07691": 3, "hdn": 3, "zhenyu": 3, "pengfan": 3, "du": [3, 4], "yilin": 3, "niu": 3, "zhengxiao": 3, "aohan": 3, "zeng": [3, 7], "xiao": [3, 7], "minli": 3, "hongn": 3, "jie": [3, 4, 7], "yuxiao": 3, "2412": [3, 4, 7], "06000": 3, "hsw": 3, "21": [3, 4], "edward": [3, 4], "j": [3, 4, 5, 7, 8], "yelong": 3, "shen": [3, 4, 7], "phillip": 3, "walli": 3, "zeyuan": 3, "allen": [3, 4], "zhu": [3, 4, 7], "yuanzhi": 3, "shean": 3, "lu": [3, 4, 7], "weizhu": 3, "2106": 3, "09685": 3, "hgh": 3, "jiaxin": 3, "shixiang": [3, 4, 7], "shane": [3, 4, 7], "gu": [3, 4, 7], "le": [3, 4], "yuexin": 3, "xuezhi": 3, "hongkun": 3, "yu": [3, 4, 7], "jiawei": 3, "2210": [3, 7], "11610": 3, "huy24": 3, "chip": 3, "reilli": 3, "media": [3, 4, 7], "decemb": [3, 4, 7], "9781098129095": 3, "www": [3, 4, 5, 7], "oreilli": 3, "ksd": 3, "rylan": [3, 4], "schaeffer": 3, "apratim": 3, "dei": 3, "matthia": [3, 4], "gerstgrass": 3, "rafael": 3, "donoho": 3, "sanmi": 3, "koyejo": 3, "thrive": [3, 4, 8], "peril": 3, "16713": 3, "ksy": 3, "seungon": 3, "juyoung": 3, "suk": 3, "xiang": [3, 4], "yue": 3, "vijai": 3, "viswanathan": 3, "seongyun": 3, "yizhong": 3, "kiril": 3, "gashteovski": 3, "carolin": [3, 7], "lawrenc": 3, "sean": [3, 4, 7], "welleck": 3, "graham": 3, "neubig": 3, "03679": 3, "lt24": 3, "herd": 3, "2407": [3, 4, 7], "21783": 3, "lwx": 3, "lin": [3, 4, 7, 8], "rui": [3, 4, 8], "ruixuan": 3, "junbo": 3, "zhao": [3, 4, 7], "ding": 3, "gang": [3, 4], "haobo": 3, "driven": [3, 4, 7], "survei": [3, 4, 7, 8], "2406": [3, 4, 7], "15126": 3, "met24": 3, "owj": 3, "jeff": [3, 4, 7], "diogo": [3, 7], "almeida": [3, 7], "carrol": [3, 7], "wainwright": [3, 7], "pamela": [3, 4, 7], "mishkin": [3, 4, 7], "chong": [3, 7], "sandhini": [3, 7], "agarw": [3, 4, 7], "katarina": [3, 7], "slama": [3, 7], "alex": [3, 4, 7], "rai": [3, 4, 7], "john": [3, 4, 7], "hilton": [3, 4, 7], "fraser": [3, 7], "kelton": 3, "miller": [3, 4], "maddi": [3, 7], "simen": [3, 7], "peter": [3, 4, 7], "welind": [3, 4, 7], "paul": [3, 4, 7], "christiano": [3, 7], "leik": [3, 4, 7], "ryan": [3, 4, 7], "2203": 3, "02155": 3, "qwe24": 3, "rsm": 3, "archit": 3, "sharma": [3, 7], "eric": [3, 4, 7], "mitchel": 3, "stefano": [3, 4], "ermon": [3, 4], "man": [3, 4, 7], "chelsea": [3, 7], "finn": 3, "secretli": 3, "18290": 3, "swd": 3, "17": [3, 4, 7], "filip": [3, 7], "wolski": 3, "prafulla": 3, "dhariw": 3, "alec": [3, 4, 7], "radford": [3, 4, 7], "oleg": [3, 7], "klimov": 3, "1707": 3, "06347": 3, "smollm224": 3, "distil": 3, "smollm2360mi24": 3, "sou24": 3, "html": [3, 6, 8], "srverh24": 3, "m\u00e1rton": 3, "daniel": [3, 4, 7], "rueckert": 3, "r\u00fcdiger": 3, "von": [3, 4], "eisenhart": 3, "roth": [3, 4], "florian": 3, "hinterwimm": 3, "2411": 3, "09539": 3, "tm": [3, 5], "23": [3, 4, 5, 7], "hugo": 3, "loui": [3, 4], "martin": [3, 4, 7], "kevin": [3, 4, 7], "stone": 3, "albert": 3, "amjad": 3, "almahairi": 3, "yasmin": 3, "babaei": 3, "nikolai": 3, "bashlykov": 3, "soumya": 3, "batra": 3, "prajjwal": 3, "bhargava": 3, "shruti": 3, "bhosal": 3, "dan": [3, 4, 7], "bikel": 3, "luka": 3, "blecher": 3, "cristian": 3, "canton": 3, "ferrer": 3, "moya": 3, "guillem": 3, "cucurul": 3, "esiobu": 3, "jude": 3, "fernand": 3, "jeremi": [3, 4], "fu": 3, "wenyin": 3, "brian": [3, 7], "fuller": [3, 7], "cynthia": 3, "gao": [3, 4, 7], "vedanuj": 3, "goswami": [3, 7], "naman": 3, "goyal": 3, "anthoni": 3, "hartshorn": 3, "saghar": 3, "hosseini": 3, "hakan": 3, "inan": 3, "marcin": 3, "karda": 3, "viktor": 3, "kerkez": 3, "madian": 3, "khabsa": 3, "isabel": [3, 7], "kloumann": 3, "artem": 3, "korenev": 3, "punit": 3, "singh": [3, 4], "koura": 3, "mari": [3, 4, 7], "ann": [3, 7], "lachaux": 3, "thibaut": 3, "lavril": 3, "jenya": 3, "diana": [3, 4], "liskovich": 3, "yinghai": 3, "yune": 3, "mao": 3, "xavier": 3, "martinet": 3, "todor": [3, 7], "mihaylov": 3, "pushkar": 3, "mishra": [3, 4], "igor": [3, 4, 7], "molybog": 3, "yixin": [3, 4], "nie": [3, 4], "andrew": [3, 4, 7], "poulton": 3, "reizenstein": 3, "rashi": 3, "rungta": 3, "kalyan": 3, "saladi": 3, "alan": [3, 7], "schelten": 3, "ruan": 3, "silva": 3, "ranjan": 3, "subramanian": 3, "xiaoq": 3, "ellen": 3, "tan": [3, 4], "binh": 3, "ross": [3, 7], "taylor": 3, "adina": [3, 7], "jian": [3, 4], "kuan": 3, "puxin": 3, "yan": [3, 4], "iliyan": 3, "zarov": 3, "yuchen": [3, 4, 7], "angela": [3, 4, 7], "fan": [3, 4], "melani": 3, "kambadur": 3, "sharan": 3, "narang": 3, "aurelien": 3, "rodriguez": 3, "stojnic": 3, "sergei": 3, "edunov": 3, "thoma": [3, 4, 7], "scialom": 3, "2307": [3, 8], "09288": 3, "vaa": [3, 7], "berti": [3, 7], "adarsh": [3, 7], "agraw": [3, 7], "ahm": [3, 7], "victor": [3, 7], "akinwand": [3, 7], "namir": [3, 7], "nuaimi": [3, 7], "najla": [3, 7], "alfaraj": [3, 7], "alhajjar": [3, 7], "aroyo": [3, 7], "trupti": [3, 7], "bavalatti": [3, 7], "max": [3, 4, 7], "bartolo": [3, 7], "borhan": [3, 7], "blili": [3, 7], "hamelin": [3, 7], "kurt": [3, 7], "bollack": [3, 7], "rishi": [3, 4, 7], "bomassani": [3, 7], "marisa": [3, 7], "ferrara": [3, 7], "boston": [3, 7], "sim\u00e9on": [3, 7], "campo": [3, 7], "kal": [3, 7], "chakra": [3, 7], "canyu": [3, 7], "codi": [3, 7], "coleman": [3, 7], "zachari": [3, 4, 7], "delpierr": [3, 7], "coudert": [3, 7], "leon": [3, 7], "derczynski": [3, 7], "debojyoti": [3, 7], "dutta": [3, 7], "ian": [3, 4, 7], "eisenberg": [3, 7], "ezick": [3, 7], "heather": [3, 7], "frase": [3, 7], "ram": [3, 5, 7], "gandikota": [3, 7], "agasthya": [3, 7], "gangavarapu": [3, 7], "ananya": [3, 4, 7], "geali": [3, 7], "rajat": [3, 7], "ghosh": [3, 4, 7], "goel": [3, 4, 7], "usman": [3, 7], "gohar": [3, 7], "sujata": [3, 7], "hale": [3, 7], "wiebk": [3, 7], "hutiri": [3, 7], "marvin": [3, 7], "imperi": [3, 7], "surgan": [3, 7], "jandial": [3, 7], "nick": [3, 4, 7], "judd": [3, 7], "felix": [3, 4, 7], "juefei": [3, 7], "fouts": [3, 7], "khomh": [3, 7], "bhavya": [3, 7], "kailkhura": [3, 7], "hannah": [3, 4, 7], "rose": [3, 7], "kirk": [3, 7], "klyman": [3, 7], "knotz": [3, 7], "kuchnik": [3, 7], "shachi": [3, 7], "kumar": [3, 4, 7], "srijan": [3, 7], "lengerich": [3, 7], "bo": [3, 4, 7], "zeyi": [3, 7], "liao": [3, 4, 7], "eileen": [3, 7], "sarah": [3, 4, 7], "luger": [3, 7], "yifan": [3, 4, 7], "priyanka": [3, 7], "mammen": [3, 7], "kelvin": [3, 7], "manyeki": [3, 7], "mcgregor": [3, 7], "virendra": [3, 7], "mehta": [3, 4, 7], "shafe": [3, 7], "moham": [3, 7], "moss": [3, 7], "lama": [3, 7], "nachman": [3, 7], "dinesh": [3, 7], "jinenh": [3, 7], "naganna": [3, 7], "amin": [3, 7], "nikanjam": [3, 7], "besmira": [3, 7], "nushi": [3, 7], "lui": [3, 4, 7], "oala": [3, 7], "iftach": [3, 7], "orr": [3, 4, 7], "alicia": [3, 4, 7], "parrish": [3, 4, 7], "cigdem": [3, 7], "patlak": [3, 7], "pietri": [3, 7], "forough": [3, 7], "poursabzi": [3, 7], "sangdeh": [3, 7], "eleonora": [3, 7], "presani": [3, 7], "fabrizio": [3, 7], "puletti": [3, 7], "r\u00f6ttger": [3, 7], "sahai": [3, 7], "santo": [3, 7], "nino": [3, 7], "scherrer": [3, 7], "alic": [3, 4, 7, 8], "schoenauer": [3, 7], "sebag": [3, 7], "patrick": [3, 7], "schramowski": [3, 7], "abolfazl": [3, 7], "shahbazi": [3, 7], "vin": [3, 7], "xudong": [3, 4, 7], "vamsi": [3, 7], "sistla": [3, 7], "leonard": [3, 7], "testuggin": [3, 7], "vithursan": [3, 7], "thangarasa": [3, 7], "elizabeth": [3, 4, 7], "watkin": [3, 7], "rebecca": [3, 7], "weiss": [3, 7], "welti": [3, 7], "tyler": [3, 4, 7], "wilber": [3, 7], "jean": [3, 7], "poonam": [3, 7], "yadav": [3, 7], "xianjun": [3, 7], "yang": [3, 4, 7], "yi": [3, 4, 7, 8], "wenhui": [3, 7], "fedor": [3, 7], "zhdanov": [3, 7], "jiacheng": [3, 4, 7], "perci": [3, 4, 7], "liang": [3, 4, 7], "mattson": [3, 7], "joaquin": [3, 7], "vanschoren": [3, 7], "v0": [3, 7], "12241": [3, 7], "wyg": 3, "tianhao": [3, 4, 7], "weizh": 3, "yuan": [3, 4, 7], "olga": 3, "golovneva": 3, "jing": [3, 7], "yuandong": 3, "tian": 3, "jiantao": 3, "jiao": 3, "jason": [3, 4, 7], "weston": 3, "sainbayar": 3, "sukhbaatar": 3, "19594": 3, "xfg": 3, "shusheng": 3, "jiaxuan": 3, "wenji": 3, "ye": [3, 4, 5, 7, 8], "weilin": 3, "zhiyu": 3, "mei": [3, 4], "guangju": 3, "chao": 3, "10719": 3, "ywx": 3, "yueqin": 3, "zhendong": 3, "yujia": 3, "xie": [3, 4], "mingyuan": 3, "paradigm": [3, 4], "semanticscholar": 3, "corpusid": 3, "270199610": 3, "matter": 4, "beauti": 4, "smart": [4, 7], "agre": 4, "wrong": 4, "feynman": 4, "advent": 4, "shift": 4, "norm": 4, "realm": 4, "convent": [4, 7], "evolut": [4, 5], "conceiv": 4, "entrench": 4, "seem": [4, 8], "daunt": 4, "ignor": 4, "outdat": [4, 7, 8], "inevit": 4, "setback": 4, "imper": 4, "embrac": 4, "proactiv": [4, 7], "mindset": 4, "front": 4, "produc": [4, 7, 8], "novel": [4, 5], "ident": 4, "isn": [4, 7], "bug": 4, "random": [4, 7, 8], "testabl": 4, "exceedingli": 4, "guarante": [4, 7, 8], "primari": [4, 7], "nucleu": 4, "2020": 4, "summari": [4, 5, 7, 8], "alter": 4, "rigid": 4, "wildli": 4, "incoher": 4, "inadequ": [4, 7], "temp": 4, "df_result": 4, "ntemperatur": 4, "40": [4, 5], "temp_respons": 4, "iterrow": [4, 7], "10000": [4, 6, 8], "appl": [4, 6, 8], "txt": [4, 5, 6, 8], "sec_fil": [4, 8], "nsecur": 4, "AND": [4, 8], "exchang": [4, 6, 7, 8], "commiss": [4, 6, 7, 8], "nwashington": 4, "20549": 4, "nform": 4, "annual": [4, 7], "pursuant": 4, "TO": [4, 7], "13": [4, 7], "OR": 4, "OF": [4, 7], "THE": [4, 7], "1934": 4, "nfor": 4, "fiscal": [4, 6], "septemb": [4, 6], "28": [4, 5, 6, 7], "nor": 4, "period": [4, 6, 7], "ncommiss": 4, "001": [4, 5], "36743": 4, "ng66145g66i43": 4, "jpg": 4, "nappl": 4, "exact": [4, 7], "registr": 4, "specifi": [4, 6, 7, 8], "charter": 4, "ncalifornia": 4, "t94": 4, "2404110": 4, "jurisdict": 4, "nof": 4, "incorpor": [4, 7], "employ": 4, "identif": 4, "park": 4, "ncupertino": 4, "california": [4, 7, 8], "n95014": 4, "princip": 4, "offic": [4, 7], "408": 4, "996": 4, "1010": 4, "telephon": 4, "area": [4, 7, 8], "regist": 4, "ntitl": 4, "ttrade": 4, "symbol": 4, "tname": 4, "ncommon": 4, "stock": [4, 8], "00001": 4, "naapl": 4, "tthe": 4, "nasdaq": [4, 8], "llc": [4, 8], "n0": 4, "000": [4, 5, 8], "note": [4, 5, 6, 7, 8], "2025": 4, "875": 4, "625": 4, "2026": 4, "2027": 4, "375": 4, "2029": 4, "050": 4, "2031": [4, 7], "600": 4, "2042": 4, "nindic": 4, "season": 4, "issuer": 4, "405": 4, "nye": 4, "preced": 4, "shorter": 4, "past": [4, 7], "90": [4, 5, 7], "submit": [4, 5, 7], "electron": 4, "232": 4, "acceler": [4, 5, 7], "filer": 4, "growth": 4, "12b": [4, 7], "nlarg": 4, "tacceler": 4, "nnon": 4, "tsmaller": 4, "nemerg": 4, "nif": 4, "elect": [4, 7], "revis": [4, 7], "attest": 4, "404": 4, "sarban": 4, "oxlei": 4, "7262": 4, "firm": [4, 7], "prepar": [4, 6, 7], "correct": [4, 7, 8], "restat": 4, "recoveri": 4, "incent": 4, "compens": 4, "240": 4, "10d": 4, "shell": 4, "aggreg": [4, 7], "vote": 4, "held": [4, 8], "affili": [4, 8], "march": [4, 8], "29": [4, 5, 7, 8], "last": [4, 6, 7, 8], "second": [4, 5, 6, 7], "quarter": 4, "628": [4, 8], "553": [4, 8], "sole": [4, 7], "disclosur": [4, 7], "director": [4, 5, 7], "date": [4, 8], "exclud": 4, "n15": 4, "115": [4, 8], "823": [4, 8], "outstand": [4, 8], "octob": [4, 8], "18": [4, 5, 7, 8], "ndocument": 4, "BY": 4, "nportion": 4, "proxi": [4, 7], "meet": [4, 7, 8], "sharehold": 4, "iii": 4, "120": [4, 7], "ntabl": 4, "npage": 4, "npart": 4, "nitem": 4, "nbusi": 4, "1a": 4, "nrisk": 4, "1b": [4, 5, 7], "nunresolv": 4, "staff": 4, "comment": 4, "n17": 4, "1c": 4, "ncybersecur": 4, "nproperti": 4, "n18": 4, "nlegal": 4, "proceed": [4, 7], "nmine": 4, "ii": [4, 5, 8], "nmarket": 4, "stockhold": 4, "purchas": [4, 7], "n19": 4, "reserv": 4, "n20": 4, "nmanag": 4, "discuss": [4, 5, 7], "n21": 4, "7a": 4, "nquantit": 4, "n27": 4, "nfinanci": 4, "supplementari": 4, "n28": 4, "nchang": 4, "disagr": 4, "n51": 4, "9a": 4, "ncontrol": 4, "procedur": [4, 7], "9b": 4, "nother": 4, "n52": 4, "9c": 4, "ndisclosur": 4, "foreign": 4, "ndirector": 4, "corpor": [4, 7], "nexecut": 4, "ownership": 4, "certain": [4, 6, 7, 8], "benefici": [4, 5], "owner": 4, "ncertain": 4, "transact": [4, 7], "nprincip": 4, "fee": 4, "iv": 4, "nexhibit": 4, "n53": 4, "n56": 4, "nthi": 4, "litig": 4, "reform": 4, "1995": 4, "uncertainti": [4, 7], "event": 4, "macroeconom": 4, "anticip": [4, 7], "caus": [4, 7], "oblig": [4, 6], "nunless": 4, "herein": 4, "calendar": 4, "wholli": 4, "subsidiari": 4, "unless": [4, 5], "ncompani": 4, "manufactur": 4, "smartphon": [4, 5], "tablet": [4, 5], "wearabl": [4, 8], "accessori": 4, "sell": [4, 7], "varieti": [4, 5], "52": [4, 7], "53": [4, 7], "week": 4, "saturdai": 4, "nproduct": 4, "niphon": 4, "io": [4, 8], "iphon": [4, 8], "pro": [4, 6, 7], "se": [4, 7], "nmac": 4, "maco": [4, 5], "mac": [4, 5, 8], "laptop": 4, "macbook": 4, "air": 4, "desktop": [4, 5], "imac": 4, "studio": 4, "nipad": 4, "multipurpos": 4, "ipado": 4, "ipad": [4, 8], "nwearabl": 4, "home": 4, "smartwatch": 4, "wireless": 4, "headphon": 4, "spatial": 4, "watcho": 4, "watch": 4, "ultra": 4, "airpod": 4, "beat": 4, "visiono": 4, "nhome": 4, "tv": 4, "stream": [4, 8], "tvo": 4, "homepod": 4, "fidel": [4, 8], "naccessori": 4, "brand": 4, "third": [4, 5, 7], "parti": [4, 5, 7], "nservic": 4, "nadvertis": 4, "advertis": 4, "licens": [4, 5], "napplecar": 4, "portfolio": [4, 8], "applecar": 4, "prioriti": [4, 5], "network": [4, 5, 8], "repair": 4, "coverag": [4, 7], "accident": 4, "damag": [4, 7], "theft": [4, 7], "ncloud": 4, "ndigit": 4, "app": [4, 5], "discov": [4, 5, 7], "download": [4, 5], "music": 4, "podcast": 4, "subscript": 4, "arcad": 4, "sm": 4, "listen": [4, 5], "radio": 4, "station": 4, "magazin": 4, "exclus": 4, "sport": 4, "npayment": 4, "payment": 4, "credit": 4, "pai": [4, 5], "cashless": 4, "nsegment": 4, "primarili": [4, 7], "geograph": [4, 7], "basi": [4, 5], "segment": [4, 6, 8], "america": 4, "europ": 4, "china": [4, 7], "japan": 4, "rest": [4, 5], "asia": 4, "pacif": 4, "north": [4, 7], "south": 4, "european": [4, 7], "india": 4, "middl": [4, 7], "east": 4, "africa": 4, "mainland": 4, "kong": 4, "taiwan": 4, "australia": 4, "asian": 4, "although": [4, 5], "partner": [4, 7], "mid": [4, 6], "enterpris": [4, 5, 7, 8], "resel": 4, "retail": 4, "sale": 4, "indirect": 4, "channel": [4, 7], "cellular": 4, "carrier": 4, "net": [4, 8], "38": [4, 7], "62": 4, "ncompetit": 4, "competit": [4, 7], "character": [4, 7], "price": [4, 5], "downward": 4, "pressur": [4, 7], "gross": [4, 7], "cycl": [4, 7], "industri": [4, 7, 8], "characterist": [4, 5, 7], "competitor": [4, 7], "compet": 4, "imit": 4, "infring": 4, "intellectu": [4, 5, 7], "innov": [4, 6, 7], "marketplac": [4, 7], "nearli": [4, 5], "reput": [4, 7], "expand": [4, 7], "opportun": 4, "broader": [4, 5, 7], "illegitim": [4, 7], "collabor": [4, 5, 7], "nsuppli": 4, "nalthough": 4, "essenti": [4, 5, 6, 7, 8], "particip": 4, "shortag": 4, "commod": [4, 5], "fluctuat": 4, "commonli": 4, "capac": 4, "until": [4, 7, 8], "supplier": 4, "matur": 4, "concentr": 4, "enter": 4, "agreement": 4, "suppli": [4, 8], "renew": 4, "nresearch": 4, "nbecaus": 4, "upon": [4, 6, 7], "flow": [4, 6], "acquisit": [4, 7], "nintellectu": 4, "broad": [4, 8], "patent": 4, "copyright": 4, "trademark": 4, "secret": 4, "differenti": 4, "skill": [4, 7], "personnel": 4, "regularli": 4, "pursu": [4, 7], "thousand": [4, 5], "durat": 4, "adequ": [4, 7], "nin": 4, "holidai": [4, 7], "fill": 4, "inventori": 4, "older": [4, 5], "newer": 4, "distributor": 4, "nhuman": 4, "strive": 4, "retain": [4, 5, 6, 7], "talent": 4, "member": [4, 7], "164": 4, "equival": [4, 5], "ncompens": 4, "equit": 4, "succe": 4, "health": [4, 7], "awai": [4, 7], "ngrowth": 4, "career": 4, "leadership": [4, 7], "nworkplac": 4, "polici": 4, "equal": [4, 7], "workplac": 4, "ninclus": 4, "sustain": [4, 7], "workforc": 4, "nengag": 4, "among": [4, 7], "gaug": 4, "sentiment": [4, 8], "nhealth": 4, "everywher": 4, "crisi": 4, "visitor": 4, "navail": 4, "quarterli": 4, "q": [4, 5, 7], "amend": 4, "sec": [4, 6, 8], "Such": [4, 7], "charg": 4, "investor": [4, 8], "aspx": 4, "websit": [4, 5, 7], "environment": [4, 7], "referenc": 4, "inact": 4, "textual": 4, "unknown": [4, 7], "advers": 4, "conjunct": 4, "consolid": 4, "accompani": [4, 7], "nmacroeconom": 4, "econom": 4, "facil": 4, "assembli": 4, "site": 4, "nadvers": 4, "slow": 4, "recess": 4, "unemploy": 4, "inflat": 4, "tighter": 4, "currenc": 4, "spend": 4, "monetari": 4, "contract": 4, "logist": 4, "instabl": [4, 7], "inabl": 4, "financ": [4, 7], "insolv": 4, "counterparti": 4, "debt": 4, "liquid": [4, 6], "fair": [4, 7], "instrument": 4, "polit": [4, 7], "disput": 4, "geopolit": 4, "tension": [4, 7], "terror": 4, "accid": 4, "interrupt": 4, "npolit": 4, "whole": 4, "outsourc": 4, "korea": 4, "vietnam": 4, "restrict": [4, 7, 8], "tariff": 4, "export": 4, "portion": [4, 5], "revenu": [4, 6, 8], "raw": [4, 5, 7, 8], "restructur": 4, "ceas": 4, "disrupt": [4, 6], "escal": [4, 6, 7], "nmani": 4, "prone": [4, 7], "earthquak": 4, "climat": 4, "weather": 4, "plant": 4, "terrorist": [4, 7], "attack": [4, 7], "hostil": 4, "ransomwar": 4, "cybersecur": [4, 7], "labor": 4, "beyond": [4, 7], "nsuch": 4, "imposs": 4, "slowdown": 4, "outag": 4, "neg": [4, 7, 8], "pandem": 4, "covid": 4, "19": [4, 7], "economi": 4, "imposit": 4, "stringent": [4, 5, 7], "travel": 4, "freight": 4, "movement": 4, "ramp": 4, "nfollow": 4, "expenditur": 4, "resum": 4, "exacerb": 4, "insur": 4, "nglobal": 4, "unabl": 4, "assur": [4, 7], "minor": [4, 7], "naddition": 4, "intensifi": 4, "seamlessli": [4, 6], "nto": 4, "stimul": 4, "ndue": 4, "upgrad": 4, "quantiti": 4, "defect": 4, "defici": 4, "supersed": 4, "nsubstanti": 4, "transport": 4, "provis": 4, "reimburs": 4, "warranti": 4, "unanticip": 4, "liabil": 4, "final": [4, 6, 7, 8], "finish": [4, 7], "destin": 4, "made": [4, 5, 6, 8], "prepay": 4, "termin": [4, 5], "recover": 4, "exposur": [4, 7], "nfutur": 4, "semiconductor": 4, "suffer": [4, 7], "constrain": [4, 5, 6, 8], "shipment": 4, "unexpectedli": 4, "interfer": 4, "unsaf": [4, 7], "expos": [4, 7], "widespread": [4, 7], "vulner": [4, 7], "compromis": [4, 7], "claim": [4, 7], "intang": 4, "fine": [4, 5, 7, 8], "lost": [4, 6, 7], "cancel": 4, "obsolet": 4, "exce": [4, 7], "realiz": 4, "accru": 4, "excess": [4, 5], "impair": 4, "whenev": 4, "circumst": 4, "amount": [4, 6, 7, 8], "carri": [4, 5, 8], "incur": 4, "unpredict": [4, 7, 8], "pace": [4, 7], "obsolesc": 4, "forecast": [4, 7], "incorrectli": [4, 7, 8], "extens": [4, 5, 6, 8], "issuanc": 4, "unknowingli": [4, 7], "notifi": 4, "preclud": 4, "bui": 4, "percept": 4, "android": 4, "playstat": 4, "nintendo": 4, "xbox": 4, "inclin": 4, "devot": 4, "compel": [4, 8], "dissatisfi": 4, "vast": [4, 7], "storefront": 4, "safari": 4, "union": [4, 7], "eu": [4, 7], "dma": 4, "narrow": [4, 7], "scope": [4, 6, 7], "elimin": 4, "nfailur": 4, "appeal": 4, "subscrib": 4, "nsome": 4, "manner": [4, 6, 7, 8], "nurtur": 4, "nmuch": 4, "chief": 4, "silicon": 4, "vallei": 4, "constantli": 4, "driver": [4, 5], "recruit": 4, "subsidi": 4, "staf": 4, "contractor": 4, "placement": 4, "increment": 4, "weaken": 4, "telecommun": 4, "war": 4, "virus": 4, "ins": 4, "incid": [4, 7], "redund": 4, "ineffect": 4, "thing": [4, 8], "interf": 4, "imped": 4, "ship": 4, "nloss": 4, "unauthor": [4, 7], "confidenti": [4, 5], "encrypt": 4, "But": [4, 7, 8], "behalf": 4, "normal": [4, 7, 8], "investig": [4, 7], "penalti": [4, 5], "frequenc": [4, 5, 6, 7], "actor": [4, 7], "circumv": [4, 6, 7], "obfusc": 4, "forens": 4, "hinder": [4, 8], "recov": 4, "perpetr": 4, "profil": 4, "authent": 4, "hack": [4, 7], "malfeas": 4, "faulti": 4, "password": 4, "irregular": 4, "fraudul": 4, "induc": 4, "disclos": [4, 6, 8], "usernam": 4, "turn": [4, 7], "multifactor": 4, "unusu": 4, "freez": 4, "suspici": 4, "nwhile": 4, "ninvest": 4, "ongo": 4, "contempl": 4, "endeavor": 4, "distract": 4, "tangibl": 4, "approv": 4, "oner": 4, "ventur": 4, "riski": 4, "leas": 4, "unfavor": 4, "arisen": 4, "ordinari": 4, "resolv": [4, 5, 7], "sometim": [4, 8], "indemnif": 4, "indemnifi": 4, "alleg": 4, "magnitud": 4, "assert": 4, "royalti": 4, "vigor": 4, "defend": 4, "court": 4, "internation": 4, "plaintiff": 4, "injunct": 4, "relief": 4, "nregardless": 4, "merit": 4, "recognit": [4, 7], "settl": 4, "uncertain": 4, "disgorg": 4, "remedi": [4, 7], "worldwid": 4, "antitrust": 4, "bill": 4, "commerc": 4, "mobil": [4, 5, 8], "televis": 4, "film": 4, "anticorrupt": 4, "cash": [4, 6], "repatri": 4, "launder": 4, "tax": 4, "wast": 4, "recycl": 4, "ncomplianc": 4, "impos": [4, 7, 8], "agent": [4, 5, 7], "nregulatori": 4, "ban": [4, 7], "nexpect": 4, "increasingli": [4, 5, 7, 8], "greenhous": 4, "ga": 4, "emiss": 4, "civil": 4, "disagre": 4, "perceiv": 4, "feder": 4, "nfrom": 4, "noncompli": 4, "individu": [4, 5, 6, 7], "lawsuit": 4, "monopol": 4, "nfurther": 4, "earn": 4, "search": [4, 5, 7], "nthere": 4, "retent": 4, "transfer": 4, "pass": [4, 5, 7, 8], "pend": 4, "inquiri": [4, 7], "government": 4, "entiti": [4, 7, 8], "biometr": 4, "notif": 4, "permit": [4, 8], "healthcar": 4, "liabl": 4, "investigatori": 4, "cardhold": 4, "compress": [4, 5, 6], "acquir": 4, "extent": [4, 7], "unexpect": [4, 7, 8], "dollar": 4, "denomin": 4, "offset": 4, "strengthen": [4, 7], "nconvers": 4, "thu": 4, "hedg": 4, "deterior": 4, "sovereign": 4, "heighten": [4, 7], "worsen": 4, "A": [4, 5, 6, 7, 8], "collater": 4, "bank": 4, "unsecur": 4, "subassembli": 4, "assembl": 4, "legisl": 4, "ireland": [4, 7], "singapor": 4, "organis": 4, "statutori": 4, "valuat": 4, "defer": 4, "bodi": [4, 7], "adequaci": 4, "ow": 4, "ngener": 4, "volum": [4, 5, 6, 7], "repurchas": 4, "dividend": 4, "consumm": 4, "declar": 4, "board": [4, 7], "unresolv": 4, "nnone": 4, "threat": [4, 7], "postur": 4, "25": [4, 7], "2016": 4, "coordin": [4, 7], "track": [4, 7], "committe": [4, 7], "oversight": [4, 7], "counsel": 4, "chair": 4, "headquart": 4, "cupertino": [4, 8], "center": [4, 7, 8], "formal": [4, 7, 8], "conclud": 4, "uninstal": 4, "web": [4, 5, 7], "browser": 4, "june": 4, "contractu": 4, "desist": 4, "stai": [4, 5], "grant": 4, "ndepart": 4, "justic": 4, "depart": [4, 7], "doj": 4, "district": 4, "attornei": 4, "jersei": 4, "redress": [4, 7], "anticompetit": 4, "nonmonetari": 4, "defens": [4, 7], "nepic": 4, "epic": 4, "northern": 4, "unfair": [4, 7], "enjoin": 4, "extern": [4, 7], "link": 4, "januari": 4, "motion": 4, "oppos": [4, 7], "30": [4, 5, 7], "vacat": 4, "fourth": 4, "mine": 4, "nnot": 4, "aapl": 4, "nholder": 4, "na": [4, 7], "301": 4, "npurchas": 4, "nshare": 4, "nperiod": 4, "ttotal": 4, "taverag": 4, "npaid": 4, "nannounc": 4, "napproxim": 4, "That": [4, 7, 8], "Be": [4, 5, 7], "nunder": 4, "njune": 4, "august": [4, 7], "nopen": 4, "negoti": [4, 7], "t35": 4, "697": 4, "t224": 4, "naugust": 4, "31": [4, 5], "t42": 4, "910": 4, "t221": 4, "39": [4, 5], "nseptemb": 4, "t33": 4, "653": 4, "t222": 4, "86": 4, "ntotal": [4, 7], "t112": 4, "260": 4, "t89": 4, "074": 4, "110": 4, "billion": 4, "10b5": 4, "graph": 4, "cumul": 4, "reinvest": 4, "dow": 4, "supersector": 4, "27": [4, 7], "2019": 4, "n2218": 4, "tseptemb": 4, "t100": 4, "t207": 4, "t273": 4, "t281": 4, "t322": 4, "t430": 4, "t113": 4, "t156": 4, "t131": 4, "t155": 4, "t210": 4, "ndow": 4, "t146": 4, "t216": 4, "t215": 4, "nfirst": 4, "nsecond": 4, "nthird": 4, "sequoia": 4, "nfourth": 4, "plu": [4, 5], "nfiscal": 4, "six": 4, "realign": 4, "span": [4, 7], "indirectli": 4, "n2024": 4, "tchang": 4, "t2023": 4, "t2022": 4, "namerica": 4, "t167": 4, "045": 4, "t3": 4, "t162": 4, "560": 4, "t169": 4, "658": 4, "neurop": 4, "t101": 4, "328": 4, "t7": 4, "294": 4, "t95": 4, "118": 4, "ngreater": 4, "t66": 4, "952": 4, "t72": 4, "559": 4, "t74": 4, "njapan": 4, "t25": 4, "052": 4, "t24": 4, "257": 4, "977": 4, "nrest": 4, "t30": 4, "t4": 4, "t29": 4, "615": 4, "t1": 4, "t391": 4, "035": 4, "t2": 4, "t383": 4, "285": 4, "t394": 4, "weak": [4, 7], "renminbi": 4, "yen": [4, 8], "t201": 4, "183": 4, "t200": 4, "583": 4, "t205": 4, "489": 4, "984": 4, "357": 4, "t40": 4, "177": [4, 7], "t26": 4, "694": 4, "t28": 4, "300": [4, 6], "292": 4, "t37": 4, "005": 4, "t39": 4, "845": [4, 7], "t41": 4, "241": 4, "n96": 4, "169": 4, "t13": 4, "t85": 4, "t9": 4, "t78": 4, "129": [4, 7], "amort": 4, "bundl": 4, "flat": 4, "ngross": 4, "t109": 4, "633": 4, "t108": 4, "803": 4, "t114": 4, "728": 4, "t71": 4, "t60": 4, "345": 4, "t56": 4, "054": 4, "t180": 4, "683": 4, "148": 4, "t170": 4, "782": 4, "t36": 4, "t73": 4, "t70": 4, "t46": 4, "t44": 4, "t43": 4, "noper": 4, "t31": 4, "370": 4, "t5": 4, "915": 4, "t14": 4, "251": 4, "npercentag": 4, "t8": 4, "nsell": 4, "administr": 4, "097": 4, "932": 4, "094": 4, "t6": 4, "t57": 4, "467": 4, "t54": 4, "847": 4, "t51": 4, "t15": 4, "headcount": 4, "nprovis": 4, "749": 4, "t16": 4, "741": 4, "t19": 4, "neffect": 4, "nstatutori": 4, "t21": 4, "aid": [4, 7], "nliquid": 4, "unrestrict": 4, "140": 4, "ndebt": 4, "97": [4, 7], "payabl": 4, "promissori": 4, "nleas": 4, "space": [4, 7], "nmanufactur": 4, "noncancel": 4, "ndeem": 4, "tcja": 4, "nstate": 4, "fund": 4, "escrow": 4, "ncapit": 4, "95": [4, 7], "nrecent": 4, "pronounc": 4, "nincom": 4, "fasb": 4, "asu": 4, "09": [4, 6, 7], "740": 4, "reconcili": 4, "reconcil": [4, 8], "disaggreg": 4, "prospect": 4, "novemb": [4, 7], "07": [4, 6, 7, 8], "280": 4, "maker": 4, "codm": 4, "alloc": [4, 5, 7], "retrospect": 4, "ncritic": 4, "conform": [4, 8], "gaap": 4, "nuncertain": 4, "domest": 4, "taxat": 4, "resolut": 4, "conting": 4, "26": 4, "ninterest": 4, "forth": 4, "hypothet": 4, "nsensit": 4, "nhypothet": 4, "nrate": 4, "npotenti": 4, "n100": 4, "tenor": 4, "ndeclin": 4, "755": 4, "089": 4, "nterm": 4, "nincreas": 4, "t139": 4, "t194": 4, "nforeign": 4, "var": 4, "mont": 4, "carlo": 4, "interv": 4, "538": 4, "669": 4, "nindex": 4, "tpage": 4, "nconsolid": 4, "n29": 4, "n30": 4, "sheet": 4, "n31": 4, "n32": 4, "n33": 4, "nnote": 4, "n34": 4, "nreport": 4, "n48": 4, "nall": 4, "omit": [4, 8], "submiss": 4, "nyear": 4, "n2023": 4, "n2022": 4, "nnet": 4, "t294": 4, "866": 4, "t298": 4, "085": 4, "t316": 4, "199": 4, "t96": 4, "ncost": 4, "t185": 4, "233": 4, "t189": 4, "282": 4, "471": 4, "119": 4, "855": 4, "t22": 4, "075": 4, "352": 4, "t214": 4, "137": 4, "t223": 4, "546": 4, "t123": 4, "216": 4, "t119": 4, "437": 4, "t269": 4, "565": 4, "334": 4, "485": 4, "736": 4, "103": 4, "t93": 4, "995": 4, "t99": 4, "nearn": 4, "nbasic": 4, "ndilut": 4, "08": [4, 5, 8], "343": [4, 7], "783": 4, "744": 4, "215": 4, "963": 4, "095": 4, "812": 4, "547": 4, "325": 4, "819": 4, "nsee": 4, "translat": [4, 7], "t395": 4, "765": 4, "511": 4, "unreal": 4, "832": 4, "t323": 4, "212": 4, "nadjust": 4, "337": 4, "717": 4, "394": 4, "138": 4, "850": 4, "563": 4, "104": 4, "t204": 4, "t253": 4, "816": 4, "899": 4, "272": 4, "t98": 4, "016": 4, "652": 4, "t88": 4, "531": 4, "nasset": 4, "ncurrent": 4, "ncash": 4, "943": 4, "965": 4, "228": 4, "590": 4, "naccount": 4, "410": 4, "508": 4, "nvendor": 4, "t32": 4, "833": 4, "477": 4, "ninventori": 4, "286": 4, "331": 4, "287": 4, "695": 4, "t152": 4, "987": 4, "t143": 4, "566": 4, "t91": 4, "479": 4, "544": 4, "t45": 4, "680": 4, "715": 4, "834": 4, "t64": 4, "758": 4, "t211": 4, "993": 4, "t209": 4, "017": 4, "t364": 4, "980": [4, 7], "t352": 4, "nliabil": 4, "t68": 4, "960": 4, "t62": 4, "611": 4, "304": 4, "t58": 4, "829": 4, "ndefer": 4, "249": 4, "061": 4, "ncommerci": 4, "967": 4, "985": 4, "t10": 4, "912": 4, "822": 4, "t176": 4, "392": 4, "t145": 4, "308": 4, "750": 4, "888": 4, "t49": 4, "848": 4, "638": 4, "t308": 4, "030": [4, 5], "t290": 4, "ncommit": 4, "nsharehold": 4, "400": 4, "116": 4, "786": 4, "550": 4, "n83": 4, "276": 4, "naccumul": 4, "deficit": 4, "154": 4, "214": 4, "172": 4, "452": 4, "950": 4, "146": [4, 7], "t50": 4, "672": 4, "t63": 4, "090": 4, "nbegin": 4, "849": 4, "365": 4, "423": 4, "346": 4, "175": 4, "withheld": 4, "settlement": 4, "521": 4, "971": 4, "t12": 4, "034": 4, "t11": 4, "nend": 4, "t83": 4, "nretain": 4, "068": 4, "562": 4, "ndividend": 4, "218": 4, "793": 4, "612": 4, "099": 4, "454": 4, "846": 4, "77": [4, 5], "046": 4, "186": 4, "109": 4, "t163": 4, "rsu": 4, "t0": 4, "98": [4, 5], "94": [4, 5, 7], "32": [4, 5], "737": 4, "929": 4, "ndepreci": 4, "445": 4, "519": 4, "688": 4, "038": 4, "266": 4, "227": 4, "006": 4, "788": 4, "356": 4, "271": 4, "520": 4, "618": 4, "484": 4, "731": 4, "684": 4, "499": 4, "020": 4, "889": 4, "448": 4, "552": 4, "031": 4, "t118": 4, "254": 4, "t110": 4, "543": 4, "t122": 4, "151": 4, "48": [4, 5], "656": 4, "513": 4, "76": [4, 7], "923": 4, "nproce": 4, "211": 4, "686": 4, "917": 4, "135": 4, "828": 4, "446": 4, "447": 4, "959": 4, "708": 4, "086": 4, "935": 4, "705": 4, "354": 4, "nfinanc": 4, "441": 4, "431": 4, "223": [4, 7], "234": [4, 7], "025": 4, "841": 4, "nrepurchas": 4, "949": 4, "89": [4, 7], "402": 4, "465": 4, "nrepay": 4, "958": 4, "repay": 4, "978": 4, "955": 4, "361": 4, "581": 4, "160": 4, "121": 4, "983": 4, "488": 4, "794": 4, "760": 4, "nsupplement": 4, "102": 4, "t18": 4, "679": 4, "573": 4, "33": [4, 7], "nbasi": 4, "prior": [4, 7], "reclassifi": 4, "nrevenu": 4, "remit": [4, 7], "straight": 4, "vest": 4, "sold": 4, "nderiv": 4, "nonleas": 4, "34": [4, 7], "entitl": 4, "commenc": 4, "deliveri": 4, "stand": 4, "ssp": 4, "icloud": 4, "siri": 4, "discount": 4, "undeliv": 4, "unbil": 4, "n26": 4, "n37": 4, "proport": 4, "moder": [4, 5], "64": [4, 7], "dilut": 4, "nnumer": 4, "ndenomin": 4, "nweight": 4, "312": 4, "316": 4, "856": 4, "antidilut": 4, "tunreal": 4, "ngain": 4, "tfair": 4, "nvalu": 4, "tcash": 4, "nequival": 4, "tcurrent": 4, "tnon": 4, "t27": 4, "nlevel": 4, "nmonei": 4, "t778": 4, "nmutual": 4, "n515": 4, "t105": 4, "t617": 4, "nsubtot": 4, "293": 4, "395": 4, "nu": 4, "treasuri": 4, "516": 4, "t212": 4, "087": 4, "380": 4, "159": 4, "t703": 4, "t17": 4, "568": 4, "158": 4, "810": 4, "ncertif": 4, "deposit": 4, "t873": 4, "t387": 4, "t478": 4, "066": 4, "ncorpor": 4, "t65": 4, "622": 4, "t270": 4, "953": 4, "939": 4, "027": 4, "t47": 4, "886": 4, "nmunicip": 4, "t412": 4, "t405": 4, "t190": 4, "nmortgag": 4, "595": 4, "t175": 4, "403": 4, "t23": 4, "367": 4, "278": [4, 7], "t132": 4, "t583": 4, "635": 4, "t128": 4, "056": 4, "966": 4, "t34": 4, "t160": 4, "t688": 4, "650": 4, "36": [4, 5, 7], "359": [4, 7], "t481": 4, "n442": 4, "t428": 4, "t923": 4, "t909": 4, "406": 4, "114": 4, "468": 4, "136": 4, "t271": 4, "533": 4, "048": 4, "491": 4, "332": 4, "t320": 4, "t608": 4, "t76": 4, "840": 4, "956": 4, "890": 4, "t20": 4, "627": 4, "243": 4, "t628": 4, "t602": 4, "t192": 4, "t410": 4, "735": 4, "636": 4, "t344": 4, "t144": 4, "470": 4, "657": 4, "831": 4, "125": 4, "162": 4, "t173": 4, "752": 4, "corrobor": 4, "mortgag": 4, "classifi": [4, 7], "37": [4, 7], "swap": 4, "remeasur": 4, "notion": 4, "069": 4, "730": 4, "575": 4, "493": 4, "t104": 4, "777": 4, "nhedg": 4, "433": 4, "505": 4, "247": [4, 7], "ntrade": 4, "41": [4, 7], "44": [4, 7], "depreci": 4, "nland": 4, "690": 4, "nmachineri": 4, "t80": 4, "205": [4, 5], "314": 4, "nleasehold": 4, "839": 4, "599": 4, "73": [4, 7], "70": 4, "884": 4, "852": 4, "t55": 4, "906": 4, "601": 4, "703": 4, "010": 4, "457": 4, "634": 4, "391": 4, "neuropean": 4, "opinion": [4, 7], "1991": 4, "2007": 4, "irish": 4, "branch": 4, "2003": 4, "2014": 4, "2015": 4, "minist": 4, "juli": [4, 7], "annul": 4, "ecj": 4, "hear": 4, "asid": 4, "confirm": 4, "unrecogn": 4, "nfeder": 4, "571": 4, "080": 4, "644": 4, "265": 4, "801": 4, "726": 4, "570": 4, "298": 4, "49": [4, 7], "t84": 4, "428": 4, "603": 4, "483": [4, 7], "t347": 4, "t669": 4, "076": 4, "830": 4, "419": 4, "072": 4, "pretax": 4, "72": [4, 7], "71": 4, "ncomput": 4, "885": 4, "012": 4, "124": 4, "518": 4, "nimpact": 4, "246": 4, "311": 4, "366": 4, "397": 4, "nexcess": 4, "893": 4, "871": 4, "192": [4, 7], "739": 4, "ntax": 4, "carryforward": 4, "302": 4, "naccru": 4, "413": [4, 7], "421": 4, "nunreal": 4, "173": 4, "168": 4, "873": 4, "743": 4, "nless": 4, "374": 4, "007": 4, "369": 4, "551": 4, "998": 4, "nright": 4, "179": 4, "nminimum": 4, "674": 4, "940": 4, "t511": 4, "t455": 4, "t490": 4, "805": 4, "202": 4, "indefinit": 4, "temporari": 4, "727": 4, "044": 4, "284": 4, "ndecreas": 4, "386": 4, "463": 4, "982": 4, "542": 4, "936": 4, "070": 4, "expir": 4, "statut": 4, "229": 4, "494": 4, "closur": 4, "intercompani": 4, "exceed": [4, 7], "multiyear": 4, "exercis": 4, "noncash": 4, "rou": 4, "tfinanci": 4, "t2024": 4, "tother": 4, "661": 4, "tproperti": 4, "015": 4, "303": 4, "676": 4, "t165": 4, "t752": 4, "t859": 4, "430": 4, "842": [4, 7], "tfinanc": 4, "n2025": 4, "820": 4, "t171": 4, "991": 4, "n2026": 4, "914": 4, "n2027": 4, "t59": 4, "733": 4, "n2028": 4, "360": 4, "t38": 4, "398": 4, "n2029": 4, "187": 4, "nthereaft": 4, "t837": 4, "undiscount": 4, "790": 4, "imput": 4, "376": 4, "534": 4, "t896": 4, "borrow": 4, "proce": 4, "nine": [4, 7], "nmatur": 4, "333": 4, "264": 4, "948": 4, "645": 4, "309": 4, "arrear": 4, "namount": 4, "n2013": 4, "nfix": 4, "2062": 4, "t97": 4, "341": 4, "03": 4, "65": [4, 7], "t106": 4, "572": 4, "n97": 4, "nunamort": 4, "premium": 4, "321": 4, "358": 4, "113": 4, "662": 4, "930": 4, "342": 4, "800": 4, "180": 4, "88": 4, "ndure": 4, "425": 4, "426": 4, "372": 4, "589": 4, "055": 4, "appreci": 4, "four": [4, 5, 7], "holder": 4, "n2014": 4, "bonu": 4, "nrestrict": 4, "nnumber": 4, "nrsu": 4, "ngrant": 4, "naggreg": 4, "nfair": 4, "nbalanc": 4, "t240": 4, "427": [4, 7], "t75": 4, "t150": 4, "861": 4, "501": 4, "768": 4, "87": [4, 5, 7], "101": [4, 7], "878": 4, "144": 4, "t127": 4, "t135": 4, "91": [4, 7], "456": 4, "78": [4, 5, 7], "59": [4, 7], "t140": 4, "326": 4, "t158": 4, "204": 4, "350": 4, "002": [4, 5, 6], "nuncondit": 4, "uncondit": 4, "206": 4, "440": 4, "156": 4, "t633": 4, "t670": 4, "226": 4, "45": 4, "nconting": 4, "accrual": 4, "nconcentr": 4, "attribut": [4, 7, 8], "46": 4, "t67": 4, "098": 4, "082": 4, "062": 4, "569": 4, "895": 4, "458": 4, "207": 4, "nonrecur": 4, "t142": 4, "196": 4, "t138": 4, "t147": 4, "859": 4, "nchina": 4, "n66": 4, "t181": 4, "887": 4, "t172": 4, "269": 4, "nlong": 4, "664": 4, "797": 4, "778": 4, "219": 4, "47": [4, 5, 7], "nopinion": 4, "nwe": 4, "fairli": 4, "pcaob": 4, "sponsor": 4, "treadwai": 4, "2013": 4, "unqualifi": 4, "thereon": 4, "nthese": 4, "misstat": 4, "fraud": [4, 7], "ndescript": 4, "naudit": 4, "nhow": 4, "nmatter": 4, "qualifi": 4, "letter": 4, "advisor": 4, "ernst": 4, "llp": 4, "auditor": 4, "2009": 4, "nsan": 4, "jose": 4, "nnovemb": 4, "coso": 4, "nour": 4, "ndefinit": 4, "mainten": [4, 7], "disposit": 4, "receipt": 4, "nevalu": 4, "nbase": 4, "supervis": [4, 7], "13a": 4, "15d": 4, "ninher": 4, "met": [4, 7], "paragraph": 4, "51": [4, 7, 8], "ninsid": 4, "deirdr": 4, "brien": 4, "vice": 4, "presid": 4, "affirm": 4, "april": 4, "withhold": 4, "remitt": 4, "mr": 4, "copi": [4, 6], "solicit": 4, "00042": 4, "nincorpor": 4, "texhibit": 4, "descript": [4, 5, 7, 8], "tform": 4, "tfile": 4, "nrestat": 4, "namend": 4, "bylaw": 4, "nindentur": 4, "york": [4, 8], "mellon": 4, "truste": 4, "noffic": 4, "certif": 4, "2018": 4, "85": [4, 5, 7], "2043": 4, "05": 4, "2044": 4, "februari": 4, "55": [4, 5], "2045": 4, "900": 4, "700": 4, "60": [4, 5, 7], "250": [4, 7], "2036": 4, "2046": 4, "450": 4, "2047": 4, "2049": 4, "2030": 4, "2050": 4, "2060": 4, "2028": 4, "2041": 4, "2051": 4, "2061": 4, "2032": 4, "2052": 4, "54": 4, "2033": 4, "2053": 4, "ceo": 4, "n12": 4, "nsubsidiari": 4, "n23": 4, "nconsent": 4, "n24": 4, "npower": 4, "signatur": 4, "nrule": 4, "nsection": 4, "1350": 4, "n101": 4, "ninlin": 4, "xbrl": 4, "n104": 4, "inlin": 4, "compensatori": 4, "herewith": 4, "furnish": 4, "herebi": 4, "undertak": 4, "56": [4, 5, 7], "nsignatur": 4, "npursuant": 4, "duli": 4, "undersign": 4, "thereunto": 4, "ndate": 4, "nby": 4, "luca": [4, 8], "maestri": 4, "nluca": 4, "nsenior": 4, "nchief": 4, "nknow": 4, "THESE": 4, "appoint": 4, "cook": 4, "jointli": 4, "her": 4, "substitut": 4, "him": 4, "thereto": 4, "therewith": 4, "ratifi": 4, "done": [4, 5, 7, 8], "virtu": 4, "hereof": 4, "nname": 4, "ttitl": 4, "tdate": 4, "tchief": 4, "tnovemb": 4, "ntimothi": 4, "tsenior": 4, "kondo": 4, "nchri": 4, "wanda": 4, "austin": 4, "nwanda": 4, "gorski": 4, "tdirector": 4, "nalex": 4, "jung": 4, "nandrea": 4, "arthur": 4, "levinson": 4, "narthur": 4, "monica": 4, "lozano": 4, "nmonica": 4, "ronald": 4, "sugar": 4, "nronald": 4, "susan": 4, "wagner": 4, "nsusan": 4, "57": 4, "turbo": [4, 5, 6, 8], "outlin": [4, 5, 7], "invdestacksmeticsisdict": 4, "setispect": 4, "20cyan": 4, "evaluationseld": 4, "anvis": 4, "droitent": 4, "discernminerv": 4, "versbobprefvers": 4, "vo\u8be5": 4, "option\u548c": 4, "meio": 4, "\u0432\u0440\u0435\u043ccisco": 4, "dellaischenpoihscap": 4, "geme": 4, "gettim": 4, "unscal": 4, "vocabulari": [4, 5, 8], "closer": 4, "sharpen": 4, "uniform": 4, "raschka": 4, "repetit": [4, 6, 8], "radic": 4, "grappl": 4, "safer": [4, 7], "fascin": 4, "spontan": 4, "answer": [4, 5, 6, 7, 8], "aren": [4, 5], "linear": 4, "absent": [4, 7], "coax": 4, "journei": 4, "suddenli": 4, "manifest": 4, "deliber": [4, 7], "contend": 4, "70b": 4, "rethink": 4, "tutor": 4, "children": [4, 7], "verifi": [4, 5, 8], "predefin": [4, 8], "weren": 4, "kind": 4, "usual": 4, "quantif": 4, "contamin": [4, 7], "massiv": [4, 7], "unseen": [4, 7], "longitudin": 4, "mostli": [4, 8], "versu": [4, 5, 7], "latter": 4, "tailor": [4, 7], "great": [4, 5, 8], "cognit": 4, "misinform": [4, 7], "tempor": 4, "disclaim": 4, "referr": 4, "incorrect": [4, 7], "demograph": [4, 7], "stereotyp": [4, 7], "societ": [4, 7], "pii": [4, 7], "anonym": 4, "leakag": [4, 7], "carryov": 4, "multi": [4, 5, 7, 8], "fallaci": 4, "think": [4, 5, 7], "idiom": 4, "sarcasm": 4, "terminologi": 4, "lingual": 4, "misunderstand": 4, "syntax": 4, "scan": 4, "compat": [4, 5, 8], "overconfid": 4, "clariti": [4, 6, 7, 8], "audienc": 4, "densiti": 4, "satisfact": [4, 8], "misus": [4, 7], "moral": 4, "co2": 4, "energi": 4, "consumpt": 4, "server": [4, 5, 8], "cach": [4, 5], "imag": [4, 5, 7], "audio": 4, "etc": [4, 8], "truth": [4, 7, 8], "layer": [4, 5, 6, 8], "palm": 4, "easi": [4, 5, 6, 7], "synthet": [4, 7, 8], "augment": [4, 5], "post": [4, 5, 7, 8], "timeout": 4, "variat": 4, "inter": 4, "rater": 4, "ti": 4, "holist": [4, 7], "fast": [4, 7, 8], "experiment": [4, 5, 8], "vi": 4, "categor": [4, 7, 8], "intrins": [4, 5], "extrins": 4, "sequenc": [4, 5, 8], "perplex": [4, 5], "downstream": [4, 8], "synthesi": 4, "discret": 4, "prefix": [4, 7], "roug": 4, "bleu": 4, "bilingu": 4, "understudi": 4, "overlap": [4, 6], "favor": [4, 8], "breviti": 4, "insensit": 4, "semant": [4, 6], "orient": [4, 7], "gist": 4, "meteor": 4, "synonym": 4, "stem": [4, 8], "paraphras": 4, "alongsid": [4, 7], "computation": [4, 6], "cider": 4, "consensu": 4, "tf": 4, "idf": 4, "caption": 4, "reliant": 4, "corpu": 4, "ter": 4, "edit": [4, 7], "hypothesi": 4, "penal": 4, "bertscor": 4, "contextu": [4, 7], "bert": 4, "spice": 4, "proposit": 4, "scene": [4, 7], "pure": [4, 5], "analyst": [4, 6], "rouge_1": 4, "rouge_2": 4, "ideal": [4, 5, 7, 8], "cheaper": 4, "setup": [4, 5, 7, 8], "evaluate_summari": 4, "unigram": 4, "bigram": 4, "absl": 4, "py": 4, "rouge_scor": 4, "generated_summari": 4, "reference_summari": 4, "google_bleu": 4, "bleu_scor": 4, "rouge1": 4, "rouge2": 4, "arbitrari": 4, "chosen": [4, 7], "sentence1": 4, "cat": [4, 7], "sat": 4, "mat": 4, "sentence2": 4, "ate": 4, "3333333333333333": 4, "7272727272727272": 4, "4444444444444445": 4, "generate_summari": 4, "summir": 4, "liner": 4, "evaluate_summary_model": 4, "model_benchmark": 4, "models_test": 4, "benchmark_summari": 4, "model_summari": 4, "evaluation_result": 4, "statu": 4, "concis": 4, "element": [4, 7, 8], "verbos": [4, 5, 7], "peripher": 4, "quit": [4, 8], "convei": [4, 6], "breadth": 4, "Of": [4, 7], "vibe": 4, "visualize_prompt_comparison": 4, "matplotlib": 4, "radar": 4, "radar_plot": 4, "tmp": 4, "ipykernel_1652501": 4, "940173201": 4, "userwarn": 4, "figurecanvasagg": 4, "largest": 4, "granular": [4, 5, 6], "tune": [4, 5, 7, 8], "likert": 4, "ensembl": 4, "repeatedli": 4, "fluenci": 4, "refin": 4, "notabl": [4, 7, 8], "integ": 4, "rubric": 4, "hollist": 4, "judgeevalu": 4, "grammar": [4, 5, 8], "evaluate_with_llm": 4, "criterion": 4, "judge_model": 4, "candidate_summari": 4, "grammat": 4, "y": [4, 7, 8], "z": 4, "w": [4, 5, 6, 7], "benchmark_model": 4, "test_model": 4, "input_text": [4, 5, 6], "trillion": [4, 8], "evals_list": 4, "1775618912": 4, "variant": [4, 5, 7], "slightli": 4, "drift": [4, 7], "lowest": [4, 5], "firstli": 4, "overhead": [4, 5], "egocentr": 4, "tight": 4, "aproach": 4, "aplic": 4, "clearli": [4, 7, 8], "earlier": [4, 7], "depict": [4, 7, 8], "multilingu": [4, 7], "golden": 4, "languang": 4, "arena": 4, "randomli": 4, "customiz": [4, 5, 7], "irrelev": 4, "unhelp": [4, 7], "occasion": 4, "rare": 4, "perfectli": 4, "cater": 4, "critiqu": [4, 7], "elo": 4, "spectrum": 4, "exam": 4, "probe": [4, 7], "certifi": 4, "began": [4, 5], "glue": 4, "entail": 4, "baselin": [4, 5, 7], "superglu": 4, "successor": 4, "grew": 4, "big": 4, "bench": 4, "srivastava": 4, "arithmet": 4, "truthfulqa": 4, "multitask": 4, "hendryck": [4, 7], "multidisciplinari": 4, "stanford": 4, "helm": 4, "multidimension": 4, "surround": [4, 7, 8], "humanev": 4, "lmsy": 4, "brought": 4, "dialogu": 4, "chiang": 4, "gather": 4, "alpacaev": 4, "duboi": 4, "mt": 4, "argilla": 4, "mila": 4, "mit": [4, 5], "contributor": [4, 5, 8], "western": 4, "centric": 4, "divid": [4, 6, 7], "subset": [4, 7], "agnost": 4, "dialect": 4, "render": [4, 7], "crowdsourc": 4, "livebench": 4, "white": [4, 7], "resili": [4, 7], "meaningfulli": 4, "zebralog": 4, "grid": 4, "puzzl": 4, "brailsford": 4, "1999": 4, "lsat": 4, "hous": 4, "clue": 4, "deduct": 4, "arriv": 4, "programmat": [4, 8], "2x2": 4, "6x6": 4, "shot": [4, 7], "reductio": 4, "ad": [4, 7, 8], "absurdum": 4, "hard": 4, "10b": 4, "counterfactu": 4, "came": 4, "arc": 4, "prize": [4, 7], "chollet": 4, "mike": [4, 7], "knoop": 4, "founder": 4, "zapier": 4, "fran\u00e7oi": 4, "creator": 4, "agi": 4, "kera": 4, "genuin": 4, "possess": 4, "elementari": 4, "novelti": 4, "wouldn": 4, "interpol": 4, "synthes": 4, "fly": 4, "retriev": [4, 5], "brute": 4, "pixel": 4, "unbeaten": 4, "win": 4, "poorli": 4, "recombin": 4, "spur": [4, 7], "takeawai": 4, "vertic": [4, 7], "finbench": 4, "legalbench": 4, "guha": 4, "berkelei": 4, "bfcl": 4, "patil": 4, "fourrier": 4, "bespok": 4, "sdk": 4, "autoregress": 4, "sub": [4, 5], "liter": 4, "disturb": 4, "zero": [4, 5, 7, 8], "varianc": [4, 7], "yt": 4, "ut": 4, "suppos": [4, 8], "ol": 4, "heteroscedast": 4, "regress": 4, "bivari": 4, "evaluation_track": 4, "evaluationtrack": 4, "model_config": 4, "basemodelconfig": 4, "parallelismmanag": 4, "pipelineparamet": 4, "envconfig": 4, "is_accelerate_avail": 4, "datetim": 4, "timedelta": 4, "initprocessgroupkwarg": 4, "create_evaluation_pipelin": 4, "cache_dir": 4, "float16": 4, "max_sampl": 4, "kwargs_handl": 4, "3000": 4, "save_detail": 4, "pipeline_param": 4, "launcher_typ": 4, "env_config": 4, "override_batch_s": 4, "use_chat_templ": 4, "trust_remote_cod": 4, "pipeline_paramet": 4, "schemat": [4, 6], "vllm": [4, 8], "tgi": 4, "storag": [4, 5, 7], "num_few_shot": 4, "bar": 4, "bigbench": 4, "winogrand": 4, "hellaswag": 4, "nlp": [4, 5, 7], "save_and_push_result": 4, "show_result": 4, "model_arg": 4, "send": [4, 5, 7, 8], "serverless": 4, "inference_server_address": 4, "inference_server_auth": 4, "model_id": 4, "null": 4, "bash": [4, 5], "command": [4, 5], "model_config_path": 4, "endpoint_model": 4, "llama3": [4, 6], "qwen2": [4, 5, 8], "smollm2": 4, "3b": 4, "alibaba": [4, 8], "5b": [4, 5, 8], "hui": 4, "allal": 4, "cluster": 4, "noteworthi": 4, "grain": [4, 5, 8], "salt": [4, 8], "exponenti": 4, "modular": 4, "offici": 4, "revisit": 4, "trace": 4, "langchain_tracing_v2": 4, "langchain_api_kei": 4, "hf_evalu": 4, "langsmith_evalu": 4, "ls_client": 4, "dataset_nam": 4, "create_dataset": 4, "create_exampl": 4, "dataset_id": 4, "calculate_scor": 4, "reference_output": 4, "oai_client": 4, "xp_model_nam": 4, "lastli": 4, "run_evalu": 4, "And": [4, 5, 7], "upload_result": 4, "experiment_prefix": 4, "num_repetit": 4, "386a3620": 4, "9e1cc3cb": 4, "9d6a": 4, "4356": 4, "ab34": 4, "138e0abe8be4": 4, "8741976e": 4, "5268": 4, "4b75": 4, "949f": 4, "99477dde5d64": 4, "selectedsess": 4, "b831dc1e": 4, "90bc": 4, "4ed8": 4, "8080": [4, 5], "fb42444724d6": 4, "4it": 4, "latest": [4, 5, 6, 7, 8], "tobia": 4, "evaluate_modul": 4, "6fc70b7be0088120a372dfdd5d320b39b8bb3630cb8029b193941d9376e86bb0": 4, "tue": 4, "nov": 4, "couldn": 4, "5it": 4, "5053784e": 4, "64445871": 4, "a53c": 4, "44b1": 4, "a422": 4, "4f49b2f9656f": 4, "69": [4, 7], "4b29f3c9": 4, "9ef7e39a": 4, "2add": 4, "410c": 4, "89f8": 4, "9f1a8b198cf1": 4, "61": [4, 7], "insert": 4, "combined_df": 4, "concat": [4, 7], "ignore_index": [4, 7], "execution_tim": 4, "example_id": 4, "333333": 4, "224388": 4, "feb10f92": 4, "3167": 4, "41f3": 4, "bb1c": 4, "d271153a31a8": 4, "5b196b22": 4, "9f4c": 4, "489c": 4, "b020": 4, "7823208b42d6": 4, "348101": 4, "722464": 4, "c310f159": 4, "064a": 4, "4035": 4, "97c3": 4, "a25bbf43abc2": 4, "386076": 4, "704104": 4, "f7f24899": 4, "dd50": 4, "409e": 4, "93cc": 4, "6fb1622b60bf": 4, "443038": 4, "725059": 4, "242856d6": 4, "efb5": 4, "4101": 4, "b1cf": 4, "5805532838ac": 4, "373418": 4, "795302": 4, "ce975169": 4, "a0ab": 4, "40ce": 4, "8e32": 4, "efa28d06079d": 4, "stat": [4, 5], "groupbi": [4, 7], "agg": [4, 7], "sort": 4, "sort_valu": 4, "subplot": 4, "pyplot": 4, "plt": 4, "numpi": 4, "np": 4, "ax1": 4, "ax2": 4, "figsiz": 4, "2ecc71": 4, "3498db": 4, "e74c3c": 4, "bleu_mean": 4, "bleu_std": 4, "enumer": [4, 6, 7], "errorbar": 4, "yerr": 4, "fmt": 4, "markers": 4, "capsiz": 4, "set_ylabel": 4, "set_titl": 4, "set_xtick": 4, "set_xticklabel": 4, "rotat": 4, "set_ylim": 4, "bottom": 4, "legend": 4, "exec_mean": 4, "exec_std": 4, "tight_layout": 4, "ndetail": 4, "4038": 4, "0453": 4, "7815": 4, "0433": 4, "3768": 4, "0424": 4, "8343": 4, "2208": 4, "3519": 4, "0775": 4, "9122": 4, "1482": 4, "377": 4, "042": 4, "078": 4, "slower": [4, 7], "04": [4, 5, 6], "latenc": [4, 5, 6, 7], "speed": [4, 5, 7], "interestingli": 4, "decoupl": 4, "reload": 4, "facilit": [4, 7], "promptfooconfig": 4, "model_comparison": 4, "pretti": [4, 7], "dump": 4, "default_flow_styl": 4, "sort_kei": 4, "prompt1": 4, "defaulttest": 4, "1000m": 4, "millisecond": 4, "eval_data": 4, "latency_m": 4, "totallatencym": 4, "token_usag": 4, "tokenusag": 4, "assert_pass": 4, "assertpasscount": 4, "assert_fail": 4, "assertfailcount": 4, "prompt_token": [4, 5], "num_request": 4, "numrequest": 4, "2463": 4, "000035": 4, "3773": 4, "004620": 4, "1669": 4, "000091": 4, "1669m": 4, "highest": [4, 5], "3773m": 4, "00462": 4, "promptfool": 4, "manual": [4, 5, 7], "redefin": 4, "prompt_comparison": 4, "prompt2": 4, "prompt3": 4, "prompt_fil": 4, "prompt_cont": 4, "BE": 4, "again": 4, "prompt_id": 4, "promptid": 4, "gradingresult": 4, "df_raw": 4, "reset_index": [4, 7], "eas": [4, 5, 7], "seamless": [4, 7], "hf": 4, "plain": [4, 5], "vanilla": 4, "defi": 4, "accustom": 4, "legaci": [4, 5], "unsustain": 4, "prd": 4, "cultiv": [4, 7], "organiz": 4, "stagnat": 4, "alb": 4, "loubna": 4, "anton": 4, "lozhkov": 4, "bakouch": 4, "gabriel": [4, 7], "mart\u00edn": [4, 7], "bl\u00e1zquez": 4, "lewi": 4, "tunstal": 4, "agust\u00edn": 4, "piquer": 4, "andr": 4, "marafioti": 4, "cyril": 4, "zakka": 4, "leandro": 4, "werra": 4, "wolf": 4, "are24": 4, "judgearena": 4, "bps99": 4, "salli": 4, "pott": 4, "barbara": 4, "557": [4, 7], "sciencedirect": 4, "s0377221798003646": 4, "doi": [4, 7, 8], "1016": 4, "s0377": 4, "2217": 4, "00364": 4, "ctj": 4, "jerri": [4, 7], "tworek": [4, 7], "heewoo": [4, 7], "jun": [4, 7], "qime": [4, 7], "henriqu": [4, 7], "pond": [4, 7], "de": [4, 7], "oliveira": [4, 7], "pinto": [4, 7], "harri": [4, 7], "yuri": 4, "burda": 4, "greg": [4, 7], "brockman": [4, 7], "raul": [4, 7], "puri": [4, 7], "gretchen": [4, 7], "krueger": [4, 7], "petrov": [4, 7], "heidi": 4, "khlaaf": 4, "girish": [4, 7], "sastri": [4, 7], "brook": [4, 7], "chan": [4, 7], "grai": [4, 7], "ryder": [4, 7], "mikhail": [4, 7], "pavlov": [4, 7], "alethea": [4, 7], "lukasz": 4, "kaiser": [4, 7], "mohammad": [4, 7], "bavarian": [4, 7], "clemen": [4, 7], "winter": [4, 7], "philipp": 4, "tillet": [4, 7], "felip": [4, 7], "petroski": [4, 7], "dave": [4, 7], "cum": [4, 7], "plappert": 4, "fotio": 4, "chantzi": [4, 7], "barn": 4, "ariel": 4, "herbert": 4, "voss": [4, 7], "hebgen": 4, "guss": 4, "nichol": 4, "paino": [4, 7], "nikola": [4, 7], "tezak": [4, 7], "babuschkin": [4, 7], "suchir": [4, 7], "balaji": [4, 7], "shantanu": [4, 7], "jain": [4, 7], "hess": [4, 7], "carr": 4, "josh": [4, 7], "achiam": [4, 7], "vedant": 4, "misra": 4, "evan": [4, 7], "morikawa": [4, 7], "matthew": 4, "knight": [4, 7], "mile": [4, 7], "brundag": [4, 7], "mira": [4, 7], "murati": [4, 7], "kati": [4, 7], "mayer": [4, 7], "bob": [4, 7, 8], "mcgrew": [4, 7], "ilya": [4, 7], "sutskev": [4, 7], "wojciech": [4, 7], "zaremba": [4, 7], "2107": 4, "03374": 4, "cz": 4, "lianmin": 4, "ying": 4, "sheng": 4, "anastasio": 4, "angelopoulo": 4, "tianl": 4, "dacheng": 4, "banghua": 4, "jordan": [4, 7], "gonzalez": 4, "ion": 4, "stoica": 4, "04132": 4, "cho24a": 4, "francoi": 4, "arcpriz": 4, "cho24b": 4, "dglh24": 4, "yann": 4, "bal\u00e1z": 4, "galambosi": 4, "tatsunori": 4, "hashimoto": 4, "debia": 4, "04475": 4, "fac24a": 4, "wiki": [4, 8], "fac24b": 4, "fac24c": 4, "model_doc": 4, "fac24d": 4, "cookbook": 4, "llm_judg": 4, "fac24f": 4, "fhwt23": 4, "cl\u00e9mentin": 4, "nathan": 4, "habib": 4, "gnh": 4, "julian": 4, "nyarko": 4, "ho": 4, "r\u00e9": 4, "adam": [4, 7], "chilton": 4, "aditya": [4, 7], "narayana": 4, "chohla": 4, "brandon": [4, 7, 8], "waldon": 4, "rockmor": 4, "diego": 4, "zambrano": 4, "dmitri": 4, "talisman": 4, "enam": 4, "hoqu": 4, "faiz": 4, "surani": 4, "frank": [4, 7], "fagan": 4, "galit": 4, "sarfati": 4, "gregori": 4, "dickinson": 4, "haggai": 4, "porat": 4, "hegland": 4, "jessica": [4, 7], "joe": [4, 7], "nudel": 4, "joel": [4, 7], "niklau": 4, "nai": 4, "jonathan": [4, 7], "choi": 4, "margaret": 4, "hagan": 4, "megan": 4, "ma": [4, 7], "livermor": 4, "nikon": 4, "rasumov": 4, "rahe": 4, "nil": 4, "holzenberg": 4, "noam": 4, "kolt": 4, "henderson": 4, "rehaag": 4, "sharad": 4, "shang": 4, "spencer": 4, "sunni": 4, "gandhi": 4, "zur": 4, "varun": 4, "iyer": 4, "zehua": 4, "2308": 4, "11462": 4, "hbb": 4, "collin": 4, "burn": 4, "steven": [4, 7], "basart": [4, 7], "zou": [4, 7], "manta": [4, 7], "mazeika": [4, 7], "03300": 4, "hbd": 4, "maxwel": 4, "forb": 4, "yejin": 4, "curiou": 4, "neural": [4, 8], "degener": 4, "1904": 4, "09751": 4, "hyc": 4, "binyuan": 4, "zeyu": 4, "cui": 4, "jiaxi": 4, "dayiheng": 4, "tianyu": 4, "jiajun": 4, "kai": [4, 7], "dang": 4, "coder": 4, "preprint": [4, 8], "2409": [4, 7], "12186": 4, "lx": 4, "zhen": 4, "xiaohan": 4, "jia": 4, "yuxuan": 4, "lai": 4, "chongyang": 4, "shuai": 4, "nlg": 4, "07103": 4, "lbl": 4, "bommasani": 4, "toni": 4, "dimitri": 4, "tsipra": 4, "dilara": 4, "soylu": 4, "michihiro": 4, "yasunaga": 4, "yian": 4, "deepak": 4, "narayanan": 4, "yuhuai": 4, "newman": 4, "binhang": 4, "bobbi": 4, "ce": 4, "christian": [4, 7], "cosgrov": 4, "acosta": 4, "nava": [4, 7], "drew": 4, "hudson": 4, "zelikman": 4, "esin": 4, "durmu": 4, "faisal": 4, "ladhak": 4, "frieda": 4, "rong": 4, "hongyu": 4, "ren": 4, "huaxiu": 4, "yao": [4, 7], "jue": 4, "keshav": 4, "santhanam": 4, "laurel": 4, "lucia": 4, "mert": 4, "yuksekgonul": 4, "mirac": 4, "suzgun": 4, "niladri": 4, "chatterji": 4, "omar": 4, "khattab": 4, "qian": [4, 7], "chi": [4, 8], "sang": 4, "shibani": [4, 7], "santurkar": [4, 7], "surya": 4, "icard": 4, "tianyi": 4, "vishrav": 4, "chaudhari": 4, "xuechen": 4, "yuhui": 4, "yuta": 4, "koreeda": 4, "2211": 4, "09110": 4, "lbc24": 4, "ronan": 4, "bra": 4, "allenai": 4, "lhe22": [4, 7], "stephani": [4, 7], "owain": [4, 7], "mimic": [4, 7], "falsehood": [4, 7], "2109": [4, 7], "07958": [4, 7], "pzwg23": 4, "shishir": 4, "tianjun": 4, "xin": [4, 7], "gorilla": 4, "15334": 4, "pro24": 4, "dev": 4, "ras24": 4, "sebastian": 4, "scratch": 4, "1633437166": 4, "srf": 4, "shivalika": 4, "angelika": 4, "roman": [4, 7], "adelani": 4, "ngui": 4, "vila": 4, "suero": 4, "peerat": 4, "limkonchotiwat": 4, "kelli": 4, "marchisio": 4, "qi": 4, "leong": 4, "yosephin": 4, "susanto": 4, "raymond": [4, 7], "ng": [4, 7], "shayn": 4, "longpr": 4, "ko": 4, "madelin": 4, "antoin": 4, "bosselut": 4, "oh": 4, "leshem": 4, "choshen": 4, "daphn": 4, "ippolito": 4, "enzo": 4, "ferrant": 4, "marzieh": 4, "fadae": 4, "beyza": 4, "ermi": 4, "sara": 4, "hooker": 4, "linguist": [4, 7], "03304": 4, "srr": 4, "aarohi": 4, "abhinav": 4, "rastogi": 4, "abhishek": 4, "rao": 4, "abu": 4, "awal": 4, "shoeb": 4, "abubakar": 4, "abid": 4, "fisch": 4, "santoro": 4, "gupta": 4, "adri\u00e0": 4, "garriga": 4, "alonso": 4, "agnieszka": 4, "kluska": 4, "aitor": 4, "lewkowycz": 4, "akshat": 4, "warstadt": 4, "alexand": [4, 7, 8], "kocurek": 4, "ali": [4, 7], "safaya": 4, "tazarv": 4, "aman": 4, "hussain": 4, "dsouza": 4, "ambros": 4, "slone": 4, "ameet": 4, "rahan": 4, "anantharaman": 4, "ander": 4, "andreassen": 4, "madotto": 4, "santilli": 4, "stuhlm\u00fcller": 4, "la": 4, "lampinen": 4, "angelica": 4, "anh": 4, "vuong": 4, "animesh": 4, "gottardi": 4, "antonio": 4, "norelli": 4, "anu": 4, "venkatesh": 4, "arash": 4, "gholamidavoodi": 4, "arfa": 4, "tabassum": 4, "arul": 4, "menez": 4, "arun": [4, 7], "kirubarajan": 4, "asher": 4, "mullokandov": 4, "ashish": 4, "sabharw": 4, "herrick": 4, "avia": 4, "efrat": 4, "aykut": 4, "erdem": 4, "ayla": 4, "karaka\u015f": 4, "bao": [4, 7], "loe": 4, "barret": [4, 7], "zoph": [4, 7], "bart\u0142omiej": 4, "bojanowski": 4, "batuhan": 4, "\u00f6zyurt": 4, "behnam": 4, "hedayatnia": 4, "neyshabur": 4, "inden": 4, "benno": 4, "stein": 4, "berk": 4, "ekmekci": 4, "blake": 4, "howald": 4, "bryan": 4, "orinion": 4, "diao": 4, "dour": 4, "stinson": 4, "cedrick": 4, "argueta": 4, "c\u00e9sar": 4, "ferri": 4, "ram\u00edrez": 4, "chandan": 4, "charl": 4, "rathkopf": 4, "chenlin": 4, "meng": 4, "chitta": 4, "baral": 4, "chiyu": 4, "callison": 4, "burch": 4, "wait": [4, 7], "voigt": 4, "cindi": 4, "ramirez": 4, "clara": 4, "rivera": 4, "clemencia": 4, "siro": 4, "colin": 4, "raffel": 4, "courtnei": 4, "ashcraft": 4, "cristina": 4, "garbacea": 4, "damien": [4, 7], "sileo": 4, "garrett": 4, "kilman": 4, "freeman": 4, "khashabi": 4, "levi": [4, 7], "mosegu\u00ed": 4, "gonz\u00e1lez": 4, "perszyk": 4, "danqi": 4, "dar": 4, "gilboa": 4, "dohan": [4, 7], "drakard": 4, "jurgen": 4, "debajyoti": 4, "datta": 4, "deni": 4, "emelin": 4, "kleyko": 4, "deniz": 4, "yuret": 4, "derek": [4, 7], "tam": [4, 8], "dieuwk": 4, "hupk": 4, "diganta": 4, "dilyar": 4, "buzan": 4, "coelho": 4, "mollo": 4, "diyi": 4, "dylan": 4, "schrader": 4, "ekaterina": 4, "shutova": 4, "ekin": 4, "dogu": 4, "cubuk": 4, "elad": 4, "segal": 4, "eleanor": 4, "hagerman": 4, "donowai": 4, "elli": 4, "pavlick": 4, "rodola": 4, "emma": 4, "lam": 4, "chu": [4, 7], "erkut": 4, "erni": 4, "dyer": 4, "jerzak": 4, "eunic": 4, "engefu": 4, "manyasi": 4, "evgenii": 4, "zheltonozhskii": 4, "fanyu": 4, "xia": 4, "fatemeh": 4, "siar": 4, "fernando": 4, "mart\u00ednez": 4, "plume": 4, "francesca": 4, "happ\u00e9": 4, "gaurav": 4, "genta": 4, "indra": 4, "winata": 4, "gerard": 4, "melo": 4, "germ\u00e1n": 4, "kruszewski": 4, "giambattista": [4, 7], "parascandolo": [4, 7], "giorgio": 4, "mariani": 4, "gloria": 4, "gonzalo": 4, "jaimovitch": 4, "l\u00f3pez": 4, "gregor": 4, "betz": 4, "gui": [4, 5], "gur": 4, "hana": 4, "galijasev": 4, "rashkin": 4, "hannaneh": 4, "hajishirzi": 4, "harsh": 4, "hayden": 4, "bogar": 4, "henri": [4, 7], "shevlin": 4, "hinrich": 4, "sch\u00fctze": 4, "hiromu": 4, "yakura": 4, "hongm": 4, "hugh": 4, "mee": 4, "wong": [4, 7], "isaac": 4, "nobl": 4, "jaap": 4, "jumelet": 4, "geissing": 4, "jaehoon": 4, "jaim": 4, "fern\u00e1ndez": 4, "fisac": 4, "simon": 4, "koppel": 4, "koco\u0144": 4, "jana": 4, "thompson": [4, 5, 7], "janel": 4, "wingfield": 4, "jarema": 4, "radom": 4, "jascha": 4, "sohl": [4, 7], "dickstein": 4, "phang": 4, "yosinski": 4, "jekaterina": 4, "novikova": 4, "jell": 4, "bosscher": 4, "jennif": 4, "marsh": 4, "jeroen": 4, "taal": 4, "engel": 4, "jesujoba": 4, "alabi": 4, "jiam": 4, "jillian": 4, "joan": 4, "waweru": 4, "burden": 4, "bali": 4, "batcheld": 4, "berant": 4, "j\u00f6rg": 4, "frohberg": 4, "jo": 4, "rozen": 4, "orallo": 4, "boudeman": 4, "guerr": 4, "tenenbaum": 4, "joyc": 4, "chua": 4, "kanclerz": 4, "karen": 4, "livescu": 4, "karl": 4, "krauth": 4, "karthik": 4, "gopalakrishnan": 4, "katerina": 4, "ignatyeva": 4, "katja": 4, "markert": 4, "kaustubh": 4, "dhole": 4, "gimpel": 4, "omondi": 4, "kori": 4, "mathewson": 4, "kristen": 4, "chiafullo": 4, "ksenia": 4, "shkaruta": 4, "shridhar": 4, "kyle": [4, 7], "mcdonel": 4, "richardson": 4, "laria": 4, "reynold": 4, "leo": [4, 7], "dugan": 4, "lianhui": 4, "lidia": 4, "contrera": 4, "ochando": 4, "morenc": 4, "moschella": 4, "luci": 4, "ludwig": 4, "schmidt": [4, 7], "luheng": 4, "olivero": 4, "col\u00f3n": 4, "metz": [4, 7], "l\u00fctfi": 4, "kerem": 4, "\u015fenel": 4, "maarten": [4, 7], "bosma": 4, "sap": [4, 7], "maartj": 4, "hoev": 4, "maheen": 4, "farooqi": 4, "manaal": 4, "faruqui": 4, "marco": 4, "baturan": 4, "marelli": 4, "maru": 4, "maria": 4, "quintana": 4, "tolkiehn": 4, "mario": [4, 7], "giulianelli": 4, "martha": 4, "potthast": 4, "leavitt": 4, "hagen": 4, "m\u00e1ty\u00e1": 4, "schubert": 4, "medina": [4, 7], "orduna": 4, "baitemirova": 4, "melodi": 4, "arnaud": 4, "melvin": 4, "mcelrath": 4, "yee": 4, "cohen": 4, "ivanitskii": 4, "starritt": 4, "strube": 4, "micha\u0142": 4, "sw\u0119drowski": 4, "michel": [4, 7], "bevilacqua": 4, "mihir": 4, "kale": 4, "cain": 4, "mime": 4, "mitch": 4, "walker": 4, "mo": 4, "tiwari": 4, "mohit": 4, "bansal": 4, "moin": 4, "aminnaseri": 4, "mor": 4, "geva": 4, "mozhdeh": 4, "gheini": 4, "mukund": 4, "varma": 4, "nanyun": 4, "peng": [4, 7], "nayeon": 4, "neta": 4, "krakov": 4, "doiron": 4, "nicol": 4, "martinez": 4, "nikita": 4, "nangia": 4, "nikla": 4, "decker": 4, "muennighoff": 4, "nitish": [4, 7], "shirish": [4, 7], "keskar": [4, 7], "niveditha": 4, "constant": 4, "fiedel": 4, "nuan": 4, "wen": 4, "oliv": [4, 7], "agha": 4, "elbaghdadi": 4, "omer": 4, "moreno": 4, "casar": 4, "parth": 4, "doshi": 4, "pascal": 4, "fung": 4, "pu": 4, "vicol": 4, "pegah": 4, "alipoormolabashi": 4, "peiyuan": 4, "eckerslei": 4, "phu": 4, "mon": 4, "htut": 4, "pinyu": 4, "hwang": 4, "piotr": 4, "mi\u0142kowski": 4, "piyush": 4, "pouya": 4, "pezeshkpour": 4, "priti": 4, "oli": 4, "qiaozhu": 4, "qing": 4, "lyu": 4, "qinlang": 4, "rabin": 4, "banjad": 4, "rachel": [4, 7], "etta": 4, "rudolph": 4, "raefer": 4, "rahel": 4, "haback": 4, "ramon": 4, "risco": 4, "rapha\u00ebl": 4, "milli\u00e8r": 4, "rhythm": 4, "garg": 4, "rif": 4, "saurou": 4, "riku": 4, "arakawa": 4, "robb": 4, "raymaek": 4, "rohan": 4, "sikand": 4, "novak": 4, "sitelew": 4, "lebra": 4, "rosann": 4, "rowan": [4, 7], "ruslan": 4, "salakhutdinov": 4, "stoval": 4, "teehan": 4, "sahib": 4, "saif": 4, "sajant": 4, "anand": [4, 7], "dillav": 4, "shleifer": 4, "wiseman": 4, "gruetter": 4, "schoenholz": 4, "sanghyun": 4, "sanjeev": 4, "kwatra": 4, "sarik": 4, "ghazarian": 4, "sayan": 4, "casei": [4, 7], "bischoff": 4, "gehrmann": 4, "schuster": 4, "sepideh": 4, "sadeghi": 4, "shadi": 4, "hamdan": 4, "sharon": 4, "shashank": 4, "sherri": 4, "shi": 4, "shikhar": 4, "shima": 4, "asaadi": 4, "shubh": 4, "pachchigar": 4, "shubham": 4, "toshniw": 4, "shyam": [4, 7], "upadhyai": 4, "shyamolima": 4, "debnath": 4, "siamak": 4, "shakeri": 4, "thormey": 4, "melzi": 4, "siva": 4, "reddi": 4, "sneha": 4, "priscilla": 4, "makini": 4, "soo": 4, "hwan": 4, "toren": 4, "sriharsha": 4, "hatwar": 4, "stanisla": 4, "dehaen": 4, "stefan": 4, "divic": 4, "stella": 4, "biderman": 4, "stephen": 4, "prasad": 4, "piantadosi": 4, "stuart": [4, 7], "shieber": 4, "summer": [4, 7], "misherghi": 4, "svetlana": 4, "kiritchenko": 4, "swaroop": 4, "tal": 4, "linzen": 4, "tariq": 4, "tatsu": 4, "te": 4, "th\u00e9o": 4, "desbord": 4, "theodor": 4, "rothschild": 4, "phan": [4, 7], "tiberiu": 4, "nkinyili": 4, "timo": 4, "schick": 4, "timofei": 4, "kornev": 4, "titu": 4, "tunduni": 4, "gerstenberg": 4, "trenton": 4, "trishala": 4, "neeraj": 4, "tushar": 4, "khot": 4, "shultz": 4, "uri": 4, "shaham": 4, "vera": 4, "demberg": 4, "victoria": [4, 7], "nyamai": 4, "vika": 4, "raunak": 4, "vinai": 4, "ramasesh": 4, "udai": 4, "prabhu": 4, "vishakh": 4, "padmakumar": 4, "vivek": 4, "srikumar": 4, "fedu": [4, 7], "wout": 4, "vossen": 4, "xiaoyu": 4, "tong": [4, 7], "xinran": 4, "xinyi": 4, "yadollah": 4, "yaghoobzadeh": 4, "yair": 4, "lakretz": 4, "yangqiu": 4, "yasaman": 4, "bahri": 4, "yichi": 4, "yide": 4, "yifu": 4, "yonatan": 4, "belinkov": 4, "yufang": 4, "seid": 4, "zhuoy": 4, "zijian": 4, "ziji": 4, "zirui": 4, "ziyi": 4, "extrapol": 4, "2206": 4, "04615": 4, "wpn": 4, "yada": 4, "pruksachatkun": 4, "amanpreet": 4, "hill": 4, "stickier": 4, "wsm": 4, "1804": 4, "07461": 4, "wtb": 4, "tai": 4, "borgeaud": 4, "dani": 4, "yogatama": 4, "denni": [4, 7], "donald": 4, "metzler": 4, "ed": 4, "oriol": 4, "vinyal": 4, "dean": 4, "07682": 4, "wdr": 4, "doolei": 4, "manlei": 4, "arka": [4, 7], "pal": 4, "feuer": 4, "siddhartha": 4, "ravid": 4, "shwartz": [4, 7], "ziv": 4, "khalid": 4, "saifullah": 4, "siddartha": 4, "naidu": 4, "chinmai": 4, "hegd": 4, "lecun": 4, "goldstein": 4, "willi": 4, "neiswang": 4, "micah": 4, "goldblum": 4, "19314": 4, "yyh": 4, "baosong": 4, "chengpeng": 4, "chengyuan": 4, "fei": 4, "guant": 4, "haoran": 4, "huan": 4, "jialong": 4, "jialin": 4, "jianhong": 4, "tu": 4, "jianwei": 4, "jianxin": 4, "jin": [4, 7], "jingren": 4, "jinz": 4, "jinzheng": 4, "junyang": 4, "keme": 4, "keqin": 4, "kexin": 4, "mingfeng": 4, "xue": [4, 7], "ni": 4, "pei": 4, "ru": 4, "men": 4, "ruiz": 4, "runji": 4, "shiji": 4, "sinan": 4, "tianhang": 4, "wenbin": 4, "ge": 4, "xiaodong": 4, "deng": 4, "xiaohuan": 4, "xingzhang": 4, "xinyu": [4, 7], "xipin": 4, "xuancheng": 4, "yichang": 4, "wan": 4, "yunfei": 4, "yuqiong": 4, "zhenru": 4, "zhihao": 4, "10671": 4, "zcl24": 4, "zhihan": 4, "cao": 4, "lizi": 4, "openreview": 4, "forum": 4, "aegrf1uy0p": 4, "zc": 4, "siyuan": 4, "zhuang": [4, 7], "zhanghao": 4, "yonghao": 4, "zi": 4, "zhuohan": 4, "xing": [4, 7], "2306": 4, "05685": 4, "huggingface24": 4, "06": [4, 8], "metaai24": 4, "di": 5, "hunter": 5, "photo": 5, "email": 5, "hipaa": 5, "properti": [5, 7], "gdpr": 5, "iot": 5, "unreli": 5, "impract": 5, "slm": 5, "viabl": 5, "sensor": 5, "programm": 5, "graphic": [5, 7], "vram": 5, "vector": [5, 7], "mathbf": 5, "x_1": 5, "x_2": 5, "x_n": 5, "x_": 5, "\u03b8": 5, "matrix": [5, 7], "concurr": 5, "mozilla": 5, "docker": 5, "gerganov": 5, "georgi": 5, "hundr": 5, "overwhelm": [5, 8], "manifesto": 5, "enjoy": 5, "bog": 5, "exploratori": 5, "hacker": 5, "Will": [5, 7], "codebas": 5, "prototyp": 5, "prematur": 5, "besid": 5, "lighter": 5, "sacrific": 5, "gguf": 5, "unifi": 5, "ggml": [5, 8], "ibm": [5, 7], "bit": 5, "metadata": 5, "disk": 5, "faster": 5, "backward": 5, "2024x": 5, "repo": 5, "easier": [5, 6, 7, 8], "compil": 5, "linux": 5, "argument": [5, 7, 8], "sudo": 5, "apt": 5, "cmake": 5, "bind": 5, "betlen": 5, "cnv": 5, "llamacpp": 5, "q8_0": 5, "succinct": 5, "ctrl": 5, "interject": 5, "philosoph": 5, "debat": 5, "fulfil": 5, "happi": 5, "responsibli": 5, "bye": 5, "goodby": 5, "port": 5, "127": 5, "curl": [5, 8], "localhost": 5, "v1": [5, 6, 7], "bearer": 5, "finish_reason": 5, "deepli": 5, "1734627879": 5, "completion_token": 5, "total_token": 5, "chatcmpl": 5, "5wl2tzjzdmzupvxwp2gcedr8xbpsyhfm": 5, "prompt_n": 5, "prompt_m": 5, "132": 5, "prompt_per_token_m": 5, "prompt_per_second": 5, "77619878666999": 5, "predicted_n": 5, "predicted_m": 5, "1700": 5, "654": 5, "predicted_per_token_m": 5, "36882142857143": 5, "predicted_per_second": 5, "92850867960208": 5, "wrestl": [5, 7], "gbnf": [5, 8], "8pm": 5, "appointmenttim": 5, "appointmentdetail": 5, "handi": 5, "modal": 5, "model_path": 5, "llama_cpp": 5, "create_chat_complet": 5, "occupi": 5, "activist": 5, "justin": [5, 7], "tunnei": 5, "ocho": 5, "appach": 5, "cosmopolitan": 5, "libc": 5, "portabl": 5, "durabl": 5, "usabl": [5, 7, 8], "tinyllama": 5, "wget": 5, "jartin": 5, "q5_k_m": 5, "renam": 5, "ex": 5, "chmod": 5, "nobrows": 5, "registri": 5, "nativ": [5, 8], "container": 5, "isol": [5, 7], "trai": 5, "familiar": 5, "permiss": [5, 7], "despit": [5, 6, 8], "ssfl": 5, "sh": [5, 8], "Or": 5, "11434": 5, "chatrespons": 5, "tradeoff": [5, 7], "easiest": 5, "rich": [5, 7], "playground": 5, "simultan": [5, 7], "verif": [5, 8], "importantli": 5, "gemma": [5, 8], "intuit": 5, "beginn": 5, "tensorrt": 5, "trt": 5, "latex": 5, "voic": 5, "pwa": 5, "rag": 5, "simpler": [5, 7, 8], "medium": [5, 7, 8], "gpt4all": 5, "rbac": 5, "fp16": 5, "q2_k": 5, "q4_k": 5, "q6_k": 5, "thoughput": 5, "evalu": [5, 6, 8], "wikitext": 5, "salesforc": 5, "wikipedia": [5, 8], "min_prompt_length": 5, "input_texts_raw": 5, "2010": 5, "valkyria": 5, "chronicl": 5, "underw": 5, "forgiv": 5, "newcom": 5, "raita": 5, "honjou": 5, "compos": [5, 7], "hitoshi": 5, "sakimoto": 5, "takeshi": 5, "ozawa": 5, "writer": 5, "theme": [5, 6, 7], "sung": 5, "escap": 5, "escaped_text": 5, "2024w": 5, "block_scal": 5, "block": [5, 7], "width": 5, "parenthes": 5, "block_min": 5, "formula": 5, "superblock": 5, "5625": 5, "ieee": 5, "754": 5, "half": [5, 7], "ppl": 5, "exp": 5, "sum_": 5, "log_2": 5, "x_i": 5, "ratio": 5, "avg": 5, "_i": 5, "corr": 5, "ln": [5, 8], "kullback": 5, "leibler": 5, "logit": [5, 8], "d_": 5, "softmax": 5, "sum": 5, "kld": 5, "q2_kresult": 5, "q6": 5, "004": 5, "q2": 5, "112": 5, "q4": 5, "smallest": 5, "390": 5, "mib": 5, "67": [5, 7], "81": [5, 7], "93": [5, 7], "462": 5, "96": [5, 7], "614": 5, "58": 5, "170": 5, "ubuntu": 5, "lt": 5, "x86_64": 5, "gnu": 5, "feasibl": [5, 6], "thank": [5, 8], "intel": 5, "i7": 5, "8550u": 5, "15gib": 5, "samsung": 5, "ssd": 5, "970": 5, "evo": 5, "500gb": 5, "bc24": 5, "andrei": [5, 7], "abetlen": 5, "gc24": 5, "ggerganov": [5, 8], "blob": [5, 8], "readm": [5, 8], "gc4a": 5, "gc4b": 5, "thread": 5, "huggingface4w": 5, "huggingface4xa": 5, "huggingface4xb": 5, "ibmthink24": 5, "lmstudio24": 5, "lmstudio": 5, "mozillaocho24": 5, "salesforce24": 5, "possibli": 6, "eliot": 6, "thumb": 6, "\u00be": 6, "max_output_token": 6, "4096": 6, "16384": 6, "contrari": 6, "surpass": 6, "truncat": 6, "max_input_token": 6, "input_cost_per_token": 6, "output_cost_per_token": 6, "11b": [6, 7], "128000": 6, "5e": 6, "20241022": [6, 7], "8192": 6, "200000": 6, "3e": 6, "0613": 6, "6e": 6, "gemini": 6, "flash": 6, "1048576": 6, "2097152": 6, "05e": 6, "incomplet": [6, 7], "abruptli": 6, "shallow": 6, "thorough": [6, 7], "dissatisfact": 6, "frustrat": [6, 7], "10k": 6, "diagram": [6, 7], "charactertextsplitt": 6, "tiktoken": [6, 7], "sequenti": 6, "newlin": 6, "broadli": [6, 8], "cheap": 6, "speciali": 6, "nltk": 6, "spaci": 6, "hierarch": [6, 7], "talk": 6, "splitter": 6, "get_chunk": 6, "chunk_siz": 6, "chunk_overlap": 6, "langchain_text_splitt": 6, "text_splitt": 6, "from_tiktoken_encod": 6, "split_text": 6, "persona": 6, "langchain_cor": [6, 8], "prompttempl": 6, "get_base_prompt_templ": 6, "base_prompt": [6, 8], "from_templ": 6, "llmchain": 6, "parser": [6, 8], "output_pars": 6, "stroutputpars": 6, "langchain_commun": 6, "chat_model": 6, "chatlitellm": 6, "get_llm_chain": 6, "prompt_templ": [6, 8], "llm_chain": [6, 8], "api_key_label": 6, "upper": 6, "_api_kei": 6, "get_dynamic_prompt_templ": 6, "get_dynamic_prompt_param": 6, "prompt_param": 6, "part_idx": 6, "total_part": 6, "chat_context": 6, "param": 6, "dynamic_prompt_param": 6, "introduct": 6, "concaten": 6, "generate_report": 6, "input_cont": 6, "llm_model_nam": 6, "report_part": 6, "num_part": 6, "dinam": 6, "priovid": 6, "invok": [6, 8], "cummul": 6, "max_chunk_s": 6, "max_chunk_overlap": 6, "readabl": 6, "apple_report": 6, "luation": 6, "disciplin": 6, "subhead": 6, "depth": [6, 7], "overlook": 6, "preprocess": [6, 8], "necessit": 6, "meticul": 6, "bottleneck": 6, "mustafa": 6, "suleyman": 6, "infinit": 6, "fewer": [6, 7], "condens": 6, "versatil": 6, "grace": 6, "fallback": 6, "empow": [6, 7], "langchain24": 6, "how_to": 6, "immens": 7, "commonplac": 7, "hartvigsen": 7, "societi": 7, "statement": 7, "alarm": 7, "openli": 7, "dolli": 7, "v2": 7, "llama2": [7, 8], "13b": 7, "emb": 7, "generalist": 7, "injustic": 7, "inequ": 7, "undermin": 7, "perpetu": 7, "displac": 7, "eros": 7, "fake": 7, "deepfak": 7, "distrust": 7, "cyberattack": 7, "spread": 7, "disinform": 7, "inadvert": 7, "interven": 7, "irrevers": 7, "uncheck": 7, "extinct": 7, "race": 7, "incentiv": 7, "shortcut": 7, "behind": 7, "stress": 7, "urgent": 7, "reorient": 7, "birth": 7, "siam": 7, "edgington": 7, "jailbreak": 7, "promptcraft": 7, "stealth": 7, "sutton": 7, "subtl": 7, "trigger": 7, "subtleti": 7, "exception": 7, "phrase": 7, "evad": 7, "hqve": 7, "frer": 7, "hplidai": 7, "pl": 7, "hyperion": 7, "coast": 7, "redwood": 7, "tallest": 7, "tree": [7, 8], "routin": 7, "prejudic": 7, "gallego": 7, "leak": 7, "poison": 7, "intention": 7, "inject": 7, "mislead": 7, "exabeam": 7, "finra": 7, "3110": 7, "mandat": 7, "supervisori": 7, "medicin": 7, "unicef": 7, "contest": 7, "congress": 7, "enact": 7, "pictur": [7, 8], "territori": 7, "oversea": 7, "chines": 7, "legitim": 7, "consent": 7, "complaint": 7, "cooper": 7, "extraterritori": 7, "offshor": 7, "draft": 7, "voluntari": 7, "neutral": 7, "player": 7, "prepared": 7, "ahead": 7, "compris": 7, "cbrn": 7, "persuas": 7, "autonomi": 7, "gradat": 7, "scorecard": 7, "elig": 7, "advisori": 7, "sag": 7, "shut": 7, "prerequisit": 7, "harden": 7, "asl": 7, "biosafeti": 7, "elev": 7, "warn": 7, "bioweapon": 7, "compartment": 7, "difficulti": 7, "4x": 7, "jump": 7, "paus": 7, "frontier": 7, "deepmind": 7, "biosecur": 7, "buffer": 7, "formul": [7, 8], "calibr": 7, "promin": 7, "taxonomi": 7, "llamaguard": 7, "leaderboard": 7, "3x": 7, "5x": 7, "alaga": 7, "substandard": 7, "rapidli": [7, 8], "oxford": 7, "wachter": 7, "blur": 7, "ill": 7, "stifl": 7, "suscept": 7, "aadc": 7, "outset": 7, "curricula": 7, "adversari": 7, "uncov": [7, 8], "appar": 7, "thoroughli": 7, "lm": [7, 8], "problemat": 7, "arrai": 7, "undergo": 7, "280b": 7, "cai": [7, 8], "utilis": 7, "minimis": 7, "enshrin": 7, "evas": 7, "resort": 7, "encod": 7, "avenu": 7, "cambria": 7, "inherit": 7, "influenti": 7, "debias": 7, "occurr": 7, "phish": 7, "clarifi": 7, "toler": 7, "checklist": 7, "abus": 7, "ux": 7, "architect": 7, "retrofit": 7, "promptli": 7, "dashboard": 7, "misalign": 7, "star": 7, "postpon": 7, "sens": 7, "combat": 7, "counter": 7, "traffic": 7, "workaround": 7, "vital": 7, "silo": 7, "hierarchi": 7, "66": 7, "mcq": 7, "regex": [7, 8], "joint": 7, "facet": 7, "purpl": 7, "circl": 7, "opensafetylab": 7, "salad_bench_dataset": 7, "base_set": 7, "gptfuzzer": 7, "auto": 7, "qid": 7, "o1": 7, "supremaci": 7, "o53": 7, "o14": 7, "o5": 7, "o65": 7, "plagiar": 7, "o16": 7, "o6": 7, "o47": 7, "campaign": 7, "o12": 7, "o52": 7, "surveil": 7, "spous": 7, "know": [7, 8], "o13": 7, "breakdown": [7, 8], "ncount": 7, "21318": 7, "8756": 7, "6486": 7, "o2": 7, "1717": 7, "o4": 7, "1477": 7, "o3": 7, "socioeconom": 7, "851": 7, "int64": 7, "gen": 7, "15433": 7, "hh": 7, "4184": 7, "659": 7, "advbench": 7, "230": 7, "189": 7, "toxicchat": 7, "anyth": 7, "817": 7, "misconcept": 7, "ingrain": 7, "mc1": 7, "singular": 7, "choices4": 7, "mc2": 7, "set4": 7, "scorer": 7, "correctli": [7, 8], "truthful_qa": 7, "truthfulqa_dataset": 7, "multiple_choic": 7, "best_answ": 7, "correct_answ": 7, "incorrect_answ": 7, "watermelon": 7, "digest": 7, "noth": 7, "stomach": 7, "sick": 7, "wonderopoli": 7, "wonder": 7, "belli": 7, "swallow": 7, "dream": 7, "die": 7, "indigest": 7, "unconsci": 7, "excret": 7, "asr": 7, "r2d2": 7, "multimod": 7, "wider": [7, 8], "mmlu": 7, "mass": 7, "destruct": 7, "asynchron": 7, "webpurifi": 7, "protectai": 7, "aw": 7, "comprehend": 7, "amazon": 7, "nemo": 7, "nvidia": 7, "keyword": 7, "toolset": 7, "nemmo": 7, "synchron": 7, "nemoguardrail": 7, "llmrail": 7, "railsconfig": 7, "from_path": 7, "rail": 7, "hello": 7, "ministr": 7, "mistralai": 7, "mistral_api_kei": 7, "moderate_chat": 7, "omni": 7, "pprint": 7, "to_json": 7, "threaten": 7, "illicit": 7, "granit": 7, "guardian": 7, "consortium": 7, "v3": 7, "begin_of_text": 7, "start_header_id": 7, "end_header_id": 7, "unsafe_categori": 7, "user_message_1": 7, "model_answer_1": 7, "comma": 7, "eot_id": 7, "eom_id": 7, "denot": 7, "s1": 7, "s2": 7, "s3": 7, "s4": 7, "s5": 7, "defam": 7, "s6": 7, "s7": 7, "s8": 7, "s9": 7, "s10": 7, "s11": 7, "s12": 7, "s13": 7, "atla": 7, "2b": 7, "hap": 7, "38m": 7, "125m": 7, "padhi": 7, "shieldgemma": 7, "exemplifi": 7, "accomplish": [7, 8], "judge_prompt": 7, "american": 7, "vandal": 7, "underag": 7, "drink": 7, "vulgar": 7, "obscen": 7, "racism": 7, "derogatori": 7, "firearm": 7, "safety_scor": 7, "IN": 7, "borderlin": 7, "verdict": 7, "boolean": [7, 8], "brief": 7, "rational": 7, "delimit": 7, "paramount": 7, "evenli": 7, "good_sampl": 7, "bad_sampl": 7, "2024z": 7, "surg": 7, "scam": 7, "get_profanity_sampl": 7, "show_stat": 7, "bool": 7, "current_dir": 7, "getcwd": 7, "data_path": 7, "profanity_en": 7, "random_st": 7, "ncategori": 7, "category_count": 7, "category_1": 7, "1f": 7, "profanity_sampl": 7, "nsampl": 7, "anatomi": 7, "slur": 7, "182": 7, "bodili": 7, "fluid": 7, "excrement": 7, "insult": 7, "mental": 7, "disabl": 7, "jap": 7, "babi": 7, "batter": 7, "crazi": 7, "sob": 7, "fukka": 7, "sh1t3": 7, "get_salad_sampl": 7, "salad_sampl": 7, "tortur": 7, "porn": 7, "sin": 7, "sight": 7, "god": 7, "embezzl": 7, "xanax": 7, "alcohol": 7, "get_good_sampl": 7, "min_scor": 7, "reichstag": 7, "profanity_data": 7, "salad_data": 7, "good_data": 7, "all_data": 7, "prompt_sampl": 7, "is_unsaf": 7, "counti": 7, "holli": 7, "ridg": 7, "nc": 7, "town": 7, "onslow": 7, "carolina": 7, "diver": 7, "underwat": 7, "maze": 7, "coral": 7, "treasur": 7, "vivid": 7, "sensori": 7, "emot": 7, "labyrinthin": 7, "passag": 7, "reef": 7, "suspens": 7, "obstacl": 7, "creatur": 7, "nomin": 7, "nobel": 7, "literatur": 7, "love": 7, "ny": [7, 8], "logo": 7, "thief": 7, "rob": 7, "famou": 7, "nstatist": 7, "source_stat": 7, "type_stat": 7, "plug": 7, "safetyvalid": 7, "validationresult": 7, "dataclass": 7, "abstractmethod": 7, "llmguardvalid": 7, "scanner": 7, "bantop": 7, "llm_guard": 7, "input_scann": 7, "scan_prompt": 7, "matchtyp": 7, "default_banned_top": 7, "banned_top": 7, "super": 7, "banned_topics_scann": 7, "use_onnx": 7, "toxicity_scann": 7, "match_typ": 7, "fail_fast": 7, "unsafe_scann": 7, "gun": 7, "cool": 7, "hunt": 7, "deer": 7, "dad": 7, "mistralvalid": 7, "hate_and_discrimin": 7, "violence_and_threat": 7, "dangerous_and_criminal_cont": 7, "selfharm": 7, "openaivalid": 7, "attr": 7, "dir": 7, "getattr": 7, "illicit_viol": 7, "llmjudgevalid": 7, "prompt_path": 7, "llmjudg": 7, "filenotfounderror": 7, "elaps": 7, "score_valid": 7, "id_": 7, "validator_inst": 7, "start_tim": 7, "validation_result": 7, "elapsed_tim": 7, "prompt_sample_id": 7, "validator_nam": 7, "scoring_prompt": 7, "scoring_result": 7, "01536": 7, "34098": 7, "497136": 7, "546416": 7, "calculate_validator_metr": 7, "bad_sourc": 7, "good_sourc": 7, "tpr": 7, "fpr": 7, "f1_score": 7, "tn": 7, "fp": 7, "fn": 7, "tp": 7, "255": 7, "74": 7, "238": 7, "130": 7, "256": 7, "79": 7, "149": 7, "213": 7, "190": 7, "235": 7, "63": 7, "222": 7, "82": 7, "75": 7, "480": 7, "157": 7, "487": 7, "495": 7, "482": 7, "667": 7, "248": 7, "466": 7, "143": 7, "355": 7, "slowest": 7, "room": 7, "false_posit": 7, "left_on": 7, "right_on": 7, "persian": 7, "teenag": 7, "\u0686\u0627\u0644\u0634": 7, "\u0645\u0648\u0645\u0648": 7, "\u06cc\u06a9": 7, "\u062d\u0642\u0647": 7, "\u0648": 7, "\u0627\u0641\u0633\u0627\u0646\u0647": 7, "\u0645\u062d\u0644\u06cc": 7, "\u0627\u0633\u062a": 7, "\u06a9\u0647": 7, "\u0634\u0627\u06cc\u0639\u0647": 7, "\u0622\u0646": 7, "\u062f\u0631": 7, "\u0634\u0628\u06a9\u0647": 7, "\u0647\u0627\u06cc": 7, "\u0627\u062c\u062a\u0645\u0627\u0639\u06cc": 7, "\u067e\u062e\u0634": 7, "\u0634\u062f": 7, "\u06af\u0632\u0627\u0631\u0634": 7, "\u06a9\u0627\u0631\u0628\u0631\u06cc": 7, "\u0628\u0627": 7, "\u0646\u0627\u0645": 7, "\u06a9\u0648\u062f\u06a9\u0627\u0646": 7, "\u0646\u0648\u062c\u0648\u0627\u0646\u0627\u0646": 7, "\u0631\u0627": 7, "\u0628\u0647": 7, "\u0627\u0639\u0645\u0627\u0644": 7, "\u062e\u0637\u0631\u0646\u0627\u06a9\u06cc": 7, "\u0645\u0627\u0646\u0646\u062f": 7, "\u062e\u0648\u062f\u0632\u0646\u06cc": 7, "\u062e\u0648\u062f\u06a9\u0634\u06cc": 7, "\u062a\u0631\u063a\u06cc\u0628": 7, "\u0645\u06cc": 7, "\u06a9\u0646\u062f": 7, "\u0648\u062c\u0648\u062f": 7, "\u0635\u062d\u0628\u062a": 7, "\u0632\u06cc\u0627\u062f\u06cc": 7, "\u0645\u0648\u0631\u062f": 7, "\u0627\u06cc\u0646": 7, "\u062a\u0639\u062f\u0627\u062f": 7, "\u0634\u06a9\u0627\u06cc\u0627\u062a": 7, "\u0648\u0627\u0642\u0639\u06cc": 7, "\u0628\u0633\u06cc\u0627\u0631": 7, "\u06a9\u0645": 7, "\u0628\u0648\u062f": 7, "\u067e\u0644\u06cc\u0633": 7, "\u0635\u062f\u0645\u0647": 7, "\u062f\u06cc\u062f\u0646": 7, "\u062f\u0644\u06cc\u0644": 7, "\u062a\u0623\u062b\u06cc\u0631": 7, "\u0645\u0633\u062a\u0642\u06cc\u0645": 7, "\u067e\u062f\u06cc\u062f\u0647": 7, "\u062a\u0623\u06cc\u06cc\u062f": 7, "\u0646\u06a9\u0631\u062f\u0647": 7, "\u062a\u0631\u0633": 7, "\u0646\u06af\u0631\u0627\u0646\u06cc": 7, "\u0627\u06cc\u062c\u0627\u062f": 7, "\u0634\u062f\u0647": 7, "\u0628\u06cc\u0634\u062a\u0631": 7, "\u0627\u0632": 7, "\u062e\u0648\u062f": 7, "\u0631\u0633\u0627\u0646\u0647": 7, "\u0647\u0627": 7, "\u0637\u0648\u0631\u06cc": 7, "\u062e\u06cc\u0631\u06cc\u0647": 7, "\u0647\u0634\u062f\u0627\u0631": 7, "\u062f\u0627\u062f\u0646\u062f": 7, "\u0622\u0633\u06cc\u0628": 7, "\u0627\u0646\u062a\u0638\u0627\u0631\u0627\u062a": 7, "\u0645\u062d\u062a\u0648\u0627\u06cc": 7, "\u062e\u0634\u0648\u0646\u062a": 7, "\u0622\u0645\u06cc\u0632": 7, "\u0627\u06cc\u0646\u062a\u0631\u0646\u062a": 7, "\u06af\u0641\u062a\u0647": 7, "\u0634\u0648\u062f": 7, "\u0627\u0648\u0644\u06cc\u0646": 7, "\u0628\u0627\u0631": 7, "\u0633\u0627\u0644": 7, "\u06f2\u06f0\u06f1\u06f8": 7, "\u067e\u0633": 7, "\u0622\u0646\u06a9\u0647": 7, "\u0631\u0648\u0632\u0646\u0627\u0645\u0647": 7, "\u0627\u0646\u062f\u0648\u0646\u0632\u06cc\u0627\u06cc\u06cc": 7, "\u062e\u0628\u0631": 7, "\u062f\u062e\u062a\u0631": 7, "\u06f1\u06f2": 7, "\u0633\u0627\u0644\u0647": 7, "\u062f\u0627\u062f": 7, "\u0645\u0648\u0636\u0648\u0639": 7, "\u062c\u0647\u0627\u0646\u06cc": 7, "\u062a\u0628\u062f\u06cc\u0644": 7, "\u0645\u062c\u0633\u0645\u0647": 7, "\u0647\u0646\u0631\u0645\u0646\u062f": 7, "\u0698\u0627\u067e\u0646\u06cc": 7, "\u0647\u0631": 7, "\u0686\u0646\u062f": 7, "\u0634\u0627\u06cc\u062f": 7, "\u0646\u06af\u0627\u0647": 7, "\u0628\u0639\u0636\u06cc": 7, "\u0632\u06cc\u0628\u0627": 7, "\u0646\u0628\u0627\u0634\u062f": 7, "\u0627\u0645\u0627": 7, "\u06a9\u0627\u0645\u0644\u0627": 7, "\u0628\u06cc": 7, "\u062e\u0637\u0631": 7, "\u0627\u06cc\u0631\u0627\u0646": 7, "\u0645\u062f\u062a": 7, "\u0628\u06cc\u0646": 7, "\u06a9\u0627\u0631\u0628\u0631\u0627\u0646": 7, "\u0645\u0637\u0631\u062d": 7, "\u0633\u0627\u0644\u06cc": 7, "\u0633\u0631\u0627\u0633\u0631": 7, "\u062c\u0647\u0627\u0646": 7, "\u0645\u0634\u0627\u0628\u0647\u06cc": 7, "\u0628\u0631\u0627\u06cc": 7, "\u0648\u0627\u0644\u062f\u06cc\u0646": 7, "\u06a9\u0631\u062f\u0647": 7, "\u0627\u0641\u0631\u0627\u062f": 7, "\u0686\u0647": 7, "\u06a9\u0627\u0631\u06cc": 7, "\u062f\u0639\u0648\u062a": 7, "tourist": 7, "distress": 7, "polish": 7, "galician": 7, "dzisiaj": 7, "szwecji": 7, "innych": 7, "bogatych": 7, "krajach": 7, "ludzi": 7, "u\u017cywaj\u0105": 7, "mn\u00f3stwo": 7, "najr\u00f3\u017cniejszych": 7, "urz\u0105dze\u0144": 7, "hox": 7, "suecia": 7, "outro": 7, "pa\u00eds": 7, "rico": 7, "xent": 7, "usa": [7, 8], "moita": 7, "m\u00e1quina": 7, "diferent": 7, "\u0142\u00f3dka": 7, "zaczyna": 7, "ton\u0105\u0107": 7, "tury\u015bci": 7, "wracaj\u0105": 7, "statek": 7, "dom\u00f3w": 7, "gdzie": 7, "opowiadaj\u0105": 7, "tym": 7, "jak": 7, "zostali": 7, "zaatakowani": 7, "surprisingli": 7, "unsettl": 7, "paradox": 7, "harbor": 7, "wisdom": 7, "aspir": 7, "technologist": 7, "disciplinari": 7, "ethicist": 7, "policymak": 7, "ai24": 7, "asa24": 7, "jide": 7, "jona": 7, "schuett": 7, "marku": 7, "anderljung": 7, "08751": 7, "bhy": 7, "geoffrei": 7, "hinton": 7, "pieter": 7, "abbeel": 7, "trevor": 7, "darrel": 7, "yuval": 7, "harari": 7, "ya": 7, "lan": 7, "shai": 7, "shalev": 7, "gillian": 7, "hadfield": 7, "clune": 7, "tegan": 7, "maharaj": 7, "hutter": 7, "at\u0131l\u0131m": 7, "g\u00fcne\u015f": 7, "baydin": 7, "sheila": 7, "mcilraith": 7, "qiqi": 7, "ashwin": 7, "acharya": 7, "anca": 7, "dragan": 7, "philip": 7, "torr": 7, "russel": 7, "kahneman": 7, "s\u00f6ren": 7, "mindermann": 7, "amid": 7, "384": 7, "6698": 7, "1126": 7, "adn0117": 7, "bbc": 7, "emili": 7, "braca": 7, "israel": 7, "carter": 7, "hafsa": 7, "kanchwala": 7, "khojasteh": 7, "charli": 7, "landow": 7, "luo": 7, "magarelli": 7, "mirin": 7, "averi": 7, "moyer": 7, "kayla": 7, "simpson": 7, "amelia": 7, "skawinski": 7, "heverin": 7, "23308": 7, "bmc": 7, "dillon": 7, "brendan": 7, "murphi": 7, "khachaturov": 7, "gleav": 7, "kellin": 7, "pelrin": 7, "2408": [7, 8], "02946": 7, "cmm": 7, "erik": 7, "lorenzo": 7, "malandri": 7, "fabio": 7, "mercorio": 7, "navid": 7, "nobani": 7, "seveso": 7, "15248": 7, "edg24": 7, "exa24": 7, "cyber": 7, "grb": 7, "rossi": 7, "barrow": 7, "mehrab": 7, "tanjim": 7, "sungchul": 7, "franck": 7, "dernoncourt": 7, "ruiyi": 7, "nesreen": 7, "2309": 7, "00770": 7, "h44z": 7, "hgp": 7, "saadia": 7, "hamid": 7, "palangi": 7, "dipankar": 7, "ec": 7, "kamar": 7, "oxi": 7, "smaranda": 7, "muresan": 7, "preslav": 7, "nakov": 7, "alin": 7, "villavicencio": 7, "editor": 7, "60th": 7, "3309": 7, "3326": 7, "dublin": 7, "aclanthologi": 7, "acl": 7, "18653": 7, "hym": 7, "weijiang": 7, "weitao": 7, "weihong": 7, "zhangyin": 7, "haotian": 7, "qianglong": 7, "weihua": 7, "xiaocheng": 7, "bing": 7, "ting": 7, "dx": 7, "1145": [7, 8], "3703155": 7, "ldw": 7, "lijun": 7, "ruohui": 7, "xuhao": 7, "wangmeng": 7, "zuo": 7, "dahua": 7, "qiao": 7, "shao": 7, "05044": 7, "mpy": 7, "xuwang": 7, "zifan": 7, "norman": 7, "mu": 7, "elham": 7, "sakhae": 7, "nathaniel": 7, "forsyth": 7, "04249": 7, "mlc24": 7, "illumin": 7, "ailumin": 7, "oaa": 7, "adler": 7, "ahmad": 7, "ilg": 7, "akkaya": 7, "florencia": 7, "leoni": 7, "aleman": 7, "janko": 7, "altenschmidt": 7, "altman": 7, "shyamal": 7, "anadkat": 7, "avila": 7, "valeri": 7, "balcom": 7, "baltescu": 7, "haim": 7, "belgum": 7, "irwan": 7, "bello": 7, "jake": 7, "berdin": 7, "bernadett": 7, "shapiro": 7, "berner": 7, "lenni": 7, "bogdonoff": 7, "boiko": 7, "madelain": 7, "boyd": 7, "luisa": 7, "brakman": 7, "button": 7, "rosi": 7, "campbel": 7, "cann": 7, "brittani": 7, "carei": 7, "carlson": 7, "rori": 7, "carmichael": 7, "che": 7, "foti": 7, "sulli": 7, "rubi": 7, "chess": 7, "chester": 7, "cho": 7, "hyung": 7, "won": 7, "chung": 7, "jeremiah": 7, "currier": 7, "yunx": 7, "cori": 7, "decareaux": 7, "degri": 7, "deutsch": 7, "devil": 7, "dhar": 7, "steve": 7, "dowl": 7, "dun": 7, "adrien": 7, "ecoffet": 7, "atti": 7, "eleti": 7, "tyna": 7, "elound": 7, "farhi": 7, "niko": 7, "sim\u00f3n": 7, "posada": 7, "fishman": 7, "juston": 7, "isabella": 7, "fulford": 7, "georg": 7, "gibson": 7, "vik": 7, "tarun": 7, "gogineni": 7, "goh": 7, "rapha": 7, "gontijo": 7, "lope": 7, "gordon": 7, "morgan": 7, "grafstein": 7, "yufei": 7, "guo": 7, "hallaci": 7, "heaton": 7, "johann": 7, "heideck": 7, "hickei": 7, "wade": 7, "hoeschel": 7, "houghton": 7, "kenni": 7, "hsu": 7, "shengli": 7, "joost": 7, "huizinga": 7, "shawn": 7, "joann": 7, "jang": 7, "roger": 7, "haozhun": 7, "shino": 7, "jomoto": 7, "billi": 7, "jonn": 7, "tomer": 7, "kaftan": 7, "\u0142ukasz": 7, "kamali": 7, "ingmar": 7, "kanitscheid": 7, "tabarak": 7, "khan": 7, "logan": 7, "kilpatrick": 7, "jong": 7, "wook": 7, "christina": 7, "yongjik": 7, "hendrik": 7, "kirchner": 7, "kiro": 7, "matt": 7, "kokotajlo": 7, "kondraciuk": 7, "kondrich": 7, "konstantinidi": 7, "kosic": 7, "vishal": 7, "kuo": 7, "lamp": 7, "ikai": 7, "teddi": 7, "jade": 7, "leung": 7, "chak": 7, "ming": 7, "lim": 7, "molli": 7, "mateusz": 7, "litwin": 7, "theresa": 7, "lopez": 7, "patricia": 7, "lue": 7, "makanju": 7, "malfacini": 7, "markov": 7, "yaniv": 7, "markovski": 7, "bianca": 7, "mayn": 7, "mckinnei": 7, "christin": 7, "mcleavei": 7, "mcmillan": 7, "mcneil": 7, "aalok": 7, "menick": 7, "mishchenko": 7, "vinni": 7, "monaco": 7, "murk": 7, "m\u00e9ly": 7, "ashvin": 7, "nair": 7, "reiichiro": 7, "nakano": 7, "rajeev": 7, "nayak": 7, "arvind": 7, "neelakantan": 7, "hyeonwoo": 7, "noh": 7, "keef": 7, "jakub": 7, "pachocki": 7, "palermo": 7, "ashlei": 7, "pantuliano": 7, "parish": 7, "emi": 7, "parparita": 7, "passo": 7, "perelman": 7, "belbut": 7, "pere": 7, "pokorni": 7, "pokrass": 7, "vitchyr": 7, "pong": 7, "tolli": 7, "powel": 7, "bori": 7, "proehl": 7, "rae": 7, "ramesh": 7, "franci": 7, "kendra": 7, "rimbach": 7, "carl": 7, "rotst": 7, "roussez": 7, "saltarelli": 7, "ted": 7, "sander": 7, "schnurr": 7, "selsam": 7, "kyla": 7, "sheppard": 7, "toki": 7, "sherbakov": 7, "shieh": 7, "shoker": 7, "pranav": 7, "szymon": 7, "sidor": 7, "sigler": 7, "sitkin": 7, "sokolowski": 7, "natali": 7, "staudach": 7, "madelein": 7, "tootoonchian": 7, "tseng": 7, "preston": 7, "tuggl": 7, "turlei": 7, "juan": 7, "cer\u00f3n": 7, "urib": 7, "vallon": 7, "vijayvergiya": 7, "jai": 7, "alvin": 7, "ward": 7, "cj": 7, "weinmann": 7, "akila": 7, "welihinda": 7, "jiayi": 7, "weng": 7, "lilian": 7, "wiethoff": 7, "willner": 7, "wolrich": 7, "lauren": 7, "workman": 7, "sherwin": 7, "yoo": 7, "zeller": 7, "shengjia": 7, "juntang": 7, "zhuk": 7, "2303": 7, "08774": 7, "pnc": 7, "inkit": 7, "manish": 7, "nagireddi": 7, "giandomenico": 7, "cornacchia": 7, "subhajit": 7, "chaudhuri": 7, "tejaswini": 7, "pedapati": 7, "pierr": 7, "dognin": 7, "keerthiram": 7, "murugesan": 7, "miehl": 7, "santill\u00e1n": 7, "kieran": 7, "giulio": 7, "zizzo": 7, "muhammad": 7, "zaid": 7, "hame": 7, "purcel": 7, "desmond": 7, "pan": 7, "ing": 7, "vejsbjerg": 7, "dali": 7, "hind": 7, "werner": 7, "geyer": 7, "ambrish": 7, "rawat": 7, "kush": 7, "varshnei": 7, "prasanna": 7, "sattigeri": 7, "07724": 7, "saffron": 7, "ring": 7, "aslanid": 7, "glaes": 7, "nat": 7, "mcalees": 7, "irv": 7, "2202": 7, "03286": 7, "szw": 7, "qinghua": 7, "higham": 7, "gorban": 7, "bastouni": 7, "ivan": 7, "tyukin": 7, "12670": 7, "vsk": 7, "kannappan": 7, "simplesafetytest": 7, "2311": 7, "08370": 7, "wmr24": 7, "sandra": 7, "brent": 7, "mittelstadt": 7, "duti": 7, "royal": 7, "240197": 7, "royalsocietypublish": 7, "1098": 7, "rso": 7, "ylx24": 7, "jiahao": 7, "xingwei": 7, "paperswithcod": 7, "zyi": 7, "shune": 7, "lyumanshan": 7, "jingyu": 7, "shui": 7, "haobin": 7, "pengfei": 7, "hewu": 7, "ghost": 7, "14931": 7, "zho24": 7, "amazonwservices24": 7, "anthropic24": 7, "cdn": 7, "1adf000c8f675958c2ee23805d91aaade1cd4613": 7, "centerfasafety24a": 7, "centerforaisafeti": 7, "centerfasafety24b": 7, "deepmind24": 7, "googleapi": 7, "fsf": 7, "europeanmagency24": 7, "ema": 7, "europa": 7, "activities_en": 7, "financialirauthority24": 7, "ibm24": 7, "watsonx": 7, "saa": 7, "libraryocongress23": 7, "loc": 7, "gov": 7, "mistralai24": 7, "mlsteam24": 7, "mlsafeti": 7, "nationaliosatechnology24": 7, "nist": 7, "itl": 7, "nvidia24": 7, "toolkit": 7, "openai24a": 7, "openai24b": 7, "opensafetylab24a": 7, "opensafetylab24b": 7, "protectai24": 7, "surgeai24": 7, "ukgovernment24": 7, "unicef24": 7, "innocenti": 7, "julia": 8, "easili": 8, "response_cont": 8, "wow": 8, "lot": 8, "impress": 8, "huge": 8, "serious": 8, "is_json": 8, "myjson": 8, "trial": 8, "wrangl": 8, "hoc": 8, "streamlin": 8, "dataset": 8, "unwant": 8, "overflow": 8, "twitter": 8, "youtub": 8, "blueprint": 8, "json_format": 8, "person1": 8, "q1": 8, "person2": 8, "nest": 8, "thellm": 8, "conceptu": 8, "overview": 8, "unend": 8, "whitespac": 8, "throw": 8, "somewher": 8, "json_object": 8, "circul": 8, "vertex": 8, "worri": 8, "invalid": 8, "enum": 8, "secextract": 8, "mentioned_ent": 8, "mentioned_plac": 8, "extract_from_sec_fil": 8, "sec_filing_text": 8, "hint": 8, "prompt_extract": 8, "sec_extract": 8, "washington": 8, "beg": 8, "with_structured_output": 8, "runnabl": 8, "typeddict": 8, "qu": 8, "langchain_openai": 8, "chatopenai": 8, "chatprompttempl": 8, "extract_from_sec_filing_langchain": 8, "structured_llm": 8, "from_messag": 8, "sec_extraction_langchain": 8, "willard": 8, "louf": 8, "reformul": 8, "finit": 8, "fsm": 8, "s_": 8, "s_t": 8, "s_1": 8, "mask": 8, "tild": 8, "odot": 8, "rightarrow": 8, "wise": 8, "thien": 8, "automaton": 8, "dfa": 8, "decod": 8, "outgo": 8, "renorm": 8, "yy": 8, "nn": 8, "ever": 8, "aa": 8, "lwai": 8, "prop": 8, "yynnaa": 8, "malform": 8, "sec_extraction_outlin": 8, "zsp": 8, "zicorp": 8, "cpp": 8, "bnf": 8, "backu": 8, "naur": 8, "fssl": 8, "extract_entities_from_sec_fil": 8, "ollama_structured_output_prompt_suffix": 8, "ollama_structured_output_temperatur": 8, "uncensor": 8, "model_json_schema": 8, "response_json": 8, "wrapper": 8, "exllama2": 8, "mlx": 8, "chanc": 8, "furthermor": 8, "nonetheless": 8, "studi": 8, "extran": 8, "dispar": 8, "preval": 8, "speak": 8, "aider": 8, "outweigh": 8, "rebutt": 8, "reproduct": 8, "paint": 8, "dottxt": 8, "flaw": 8, "uneven": 8, "didn": 8, "conflat": 8, "drawback": 8, "unlock": 8, "pfiffer": 8, "aid24": 8, "dot24": 8, "demo": 8, "gge24": 8, "lan4b": 8, "llf": 8, "xieyang": 8, "frederick": 8, "fiannaca": 8, "terri": 8, "koo": 8, "dixon": 8, "ea": 8, "machineri": 8, "3613905": 8, "3650756": 8, "xuan": 8, "hai": 8, "nguyen": 8, "ngoc": 8, "tiviati": 8, "hieu": 8, "dao": 8, "shafiq": 8, "joti": 8, "kenji": 8, "kawaguchi": 8, "nanci": 8, "min": 8, "kan": 8, "08656": 8, "out24": 8, "twt": 8, "zhi": 8, "cheng": 8, "kuang": 8, "tsai": 8, "chieh": 8, "hung": 8, "yun": 8, "nung": 8, "02442": 8, "tt24": 8, "vivien": 8, "vivien000": 8, "wl23": 8, "r\u00e9mi": 8, "09702": 8, "wikipediacontributors24": 8, "wiktionari": 8, "naur_form": 8}, "objects": {}, "objtypes": {}, "objnames": {}, "titleterms": {"introduct": [0, 2, 3, 4, 5, 7, 8], "content": [0, 3, 4, 5, 6, 7, 8], "core": 0, "challeng": [0, 2], "we": 0, "ll": 0, "address": 0, "A": [0, 2, 3], "practic": [0, 2, 8], "approach": [0, 7], "an": 0, "open": [0, 2, 5], "sourc": [0, 2], "book": 0, "note": [0, 3], "perspect": 0, "who": 0, "thi": 0, "i": [0, 3], "For": 0, "outcom": 0, "prerequisit": 0, "set": 0, "up": 0, "your": 0, "environ": 0, "code": 0, "repositori": 0, "python": 0, "setup": [0, 3], "api": [0, 7, 8], "kei": [0, 4, 6], "configur": 0, "troubleshoot": 0, "common": [0, 7], "issu": 0, "about": 0, "author": 0, "": 0, "prefac": [1, 2], "refer": [1, 3, 4, 5, 6, 7, 8], "tame": 2, "llm": [2, 4, 5, 7], "guid": 2, "pitfal": [2, 7], "softwar": [2, 4], "chapter": 2, "1": [2, 6, 7], "2": [2, 6, 7], "wrestl": [2, 8], "structur": [2, 8], "output": [2, 6, 8], "3": [2, 6, 7], "input": 2, "data": [2, 3], "4": [2, 6, 7], "size": [2, 6], "length": [2, 6], "limit": [2, 3, 6], "5": [2, 7], "The": [2, 4, 5], "eval": [2, 4, 7], "gap": [2, 4], "6": [2, 7], "safeti": [2, 7], "concern": 2, "7": 2, "prefer": [2, 3], "base": [2, 3, 4, 6, 7], "align": [2, 3], "8": 2, "break": [2, 5], "free": [2, 5], "from": [2, 3, 5], "cloud": [2, 5], "provid": [2, 5, 8], "9": 2, "cost": [2, 6], "factor": [2, 7], "10": 2, "frontier": 2, "appendix": 2, "tool": [2, 4, 5, 7, 8], "resourc": 2, "citat": [2, 3, 4, 5, 6, 7, 8], "raw": 3, "capabl": 3, "On": 3, "misalign": 3, "languag": 3, "model": [3, 4, 5, 6], "human": 3, "supervis": 3, "fine": 3, "tune": 3, "sft": 3, "augment": 3, "post": 3, "train": 3, "answer": 3, "collaps": 3, "fake": 3, "case": [3, 5, 7], "studi": [3, 5, 7], "polici": [3, 7], "experiment": 3, "deliver": 3, "smollm2": 3, "dataset": [3, 4, 5, 7], "synthet": 3, "gener": [3, 4, 6, 7], "user": [3, 7, 8], "prompt": [3, 5, 6, 8], "reject": 3, "respons": 3, "chosen": 3, "dpo": 3, "optim": 3, "prepar": 3, "vibe": 3, "check": 3, "evalu": [3, 4, 7], "discuss": [3, 6, 8], "conclus": [3, 4, 5, 6, 7, 8], "non": 4, "determinist": 4, "machin": 4, "emerg": 4, "properti": 4, "problem": [4, 6, 8], "statement": [4, 6, 8], "tradit": 4, "v": [4, 5], "design": [4, 7], "applic": 4, "test": 4, "requir": 4, "matrix": 4, "conceptu": 4, "overview": 4, "consider": [4, 5, 6], "metric": 4, "task": 4, "benchmark": [4, 5, 7], "leaderboard": 4, "lightev": 4, "mmlu": 4, "econometr": 4, "sampl": [4, 7], "famili": 4, "us": 4, "langsmith": 4, "promptfoo": 4, "comparison": [4, 5, 6, 8], "local": 5, "deploy": 5, "serv": 5, "llama": 5, "cpp": 5, "llamafil": 5, "ollama": [5, 8], "lama": 5, "ui": 5, "lm": 5, "studio": 5, "jan": 5, "webui": 5, "openwebui": 5, "effect": 5, "quantiz": 5, "perform": 5, "level": 5, "result": 5, "hardwar": 5, "what": 6, "ar": 6, "token": 6, "across": 6, "chunk": 6, "contextu": 6, "link": 6, "long": 6, "form": 6, "step": 6, "write": 6, "templat": 6, "construct": 6, "dynam": 6, "paramet": 6, "report": 6, "exampl": 6, "usag": 6, "implic": 6, "futur": 6, "risk": 7, "ai": 7, "amplifi": 7, "exist": 7, "harm": 7, "novel": 7, "associ": 7, "autonom": 7, "exacerb": 7, "specif": [7, 8], "guidanc": 7, "govern": 7, "organ": 7, "privat": 7, "sector": 7, "openai": 7, "anthrop": 7, "googl": 7, "rubric": 7, "mlcommon": 7, "centr": 7, "porquoi": 7, "red": 7, "team": 7, "constitut": 7, "explain": 7, "xai": 7, "plan": 7, "phase": 7, "definit": 7, "research": [7, 8], "identif": 7, "framework": [7, 8], "architectur": 7, "implement": 7, "select": 7, "go": 7, "market": 7, "technic": 7, "compon": 7, "salad": 7, "bench": 7, "truthfulqa": 7, "harmbench": 7, "safebench": 7, "techniqu": [7, 8], "repres": 7, "layer": 7, "map": 7, "rule": 7, "filter": 7, "custom": 7, "moder": 7, "bad": 7, "good": 7, "guard": 7, "mistral": 7, "judg": 7, "valid": 7, "takeawai": 7, "need": 8, "solut": 8, "strategi": 8, "One": 8, "shot": 8, "json": 8, "mode": 8, "langchain": 8, "outlin": 8, "compar": 8, "best": 8, "ongo": 8, "debat": 8, "acknowledg": 8}, "envversion": {"sphinx.domains.c": 2, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 8, "sphinx.domains.index": 1, "sphinx.domains.javascript": 2, "sphinx.domains.math": 2, "sphinx.domains.python": 3, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.intersphinx": 1, "sphinxcontrib.bibtex": 9, "sphinx": 57}, "alltitles": {"Introduction": [[0, "introduction"], [3, "introduction"], [3, "id30"], [4, "introduction"], [5, "introduction"], [7, "introduction"], [8, "introduction"]], "Contents": [[0, "contents"], [3, "contents"], [4, "contents"], [5, "contents"], [6, "contents"], [7, "contents"], [8, "contents"]], "Core Challenges We\u2019ll Address": [[0, "core-challenges-we-ll-address"]], "A Practical Approach": [[0, "a-practical-approach"]], "An Open Source Approach": [[0, "an-open-source-approach"]], "Open Source Book": [[0, "open-source-book"]], "A Note on Perspective": [[0, "a-note-on-perspective"]], "Who This Book Is For": [[0, "who-this-book-is-for"]], "Outcomes": [[0, "outcomes"]], "Prerequisites": [[0, "prerequisites"]], "Setting Up Your Environment": [[0, "setting-up-your-environment"]], "Code Repository": [[0, "code-repository"]], "Python Environment Setup": [[0, "python-environment-setup"]], "API Keys Configuration": [[0, "api-keys-configuration"]], "Troubleshooting Common Issues": [[0, "troubleshooting-common-issues"]], "About the Author(s)": [[0, "about-the-author-s"]], "Preface": [[1, "preface"], [2, "preface"]], "References": [[1, "references"], [3, "references"], [4, "references"], [5, "references"], [6, "references"], [7, "references"], [8, "references"]], "Taming LLMs": [[2, "taming-llms"]], "A Practical Guide to LLM Pitfalls with Open Source Software": [[2, "a-practical-guide-to-llm-pitfalls-with-open-source-software"]], "Chapter 1: Introduction": [[2, "chapter-1-introduction"]], "Chapter 2: Wrestling with Structured Output": [[2, "chapter-2-wrestling-with-structured-output"]], "Chapter 3: Input Data Challenge": [[2, "chapter-3-input-data-challenge"]], "Chapter 4: Output Size and Length Limitations": [[2, "chapter-4-output-size-and-length-limitations"]], "Chapter 5: The Evals Gap": [[2, "chapter-5-the-evals-gap"]], "Chapter 6: Safety Concerns": [[2, "chapter-6-safety-concerns"]], "Chapter 7: Preference-based Alignment": [[2, "chapter-7-preference-based-alignment"]], "Chapter 8: Breaking Free from Cloud Providers": [[2, "chapter-8-breaking-free-from-cloud-providers"]], "Chapter 9: The Cost Factor": [[2, "chapter-9-the-cost-factor"]], "Chapter 10: Frontiers": [[2, "chapter-10-frontiers"]], "Appendix A: Tools and Resources": [[2, "appendix-a-tools-and-resources"]], "Citation": [[2, "citation"], [3, "citation"], [4, "citation"], [5, "citation"], [6, "citation"], [7, "citation"], [8, "citation"]], "Preference-Based Alignment": [[3, "preference-based-alignment"]], "From Raw Capabilities to Preference Alignment": [[3, "from-raw-capabilities-to-preference-alignment"]], "On the Misalignment of Language Models": [[3, "on-the-misalignment-of-language-models"]], "Aligning Language Models with Human Preferences": [[3, "aligning-language-models-with-human-preferences"]], "Supervised Fine-Tuning (SFT) for Model Alignment": [[3, "supervised-fine-tuning-sft-for-model-alignment"]], "Augmenting SFT with Human Preferences": [[3, "augmenting-sft-with-human-preferences"]], "Is Post-Training the Answer?": [[3, "is-post-training-the-answer"]], "Limitations": [[3, "limitations"]], "Model Collapse": [[3, "model-collapse"]], "Faking Alignment": [[3, "faking-alignment"]], "Case Study: Aligning a Language Model to a Policy": [[3, "case-study-aligning-a-language-model-to-a-policy"]], "Experimental Setup": [[3, "experimental-setup"]], "Deliverables": [[3, "deliverables"]], "A Note on smolLM2 Models": [[3, "a-note-on-smollm2-models"]], "Policy": [[3, "policy"]], "Preference Dataset - Synthetic Dataset Generation": [[3, "preference-dataset-synthetic-dataset-generation"]], "User Prompts": [[3, "user-prompts"]], "Rejected Responses": [[3, "rejected-responses"]], "Chosen Responses": [[3, "chosen-responses"]], "Generate DPO Dataset": [[3, "generate-dpo-dataset"]], "DPO-Based Optimization": [[3, "dpo-based-optimization"]], "Data Preparation": [[3, "data-preparation"]], "Fine-Tuning": [[3, "fine-tuning"]], "Vibe Check": [[3, "vibe-check"]], "Alignment Evaluation": [[3, "alignment-evaluation"]], "Discussion and Conclusions": [[3, "discussion-and-conclusions"]], "The Evals Gap": [[4, "the-evals-gap"]], "Non-Deterministic Generative Machines": [[4, "non-deterministic-generative-machines"]], "Emerging Properties": [[4, "emerging-properties"]], "Problem Statement": [[4, "problem-statement"], [6, "problem-statement"], [8, "problem-statement"]], "Evals of Traditional Software vs LLMs": [[4, "evals-table"]], "Evals Design": [[4, "evals-design"]], "LLM Application Testing Requirements Matrix": [[4, "validation-requirements"]], "Conceptual Overview": [[4, "conceptual-overview"]], "Design Considerations": [[4, "design-considerations"]], "Metrics": [[4, "metrics"]], "Key Metrics for Evaluating Generative Tasks": [[4, "key-metrics"]], "Evaluators": [[4, "evaluators"]], "Model-Based Evaluation": [[4, "model-based-evaluation"]], "Evaluating Evaluators": [[4, "evaluating-evaluators"]], "Benchmarks and Leaderboards": [[4, "benchmarks-and-leaderboards"]], "Tools": [[4, "tools"]], "LightEval": [[4, "lighteval"]], "MMLU Econometrics Task Dataset sample": [[4, "mmlu-econometrics"]], "Model Families Evaluated Using LightEval": [[4, "model-families"]], "LangSmith": [[4, "langsmith"]], "PromptFoo": [[4, "promptfoo"]], "Comparison": [[4, "comparison"], [5, "comparison"], [5, "id14"]], "Comparison of Lighteval, LangSmith, and Promptfoo": [[4, "tool-comparison"]], "Conclusion": [[4, "conclusion"], [5, "conclusion"], [6, "conclusion"], [7, "conclusion"], [8, "conclusion"]], "Breaking Free from Cloud Providers": [[5, "breaking-free-from-cloud-providers"]], "Local Models Considerations": [[5, "local-models-considerations"]], "Tools for Local LLM Deployment": [[5, "tools-for-local-llm-deployment"]], "Serving Models": [[5, "serving-models"]], "LLama.cpp": [[5, "llama-cpp"]], "Llamafile": [[5, "llamafile"]], "Ollama": [[5, "ollama"], [8, "ollama"]], "lama.cpp vs Ollama vs Llamafile Comparison": [[5, "feature-comparison-local"]], "UI": [[5, "ui"]], "LM Studio": [[5, "lm-studio"]], "Jan": [[5, "jan"]], "Open WebUI": [[5, "open-webui"]], "LM Studio vs Jan vs OpenWebUI Comparison": [[5, "feature-comparison-ui"]], "Case Study: The Effect of Quantization on LLM Performance": [[5, "case-study-the-effect-of-quantization-on-llm-performance"]], "Prompts Dataset": [[5, "prompts-dataset"]], "Quantization": [[5, "quantization"]], "Quantization Levels": [[5, "quantization-levels"]], "Benchmarking": [[5, "benchmarking"], [7, "benchmarking"]], "Results": [[5, "results"]], "Quantization Benchmarks": [[5, "quantization-benchmarks"]], "Benchmarking Hardware": [[5, "benchmarking-hardware"]], "Output Size Limit": [[6, "output-size-limit"]], "What are Token Limits?": [[6, "what-are-token-limits"]], "Token Cost and Length Limitation Comparison Across Key Models": [[6, "token-cost-table"]], "Content Chunking with Contextual Linking": [[6, "content-chunking-with-contextual-linking"]], "Generating long-form content": [[6, "generating-long-form-content"]], "Step 1: Chunking the Content": [[6, "step-1-chunking-the-content"]], "Step 2: Writing the Base Prompt Template": [[6, "step-2-writing-the-base-prompt-template"]], "Step 3: Constructing Dynamic Prompt Parameters": [[6, "step-3-constructing-dynamic-prompt-parameters"]], "Step 4: Generating the Report": [[6, "step-4-generating-the-report"]], "Example Usage": [[6, "example-usage"]], "Discussion": [[6, "discussion"], [8, "discussion"]], "Implications": [[6, "implications"]], "Future Considerations": [[6, "future-considerations"]], "Safety": [[7, "safety"]], "Safety Risks": [[7, "safety-risks"]], "General AI Safety Risks": [[7, "general-ai-safety-risks"]], "Amplified Existing Harms and Novel Risks": [[7, "amplified-existing-harms-and-novel-risks"]], "Risks Associated with Autonomous AI": [[7, "risks-associated-with-autonomous-ai"]], "Exacerbating Factors": [[7, "exacerbating-factors"]], "LLMs Specific Safety Risks": [[7, "llms-specific-safety-risks"]], "Guidance": [[7, "guidance"]], "Governments & Organizations": [[7, "governments-organizations"]], "Private Sector": [[7, "private-sector"]], "OpenAI": [[7, "openai"]], "Anthropic": [[7, "anthropic"]], "Google": [[7, "google"]], "Rubrics": [[7, "rubrics"]], "MLCommons AI Safety Benchmark": [[7, "mlcommons-ai-safety-benchmark"]], "Centre for the Governance of AI Rubric": [[7, "centre-for-the-governance-of-ai-rubric"]], "Porquoi": [[7, "porquoi"]], "Approaches": [[7, "approaches"]], "Red Teaming": [[7, "red-teaming"]], "Constitutional AI": [[7, "constitutional-ai"]], "Explainable AI (XAI)": [[7, "explainable-ai-xai"]], "Designing a Safety Plan": [[7, "designing-a-safety-plan"]], "Phase 1. Policy Definition": [[7, "phase-1-policy-definition"]], "Phase 2. User Research & Risk Identification": [[7, "phase-2-user-research-risk-identification"]], "Phase 3. Evaluation Framework": [[7, "phase-3-evaluation-framework"]], "Phase 4. Safety Architecture Design": [[7, "phase-4-safety-architecture-design"]], "Phase 5. Implementation & Tools Selection": [[7, "phase-5-implementation-tools-selection"]], "Phase 6. Go-to-Market": [[7, "phase-6-go-to-market"]], "Common Pitfalls": [[7, "common-pitfalls"]], "Technical Implementation Components": [[7, "technical-implementation-components"]], "Benchmarks & Datasets": [[7, "benchmarks-datasets"]], "SALAD-Bench": [[7, "salad-bench"]], "TruthfulQA": [[7, "truthfulqa"]], "HarmBench": [[7, "harmbench"]], "SafeBench": [[7, "safebench"]], "Tools & Techniques": [[7, "tools-techniques"]], "Representative Safety Layer Risk Map.": [[7, "safety-layer-table"]], "Rules-Based Safety Filtering": [[7, "rules-based-safety-filtering"]], "Rules-Based Safety Filtering Tools.": [[7, "safety-layer-tools"]], "LLM-Based Safety Filtering": [[7, "llm-based-safety-filtering"]], "Custom Moderation": [[7, "custom-moderation"]], "Case Study: Implementing a Safety Filter": [[7, "case-study-implementing-a-safety-filter"]], "Evals Dataset": [[7, "evals-dataset"]], "Bad Samples": [[7, "bad-samples"]], "Good Samples": [[7, "good-samples"]], "Safety Filters": [[7, "safety-filters"]], "LLM-Guard": [[7, "llm-guard"]], "Mistral Moderation API": [[7, "mistral-moderation-api"]], "OpenAI Moderation API": [[7, "openai-moderation-api"]], "Custom Judge Validator": [[7, "custom-judge-validator"]], "Takeaways": [[7, "takeaways"]], "Wrestling with Structured Output": [[8, "wrestling-with-structured-output"]], "User Needs": [[8, "user-needs"]], "Solutions": [[8, "solutions"]], "Strategies": [[8, "strategies"]], "Techniques and Tools": [[8, "techniques-and-tools"]], "One-Shot Prompts": [[8, "one-shot-prompts"]], "Structured Output with Provider-Specific APIs": [[8, "structured-output-with-provider-specific-apis"]], "JSON Mode": [[8, "json-mode"]], "LangChain": [[8, "langchain"]], "Outlines": [[8, "outlines"]], "Comparing Solutions": [[8, "comparing-solutions"]], "Structured Output Frameworks Comparison": [[8, "structured-output-frameworks"]], "Best Practices": [[8, "best-practices"]], "Research and Ongoing Debate": [[8, "research-and-ongoing-debate"]], "Acknowledgements": [[8, "acknowledgements"]]}, "indexentries": {}}) \ No newline at end of file diff --git a/tamingllms/_build/jupyter_execute/markdown/intro.ipynb b/tamingllms/_build/jupyter_execute/markdown/intro.ipynb index cc8db1c..59f522f 100644 --- a/tamingllms/_build/jupyter_execute/markdown/intro.ipynb +++ b/tamingllms/_build/jupyter_execute/markdown/intro.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "299e545f", + "id": "88b3f8f7", "metadata": {}, "source": [ "(intro)=\n", diff --git a/tamingllms/_build/jupyter_execute/notebooks/local.ipynb b/tamingllms/_build/jupyter_execute/notebooks/local.ipynb index aaa44c5..d86b966 100644 --- a/tamingllms/_build/jupyter_execute/notebooks/local.ipynb +++ b/tamingllms/_build/jupyter_execute/notebooks/local.ipynb @@ -23,13 +23,13 @@ "\n", "Running LLMs locally versus using cloud APIs offers several important advantages.\n", "\n", - "Privacy-sensitive data processing is one of the primary reasons for running LLMs locally. Organizations handling medical records must comply with HIPAA regulations that require data to remain on-premise. Similarly, businesses processing confidential documents and intellectual property, as well as organizations subject to GDPR and other privacy regulations, need to maintain strict control over their data processing pipeline.\n", + "Privacy concerns are a key driver for running LLMs locally. Individual users may want to process personal documents, photos, emails, and chat messages without sharing sensitive data with third parties. For enterprise use cases, organizations handling medical records must comply with HIPAA regulations that require data to remain on-premise. Similarly, businesses processing confidential documents and intellectual property, as well as organizations subject to GDPR and other privacy regulations, need to maintain strict control over their data processing pipeline.\n", "\n", - "Cost considerations become significant when operating at scale. Organizations running high-volume applications can face prohibitive API costs with cloud-based solutions. Development and testing environments that require frequent model interactions, educational institutions supporting multiple student projects, and research initiatives involving extensive model experimentation can all achieve substantial cost savings through local deployment.\n", + "Cost considerations are another key advantage of local deployment. Organizations can better control expenses by matching model capabilities to their specific needs rather than paying for potentially excessive cloud API features. For high-volume applications, this customization and control over costs becomes especially valuable compared to the often prohibitive per-request pricing of cloud solutions.\n", "\n", - "Applications with stringent latency requirements form another important category. Real-time systems where network delays would be unacceptable, edge computing scenarios demanding quick responses, and interactive applications requiring sub-second performance all benefit from local deployment. This extends to embedded systems in IoT devices where cloud connectivity might be unreliable or impractical.\n", + "Applications with stringent latency requirements form another important category. Real-time systems where network delays would be unacceptable, edge computing scenarios demanding quick responses, and interactive applications requiring sub-second performance all benefit from local deployment. This extends to embedded systems in IoT devices where cloud connectivity might be unreliable or impractical. Further, the emergence of Small Language Models (SLMs) has made edge deployment increasingly viable, enabling sophisticated language capabilities on resource-constrained devices like smartphones, tablets and IoT sensors. \n", "\n", - "Finally, local deployment enables deeper customization and fine-tuning capabilities. Organizations can perform specialized domain adaptation through model modifications, experiment with different architectures and parameters, and integrate models with proprietary systems and workflows. This flexibility is particularly valuable for developing novel applications that require direct model access and manipulation." + "Running locally also enables fine-grained optimization of resource usage and model characteristics based on target use case. Organizations can perform specialized domain adaptation through model modifications, experiment with different architectures and parameters, and integrate models with proprietary systems and workflows. This flexibility is particularly valuable for developing novel applications that require direct model access and manipulation. " ] }, { diff --git a/tamingllms/notebooks/local.ipynb b/tamingllms/notebooks/local.ipynb index 885074e..ae64b3f 100644 --- a/tamingllms/notebooks/local.ipynb +++ b/tamingllms/notebooks/local.ipynb @@ -23,13 +23,13 @@ "\n", "Running LLMs locally versus using cloud APIs offers several important advantages.\n", "\n", - "Privacy-sensitive data processing is one of the primary reasons for running LLMs locally. Organizations handling medical records must comply with HIPAA regulations that require data to remain on-premise. Similarly, businesses processing confidential documents and intellectual property, as well as organizations subject to GDPR and other privacy regulations, need to maintain strict control over their data processing pipeline.\n", + "Privacy concerns are a key driver for running LLMs locally. Individual users may want to process personal documents, photos, emails, and chat messages without sharing sensitive data with third parties. For enterprise use cases, organizations handling medical records must comply with HIPAA regulations that require data to remain on-premise. Similarly, businesses processing confidential documents and intellectual property, as well as organizations subject to GDPR and other privacy regulations, need to maintain strict control over their data processing pipeline.\n", "\n", - "Cost considerations become significant when operating at scale. Organizations running high-volume applications can face prohibitive API costs with cloud-based solutions. Development and testing environments that require frequent model interactions, educational institutions supporting multiple student projects, and research initiatives involving extensive model experimentation can all achieve substantial cost savings through local deployment.\n", + "Cost considerations are another key advantage of local deployment. Organizations can better control expenses by matching model capabilities to their specific needs rather than paying for potentially excessive cloud API features. For high-volume applications, this customization and control over costs becomes especially valuable compared to the often prohibitive per-request pricing of cloud solutions.\n", "\n", - "Applications with stringent latency requirements form another important category. Real-time systems where network delays would be unacceptable, edge computing scenarios demanding quick responses, and interactive applications requiring sub-second performance all benefit from local deployment. This extends to embedded systems in IoT devices where cloud connectivity might be unreliable or impractical.\n", + "Applications with stringent latency requirements form another important category. Real-time systems where network delays would be unacceptable, edge computing scenarios demanding quick responses, and interactive applications requiring sub-second performance all benefit from local deployment. This extends to embedded systems in IoT devices where cloud connectivity might be unreliable or impractical. Further, the emergence of Small Language Models (SLMs) has made edge deployment increasingly viable, enabling sophisticated language capabilities on resource-constrained devices like smartphones, tablets and IoT sensors. \n", "\n", - "Finally, local deployment enables deeper customization and fine-tuning capabilities. Organizations can perform specialized domain adaptation through model modifications, experiment with different architectures and parameters, and integrate models with proprietary systems and workflows. This flexibility is particularly valuable for developing novel applications that require direct model access and manipulation." + "Running locally also enables fine-grained optimization of resource usage and model characteristics based on target use case. Organizations can perform specialized domain adaptation through model modifications, experiment with different architectures and parameters, and integrate models with proprietary systems and workflows. This flexibility is particularly valuable for developing novel applications that require direct model access and manipulation. " ] }, {