Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test suite fails #87

Open
jappeace opened this issue Mar 6, 2019 · 4 comments
Open

Test suite fails #87

jappeace opened this issue Mar 6, 2019 · 4 comments

Comments

@jappeace
Copy link

jappeace commented Mar 6, 2019 •
edited
Loading

Hi, I'm trying to build grenade with nix, the test suite fails however:

━━━ Test.Grenade.Recurrent.Layers.LSTM ━━━
  ✗ prop_lstm_reference_forwards failed after 54 tests and 55 shrinks.

       ┏━━ test/Test/Grenade/Recurrent/Layers/LSTM.hs ━━━
    44 ┃ prop_lstm_reference_forwards =
    45 ┃   property $ do
    46 ┃     input :: S.R 3                       <- forAll randomVector
       ┃     │ (vector54319561065,0.7925737992825796,-6.308042121263235e-2] :: R 3)
    47 ┃     cell :: S.R 2                        <- forAll randomVector
       ┃     │ (vector102552342276,-0.5733412138993579] :: R 2)
    48 ┃     net@(LSTM lstmWeights _) :: LSTM 3 2 <- forAll genLSTM
       ┃     │ LSTM
    49 ┃
    50 ┃     let actual          = runRecurrentForwards net (S1D cell) (S1D input)
    51 ┃     case actual of
    52 ┃       (_, (S1D cellOut) :: S ('D1 2), (S1D output) :: S ('D1 2)) ->
    53 ┃         let cellOut'        = Reference.Vector . H.toList . S.extract $ cellOut
    54 ┃             output'         = Reference.Vector . H.toList . S.extract $ output
    55 ┃             refNet          = Reference.lstmToReference lstmWeights
    56 ┃             refCell         = Reference.Vector . H.toList . S.extract $ cell
    57 ┃             refInput        = Reference.Vector . H.toList . S.extract $ input
    58 ┃             (refCO, refO)   = Reference.runLSTM refNet refCell refInput
    59 ┃         in do toList refCO ~~~ toList cellOut'
       ┃         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       ┃         │ Failed (- lhs ~/~ + rhs)
       ┃         │ - [ 8.600932134870043e-2 , 0.10405281905297237 ]
       ┃         │ + [ 8.016375201087303e-2 , 7.653272311821224e-2 ]
    60 ┃               toList refO ~~~ toList output'

    This failure can be reproduced by running:
    > recheck (Size 53) (Seed 10790320388850313198 5063647523751707663) prop_lstm_reference_forwards

  ✗ prop_lstm_reference_backwards failed after 58 tests and 36 shrinks.

       ┏━━ test/Test/Grenade/Recurrent/Layers/LSTM.hs ━━━
    63 ┃ prop_lstm_reference_backwards =
    64 ┃   property $ do
    65 ┃     input :: S.R 3                       <- forAll randomVector
       ┃     │ (vector55046569861,0.7353593240191039,0.10220169420456582] :: R 3)
    66 ┃     cell :: S.R 2                        <- forAll randomVector
       ┃     │ (vector94712659202,8.446059985294041e-2] :: R 2)
    67 ┃     net@(LSTM lstmWeights _) :: LSTM 3 2 <- forAll genLSTM
       ┃     │ LSTM
    68 ┃     let (tape, _ :: S ('D1 2), _ :: S ('D1 2))
    69 ┃                                           = runRecurrentForwards  net (S1D cell) (S1D input)
    70 ┃         actualBacks                       = runRecurrentBackwards net tape (S1D (S.konst 1) :: S ('D1 2)) (S1D (S.konst 1) :: S ('D1 2))
    71 ┃     case actualBacks of
    72 ┃       (actualGradients, _, _ :: S ('D1 3)) ->
    73 ┃         let refNet          = Reference.lstmToReference lstmWeights
    74 ┃             refCell         = Reference.Vector . H.toList . S.extract $ cell
    75 ┃             refInput        = Reference.Vector . H.toList . S.extract $ input
    76 ┃             refGradients    = Reference.runLSTMback refCell refInput refNet
    77 ┃         in toList refGradients ~~~ toList (Reference.lstmToReference actualGradients)
       ┃         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       ┃         │ Failed (- lhs ~/~ + rhs)
       ┃         │   [
       ┃         │ -   -0.2833762365900625
       ┃         │ +   -0.2772291966623182
       ┃         │ - , 0.2792387920721928
       ┃         │ + , 0.273181502212966
       ┃         │ - , 3.8809159964732e-2
       ┃         │ + , 3.796730583209162e-2
       ┃         │ - , -1.8789963351874522e-2
       ┃         │ + , -1.9046776736865983e-2
       ┃         │ - , 1.851561984376432e-2
       ┃         │ + , 1.8768683616069006e-2
       ┃         │ - , 2.5733374902189056e-3
       ┃         │ + , 2.60850879413327e-3
       ┃         │ - , 0.3522269593369693
       ┃         │ + , 0.34458639918017514
       ┃         │ - , 3.2072315003399895e-2
       ┃         │ + , 3.1376597524498194e-2
       ┃         │ - , 2.3355281081871578e-2
       ┃         │ + , 2.36744913261781e-2
       ┃         │ - , 2.1266342964228386e-3
       ┃         │ + , 2.1557002473284216e-3
       ┃         │ - , 0.3797310824129001
       ┃         │ + , 0.3714938986832906
       ┃         │ - , 2.51790100961354e-2
       ┃         │ + , 2.5523146308241297e-2
       ┃         │ - , -0.14407943439166543
       ┃         │ + , -0.14095404163950748
       ┃         │ - , 0.14197579763956925
       ┃         │ + , 0.13889603729210484
       ┃         │ - , 1.9732077340779888e-2
       ┃         │ + , 1.930404615252405e-2
       ┃         │ - , 0.10233133248392495
       ┃         │ + , 0.10372995447129899
       ┃         │ - , -0.10083724033390323
       ┃         │ + , -0.1022154416927028
       ┃         │ - , -1.401455922896567e-2
       ┃         │ + , -1.4206104381415103e-2
       ┃         │ - , 0.17908580369842605
       ┃         │ + , 0.1752010475203034
       ┃         │ - , 1.6306804906884104e-2
       ┃         │ + , 1.5953075243230058e-2
       ┃         │ - , -0.12719434247358946
       ┃         │ + , -0.12893278171536413
       ┃         │ - , -1.1581785295885263e-2
       ┃         │ + , -1.1740080308514117e-2
       ┃         │ - , 0.19306996321689512
       ┃         │ + , 0.1888818605480774
       ┃         │ - , -0.13712648638597202
       ┃         │ + , -0.13900067402973101
       ┃         │ - , -0.11252340904695016
       ┃         │ + , -0.131574427655387
       ┃         │ - , 0.11088050713147728
       ┃         │ + , 0.12965337068552493
       ┃         │ - , 1.5410392325159424e-2
       ┃         │ + , 1.80194820553455e-2
       ┃         │ - , 4.787291093695013e-2
       ┃         │ + , 4.6151157876618276e-2
       ┃         │ - , -4.71739408493592e-2
       ┃         │ + , -4.5477326303979465e-2
       ┃         │ - , -6.556327661366858e-3
       ┃         │ + , -6.320528814073785e-3
       ┃         │ - , 0.1398627446668094
       ┃         │ + , 0.16354250849410923
       ┃         │ - , 1.2735317059880666e-2
       ┃         │ + , 1.489149739912809e-2
       ┃         │ - , -5.9504389135933686e-2
       ┃         │ + , -5.736430903441441e-2
       ┃         │ - , -5.418221013079177e-3
       ┃         │ + , -5.22335426217051e-3
       ┃         │ - , 0.1507841180627999
       ┃         │ + , 0.17631294858261248
       ┃         │ - , -6.415087061319925e-2
       ┃         │ + , -6.184367943473307e-2
       ┃         │ - , -0.6893313089947273
       ┃         │ + , -0.6743782306038787
       ┃         │ - , 0.6792667034381152
       ┃         │ + , 0.6645319479841436
       ┃         │ - , 9.440583078310466e-2
       ┃         │ + , 9.235796530850202e-2
       ┃         │ - , -0.7491175754191286
       ┃         │ + , -0.7593562021102653
       ┃         │ - , 0.7381800584171601
       ┃         │ + , 0.7482691956860943
       ┃         │ - , 0.10259372545373369
       ┃         │ + , 0.10399593371881949
       ┃         │ - , 0.9237208005000677
       ┃         │ + , 0.9036833100206666
       ┃         │ - , 1.00383585861486
       ┃         │ + , 1.0175558686009876
       ┃         │   ]

    This failure can be reproduced by running:
    > recheck (Size 57) (Seed 15557724053009315116 9838685224472302247) prop_lstm_reference_backwards

  ✗ prop_lstm_reference_backwards_input failed after 73 tests and 51 shrinks.

       ┏━━ test/Test/Grenade/Recurrent/Layers/LSTM.hs ━━━
    79 ┃ prop_lstm_reference_backwards_input =
    80 ┃   property $ do
    81 ┃     input :: S.R 3                       <- forAll randomVector
       ┃     │ (vector132148291512,0.2850784935453341,-1.40887764348131e-2] :: R 3)
    82 ┃     cell :: S.R 2                        <- forAll randomVector
       ┃     │ (vector429500642426,0.7481729899291754] :: R 2)
    83 ┃     net@(LSTM lstmWeights _) :: LSTM 3 2 <- forAll genLSTM
       ┃     │ LSTM
    84 ┃     let (tape, _ :: S ('D1 2), _ :: S ('D1 2))
    85 ┃                                           = runRecurrentForwards  net (S1D cell) (S1D input)
    86 ┃         actualBacks                       = runRecurrentBackwards net tape (S1D (S.konst 1) :: S ('D1 2)) (S1D (S.konst 1) :: S ('D1 2))
    87 ┃     case actualBacks of
    88 ┃       (_, _, S1D actualGradients :: S ('D1 3)) ->
    89 ┃         let refNet          = Reference.lstmToReference lstmWeights
    90 ┃             refCell         = Reference.Vector . H.toList . S.extract $ cell
    91 ┃             refInput        = Reference.Vector . H.toList . S.extract $ input
    92 ┃             refGradients    = Reference.runLSTMbackOnInput refCell refNet refInput
    93 ┃         in toList refGradients ~~~ H.toList (S.extract actualGradients)
       ┃         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       ┃         │ Failed (- lhs ~/~ + rhs)
       ┃         │   [
       ┃         │ -   0.41783498961490245
       ┃         │ +   -7.144093770897443e-2
       ┃         │ - , -0.643583384088483
       ┃         │ + , -0.5033482252382088
       ┃         │ - , 0.24741239536602957
       ┃         │ + , 0.2196259366479876
       ┃         │   ]

    This failure can be reproduced by running:
    > recheck (Size 72) (Seed 15511439570864369243 14061565670218007919) prop_lstm_reference_backwards_input

  ✗ prop_lstm_reference_backwards_cell failed after 45 tests and 31 shrinks.

        ┏━━ test/Test/Grenade/Recurrent/Layers/LSTM.hs ━━━
     95 ┃ prop_lstm_reference_backwards_cell =
     96 ┃   property $ do
     97 ┃     input :: S.R 3                       <- forAll randomVector
        ┃     │ (vector957622892669,-0.26370740740732634,-0.8739678849810585] :: R 3)
     98 ┃     cell :: S.R 2                        <- forAll randomVector
        ┃     │ (vector544704658233,-0.82774989764567] :: R 2)
     99 ┃     net@(LSTM lstmWeights _) :: LSTM 3 2 <- forAll genLSTM
        ┃     │ LSTM
    100 ┃     let (tape, _ :: S ('D1 2), _ :: S ('D1 2))
    101 ┃                                           = runRecurrentForwards  net (S1D cell) (S1D input)
    102 ┃         actualBacks                       = runRecurrentBackwards net tape (S1D (S.konst 1) :: S ('D1 2)) (S1D (S.konst 1) :: S ('D1 2))
    103 ┃     case actualBacks of
    104 ┃       (_, S1D actualGradients, _ :: S ('D1 3)) ->
    105 ┃         let refNet          = Reference.lstmToReference lstmWeights
    106 ┃             refCell         = Reference.Vector . H.toList . S.extract $ cell
    107 ┃             refInput        = Reference.Vector . H.toList . S.extract $ input
    108 ┃             refGradients    = Reference.runLSTMbackOnCell refInput refNet refCell
    109 ┃         in toList refGradients ~~~ H.toList (S.extract actualGradients)
        ┃         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        ┃         │ Failed (- lhs ~/~ + rhs)
        ┃         │ - [ 0.3555130556015041 , 4.6592664606872156e-2 ]
        ┃         │ + [ 0.4057801473816573 , -1.403610494537344e-2 ]
    110 ┃
    111 ┃ (~~~) :: (Monad m, Eq a, Ord a, Num a, Fractional a, Show a, HasCallStack) => [a] -> [a] -> PropertyT m ()
    112 ┃ (~~~) x y =
    113 ┃   if all (< 1e-8) (zipWith (-) x y) then
    114 ┃     success
    115 ┃   else
    116 ┃     case valueDiff <$> mkValue x <*> mkValue y of
    117 ┃       Nothing ->
    118 ┃         withFrozenCallStack $
    119 ┃           failWith Nothing $ unlines [
    120 ┃               "━━━ Not Simliar ━━━"
    121 ┃             , showPretty x
    122 ┃             , showPretty y
    123 ┃             ]
    124 ┃       Just diff ->
    125 ┃         withFrozenCallStack $
    126 ┃           failWith (Just $ Diff "Failed (" "- lhs" "~/~" "+ rhs" ")" diff) ""

    This failure can be reproduced by running:
    > recheck (Size 44) (Seed 17734744653642261044 4653266027254713753) prop_lstm_reference_backwards_cell

  ✗ 4 failed.
Test suite test: FAIL
Test suite logged to: dist/test/grenade-0.1.0-test.log
0 of 1 test suites (0 of 1 test cases) passed.
builder for '/nix/store/g2ishp6kxmakfxpxxl0l58m0n6w7dfzi-grenade-0.1.0.drv' failed with exit code 1
cannot build derivation '/nix/store/djy93n6n0wjvbv85a1g9s6nc7hh0nxq9-ghc-8.4.3-with-packages.drv': 1 dependencies couldn't be built
error: build of '/nix/store/djy93n6n0wjvbv85a1g9s6nc7hh0nxq9-ghc-8.4.3-with-packages.drv' failed
make[1]: *** [makefile:70: enter] Fout 100
make[1]: Map '/home/jappie/projects/raster' wordt verlaten

I used cabal2nix to just grab the latest version from github, this one to be precise:

{ mkDerivation, ad, base, bytestring, cereal, constraints
, containers, criterion, deepseq, fetchgit, hedgehog, hmatrix
, MonadRandom, mtl, primitive, random, reflection, singletons
, stdenv, text, transformers, typelits-witnesses, vector
}:
mkDerivation {
  pname = "grenade";
  version = "0.1.0";
  src = fetchgit {
    url = "https://github.com/HuwCampbell/grenade";
    sha256 = "1nzzacpiz86z3svvi2lpczzkv4algjfw3m2h2pqsw5n44xp8b320";
    rev = "f729b291e40a9044d95d49d31bce3149a5b9d599";
    fetchSubmodules = true;
  };
  libraryHaskellDepends = [
    base bytestring cereal containers deepseq hmatrix MonadRandom
    primitive singletons vector
  ];
  testHaskellDepends = [
    ad base constraints hedgehog hmatrix MonadRandom mtl random
    reflection singletons text transformers typelits-witnesses vector
  ];
  benchmarkHaskellDepends = [ base bytestring criterion hmatrix ];
  description = "Practical Deep Learning in Haskell";
  license = stdenv.lib.licenses.bsd2;
}

I'll disable the tests for now so it's not a big deal, but I just wanted to let you know.
@erikd
Copy link
Collaborator

erikd commented Mar 7, 2019

I don't use Nix, but i just tested building this with ghc-8.6.3 and cabal-install 2.0.0.1.

I built it and ran the tests using:

./mafia build -w
./mafia test

Unless this can be reproduced outside Nix, I can't work on it.

@HuwCampbell
Copy link
Owner

HuwCampbell commented Mar 7, 2019 via email

@jappeace
Copy link
Author

jappeace commented Mar 7, 2019

This project fails reproducibly: https://github.com/jappeace/awesome-project-name/tree/show-grenade-fails

You can't enter the shell unless you disable the tests.
Enter shell with:

   nix-shell -A shells.ghc

@jappeace
Copy link
Author

jappeace commented Mar 7, 2019 •
edited
Loading

I'm not sure how to make a pure derivation, but since the project I linked is pinned, that should be good enough?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@erikd @jappeace @HuwCampbell