gtools-1.0.3 (2018-08-18); exit if too many obs: SPI 32-bit integer bug

* Closes #43 * Gtools exits with error if `_N > 2^31-1` and points the user to the pertinent bug report. The SPI uses long integers (32-bit) instead of long long integers (64-bit) for its observation numbers (not to mention theyare signed integers, but alas). At any rate, this de facto limits the number of observations to 2^31-1.
mcaceresb · Aug 19, 2018 · 0fe7a65 · 0fe7a65
1 parent 71f7859
commit 0fe7a65
Show file tree

Hide file tree

Showing 20 changed files with 95 additions and 50 deletions.
diff --git a/.appveyor.yml b/.appveyor.yml
@@ -1,4 +1,4 @@
-version: "generic-1.0.0-{build}"
+version: "generic-1.0.3-{build}"
 
 environment:
   matrix:

diff --git a/README.md b/README.md
@@ -13,7 +13,7 @@ implementation of collapse, pctile, xtile, contract, egen, isid,
 levelsof, duplicates, and unique/distinct using C plugins for a massive
 speed improvement.
 
-`version 1.0.2 08Aug2018`
+`version 1.0.3 18Aug2018`
 Builds: Linux, OSX [![Travis Build Status](https://travis-ci.org/mcaceresb/stata-gtools.svg?branch=develop)](https://travis-ci.org/mcaceresb/stata-gtools),
 Windows (Cygwin) [![Appveyor Build status](https://ci.appveyor.com/api/projects/status/2bh1q9bulx3pl81p/branch/develop?svg=true)](https://ci.appveyor.com/project/mcaceresb/stata-gtools)
 

diff --git a/build/_gtools_internal.ado b/build/_gtools_internal.ado
@@ -1,5 +1,5 @@
-*! version 1.0.1 23Jul2018 Mauricio Caceres Bravo, [email protected]
-*! Encode varlist using Jenkin's 128-bit spookyhash via C plugins
+*! version 1.0.3 18Aug2018 Mauricio Caceres Bravo, [email protected]
+*! gtools function internals
 
 * rc 17000
 * rc 17001 - no observations
@@ -8,6 +8,7 @@
 * rc 17004 - strL variables could not be compressed
 * rc 17005 - strL contains binary data
 * rc 17006 - strL variables uknown error
+* rc 17800 - More than 2^31-1 obs
 * rc 17459
 * rc 17900
 * rc 17999
@@ -47,6 +48,17 @@ program _gtools_internal, rclass
         exit 17001
     }
 
+    if ( `=_N > 2^31-1' ) {
+        local nmax = trim("`: disp %21.0gc 2^31-1'")
+        di as err `"too many observations"'
+        di as err `""'
+        di as err `"A Stata bug prevents gtools from working with more than `nmax' observations."'
+        di as err `"See {browse "https://www.statalist.org/forums/forum/general-stata-discussion/general/1457637"}"'
+        di as err `"and {browse "https://github.com/mcaceresb/stata-gtools/issues/43"}"'
+        clean_all 17800
+        exit 17800
+    }
+
     local 00 `0'
 
     * Time the entire function execution

diff --git a/build/changelog.md b/build/changelog.md
@@ -1,6 +1,13 @@
 Change Log
 ==========
 
+## gtools-1.0.3 (2018-08-18)
+
+### Bug fixes
+
+* Gtools exits with error if `_N > 2^31-1` and points the user to the
+  pertinent bug report.
+
 ## gtools-1.0.2 (2018-08-08)
 
 ### Enhancements

diff --git a/build/gtools.ado b/build/gtools.ado
@@ -1,4 +1,4 @@
-*! version 1.0.2 08Aug2018 Mauricio Caceres Bravo, [email protected]
+*! version 1.0.3 18Aug2018 Mauricio Caceres Bravo, [email protected]
 *! Program for managing the gtools package installation
 
 capture program drop gtools

diff --git a/build/gtools.pkg b/build/gtools.pkg
@@ -1,4 +1,4 @@
-v 1.0.2
+v 1.0.3
 d
 d 'GTOOLS': Faster implementation of common Stata commands optimized for large datasets
 d
@@ -45,7 +45,7 @@ d
 d Author: Mauricio Caceres Bravo
 d Support: email [email protected]
 d
-d Distribution-Date: 20180808
+d Distribution-Date: 20180818
 d
 f _gtools_internal.ado
 f gcollapse.ado

diff --git a/build/gtools.sthlp b/build/gtools.sthlp
@@ -1,5 +1,5 @@
 {smcl}
-{* *! version 1.0.2 08Aug2018}{...}
+{* *! version 1.0.3 18Aug2018}{...}
 {viewerdialog gtools "dialog gtools"}{...}
 {vieweralsosee "[R] gtools" "mansection R gtools"}{...}
 {viewerjumpto "Syntax" "gtools##syntax"}{...}

diff --git a/build/gtools_unix_v2.plugin b/build/gtools_unix_v2.plugin
diff --git a/build/gtools_unix_v3.plugin b/build/gtools_unix_v3.plugin
diff --git a/build/stata.toc b/build/stata.toc
@@ -1,3 +1,3 @@
-v 1.0.0
+v 1.0.3
 d Mauricio Caceres Bravo, [email protected]
 p 'GTOOLS': Generic stata repo that uses C plugins
diff --git a/changelog.md b/changelog.md
@@ -1,6 +1,13 @@
 Change Log
 ==========
 
+## gtools-1.0.3 (2018-08-18)
+
+### Bug fixes
+
+* Gtools exits with error if `_N > 2^31-1` and points the user to the
+  pertinent bug report.
+
 ## gtools-1.0.2 (2018-08-08)
 
 ### Enhancements

diff --git a/docs/index.md b/docs/index.md
@@ -5,7 +5,7 @@ implementation of collapse, pctile, xtile, contract, egen, isid,
 levelsof, duplicates, and unique/distinct using C plugins for a massive
 speed improvement.
 
-`version 1.0.2 08Aug2018`
+`version 1.0.3 18Aug2018`
 Builds: Linux, OSX [![Travis Build Status](https://travis-ci.org/mcaceresb/stata-gtools.svg?branch=master)](https://travis-ci.org/mcaceresb/stata-gtools),
 Windows (Cygwin) [![Appveyor Build status](https://ci.appveyor.com/api/projects/status/2bh1q9bulx3pl81p/branch/master?svg=true)](https://ci.appveyor.com/project/mcaceresb/stata-gtools)
 

diff --git a/docs/stata/gtools.sthlp b/docs/stata/gtools.sthlp
@@ -1,5 +1,5 @@
 {smcl}
-{* *! version 1.0.2 08Aug2018}{...}
+{* *! version 1.0.3 18Aug2018}{...}
 {viewerdialog gtools "dialog gtools"}{...}
 {vieweralsosee "[R] gtools" "mansection R gtools"}{...}
 {viewerjumpto "Syntax" "gtools##syntax"}{...}

diff --git a/lib/plugin/gtools_unix_v2.plugin b/lib/plugin/gtools_unix_v2.plugin
diff --git a/lib/plugin/gtools_unix_v3.plugin b/lib/plugin/gtools_unix_v3.plugin
diff --git a/lib/update_version.py b/lib/update_version.py
@@ -43,44 +43,51 @@
     minor = int(args['minor'])
     patch = int(args['patch'])
 
-main = [path.join("src", "gtools.pkg"),
-        path.join("src", "stata.toc"),
-        path.join("src", "ado", "_gtools_internal.ado"),
-        path.join("src", "ado", "gcollapse.ado"),
-        path.join("src", "ado", "gcontract.ado"),
-        path.join("src", "ado", "gegen.ado"),
-        path.join("src", "ado", "gunique.ado"),
-        path.join("src", "ado", "gdistinct.ado"),
-        path.join("src", "ado", "glevelsof.ado"),
-        path.join("src", "ado", "gtop.ado"),
-        path.join("src", "ado", "gtoplevelsof.ado"),
-        path.join("src", "ado", "gisid.ado"),
-        path.join("src", "ado", "gduplicates.ado"),
-        path.join("src", "ado", "gquantiles.ado"),
-        path.join("src", "ado", "fasterxtile.ado"),
-        path.join("src", "ado", "hashsort.ado"),
-        path.join("src", "ado", "gtools.ado"),
-        path.join("docs", "stata", "gcollapse.sthlp"),
-        path.join("docs", "stata", "gcontract.sthlp"),
-        path.join("docs", "stata", "gegen.sthlp"),
-        path.join("docs", "stata", "gunique.sthlp"),
-        path.join("docs", "stata", "gdistinct.sthlp"),
-        path.join("docs", "stata", "glevelsof.sthlp"),
-        path.join("docs", "stata", "gtoplevelsof.sthlp"),
-        path.join("docs", "stata", "gisid.sthlp"),
-        path.join("docs", "stata", "gduplicates.sthlp"),
-        path.join("docs", "stata", "gquantiles.sthlp"),
-        path.join("docs", "stata", "hashsort.sthlp"),
-        path.join("docs", "stata", "gtools.sthlp"),
-        path.join(".appveyor.yml"),
-        path.join("README.md"),
-        path.join("docs", "index.md")]
+base = [
+    path.join("src", "gtools.pkg"),
+    path.join("src", "stata.toc"),
+    path.join("src", "ado", "_gtools_internal.ado"),
+    path.join("src", "ado", "gtools.ado"),
+    path.join("docs", "stata", "gtools.sthlp"),
+    path.join(".appveyor.yml"),
+    path.join("README.md"),
+    path.join("docs", "index.md")]
+main = [
+    path.join("src", "ado", "gcollapse.ado"),
+    path.join("src", "ado", "gcontract.ado"),
+    path.join("src", "ado", "gegen.ado"),
+    path.join("src", "ado", "gunique.ado"),
+    path.join("src", "ado", "gdistinct.ado"),
+    path.join("src", "ado", "glevelsof.ado"),
+    path.join("src", "ado", "gtop.ado"),
+    path.join("src", "ado", "gtoplevelsof.ado"),
+    path.join("src", "ado", "gisid.ado"),
+    path.join("src", "ado", "gduplicates.ado"),
+    path.join("src", "ado", "gquantiles.ado"),
+    path.join("src", "ado", "fasterxtile.ado"),
+    path.join("src", "ado", "hashsort.ado"),
+    path.join("docs", "stata", "gcollapse.sthlp"),
+    path.join("docs", "stata", "gcontract.sthlp"),
+    path.join("docs", "stata", "gegen.sthlp"),
+    path.join("docs", "stata", "gunique.sthlp"),
+    path.join("docs", "stata", "gdistinct.sthlp"),
+    path.join("docs", "stata", "glevelsof.sthlp"),
+    path.join("docs", "stata", "gtoplevelsof.sthlp"),
+    path.join("docs", "stata", "gisid.sthlp"),
+    path.join("docs", "stata", "gduplicates.sthlp"),
+    path.join("docs", "stata", "gquantiles.sthlp"),
+    path.join("docs", "stata", "hashsort.sthlp")]
 test = [path.join("src", "test", "gtools_tests.do")]
 plug = [path.join("src", "plugin", "gtools.c")]
 
 callok = False
+todo   = base
+if "base" in args['set']:
+    callok = True
+
 todo   = main
 if "main" in args['set']:
+    todo += main
     callok = True
 
 if "test" in args['set']:
@@ -91,7 +98,7 @@
     callok = True
 
 if "all" in args['set']:
-    todo = main + test + plug
+    todo = base + main + test + plug
     callok = True
 
 if not callok:

diff --git a/src/ado/_gtools_internal.ado b/src/ado/_gtools_internal.ado
@@ -1,5 +1,5 @@
-*! version 1.0.1 23Jul2018 Mauricio Caceres Bravo, [email protected]
-*! Encode varlist using Jenkin's 128-bit spookyhash via C plugins
+*! version 1.0.3 18Aug2018 Mauricio Caceres Bravo, [email protected]
+*! gtools function internals
 
 * rc 17000
 * rc 17001 - no observations
@@ -8,6 +8,7 @@
 * rc 17004 - strL variables could not be compressed
 * rc 17005 - strL contains binary data
 * rc 17006 - strL variables uknown error
+* rc 17800 - More than 2^31-1 obs
 * rc 17459
 * rc 17900
 * rc 17999
@@ -47,6 +48,17 @@ program _gtools_internal, rclass
         exit 17001
     }
 
+    if ( `=_N > 2^31-1' ) {
+        local nmax = trim("`: disp %21.0gc 2^31-1'")
+        di as err `"too many observations"'
+        di as err `""'
+        di as err `"A Stata bug prevents gtools from working with more than `nmax' observations."'
+        di as err `"See {browse "https://www.statalist.org/forums/forum/general-stata-discussion/general/1457637"}"'
+        di as err `"and {browse "https://github.com/mcaceresb/stata-gtools/issues/43"}"'
+        clean_all 17800
+        exit 17800
+    }
+
     local 00 `0'
 
     * Time the entire function execution

diff --git a/src/ado/gtools.ado b/src/ado/gtools.ado
@@ -1,4 +1,4 @@
-*! version 1.0.2 08Aug2018 Mauricio Caceres Bravo, [email protected]
+*! version 1.0.3 18Aug2018 Mauricio Caceres Bravo, [email protected]
 *! Program for managing the gtools package installation
 
 capture program drop gtools

diff --git a/src/gtools.pkg b/src/gtools.pkg
@@ -1,4 +1,4 @@
-v 1.0.2
+v 1.0.3
 d
 d 'GTOOLS': Faster implementation of common Stata commands optimized for large datasets
 d
@@ -45,7 +45,7 @@ d
 d Author: Mauricio Caceres Bravo
 d Support: email [email protected]
 d
-d Distribution-Date: 20180808
+d Distribution-Date: 20180818
 d
 f _gtools_internal.ado
 f gcollapse.ado

diff --git a/src/stata.toc b/src/stata.toc
@@ -1,3 +1,3 @@
-v 1.0.0
+v 1.0.3
 d Mauricio Caceres Bravo, [email protected]
 p 'GTOOLS': Generic stata repo that uses C plugins