-
Notifications
You must be signed in to change notification settings - Fork 3
/
CHANGES
3771 lines (3041 loc) · 167 KB
/
CHANGES
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
New features as of version 1.7.0
Sept. 2016
***SOME BREAKAGE MAY OCCUR!! (See below)***
Top-Level:
* Printing more resolution of floats and doubles (so doesn't lose precision)
* The op[] for Val is having trouble when using const Val objects.
* InformativeError in occonforms.h needs inline marker
Discussion:
******* PRECISION OF FLOATS AND DOUBLES WHEN PRINTING *****************
**As of PicklingTools 1.7.0 (this release) we are printing floats/doubles
in higher precision, so some outputs may have more significant figures
when printed. This is so that going through a string WON'T lose any
precision. The example below shows going from
real_8 -> string -> real_8
can lose precision (unless we change the precision to 17 places).
real_8 a = 1.2345678912345654321; // uses all bits
Val v = a;
char filename[] = "/tmp/junkreal";
WriteValToFile(v, filename); // Writes out human readable string
Val vresult;
ReadValFromFile(filename, vresult); // Reads human readable string
real_8 result = vresult;
if (a==result) {
cout << "OK" << endl;
} else {
cout << "Lost precision going to the File and string!" << endl;
}
We had to extend to precision for doubles to 17 digits, and floats
to 9 digits so we don't lose any information going to a string file.
See the test C++/opencontainers_1_8_5/tests/precision_test.cc for
more information.
YOU CAN STILL GET OLD BEHAVIOUR BY SETTING TWO MACROS FROM THE COMMAND
LINE (Changing the macros OC_DBL_DIGITS and OC_FLT_DIGITS):
-DOC_DBL_DIGITS=16
-DOC_FLT_DIGITS=7
We feel the newer behavior is more correct because you won't lose
any information going back and forth between strings and reals.
************* CALLING OPERATOR[] ON A CONST VAL *******************
**Using non-const methods on a const object should give a compiler
error. As expected, line 4 below (t["a"]) fails to compile
because operator[] is a non-const method (because it can CHANGE the Val!)
// This all works as expected
const Tab tt("{'a':666}");
cout << tt("a") << std::endl; /// compiles okay
cout << tt["a"] << endl; /// FAILS to compile, but that's correct
We expect similar behavior with Val:
// This all compiles??? Check line 4!
const Val t = Tab("{'a':666}");
cout << t("a") << std::endl; /// compiles okay
cout << t["a"] << endl; /// compiles OKAY??? NOT GOOD!!
What happens is that the compiler looks for a const method on Val so it can
find something to make this compile. And it finds "operator int_8() const"
conversion (which is a const method). So:
cout << t["a"] << endl; // is equivalent to ...
int_8 tmp = t.operator int_8();
cout << tmp["a"] << endl; // This compiles?
How can that compile, you ask? An old, sometimes forgotten piece of
C lore helps explain:
char *a = "abc";
int i;
cout << a[i] << endl; // These both compile??
cout << i[a] << endl;
Because: a[i] == *(a + i) == *(i + a) == i[a];
In other words, (const char* + int) is the same operation
as (int + const char*): it's just pointer arithmetic.
We WANT a compiler error! So, there are two ways to do this.
operator [] (int) const = delete; // Newer C++
operator [] (int const; // NO IMPLEMENTATION PROVIDED
So, if we provide a prototype operator[] that's const. then
the compiler has no choice but to match that method (and it
won't match the "operator int_8() const" method). However,
since there's no implementation, a link error will occur, basically
saying "Hey, you are trying to call [] on a const object! No implementation
provided!". The message looks something like:
val_test.cc:28: undefined reference to `OC::Val::operator[](char const*) const'
The "=delete" gives a better error message, and happens
at compile time rather than link-time (but it's still definitely not
run-time). The only problem: the "=delete" only works on newer
C++ compilers. We still have just enough users at RedHat 6 with an
older compiler that we have to support the less optimal "NO IMPL PROVIDED"
way. But, the most important thing: WE GET AN ERROR BEFORE RUN-TIME
on either platform.
With the OC_DELETE_OPBRACKET_CONST macro, you can choose to select the
=delete version. From the compile line:
-DOC_DELETE_OPBRACKET
Details:
C++:
* Moved to opencontainers 1.8.5
* Updated version numbers
* Updated resolution of double and float strinigize to 17 and 9 (resp.).
* Added tests/precision_test.cc to show how the problem can manifest
(if you don't have 17 and 9 digits). Affects a number of tests.
* Updated ocval.h to provide "empty" implementations for
operator[] const (for Val). See Discussion above.
Allow using =delete (instead of empty impl) with a macro
for people with newer compilers: OC_DELETE_OPBRACKET
* Updated ocforms.h so InformativeError is inline
M C++/opencontainers_1_8_5/include/ocport.h
M C++/opencontainers_1_8_5/include/ocval.h
M C++/opencontainers_1_8_5/tests/conform_test.output
M C++/opencontainers_1_8_5/tests/items_test.output
M C++/opencontainers_1_8_5/tests/iter_test.output
M C++/opencontainers_1_8_5/tests/maketab_test.output
M C++/opencontainers_1_8_5/tests/otab_test.output
M C++/opencontainers_1_8_5/tests/pretty_test.output
M C++/opencontainers_1_8_5/tests/proxy_test.output
M C++/opencontainers_1_8_5/tests/runall
M C++/opencontainers_1_8_5/tests/ser_test.output
M C++/opencontainers_1_8_5/tests/tab_test.output
M C++/opencontainers_1_8_5/tests/valbigint_test.output
M C++/opencontainers_1_8_5/tests/valreader_test.output
M CHANGES
A C++/opencontainers_1_8_5/tests/precision_test
A C++/opencontainers_1_8_5/tests/precision_test.cc
A C++/opencontainers_1_8_5/tests/precision_test.output
New features as of version 1.6.3
July 2016
Top-Level :
* Added keys(), values(), items() to Tab and OTab (like Python)
* Fixed bug with Python C Extension modules not building
* Making sure release works with CentOS 7
Details:
C++:
* Moved to opencontainers 1.8.4
* Updated version numbers
* Added items_test.cc, .output (tests for .keys(), .values(), .items())
* Updated ocval.cc, .h for for .keys(), .values(), .items()
* Added ocitems.h (for templatized code)
* Updated port_test to get rid of a warning in CentOS 7
* Updated all Makefiles to work with OpenContainers 1.8.4
PythonCExt:
* Fixed bug that setup.py wouldn't build (wrong dir in setup.py)
* Verified builds/works with CentOS 7
X-Midas:
* Updated to use opencontainers 1.8.4 code
New features as of version 1.6.2
December 2015
Top-Level :
* updated Python cxmltools to avoid some crashes
* updated CQ includes (needed circular buffer include)
* updated Array to allow it to reference (not adopt) some memory
* minor doc and bug fixes
Details:
X-Midas:
* Updated explain page for midasyeller (had incorrect notion of host)
Python:
* updated incorret comment in midastalker.py about the naming of the
ARRAYDISPOSITION enums
PythonCExt:
* pyocsermodule.cc: certain illegal XML would cause the cxmltools
to crash:
>>> import cxmltools
>>> s = '<?xml><tag>vvv</tag>'
>>> cxmltools.ReadFromXMLString(s)
Problem: Was only catching runtime_error, not all generic
exceptions (so logic_error would propagate out and kill
the Python interpreter).
Fixed: by catching exception appropriately rather than just runtime_error
PythonModule:
* PythonModule: updated to have latest version of Python code
(this is included so you can just import ptools)
C++:
* httplib.h: Extraneous cerr commented out
* midasyeller.h : Added some code in sendto to catch EAGAIN
and EINTR if sendto fails for sprurious wakeup.
Should fix an error JG is seeing.
Open Containers 1.8.3
- occq.h, occqts.h:
Updated includes to contain #include "occircularbuffer.h
- ocport.h: updated to 1.8.3
- ocarray.h : updated Array so that you can refer to a piece of
memory (assuming some other entity is responsible
for its maintenance) so that pieces of memory
can be passed around using Array interfaces.
Added a new constructor and a new memory policy.
New Features as of version 1.6.1
November 2015
This is just a bug-fix release. A minor change to fix
the FixedSizeAllocator to work with cross process shared memory
C++: Updated to opencontainers_1_8_2
* Updated ocstreamingpool.cc so it specifically requests a
FixedSizeAllocator to work in cross process shared memory
* Updated FixedSizeAllocator so you can request the lock (used
to ensure thread-safety) to work in cross process shared memory
New Features as of version 1.6.0
August 2015
BIG NEWS:
Major updates across libraries to support very large serializations:
greater than 2Gig!
New Python C Extension Module for supporting OC Serialization in Python.
Why? Pickling can't support large serialization (>2G), but OC Ser can.
All MidasTalker, MidasServer, etc. can use SERIALIZE_OC in Python and C++.
For example:
>>> import numpy
>>> from pyoceser import ocdumps, odloads # Python C Extension module
>>> huge_array = numpy.zeros(2**32+1, 'b')
>>> serialized_string = ocdumps(huge_array)
>>> get_back_huge_array = odloads(serialized_string)
Suport for cx_t<int> for all int types. For DSP, complex ints
can sometimes be the way data is sampled off an antenna.
Top-Level
* Adding timeout capability to TransactionLock
* Added external break checker/timeouts for shared memory routines
* Added fixed size allocator, and augment StreamingPool to possibly use
* CircularQueue now cleans up after a get() to force reclamation
* Clean up Array to allow 64-bit lengths
* Minor change to BigInt and BigUInt interface (sign) because of Array fixes
* Change length()/entries() to return size_t throughout baseline
* Updated Val to support cx_t<> for int_1,..int_u8
* Updated ocser, ocserialize, xmldumper, xmlloader, convert to support cx_t<>
* Updated MidasTalker/MidasServer to allow sockets to have very large msgs
* Updated the OpenContainers serialization to allow very large msgs (>2*32)
* Updated the PickleLoader and PickleDumper to allow very large msgs (>2*32)
* Added a new Python C Extension module "pyocser" for Python OC Serialization
- This is needed to do serialization of very large objects
* Updated OpalPythonDaemon and related code to work w/large serialized data
Details:
* Python C Extension
Because Python Pickling 2 cannot handle very large arrays or strings,
we needed to create a serialization that will work with very large
arrays and strings (byte lengths of over 2**32). To this end,
we have added a new Python C Extension module to the baseline called
'pyocser' with two functions:
ocdumps(po) : Dumps Python Object po, returning the serialized string
ocloads(str) : Load a Python Object from the given string (str),
returning the deserialized Python Object
- pyocser.h,.cc : Implementation of OC Serialization in Python
- pyocsermodule.cc : The binding code to make a Python module
- check_ocser.py : A python script to verify the ocdumps/loads works
- psuedonumeric.h : Since we may not have Numeric or numpy, pull
the definition (refactored from pyobjconvert)
- pyobjconverter.cc:
- setup.py : Added the new pyocser module setup and build
A few fixes to occomplex.h, ocnumerictools.h, ocnumpy.tools.h &ocval.cc
so that the Python C Extension module would build correctly on
RedHat 5, 32-bit, Python 2.4 systems . Note that the default testing
environment is RedHat 6, 64-bit Python 2.6 systems.
* Python
midaslistener_ex.py, midasserver_ex.py, permutation_client.py,
permutation_server.py, midasyeller_ex.py
README: Updates to show you can use OC Serialization (--ser=5)
if you have the Python C Extension module pyocser built.
Also recommends NumPy (--arrdisp=4) going forward.
midassocket.py:
* Changes for SERIALIZE_OC is supported, plus the capability
to try to give good error messages.
* Changes so can have either 4 byte header (normal messages
with length of message under 4G),
or 12-byte header (0xFFFFFFFF and length in int_u8)
* Updates to send and recv
send: Don't use sendall anymore: actually break apart
buffers because may run into problems with non-blocking
sockets, and also buffers that are too big
recv: Not allowed to use very large sizes, so we have
to break buffers up into chunks
midastalker.py:
* Updates to documentation for SERIALIZE_OC
speed_test.py:
* updates to allow ocdumps and ocloads. Surprisingly, ocdumps
and ocloads are 50%-100% faster than pickle2/unpickle2
* Updated to OpenContainers 1.8.1
- ocport.h, Makfile, Makefile.Linux, Makefile.Linux.factored, Makefile.icc
: updated version numbers
- ocproxy.h : Added a timeout capability for TransactionLock.
By default, there is no timeout, so it can try to
get in forever. If the timeout expires, however,
a runtime_error is thrown, and the user can adjust
accordingly.
- ocspinfo.h,
ocstreamingpool.h,cc
: Updated so will work with pools larger than 2G
(some ints returned change size to int_ptr, which
is 4 bytes on a 32-bit machimne, 8 bytes on a
64-bit machine).
Also, modified so can have a fixed size
allocator section, if desired.
- ocfixedsizeallocator.h
fsa_test.cc, fsa_test.output
: A small, low overhead allocator for small pieces.
And a test.
- run_all : Added fsa_test
- occircularbuffer.h
: When a get happens, it leaves the old value in
there (fully constructed). For the OCCQ, this means
a full copy of the packet stays in the queue, when it needs
to immediately get reclaimed by the memory allocator.
- ocarray.h, array_test.cc, array_test.output
: length() and capacity() now return size_t.
Restructured Array so that length and capacity are
size_t (so potentially 64-bit quantities on 64-bit machines)
so Array can hold very large sizes--This meant restructuring
Array, which meant getting rid of the reservedSpace
(replacing it with a bit in the capacity)
and usenewandDlete field (got subsumed under the
allocator). All of this was necessary to keep the
sizeof(Array) to 32, but still allow 64-bit quantities.
Updated the test to make sure setBit/getBit work.
- ocbigint.h, ocbigint_test.cc, ocbigint_test.output
: Because the "reservedSpace" is gone from Array,
we have to use the new interface (getBit/setBit)
of array which only gives us one bit of information
for the sign bit. Also updated the code to use size_t
for all the new sizes. Updated the test to make
sure negate works.
- ocbigunit.h : Updated code to propagate the extra bit along
(mostly for BigUInt). Updated to use size_t
in a few places (because of Array's change),
Also moved length and expandTo to the class
(instead of always deferring to the impl).
- ocavlhasht.h, ocavltreet.h, ochashtablet.h, ocordavlhasht.h
: Made entries return a size_t instead of an int_u4
(which is what it is stored as internally anyways).
Also had the iterators use size_t when using an index
to iterate.
- ocval.cc, ocval.cc:
: Everything that returned a size_t: updated the
code to use size_t rather than int when
appropriate. Also cleaned up iterators
to use size_t.
Added .length() method to Tab
- ocstring_impl.h: Use size_t for internal length store for large strs
- occomplex.h
: Updated so better support for printing cx_t<INTTYPE>,
and added MOVEARRAYPOD for cx_t<INTTYPE>
- ocnumerictools.h, ocnumpytools.h
: NumPy and Numeric DO NOT support complex ints,
so for output we just use the PTOOLS char tags
(which are distinct from Numeric tags). Also updated
Array conversions to support cx_t<INTYPES>.
- ocproxy.h, ocproxy.cc: Updated to support cx_t<INTTYPES>
- ocval.h, ocval.cc: Updated to support cx_t<INTYPES>. Updated TagFor
to support all new types, including Array<numeric>
- complex_test.cc, complex_test.output, runall:
: New test: make sure the cx_t<INTTYPES> seems to be working
- ocser.cc, ocserialize.cc, ocval.cc, ser_test.cc,.output, occonvert.h
: Updated the built-in serializations to handle
serialization of cx_t<INTTYPES> and Array<cx_t<>>
Also updated to allow very large tables, arrays
to be able to be serialized. There are two tags
for most large containers:
'n''t''o''u''q''Q' for int_u4 lengths
'N''T''O''U''y''Y' for int_u8 lengths.
The new tags are only seen by the serialization routines.
Also updated the tests to check these still work.
- ocserialize.cc, ocloadumpcontext.h :
Factored out the load and dump contexts so the Python C Ext
modules can reuse.
* C++:
Added the ability to pass in an external error check (pointer to
function) and set a timeout. Inside the shared memory code,
many wait for pipes to be available/be created have a loop with a
small timeout. With this change, the user can (a) specify the
timeout and (b) force immediate failure by checking an external
break checker. These changes are important for allowing the
shared memory routines to do a clean shutdown in the face of
error conditions.
Also allows the SHMMain to use small allocators. This allows the
little chunks of memory (mostly Vals) to not clutter memory so much.
Also deferring to 16 byte alignment (SSE instructions, esp. FFTW
require 16 byte alignment).
Update memdump to use size_t
- sharedmem.h,.cc:
- shmboot.h,.cc See above.
- xmlloader.h, xmldumper.h, xmlload_test.cc,.output
: Updated XML tools to make sure they work with cx_t<INTTYPES>
Updated the socket code (MidasTalker and the ilk) to allow very
large messages (greater than 2*32 bytes)
- fdtools.h:
* Added htonll/ntohll for large ints in Network Byte Order
* updated methods ReadExact, WriteExact, clientReadExact, and
readUntilFull, and data members of FDTools class to use
size_t instead of int
* Updated the number of bytes written/read routines so that
if the data is larger than int_u4, will write special escape
sequence 0xFFFFFFFF and then another 8 bytes for the real
bytelength
- midastalker.h :
* Updates to size_t for byte counts (from int)
- midassocket.h :
* Updates so uses 4 bytes for normal message counts,
for > 0xFFFFFFFF, uses 4 bytes ESC count of 0XFFFFFFF
and then an 8 byte length. Also udpates for size_t when
needed
- xmldump_test.cc : updated to size_t for byte counts
- valprotocol.h, valprotocol.cc : updated to use size_t for lengths
- pickleloader_test.cc, .output: updated for size_t for lengths
- pickleloader.h:
* Significant updates for size_t for byte counts to support
very large pickles
- p2_test.cc, .output, p2common.h
* Updated for size_t interface changes, and updated the p2_test
to be current and correct
- chooseser.h:
* Updated to use size_t instead of int for all lengths
- README: Show that NumPy is preferred, and there is a new Python
C-extension module for pyocser (OC serialization in Python).
- midastalker_ex2.cc:
* Left some code in to try very large arrays and strings to test
new large facilities
M2K:
Updated a bunch of files so that the OpalPythonDaemon works with
serialized data of greater than 2G. NOTE! These only work if
you update M2k Vector and string to use size_t for lengths/capacities.
By default, early MITE/M2k use unsigned int/int_u4, which does
not work for data > 4G. At this time, only OC (OpenContainers
serialization) works with data gerater
- m2openconser.cc: Updating protocol for int_u8 for some lengths
- m2opalmsgextnethdr.h.,cc:
* Substantial changes so works with underlying
Midas 2k infrastructure and still work with
int_u8 sizes. Added
- m2opalpythondaemon.cc: uses size_t for lengths instead of int
- m2opalpythontablewriter.h,cc: Allows to use OC Serialization
New Features as of version 1.5.4
July 2015
Top-Level
* Minor fixes so Proxys compile with templatized arrays
* Some X-Midas minor bug-fixes for T4000 files
* Bug fix to CQ in timed enqueue
* Minor bugfixes to message printed out in shmboot, sharedmem modules
Details:
* Updated to OpenContainers 1.8.0
- ocport.h : updated version numbers
- ocproxy.cc : template instantiation error, was instantiation Array<T>
wrong with Proxys
- occq.h,
occqts.h : When dequeueing with a timeout, returned the wrong thing,
potential making it look like enqueue would succeed
or fail incorrectly.
* Shared memory: cleaning up output messages
- shmboot.h : Make it so you can turn off the "forced unlink"
message (when you start up and "unlink" the old one).
- shmboot.cc : Fixed the Address randomization message for 64 bit
- sharedmem.h, cc: Added flag to interface for to control whether unlink
failure message is shown.
* X-Midas:
- t4val.cc : Allow users to use non-blocking
New Features as of version 1.5.3
Feb. 2015
Top-Level
* Better error messages if mixing single socket and dual socket
* Allow NORMAL_SOCKET in all Python, C++ MidasTalker/MidasSocket
* Added in a second permutation generator (pure C, no dependencies)
* Fix up some NAMESPACE issues (so everything works with OC_FORCE_NAMESPACE)
* More compiler warnings for ARM eliminated
* Preliminary packaging as a Python module
* Fixed X-Midas T4Val to handle all serializations to all Python primitives
* Minor fix to UDP MidasListener/Yeller documentation
Details:
* X-Midas: Allow reading t4000 pipes from XMPY Python primitives
* Made it so T4Tab/Val routines can choose their serialization
(by default it was SERIALIZE_OC): this allows Python primitives
to be able to get T4000 data as dicts from the pipe directly.
* Added C++ primitive "tabgenout" to show how you can use
the new routines above to serialize T4000 (in Python serialization)
to the pipes (also added explain files)
* Added Python primitive "tabinpytester.py" to show how to read
from a T4000 from a pipe (i.e, tabgenout) (also added explain files)
* Added a silly macro "t4pytest.txt" in the mcr area showing
how to run the two primitives above (tabgenout and tabinpytester)
* Added the "t4val.py" to the python area: it makes it easier
to read dicts from a t4000 pipe (and is used by tabinpytester)
* Separated the serialization_e into its own file so we don't
have #include m2chooser.h inside of t4val.h (just serialization.h)
* Added in a new all C permutation generator. It works in both
C and C++, generates all permutations in order (similar to
the C++ STL next_permutation ... Last check it was about 10%
faster than the C++ version. Your mileage may vary). Added
a new example program as well.
* Fixed up and made sure tests and examples in OpenContainers
area all worked.
* Better error messages if mixing single socket and dual socket;
The better error messages come from the *client* when they try to
connect and "can't" for some reason.
Plus, added better support for NORMAL_SOCKET (which essentially
just doesn't do the 16 byte exchange at the start, and only uses
one socket: this is the most compatible with other sockets).
* C++:
* MidasTalker: updated error reporting, NORMAL_SOCKET support.
* midastalker_ex2.cc: Example allows NORMAL_SOCKET now
* MidasServer: a little extra code for handling NORMAL_SOCKET
* midasserver_ex.cc: Example allows NORMAL_SOCKET now
* permutation_client.cc, server.cc: Inheriting updated error messages
plus better NORMAL_SOCKET handling
* README: updated group documention to mention NORMAL_SOCKET
* Python
* midassocket.py: Added NORMAL_SOCKET (didn't have in Python before!)
* MidasTalker: updated error reporting, NORMAL_SOCKET support.
* midastalker_ex2.py: Example allows NORMAL_SOCKET now
* MidasServer: a little extra code for handling NORMAL_SOCKET
* midasserver_ex.py: Example allows NORMAL_SOCKET now
* permutation_client.py, server.py: Inheriting updated error messages
plus better NORMAL_SOCKET handling
* README: updated group documention to mention NORMAL_SOCKET
* X-Midas
* Added all Python/C++ code (above) dealing with better error
messages and NORMAL_SOCKET
* Updated the Host primitives/explain pages
valpipe, permclient, permserver, xmclient, xmserver
So they deal with NORMAL_SOCKET and through better errors.
* Java
* Updated README, Makefiles
* Updated Midastalker to give better error messages when
mixing single and dual socket.
* Namespace issues:
In an attempt to make sure the OC Namespace still is in good shape,
and everything still works with OC_FORCE_NAMESPACE,
we discovered a few issues in:
C++: arraydisposition.h, httplib.h chooseser.h fdtools.cc httptools.h
PythonCExt: pyobjconverter.h
The PTOOLS option tree compiles and should work with either
#define OC_FORCE_NAMESPACE on or off
HOST Primitives changed: tab2xml, httpclient, keywords2tab, xmclient
* Few more warnings fixed (unused variables) in shmboot.c, p2_test.cc
* Added the new PythonModule/ptools dir, which contains most of the
working code (minus examples and tests) for people who want to
use the code more like a Python module/make it more Pythonic.
We DO NOT want to break backwards compatibility (for people who depend
on the previous way thinga are packaged), but this is a first step
to trying out a new module.
List of files:
M Python/permutation_client.py
M Python/midasserver_ex.py
M Python/midasserver.py
M Python/permutation_server.py
M Python/midastalker.py
M Python/midassocket.py
M Python/README
M Python/midastalker_ex2.py
D Xm/ptools152
A + Xm/ptools153
M + Xm/ptools153/python/permutation_client.py
M + Xm/ptools153/python/midasserver_ex.py
M + Xm/ptools153/python/midasserver.py
M + Xm/ptools153/python/permutation_server.py
M + Xm/ptools153/python/midastalker.py
M + Xm/ptools153/python/README
M + Xm/ptools153/python/midassocket.py
M + Xm/ptools153/python/midastalker_ex2.py
M + Xm/ptools153/inc/midastalker.h
M + Xm/ptools153/inc/midasserver.h
M + Xm/ptools153/exp/permserver.exp
M + Xm/ptools153/exp/valpipe.exp
M + Xm/ptools153/exp/permclient.exp
M + Xm/ptools153/exp/xmserver.exp
M + Xm/ptools153/exp/xmclient.exp
M + Xm/ptools153/host/valpipe.cc
M + Xm/ptools153/host/permclient.cc
M + Xm/ptools153/host/xmserver.cc
M + Xm/ptools153/host/xmclient.cc
M + Xm/ptools153/host/permserver.cc
M Xm/README
M C++/midastalker.h
M C++/permutation_client.cc
M C++/midasserver_ex.cc
M C++/permutation_server.cc
M C++/midastalker_ex2.cc
M C++/README
M C++/midasserver.h
M C++/chooseser.h
A C++/serialization.h
M CHANGES
M README
M Xm/ptools153/pythoncext/pyobjconverter.h
M Xm/ptools153/lib/ptools/fdtools.cc
M Xm/ptools153/inc/occomplex.h
A Xm/ptools153/inc/ocpermute.h
M Xm/ptools153/inc/arraydisposition.h
M Xm/ptools153/inc/ocport.h
M Xm/ptools153/inc/httplib.h
M Xm/ptools153/inc/chooseser.h
A Xm/ptools153/inc/serialization.h
M Xm/ptools153/inc/httptools.h
M Xm/ptools153/host/tab2xml.cc
M Xm/ptools153/host/httpclient.cc
M Xm/ptools153/host/keywords2tab.cc
M Xm/ptools153/host/xmclient.cc
M PythonCExt/pyobjconverter.h
D C++/opencontainers_1_7_8
A + C++/opencontainers_1_7_9
M + C++/opencontainers_1_7_9/tests/bag_test.output
M + C++/opencontainers_1_7_9/include/occomplex.h
A C++/opencontainers_1_7_9/include/ocpermute.h
M + C++/opencontainers_1_7_9/include/ocport.h
M + C++/opencontainers_1_7_9/CHANGES
M + C++/opencontainers_1_7_9/examples/runall
A C++/opencontainers_1_7_9/examples/permute_ex.cc
M C++/arraydisposition.h
M C++/httplib.h
M C++/chooseser.h
M C++/fdtools.cc
M C++/httptools.h
M Xm/ptools153/inc/shmboot.h
M C++/Makefile.Linux
M C++/Makefile.Linux.factored
M C++/p2_test.cc
M C++/shmboot.cc
M Java/Makefile
M Java/README
M Java/com/picklingtools/pickle/Makefile
M Java/com/picklingtools/midas/MidasTalker.java
A PythonModule
A PythonModule/ptools
A PythonModule/ptools/xmlloader_defs.py
A PythonModule/ptools/simplearray.py
A PythonModule/ptools/opalfile.py
A PythonModule/ptools/parsereader.py
A PythonModule/ptools/circularbuffer.py
A PythonModule/ptools/__init__.py
A PythonModule/ptools/midastalker.py
A PythonModule/ptools/occonvert.py
A PythonModule/ptools/XMTime.py
A PythonModule/ptools/prettyopal.py
A PythonModule/ptools/midaslistener.py
A PythonModule/ptools/arraydisposition.py
A PythonModule/ptools/seriallib.py
A PythonModule/ptools/midassocket.py
A PythonModule/ptools/midasyeller.py
A PythonModule/ptools/xmldumper.py
A PythonModule/ptools/opalfile_numeric.py
A PythonModule/ptools/xmldumper_defs.py
A PythonModule/ptools/xmltools.py
A PythonModule/ptools/opalfile_numpy.py
A PythonModule/ptools/midasserver.py
A PythonModule/ptools/cxmltools.py
A PythonModule/ptools/conforms.py
A PythonModule/ptools/pretty.py
A PythonModule/ptools/xmlloader.py
M Xm/ptools153/cfg/commands.cfg
M Xm/ptools153/cfg/primitives.cfg
A Xm/ptools153/mcr/t4pytest.txt
A Xm/ptools153/python/t4val.py
M Xm/ptools153/lib/ptools/t4val.cc
A Xm/ptools153/exp/tabgenout.exp
A Xm/ptools153/exp/tabinpytester.exp
A Xm/ptools153/host/tabinpytester.py
A Xm/ptools153/host/tabgenout.cc
A Xm/ptools153/host/tabinpytester.exe
M Docs/faq.pdf
M Docs/shm.pdf
M Docs/html/_sources/xmldoc.txt
M Docs/html/_sources/faq.txt
M Docs/html/_sources/usersguide.txt
M Docs/html/usersguide.html
M Docs/html/genindex.html
M Docs/html/java.html
M Docs/html/objects.inv
M Docs/html/_static/jquery.js
M Docs/html/_static/pygments.css
M Docs/html/_static/doctools.js
M Docs/html/_static/searchtools.js
M Docs/html/_static/basic.css
M Docs/html/_static/default.css
M Docs/html/search.html
M Docs/html/searchindex.js
M Docs/html/xmldoc.html
M Docs/html/faq.html
M Docs/html/index.html
M Docs/html/shm.html
M Docs/xmldoc.txt
M Docs/faq.txt
M Docs/usersguide.pdf
M Docs/PicklingTools.pdf
M Docs/java.pdf
M Docs/usersguide.txt
M Docs/xmldoc.pdf
New Features as of version 1.5.2
Sept. 2014
Top-Level
* Updated some more documentation for NumPy
* Updated Makefiles to work with ARM Linux
* Fixed some XML processing with & in attributes
* Fixed problem when MidasServer shuts down
* Added templatized thread-safe, Circular Queue class
* Added a bunch of new primitives for X-Midas for Tab conversions
* Added seriallib.py (for converting between Windows INI files and dicts)
Details:
* Updates so PicklingTools will build on ARM with Ubuntu Linux
* Need unistd.h in sharedmem_test.cc, shmboot.cc
* Line 385 and 386 of xmldump_test.cc, change s to s1
* Added serialib.py to Python area
* Updated documentation
* Allows conversions between Windows INI files and Python
dictionaries
* Bug in handling of & in xmlloader.cc, .py:
* Any special escape sequence wasn't recognized in attributes
* Added xmlload_ex.py for another example
* Updated OpenContainers to 1.7.8:
* Added occqts.h: A templatized version of occq.h, so you can
have a synchronized (threadsafe) circular queue of any type
(rather than just Vals from occq.h).
* Updated occq.h to have an empty() method.
* Made it so once an OCThread is started, can't keep creating threads
* Updated MidasServer (C++) so shutsdown correctly even when not started.
* X-Midas:
* Added keywords2tab primitive (and explain page): allows converting
between keywords in files to Python dicts (Tab) and vice-verse
* Added tab2xml primitive (and explain page): allows converting
between tabs and XML (as an X-Midas primitive)
* Updated commands.cfg/primitives.cfg
* Updated tab2opal to handle more cases back and forth
* Doc update
* Updates so that NumPy support is more prevelant throughout
* Clarified many points in usersguide, updated XML conversion guide,
added serialib
New Features as of version 1.5.1
March 2014
Top-Level
* Complete NumPy transition (Python)
* Move timeout send/recv code from child MidasTalker to parent MidasSocket
so MidasServers can have access to timed send/recv code.
(only MidasTalkers had access to those routines before)
* Updated VALPIPE primitive so wouldn't seg-fault upon exit
* Updated documentation to fix spelling errors
* Updated C++ so can compile with -Wextra for better warning coverage
Details:
* prettyopal.py :
* fixed long-standing problem with opal tables having too much space by ,
* fixed so works with numpy
* pretty.py, xmlloader.py, xmldumper.py
* prefer numpy (only import numpy even if Numeric is present)
* midassocket.py
* small doc update, using numpy instead
* opalfile.py
* added opalfile_numpy.py, opalfile_numeric.py
* kept old modulem, with Numeric hard-coded and add
new version with numpy hard-coded. The opalfile.py
simply chooses the right one.
* midassocket.h, midassocket.cc
* Moved the guts of send/recv (with the timeouts)
up into MidasSocket so that MidasServers can
use the timeout code as well. (No interface changes,
just augmented MidasSocket with recvTimed_ and sendTimed_)
* Files changed for get rid of -Wextra warnings
M + opencontainers_1_7_7/include/ocproxy.h
M + opencontainers_1_7_7/include/ocnumerictools.h
M + opencontainers_1_7_7/include/occonvert.h
M + opencontainers_1_7_7/include/ocval.h
M + opencontainers_1_7_7/include/ocbiguint.h
M + opencontainers_1_7_7/CHANGES
M m2convertrep.cc
M Makefile.Linux
M xmlload_test.cc
M httplib.h
M valprotocol2.cc
M sharedmem_test.cc
M fdtools.cc
M p2_test.cc
M pickleloader.h
M httptools.h
M xmlloader.h
New Features as of version 1.5.0
Feb 2013
Top-Level:
* Java Support
* Documentation for new Java support
Details:
* Added new Java directory containing the first version of the
PicklingTools for Java.
- Added MidasTalker_ex.java showing a simple version of
the Java Midastalker
- Cleaned up Java structure:
- pythonesque: gives you python-like data structures from Java
- pickle: pickle libraries from pyro-lite
- midas: socket comms for talking like a MidasTalker
- Added command-line java testing suite with Makefile
* The "java.pdf" covers most of the Java
* Minor fix to XMLDumper: wasn't outputting _ on tag correctly
if we were dumping numbers.
* Fixed a few spelling errors in documentation.
New Features as of version 1.4.2
December 2012
Top-Level:
* Added a thread-safe "Bag" data structure
* Change so MidasTalker works with Jython
* Using Py_ssize_t correctly (previously would have compile error on newer Pythons)
Details:
* MidasTalker can use non-blocking sockets to
make sure an "open" can have a reasonable timeout,
but the Jython socket module (by default) requires
the non-blocking socket for select to work. There is a
CPython compatible select (called cpython_compatible_select)
in Jython that allows things to work. To see which select
to use, we look for "Java" in the sys.version string.
* Added several abstractions for a thread-safe Bag:
the idea is that several threads are reaching into a bag,
all at the same time, and we want to allow each thread
to pull out one piece of work at a time. A simple idea,
but making it thread-safe and thread-hot is the challenge.
The first key aha! is that we only need to supply a bag of
ints from [0..n) : By putting everything we want served out
into an array, we simply grab the int we want from the bag
and index into the array. With this key simplification, we
can take advantage of atomic increment instructions to make
Bags easy, thread-safe, quick, and (most important) avoid
excessive collateral damage (collateral damage here would
be preventing others from enetering the bag).
iDrawer: The first abstraction is the primitive on which all other
Bag abstractions are built-on: the iDrawer: it is a
thread-safe bag that deals out integers from start to start+length.
It has no notion of how many threads are looking in the bag:
it simply locks a counter and only increments it if there