-
Notifications
You must be signed in to change notification settings - Fork 0
/
README-content.html
178 lines (158 loc) · 17.9 KB
/
README-content.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
<p><a href="https://pypi.org/project/FIFOStr/"><img src="https://img.shields.io/pypi/v/fifostr.svg" alt="PyPi version" /></a> <a href="https://travis-ci.com/deftio/fifostr"><img src="https://travis-ci.com/deftio/fifostr.svg?branch=master" alt="Build Status" /></a> <a href="https://coveralls.io/github/deftio/fifostr?branch=master"><img src="https://coveralls.io/repos/github/deftio/fifostr/badge.svg?branch=master" alt="Coverage Status" /></a> <a href="https://opensource.org/licenses/BSD-2-Clause"><img src="https://img.shields.io/badge/License-BSD%202--Clause-blue.svg" alt="License" /></a></p>
<h1 id="fifostr.py">fifostr.py</h1>
<p>FIFOStr - A python language string library designed to look for patterns in a stream, such as serial connection or look back parser. FIFOStr works by allowing character(s) to be inserted in to a FIFOstr object which is treated as FIFO (First-In-First-Out) buffer. The FIFOstr object can be set to a fixed size so that when a new character is inserted the last character is removed.</p>
<p>Trigger patterns (either strings, regular experssions, or customer parsers written in Python) can be attached to a FIFOstr object. If that pattern is deteced in a string, a callback function will is called to trigger an action. There is no limit to the number of patterns that can be deteted and each pattern can have a separate callback.</p>
<p>Since a FIFOstr is a fixed length buffer it can be attached to any stream with predictable resource consumption. FIFOstr also supports mutable character injection / mutable string support (used to inject / to test patterns).</p>
<p>Example code and a full test suite with CI are provided. See coverage and test section below.</p>
<p>Logically FIFOstr works as follows:</p>
<div class="sourceCode"><pre class="sourceCode python"><code class="sourceCode python"><span class="im">from</span> fifostr <span class="im">import</span> FIFOStr
<span class="co">#initialize from a given string</span>
myString <span class="op">=</span> FIFOStr(<span class="st">"this is a test"</span>) <span class="co"># initialize with a string</span>
<span class="bu">len</span>(myString) <span class="op">==</span> <span class="dv">14</span> <span class="co">#true</span>
<span class="co"># The FIFOStr function head( int N) returns the the first N characters</span>
myString.head(<span class="dv">4</span>) <span class="op">==</span> <span class="st">"this"</span> <span class="co">#true</span>
myString<span class="op">+=</span> <span class="st">" more"</span> <span class="co"># this addes the string " more" to myString, each character is fed in and the last character in the FIFOstr is removed.</span>
<span class="co">#initialize with max fixed length</span>
myString.head(<span class="dv">4</span>) <span class="op">==</span> <span class="st">"is a"</span> <span class="co">#true because the string is also a true FIFO and keeps fixed length. Look at example below for more on pattern matching (and multiple pattern matching for more details)</span>
myString <span class="op">=</span> FIFOStr(<span class="dv">10</span>) <span class="co"># creates a blank FIFOStr max length FIFOStr of 10 chars</span>
<span class="bu">len</span>(myString) <span class="co"># returns 0, there is no data in FIFOstr yet</span>
myString <span class="op">+=</span> <span class="st">"This"</span>
<span class="bu">len</span>(myString) <span class="op">==</span> <span class="dv">4</span> <span class="co"># true</span>
myString <span class="op">+=</span> <span class="st">" is more stuff"</span> <span class="co"># add characters to end of string</span>
<span class="bu">len</span>(myString) <span class="op">==</span> <span class="dv">10</span> <span class="co">#true</span>
myString <span class="op">==</span> <span class="st">"more stuff"</span> <span class="co"># true, string is at max len of 10 chars</span>
<span class="co">#mutable support</span>
myString[<span class="dv">2</span>] <span class="op">==</span> <span class="st">'d'</span> <span class="co"># in place modification (mutable string)</span>
myString <span class="op">==</span> <span class="st">"mode stuff"</span> <span class="co"># string position[2] is changed </span>
<span class="co">#For pattern matching & triggers see examples section.</span></code></pre></div>
<p>Originally a lighter version of this was used in a python serial terminal program which allowed the serial terminal to parse commands sent/received by both sides.</p>
<h2 id="pattern-triggering-features">Pattern Triggering Features</h2>
<p>Built-in pattern matching and triggering: simply add / remove patterns which then call a callback function (E.g. if the pattern is "seen" then trigger the function). Patterns can be strings, regexes or user-supplied-functions (parsers written in python). A pattern consists of:<br />
* pattern: string <em>or</em> compiled regex <em>or</em> user-supplied-parser-function<br />
* label: user supplied 'name' for this pattern<br />
* start index : position in fifostr to begin pattern match. default is 0 (also accepts the character '^' as start anchor for those familiar with regexes) * stop index : position in fifostr to end pattern match. default is end of fifostr. the letter '$' has special meaning as end of string no matter the length (again regex) * callback_fn : called if pattern is found, fifostr(start:end) and the label are passed to the callback function (callback('thematchingstring','label')) * active : default is True, sets whether this pattern should be actively looked for<br />
* mutable string support for changing the contents middle position characters to look at callback responses.</p>
<h3 id="installation">Installation</h3>
<pre><code>pip install fifostr # or just pull fifostr.py from the source repository and put in your source path </code></pre>
<h3 id="original-usage">Original Usage</h3>
<p>Originally part of a terminal program called 'dioterm' (albeit in much more compact form), this library was used used to 'listen' to traffic in either direction on a serial port. When certain patterns were found such as a command sent from the host or a special piece of data from the embedded microntroller client, fifostr would trigger a callback to do something. This was very useful when sequences of commands had to be set up between the host and client. Many of these sequences where conditional based on what either the host or client sent resulting in many variations of sequence-test cases, especially if this results in the host then having to make some other call to an unrelated process or hardware to reply correctly.</p>
<h3 id="functionality">Functionality</h3>
<p>FIFOStr is a string which is (derived from deque) with these properties:<br />
* add/remove chars or strings at either end<br />
* mutable (can set a char to any value like an array with []) * use slices, lists, or tuples to retrieve members (just like a real str object)<br />
* get head/tail (returns as a str)<br />
* match head/tail --> match a supplied string to either the head or tail<br />
* use patterns to trigger callbacks --> pattern can be string | regex | user_supplied_parser any of which triggers user supplied callback_fn<br />
* all patterns can look at either the whole fifostr or any subset e.g. addPattern("foo",myCallback,2,5,"bar") --> only looks for "foo" between positions 2 and 5 in the fifostr and will call myCallback with ("foo","bar") if found * all patterns have optional label which can be used for logging purposes (eg. when pattern found, in addition to callback, emit label)<br />
* user supplied callback_fn is called with both the string-match section and the label<br />
* patterns can be added/deleted from the list of patterns "watching" the fifostr content * all (active) patterns are always matched. fifostr matches multiple different patterns over the same string.<br />
* clear all patterns --> removes patterns from processing<br />
* get/setPattern Active/Inactive --> allows a stored pattern to set on or off<br />
* Python 2.7+, Python 3+ support, derived from built-in deque package<br />
* 100% test coverage in both 2.7 and 3.x</p>
<h3 id="usage-example">Usage example</h3>
<p>See example.py to run in tests dir -- same examples as here but more comments, more use cases</p>
<div class="sourceCode"><pre class="sourceCode python"><code class="sourceCode python"><span class="im">from</span> fifostr <span class="im">import</span> FIFOStr
<span class="kw">def</span> main():
myFifoStr<span class="op">=</span>FIFOStr(<span class="dv">5</span>) <span class="co">#make a fifostr of length 5 (for unlimited length omit number)</span>
myFifoStr<span class="op">+=</span><span class="st">'1234567'</span> <span class="co">#adds 1234567 to fifostr ... but len of fifostr is 5</span>
<span class="co"># so only 34567 is retained</span>
<span class="bu">print</span> <span class="st">"myFifoStr.head(3)= "</span>,myFifoStr.head(<span class="dv">3</span>) <span class="co">#shows 345</span>
<span class="bu">print</span> <span class="st">"myFifoStr.tail(4)= "</span>,myFifoStr.tail(<span class="dv">4</span>) <span class="co">#shows 4567</span>
<span class="co"># the eqhead and eqtail functions allow string compares against</span>
<span class="co"># the head or the tail</span>
myFifoStr.eqhead(<span class="st">"3456"</span>) <span class="co">#True</span>
myFifoStr.eqhead(<span class="st">"567"</span>) <span class="co">#False</span>
myFifoStr.eqtail(<span class="st">"4567"</span>) <span class="co">#True</span>
myFifoStr.eqtail(<span class="st">"abc"</span>) <span class="co">#False</span>
<span class="co">#fifostr.testPattern() allows you to test if the pattern is present in the fifostr object</span>
<span class="co">#test a string pattern directly</span>
myFifoStr.testPattern(<span class="st">'67890'</span>) <span class="co">#False</span>
<span class="co">#test a regex pattern directly. to do this pass any valid regex in compiled form</span>
r1<span class="op">=</span>re.<span class="bu">compile</span>(<span class="st">"[0-9]+"</span>)
myFifoStr.testPattern(r1) <span class="co">#True</span>
r2<span class="op">=</span>re.<span class="bu">compile</span>(<span class="st">"[a-z]+"</span>)
myFifoStr.testPattern(r2) <span class="co">#False</span>
<span class="co">#more generally we can add (and remove) patterns which will scan and trigger a call back everytime the fifostr </span>
<span class="co">#internal content changes (whether adding or deleting chars from either end or even rotating/reversing the fifstr object)</span>
<span class="co">#adding patterns</span>
p1 <span class="op">=</span> myFifoStr.addPattern(<span class="st">"234"</span>,logf,label<span class="op">=</span><span class="st">"234 was here"</span>) <span class="co">#integer index returned managing pattern </span>
p2 <span class="op">=</span> myFifoStr.addPattern(<span class="st">"67890"</span>,logf,label<span class="op">=</span><span class="st">"67890 detected"</span>)
p3 <span class="op">=</span> myFifoStr.addPattern(r1,logf,label<span class="op">=</span><span class="st">"r1 detected"</span>)
myFifoStr.addPattern(r2,logf,label<span class="op">=</span><span class="st">"r2 hit"</span>)
myFifoStr.addPattern(f1,logf,label<span class="op">=</span><span class="st">"f1 hit"</span>)
myFifoStr.addPattern(f2,logf,label<span class="op">=</span><span class="st">"f2 hit"</span>)
<span class="co">#patterns can be set active/inactive via pattern management fns </span>
myFifoStr.setPatternActiveState(p1,<span class="va">False</span>) <span class="co">#based on index returned from addPattern</span>
<span class="co">#now show searching for stored pattern matchers in the pattern dict</span>
<span class="co">#this is not searching the fifo-string itself, just the stored patterns that we have entered</span>
<span class="bu">print</span>(<span class="st">"find pattern by label 'foo':"</span>,myFifoStr.findPatternByLabel(<span class="st">"foo"</span>)) <span class="co">#no matches returns empty list</span>
<span class="bu">print</span>(<span class="st">"find pattern by label '234 hit':"</span>,myFifoStr.findPatternByLabel(<span class="st">"234 hit"</span>)) <span class="co">#shows match</span>
<span class="bu">print</span>(<span class="st">"find pattern by label using regex '[rf][0-9]':"</span>)
pp.pprint(myFifoStr.findPatternByLabel(re.<span class="bu">compile</span>(<span class="st">"[rf][0-9]"</span>)))
<span class="co">#and finally demonstrate that patterns auto-trigger when items inserted in fifostr .. which afterall</span>
<span class="co">#is the point of the whole thing.. ;)</span>
<span class="bu">print</span>(<span class="st">"</span><span class="ch">\n</span><span class="st"> fifo operations ============"</span>)
<span class="cf">for</span> c <span class="kw">in</span> <span class="st">'01234567890abcdefghijklmnop'</span>: <span class="co">#show using inc which accomplishes same thing</span>
myFifoStr <span class="op">+=</span> c
myFifoStr<span class="op">+=</span> <span class="st">'abcdefghi'</span>
<span class="bu">print</span> (myFifoStr.<span class="bu">all</span>())</code></pre></div>
<h3 id="notes">Notes</h3>
<p>FIFOstr is not meant replacement for a compiler/parser front end though it can be used as complex tokenizer. Internally just iterates over stored patterns every time something is added to the fifostr object. If you do have a parser you wish to be called then just add it as a callback function so that every time the fifostr is updated with a char(s) it will call your parser to do the work. Your parser must return a boolean result if you wish to use the callback based triggering. Multiple custom parsers can be run in along with static string patterns or regexes.</p>
<div class="sourceCode"><pre class="sourceCode python"><code class="sourceCode python"><span class="co">#let your own parser do the work </span>
myFifo <span class="op">=</span> fifostr(<span class="dv">20</span>) <span class="co"># make a 20 char fifostr</span>
myFifo.addPattern(myParser,myCallbk) <span class="co">#myParser passed entire fifostr (as str) when char(s) added</span>
myFifo.addPattern(myParser,myCallbk2,<span class="dv">3</span>,<span class="dv">5</span>) <span class="co">#myParser passed fifostr btw (3,5). My Parser must return True if match found for callback to be invoked</span></code></pre></div>
<h3 id="source-code">Source code</h3>
<p>all source is at github:<br />
https://github.com/deftio/fifostr</p>
<h3 id="project-home">Project Home</h3>
<p>https://deftio.com/fifostr</p>
<h3 id="company-home">Company Home</h3>
<p>docs and other projects at<br />
https://deftio.com/</p>
<h3 id="tests-coverage">Tests & Coverage</h3>
<p>for quick usage see<br />
see <strong>main</strong> in example.py file</p>
<p>for test coverage look in the /tests directory<br />
to run tests pytest needs to be installed.</p>
<h4 id="on-ubuntu">on Ubuntu</h4>
<div class="sourceCode"><pre class="sourceCode bash"><code class="sourceCode bash"><span class="ex">pip</span> install -U pytest pytest-cov
<span class="ex">pip</span> install coveralls </code></pre></div>
<p>note: more info at pytest.org for installation on other OSes</p>
<div class="sourceCode"><pre class="sourceCode bash"><code class="sourceCode bash"><span class="co"># running basic tests</span>
<span class="bu">cd</span> tests
<span class="ex">pytest</span> #or py.test
<span class="co"># coverage stats below</span>
<span class="ex">coverage</span> run --source fifostr -m pytest
<span class="ex">coverage</span> report -m</code></pre></div>
<h3 id="generating-docs">Generating docs</h3>
<p>Documenation is generated using pandoc and pydoc from the build scripts.</p>
<div class="sourceCode"><pre class="sourceCode bash"><code class="sourceCode bash"><span class="fu">sudo</span> apt-get install pandoc</code></pre></div>
<p>documentation is in /docs directory (generated by pydoc) to (re)generate the docs. cd to the docs directory. then type:</p>
<div class="sourceCode"><pre class="sourceCode bash"><code class="sourceCode bash"><span class="ex">pydoc</span> -w ../fifostr.py </code></pre></div>
<p>note that as of this writing pydoc generates its output in the current directory and doesn't seem to be pipeable to another.</p>
<h3 id="release-history">Release History</h3>
<ul>
<li>1.1.19 added sys.displayhook chaining for interactive mode display</li>
<li>1.1.18 added Str constructor</li>
<li>1.1.17 updated to support 3.9x</li>
<li>1.1.16 updated to support python 3.7 3.8 3.9 in tests</li>
<li>1.1.15 updated PyPi to use README.md instead of README.rst (no other changes)</li>
<li>1.1.10 Updated docs and related usage info for repo</li>
<li>1.1.9 rebuild for README.md to README.rst conversion using pandoc (no code changes) for PyPi</li>
<li>1.1.8 rebuild to make sure proper pkg loaded to PyPi (no code changes)</li>
<li>1.1.7 updated MANIFEST.in to use README.rst</li>
<li>1.1.6 added PyPi version badge in README.md</li>
<li>1.1.5 coverage to 100%, added badging, added README.rst</li>
<li>1.1.x changed class name from fifostr to FIFOStr to make PEP8 compliant. fixed bug in setup.py (package_dir)</li>
<li>1.0.x documentation clean up</li>
<li>1.0.0 Initial release</li>
</ul>
<h3 id="readme.md-vs-readme.rst">README.md vs README.rst</h3>
<p>The README.rst is generated from the README.md using pandoc but the content is identical. (used for PyPi in earlier releases)</p>
<h3 id="license">License</h3>
<p>See LICENSE.txt file in this directory. The license is the OSI approved "FreeBSD" 2 clause license.</p>
<ol start="3" style="list-style-type: lower-alpha">
<li>2018 m a chatterjee</li>
</ol>