From b2fe1ac666639ae702633fb31714f06c9094b8d8 Mon Sep 17 00:00:00 2001 From: LaunchDarklyReleaseBot <86431345+LaunchDarklyReleaseBot@users.noreply.github.com> Date: Fri, 22 Apr 2022 09:54:50 -0700 Subject: [PATCH] prepare 7.4.1 release (#168) * Allow explicitly proxying only ld requests (#130) * fix broken indirect/patch request, add tests for feature requestor * Python 2/3 compatibility for HTTPServer * Py2/3 compatibility: queue * more Py3 compatibility * don't need import of builtins * fix string encoding * implement setting proxy URL by environment variable * rm debugging * fix autodoc options to exclude magic methods * comment * add end-to-end unit tests for proxy config * indents * add 3.8 build * image name * fail on SyntaxWarning * typo * command syntax * pin expiringdict dependency for Python 3.3 compatibility * add Windows CircleCI job * periods are no longer valid in CircleCI job names * syntax fix * install Python in Windows * set path * move command * turn off debug logging * Py3 in Windows * config param * rm redundant step * choco switch * refactor Linux jobs using CircleCI 2.1 features * set log level before anything else * rm Azure config * use yaml.safe_load() to avoid code execution vulnerability in file data source * Initial work on wrapper_name, wrapper_version, diagnostic config options and start of diagnostic config event creation. * Python 2 compat changes. * More event generation code and starting to integrate tracking diagnostic values. * Add minimum diagnostic recording interval. Fix diagnostic.py to be importable. Add more diagnostic event fields. * don't let user fall outside of last bucket in rollout * fixing conditional logic * Add docstrings for diagnostic configuration options. * fix off-by-1 error * avoid redundant dict lookups * add unit tests for basic bucketing logic and edge case * Stream init tracking. Feeding of accumulator object through SDK. Various fixes. * Track events in last batch. * Fix sdk version field, some stylistic improvements. * Last of diagnostic configuration object fields. * Fill out rest of platform fields. * Cleanup and failed stream initialization tracking. * Add diagnostic config option test. * Add tests for diagnostics.py * Testing rest of diagnostic fields. * Test that streaming update processor records successful and unsuccessful connection attempts in the diagnostic accumulator when available. * Improvements to testability of event processor. * Rest of event processor tests. * Remove janky reflection. * Test change to filesource optional test requirements. * [ch61092] Add event payload ID on event requests. * normalize data store type and OS name in diagnostic events * gitignore * copyedit to diagnostic event config property comment * fix spurious error after sending diagnostic event * make verify_ssl=False turn off certificate verification too (#129) * add more TLS config options and collect HTTP/HTTPS config options in a class (#130) * make stream retry/backoff/jitter behavior consistent with other SDKs + improve testing (#131) * streams shouldn't use the same read timeout as the rest of the SDK (#132) * implement our own retry logic & logging for event posts, don't use urllib3.Retry (#133) * remove support for indirect/patch and indirect/put * remove unused logic for individual flag/segment poll for indirect/patch * Ehaisley/84082/remove python2 (#136) * remove all references to six and remove queue fallback imports * remove NullHandler logger backwards compat * update circleci config to remove python 2.7 tests * remove ordereddict backwards compat * update setup.py to no longer list python 2.7 as compatible * no longer inherit from object for python 2 backwards compat * update readme and manifest to reflect python 2.7 removal * remove unicode type compatibility * remove 2.7 support from circleci * Allow authenticating with proxy This commit allows for authenticating with a proxy configured with the `http_proxy` environment variable. Authentication requires passing a header, and is not parsed by urllib3 from the proxy_url. * reimplement proxy tests for DRY and add test of proxy auth params * doc comment on auth params in proxy URL * add type hints to some of the public facing api. update some docs * Revert "add type hints to some of the public facing api." This reverts commit c35fa6184ce1a274fd5c6d226cb3f1f7a795901a. * Ehaisley/ch86857/type hints (#138) * add typehints to the public API * validate typehints in the public api and tests with mypy * remove all current deprecations (#139) * remove all currently deprecated classes, methods, arguments, and tests * also update semver usage to remove calls to deprecated functions and classes * remove global set_sdk_key, make SDK key required in Config (#140) * Removed the guides link * Pinning mypy and running it against different python versions (#141) * fix time zone mishandling that could make event debugging not work (#142) * fix 6.x build (#143) * fix time zone mishandling that could make event debugging not work (6.x) (#144) * prepare 6.13.3 release (#154) * Releasing version 6.13.3 * [ch99756] Add alias events (#145) * add support for experiment rollouts * fix unit test * address PR comments * use Releaser v2 config * Use newer docker images (#147) * Updates docs URLs * Add support for 3.10 (#150) * started work on FlagBuilder in as part of test data source implementation * finished FlagBuilder implementation and added FlagRuleBuilder implementation * added initial TestData interface and updated tests to not rely on test data internals * started data source implementation * changed FlagBuilder to public class; changed FlagBuilder attributes to be initialized in __init__ and eliminated use of try ... except: pass for handling empty attributes * (big segments 1) add public config/interface types * added implementation of test data source * docstring * formatting * ensure property doesn't return None * (big segments 2) implement evaluation, refactor eval logic & modules * linting * (big segments 3) implement big segment status tracking, wire up components * typing fixes * typing fixes * implement SSE contract tests * fix CI * fix CI again * fix CI * disable SSE tests in Python 3.5 * make test service port configurable * better SSE implementation that fixes linefeed and multi-byte char issues * fix constructor parameters in test service * comment * test improvements * rm obsolete default config logic * (big segments 4) implement big segment stores in Redis+DynamoDB, refactor db tests (#158) * converted ldclient.integrations module from file to directory; started moving public classes out of ldclient.impl.integrations.test_data* and instead into ldclient.integrations.test_data*; started adding TestData documentation * removed setup/teardown functions leftover from test scaffold * added TestData, FlagBuilder, and FlagRuleBuilder documentation; minor adjustments to implementation details * removed warning supression from TestData tests * fix big segments user hash algorithm to use SHA256 * update mypy version * updates to tests and related bug fixes * always cache Big Segment query result even if it's None * fix test assertion * lint * fix big segment ref format * fix big segments cache TTL being set to wrong value * fixed structure of fallthrough variation in result of FlagBuilder.build() * moved __test__ attribute into TestData class definition to prevent mypy from complaining about a missing class attribute * minor doc comment fix * Apply suggestions related to Sphinx docstring formatting from code review Co-authored-by: Eli Bishop * fixed errors in the implementation of FlagBuilder's fallthrough_variation and off_variation when passing boolean variation values; updated tests to assert the expected behavior * added missing value_for_all_users() method to FlagBuilder class * Fix operator parsing errors (#169) * identify should not emit event if user key is empty (#164) * secondary should be treated as built-in attribute (#168) * URIs should have trailing slashes trimmed (#165) * all_flags_state should always include flag version (#166) * output event should not include a null prereqOf key (#167) * Account for traffic allocation on all flags (#171) * Add SDK contract tests (#170) * misc fixes to test data docs + add type hints * more type hints * remove some methods from the public test_data API * can't use "x|y" shortcut in typehints in older Pythons; use Union * fix misc type mistakes because I forgot to run the linter * update CONTRIBUTING.md and provide make targets * fixed a bug with flag rule clause builder internals; added unit test to verify rule evaluation * added ready argument to _TestDataSource class and indicated ready upon start to avoid delays in TestData initialization * Update contract tests to latest flask version (#176) Our contract tests depend on flask v1, which in turn depends on Jinja 2. Both of these are terribly dated and no longer supported. Jinja depends on markupsafe. markupsafe recently updated its code to no longer provide soft_unicode which in turn broke Jinja. Updating to the latest flask keeps all transitive dependencies better aligned and addresses this mismatch. * Handle explicit None values in test payload (#179) The test harness may send explicit None values which should be treated the same as if the value was omitted entirely. * Fix "unhandled response" error in test harness (#180) When we return a `('', 204)` response from the flask handler, [Werkzeug intentionally removes the 'Content-Type' header][1], which causes the response to be created as a chunked response. The test harness is likely seeing a 204 response and isn't trying to read anything more from the stream. But since we are re-using connections, the next time it reads from the stream, it sees the `0\r\n\r\n` chunk and outputs an error: > 2022/04/20 14:23:39 Unsolicited response received on idle HTTP channel starting with "0\r\n\r\n"; err= Changing this response to 202 causes Werkzeug to return an empty response and silences the error. [1]: https://github.com/pallets/werkzeug/blob/560dd5f320bff318175f209595d42f5a80045417/src/werkzeug/wrappers/response.py#L540 * Exclude booleans when getting bucketable value (#181) When calculating a bucket, we get the bucketable value from the specified bucket by attribute. If this value is a string or an int, we can use it. Otherwise, we return None. Python considers a bool an instance of an int, which isn't what we want. So we need to add an explicit exclusion for this. Co-authored-by: Eli Bishop Co-authored-by: Gabor Angeli Co-authored-by: LaunchDarklyCI Co-authored-by: Ben Woskow Co-authored-by: Gavin Whelan Co-authored-by: Elliot <35050275+Apache-HB@users.noreply.github.com> Co-authored-by: Gabor Angeli Co-authored-by: Elliot Co-authored-by: Ben Woskow <48036130+bwoskow-ld@users.noreply.github.com> Co-authored-by: LaunchDarklyCI Co-authored-by: hroederld Co-authored-by: Robert J. Neal Co-authored-by: Robert J. Neal Co-authored-by: Ember Stevens Co-authored-by: ember-stevens <79482775+ember-stevens@users.noreply.github.com> Co-authored-by: Matthew M. Keeler Co-authored-by: charukiewicz Co-authored-by: LaunchDarklyReleaseBot Co-authored-by: Christian Charukiewicz Co-authored-by: Matthew M. Keeler --- .circleci/config.yml | 21 ++++++++---- contract-tests/client_entity.py | 19 +++++------ contract-tests/requirements.txt | 2 +- contract-tests/service.py | 4 +-- ldclient/impl/evaluator.py | 7 +++- .../redis/redis_big_segment_store.py | 5 ++- .../test_data/test_data_source.py | 4 ++- ldclient/integrations/test_data.py | 6 ++-- sse-contract-tests/requirements.txt | 2 +- sse-contract-tests/service.py | 2 +- testing/integrations/test_test_data_source.py | 34 ++++++++++++++++++- 11 files changed, 76 insertions(+), 30 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index e3d5b29c..007b5fb2 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -10,6 +10,7 @@ workflows: name: Python 3.5 docker-image: cimg/python:3.5 skip-sse-contract-tests: true # the test service app has dependencies that aren't available in 3.5, which is EOL anyway + skip-contract-tests: true # the test service app has dependencies that aren't available in 3.5, which is EOL anyway - test-linux: name: Python 3.6 docker-image: cimg/python:3.6 @@ -46,6 +47,9 @@ jobs: skip-sse-contract-tests: type: boolean default: false + skip-contract-tests: + type: boolean + default: false docker: - image: <> - image: redis @@ -109,13 +113,16 @@ jobs: name: run SSE contract tests command: cd sse-contract-tests && make run-contract-tests - - run: make build-contract-tests - - run: - command: make start-contract-test-service - background: true - - run: - name: run contract tests - command: TEST_HARNESS_PARAMS="-junit test-reports/contract-tests-junit.xml" make run-contract-tests + - unless: + condition: <> + steps: + - run: make build-contract-tests + - run: + command: make start-contract-test-service + background: true + - run: + name: run contract tests + command: TEST_HARNESS_PARAMS="-junit test-reports/contract-tests-junit.xml" make run-contract-tests - store_test_results: path: test-reports diff --git a/contract-tests/client_entity.py b/contract-tests/client_entity.py index f3bf22fc..5d2d5220 100644 --- a/contract-tests/client_entity.py +++ b/contract-tests/client_entity.py @@ -6,39 +6,36 @@ sys.path.insert(1, os.path.join(sys.path[0], '..')) from ldclient import * -def millis_to_seconds(t): - return None if t is None else t / 1000 - class ClientEntity: def __init__(self, tag, config): self.log = logging.getLogger(tag) opts = {"sdk_key": config["credential"]} - if "streaming" in config: + if config.get("streaming") is not None: streaming = config["streaming"] - if "baseUri" in streaming: + if streaming.get("baseUri") is not None: opts["stream_uri"] = streaming["baseUri"] if streaming.get("initialRetryDelayMs") is not None: opts["initial_reconnect_delay"] = streaming["initialRetryDelayMs"] / 1000.0 - if "events" in config: + if config.get("events") is not None: events = config["events"] - if "baseUri" in events: + if events.get("baseUri") is not None: opts["events_uri"] = events["baseUri"] - if events.get("capacity", None) is not None: + if events.get("capacity") is not None: opts["events_max_pending"] = events["capacity"] opts["diagnostic_opt_out"] = not events.get("enableDiagnostics", False) opts["all_attributes_private"] = events.get("allAttributesPrivate", False) opts["private_attribute_names"] = events.get("globalPrivateAttributes", {}) - if "flushIntervalMs" in events: + if events.get("flushIntervalMs") is not None: opts["flush_interval"] = events["flushIntervalMs"] / 1000.0 - if "inlineUsers" in events: + if events.get("inlineUsers") is not None: opts["inline_users_in_events"] = events["inlineUsers"] else: opts["send_events"] = False - start_wait = config.get("startWaitTimeMs", 5000) + start_wait = config.get("startWaitTimeMs") or 5000 config = Config(**opts) self.client = client.LDClient(config, start_wait / 1000.0) diff --git a/contract-tests/requirements.txt b/contract-tests/requirements.txt index f55a4204..0018e4c8 100644 --- a/contract-tests/requirements.txt +++ b/contract-tests/requirements.txt @@ -1,2 +1,2 @@ -Flask==1.1.4 +Flask==2.0.3 urllib3>=1.22.0 diff --git a/contract-tests/service.py b/contract-tests/service.py index b4728867..d9f8e0a5 100644 --- a/contract-tests/service.py +++ b/contract-tests/service.py @@ -4,7 +4,7 @@ import logging import os import sys -from flask import Flask, request, jsonify +from flask import Flask, request from flask.logging import default_handler from logging.config import dictConfig from werkzeug.exceptions import HTTPException @@ -132,7 +132,7 @@ def delete_client(id): return ('', 404) client.close() - return ('', 204) + return ('', 202) if __name__ == "__main__": port = default_port diff --git a/ldclient/impl/evaluator.py b/ldclient/impl/evaluator.py index d019f10d..0fa9f088 100644 --- a/ldclient/impl/evaluator.py +++ b/ldclient/impl/evaluator.py @@ -243,7 +243,12 @@ def _bucket_user(seed, user, key, salt, bucket_by): return result def _bucketable_string_value(u_value): - return str(u_value) if isinstance(u_value, (str, int)) else None + if isinstance(u_value, bool): + return None + elif isinstance(u_value, (str, int)): + return str(u_value) + + return None def _clause_matches_user_no_segments(clause, user): u_value, should_pass = _get_user_attribute(user, clause.get('attribute')) diff --git a/ldclient/impl/integrations/redis/redis_big_segment_store.py b/ldclient/impl/integrations/redis/redis_big_segment_store.py index 35b42b71..d3b4b767 100644 --- a/ldclient/impl/integrations/redis/redis_big_segment_store.py +++ b/ldclient/impl/integrations/redis/redis_big_segment_store.py @@ -26,7 +26,10 @@ def __init__(self, url: str, prefix: Optional[str], max_connections: int): def get_metadata(self) -> BigSegmentStoreMetadata: r = redis.Redis(connection_pool=self._pool) value = r.get(self._prefix + self.KEY_LAST_UP_TO_DATE) - return BigSegmentStoreMetadata(None if value is None else int(value)) + if value is None: + return BigSegmentStoreMetadata(None) + + return BigSegmentStoreMetadata(int(value)) def get_membership(self, user_hash: str) -> Optional[dict]: r = redis.Redis(connection_pool=self._pool) diff --git a/ldclient/impl/integrations/test_data/test_data_source.py b/ldclient/impl/integrations/test_data/test_data_source.py index db3ac729..e6272925 100644 --- a/ldclient/impl/integrations/test_data/test_data_source.py +++ b/ldclient/impl/integrations/test_data/test_data_source.py @@ -5,11 +5,13 @@ class _TestDataSource(): - def __init__(self, feature_store, test_data): + def __init__(self, feature_store, test_data, ready): self._feature_store = feature_store self._test_data = test_data + self._ready = ready def start(self): + self._ready.set() self._feature_store.init(self._test_data._make_init_data()) def stop(self): diff --git a/ldclient/integrations/test_data.py b/ldclient/integrations/test_data.py index a159eb12..0030cde6 100644 --- a/ldclient/integrations/test_data.py +++ b/ldclient/integrations/test_data.py @@ -51,7 +51,7 @@ def __init__(self): self._instances = [] def __call__(self, config, store, ready): - data_source = _TestDataSource(store, self) + data_source = _TestDataSource(store, self, ready) try: self._lock.lock() self._instances.append(data_source) @@ -485,7 +485,7 @@ def and_match(self, attribute: str, *values) -> 'FlagRuleBuilder': """ self._clauses.append({ 'attribute': attribute, - 'operator': 'in', + 'op': 'in', 'values': list(values), 'negate': False }) @@ -508,7 +508,7 @@ def and_not_match(self, attribute: str, *values) -> 'FlagRuleBuilder': """ self._clauses.append({ 'attribute': attribute, - 'operator': 'in', + 'op': 'in', 'values': list(values), 'negate': True }) diff --git a/sse-contract-tests/requirements.txt b/sse-contract-tests/requirements.txt index 2d1d2a7b..0018e4c8 100644 --- a/sse-contract-tests/requirements.txt +++ b/sse-contract-tests/requirements.txt @@ -1,2 +1,2 @@ -Flask==2.0.2 +Flask==2.0.3 urllib3>=1.22.0 diff --git a/sse-contract-tests/service.py b/sse-contract-tests/service.py index 6d07fc59..389b1a1f 100644 --- a/sse-contract-tests/service.py +++ b/sse-contract-tests/service.py @@ -81,7 +81,7 @@ def delete_stream(id): if stream is None: return ('', 404) stream.close() - return ('', 204) + return ('', 202) if __name__ == "__main__": port = default_port diff --git a/testing/integrations/test_test_data_source.py b/testing/integrations/test_test_data_source.py index e0db1208..47f0d025 100644 --- a/testing/integrations/test_test_data_source.py +++ b/testing/integrations/test_test_data_source.py @@ -285,7 +285,7 @@ def test_flagbuilder_can_build(): 'clauses': [ {'attribute': 'country', 'negate': False, - 'operator': 'in', + 'op': 'in', 'values': ['fr'] } ], @@ -297,3 +297,35 @@ def test_flagbuilder_can_build(): } assert flag._build(1) == expected_result + +def test_flag_can_evaluate_rules(): + td = TestData.data_source() + store = InMemoryFeatureStore() + + client = LDClient(config=Config('SDK_KEY', + update_processor_class = td, + send_events = False, + feature_store = store)) + + td.update(td.flag(key='test-flag') + .fallthrough_variation(False) + .if_match('firstName', 'Mike') + .and_not_match('country', 'gb') + .then_return(True)) + + # user1 should satisfy the rule (matching firstname, not matching country) + user1 = { 'key': 'user1', 'firstName': 'Mike', 'country': 'us' } + eval1 = client.variation_detail('test-flag', user1, default='default') + + assert eval1.value == True + assert eval1.variation_index == 0 + assert eval1.reason['kind'] == 'RULE_MATCH' + + # user2 should NOT satisfy the rule (not matching firstname despite not matching country) + user2 = { 'key': 'user2', 'firstName': 'Joe', 'country': 'us' } + eval2 = client.variation_detail('test-flag', user2, default='default') + + assert eval2.value == False + assert eval2.variation_index == 1 + assert eval2.reason['kind'] == 'FALLTHROUGH' +