Integration tests executed on a real deployment as part of the CICD #1220
Labels
Comments
dlpzx
added
type: newfeature
|
In PR #1219 the design for integration tests in AWS is introduced. In that PR only tests for Organizations are included. We slowly need to add tests for other core and modules APIs
Modules. ---> they depend on core modules, it might be difficult to test them without environments
|
dlpzx
added a commit
that referenced
this issue
Apr 30, 2024
… CICD (#1219) ### Feature or Bugfix - Feature ### Detail Add integration tests that use a real Client to execute different validation actions. - Define the Client and the way API calls are posted to API Gateway in the conftest - Define the Cognito users and the different fixtures needed for all tests - Write tests for the Organization core module as example - Add feature flag in `cdk.json` called `with_approval_tests` that can be defined at the deployment environment level. If set to True, a CodeBuild stage running the tests is created. ### Relates - #1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
noah-paige
added a commit
that referenced
this issue
Jun 25, 2024
commit 6968e67c Author: Noah Paige <[email protected]> Date: Fri May 17 2024 16:12:45 GMT-0400 (Eastern Daylight Time) Get to v2.5.0 commit 93ff7725 Author: Sofia Sazonova <[email protected]> Date: Mon May 13 2024 08:00:38 GMT-0400 (Eastern Daylight Time) Update version.json (#1264) Release info update commit e718d861 Author: Sofia Sazonova <[email protected]> Date: Mon May 13 2024 07:29:27 GMT-0400 (Eastern Daylight Time) fix permission query (#1263) ### Feature or Bugfix - Bugfix ### Detail - The filter -- array of permissions' NAMES, so in order to query policies correctly we need to add join - The filter 'share_type' and 'share_item_status' must be string - IMPORTANT: in block "finally" the param session was used, but session was defined only in "try" block. So, the lock failed to be released. ### Relates - <URL or Ticket> ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <[email protected]> commit 479b8f3f Author: mourya-33 <[email protected]> Date: Wed May 08 2024 10:29:36 GMT-0400 (Eastern Daylight Time) Add encryption and tag immutability to ECR repository (#1224) ### Feature or Bugfix - Bugfix ### Detail - Currently the ecr repository created do not have encryption and tag immutability enabled which is identified by checkov scans. This fix is to enable both. ### Relates [- <URL or Ticket>](https://github.com/data-dot-all/dataall/issues/1200) ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes N/A - Is the input sanitized? N/A - What precautions are you taking before deserializing the data you consume? N/A - Is injection prevented by parametrizing queries? N/A - Have you ensured no `eval` or similar functions are used? N/A - Does this PR introduce any functionality or component that requires authorization? N/A - How have you ensured it respects the existing AuthN/AuthZ mechanisms? N/A - Are you logging failed auth attempts? N/A - Are you using or adding any cryptographic features? N/A - Do you use a standard proven implementations? N/A - Are the used keys controlled by the customer? Where are they stored? No. This is with default encryption - Are you introducing any new policies/roles/users? N/A - Have you used the least-privilege principle? How? N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 2f885773 Author: Sofia Sazonova <[email protected]> Date: Wed May 08 2024 09:22:40 GMT-0400 (Eastern Daylight Time) Multiple permission roots (#1259) ### Feature or Bugfix - Bugfix ### Detail - GET_DATASET_TABLE (FOLDER) permissions are granted to the group only if they are not granted already - these permissions are removed if group is not admin|steward and there are no other shares of this item. ### Relates - #1174 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <[email protected]> commit c4cc07ee Author: Petros Kalos <[email protected]> Date: Wed May 08 2024 08:54:02 GMT-0400 (Eastern Daylight Time) explicitly specify dataset_client s3 endpoint_url (#1260) * AWS requires that the endpoint_url should be explicitly specified for some regions * Remove misleading CORS error message, the upload step can fail for many reason ### Feature or Bugfix - Bugfix ### Detail Resolves #778 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 40defe8e Author: dlpzx <[email protected]> Date: Tue May 07 2024 11:52:17 GMT-0400 (Eastern Daylight Time) Generic dataset module and specific s3_datasets module - part 1 (Rename datasets as s3_datasets) (#1250) ### Feature or Bugfix - Refactoring ### Detail - Rename `datasets` module to `s3_datasets` module This PR is the first step to extract a generic datasets_base module that implements the undifferentiated concepts of Dataset in data.all. s3_datasets will use this base module to implement the specific implementation for S3 datatasets. ### Relates - #1123 - #955 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 74a303cb Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue May 07 2024 02:26:09 GMT-0400 (Eastern Daylight Time) Bump werkzeug from 3.0.1 to 3.0.3 in /tests (#1253) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.1 to 3.0.3. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/releases">werkzeug's releases</a>.</em></p> <blockquote> <h2>3.0.3</h2> <p>This is the Werkzeug 3.0.3 security release, which fixes security issues and bugs but does not otherwise change behavior and should not result in breaking changes.</p> <p>PyPI: <a href="https://pypi.org/project/Werkzeug/3.0.3/">https://pypi.org/project/Werkzeug/3.0.3/</a> Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3</a> Milestone: <a href="https://github.com/pallets/werkzeug/milestone/35?closed=1">https://github.com/pallets/werkzeug/milestone/35?closed=1</a></p> <ul> <li>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. GHSA-2g68-c3qc-8985</li> <li>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2823">#2823</a></li> <li>Better TLS cert format with <code>adhoc</code> dev certs. <a href="https://redirect.github.com/pallets/werkzeug/issues/2891">#2891</a></li> <li>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. <a href="https://redirect.github.com/pallets/werkzeug/issues/2828">#2828</a></li> <li>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2836">#2836</a></li> </ul> <h2>3.0.2</h2> <p>This is a fix release for the 3.0.x feature branch.</p> <ul> <li>Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/blob/main/CHANGES.rst">werkzeug's changelog</a>.</em></p> <blockquote> <h2>Version 3.0.3</h2> <p>Released 2024-05-05</p> <ul> <li> <p>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. :ghsa:<code>2g68-c3qc-8985</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> <li> <p>Better TLS cert format with <code>adhoc</code> dev certs. :pr:<code>2891</code></p> </li> <li> <p>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. :issue:<code>2828</code></p> </li> <li> <p>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. :issue:<code>2836</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> </ul> <h2>Version 3.0.2</h2> <p>Released 2024-04-01</p> <ul> <li>Ensure setting <code>merge_slashes</code> to <code>False</code> results in <code>NotFound</code> for repeated-slash requests against single slash routes. :issue:<code>2834</code></li> <li>Fix handling of <code>TypeError</code> in <code>TypeConversionDict.get()</code> to match <code>ValueError</code>. :issue:<code>2843</code></li> <li>Fix <code>response_wrapper</code> type check in test client. :issue:<code>2831</code></li> <li>Make the return type of <code>MultiPartParser.parse</code> more precise. :issue:<code>2840</code></li> <li>Raise an error if converter arguments cannot be parsed. :issue:<code>2822</code></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pallets/werkzeug/commit/f9995e967979eb694d6b31536cc65314fd7e9c8c"><code>f9995e9</code></a> release version 3.0.3</li> <li><a href="https://github.com/pallets/werkzeug/commit/3386395b24c7371db11a5b8eaac0c91da5362692"><code>3386395</code></a> Merge pull request from GHSA-2g68-c3qc-8985</li> <li><a href="https://github.com/pallets/werkzeug/commit/890b6b62634fa61224222aee31081c61b054ff01"><code>890b6b6</code></a> only require trusted host for evalex</li> <li><a href="https://github.com/pallets/werkzeug/commit/71b69dfb7df3d912e66bab87fbb1f21f83504967"><code>71b69df</code></a> restrict debugger trusted hosts</li> <li><a href="https://github.com/pallets/werkzeug/commit/d2d3869525a4ffb2c41dfb2c0e39d94dab2d870c"><code>d2d3869</code></a> endpoint type is Any (<a href="https://redirect.github.com/pallets/werkzeug/issues/2895">#2895</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/7080b55acd48b68afdda65ee6c7f99e9afafb0ba"><code>7080b55</code></a> endpoint type is Any</li> <li><a href="https://github.com/pallets/werkzeug/commit/7555eff296fbdf12f2e576b6bbb0b506df8417ed"><code>7555eff</code></a> remove iri_to_uri redirect workaround (<a href="https://redirect.github.com/pallets/werkzeug/issues/2894">#2894</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/97fb2f722297ae4e12e36dab024e0acf8477b3c8"><code>97fb2f7</code></a> remove _invalid_iri_to_uri workaround</li> <li><a href="https://github.com/pallets/werkzeug/commit/249527ff981e7aa22cd714825c5637cc92df7761"><code>249527f</code></a> make cn field a valid single hostname, and use wildcard in SANs field. (<a href="https://redirect.github.com/pallets/werkzeug/issues/2892">#2892</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/793be472c9d145eb9be7d4200672d1806289d84a"><code>793be47</code></a> update adhoc tls dev cert format</li> <li>Additional commits viewable in <a href="https://github.com/pallets/werkzeug/compare/3.0.1...3.0.3">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/data-dot-all/dataall/network/alerts). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 2f33320c Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue May 07 2024 02:25:03 GMT-0400 (Eastern Daylight Time) Bump werkzeug from 3.0.1 to 3.0.3 in /backend/dataall/base/cdkproxy (#1252) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.1 to 3.0.3. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/releases">werkzeug's releases</a>.</em></p> <blockquote> <h2>3.0.3</h2> <p>This is the Werkzeug 3.0.3 security release, which fixes security issues and bugs but does not otherwise change behavior and should not result in breaking changes.</p> <p>PyPI: <a href="https://pypi.org/project/Werkzeug/3.0.3/">https://pypi.org/project/Werkzeug/3.0.3/</a> Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3</a> Milestone: <a href="https://github.com/pallets/werkzeug/milestone/35?closed=1">https://github.com/pallets/werkzeug/milestone/35?closed=1</a></p> <ul> <li>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. GHSA-2g68-c3qc-8985</li> <li>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2823">#2823</a></li> <li>Better TLS cert format with <code>adhoc</code> dev certs. <a href="https://redirect.github.com/pallets/werkzeug/issues/2891">#2891</a></li> <li>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. <a href="https://redirect.github.com/pallets/werkzeug/issues/2828">#2828</a></li> <li>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2836">#2836</a></li> </ul> <h2>3.0.2</h2> <p>This is a fix release for the 3.0.x feature branch.</p> <ul> <li>Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/blob/main/CHANGES.rst">werkzeug's changelog</a>.</em></p> <blockquote> <h2>Version 3.0.3</h2> <p>Released 2024-05-05</p> <ul> <li> <p>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. :ghsa:<code>2g68-c3qc-8985</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> <li> <p>Better TLS cert format with <code>adhoc</code> dev certs. :pr:<code>2891</code></p> </li> <li> <p>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. :issue:<code>2828</code></p> </li> <li> <p>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. :issue:<code>2836</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> </ul> <h2>Version 3.0.2</h2> <p>Released 2024-04-01</p> <ul> <li>Ensure setting <code>merge_slashes</code> to <code>False</code> results in <code>NotFound</code> for repeated-slash requests against single slash routes. :issue:<code>2834</code></li> <li>Fix handling of <code>TypeError</code> in <code>TypeConversionDict.get()</code> to match <code>ValueError</code>. :issue:<code>2843</code></li> <li>Fix <code>response_wrapper</code> type check in test client. :issue:<code>2831</code></li> <li>Make the return type of <code>MultiPartParser.parse</code> more precise. :issue:<code>2840</code></li> <li>Raise an error if converter arguments cannot be parsed. :issue:<code>2822</code></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pallets/werkzeug/commit/f9995e967979eb694d6b31536cc65314fd7e9c8c"><code>f9995e9</code></a> release version 3.0.3</li> <li><a href="https://github.com/pallets/werkzeug/commit/3386395b24c7371db11a5b8eaac0c91da5362692"><code>3386395</code></a> Merge pull request from GHSA-2g68-c3qc-8985</li> <li><a href="https://github.com/pallets/werkzeug/commit/890b6b62634fa61224222aee31081c61b054ff01"><code>890b6b6</code></a> only require trusted host for evalex</li> <li><a href="https://github.com/pallets/werkzeug/commit/71b69dfb7df3d912e66bab87fbb1f21f83504967"><code>71b69df</code></a> restrict debugger trusted hosts</li> <li><a href="https://github.com/pallets/werkzeug/commit/d2d3869525a4ffb2c41dfb2c0e39d94dab2d870c"><code>d2d3869</code></a> endpoint type is Any (<a href="https://redirect.github.com/pallets/werkzeug/issues/2895">#2895</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/7080b55acd48b68afdda65ee6c7f99e9afafb0ba"><code>7080b55</code></a> endpoint type is Any</li> <li><a href="https://github.com/pallets/werkzeug/commit/7555eff296fbdf12f2e576b6bbb0b506df8417ed"><code>7555eff</code></a> remove iri_to_uri redirect workaround (<a href="https://redirect.github.com/pallets/werkzeug/issues/2894">#2894</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/97fb2f722297ae4e12e36dab024e0acf8477b3c8"><code>97fb2f7</code></a> remove _invalid_iri_to_uri workaround</li> <li><a href="https://github.com/pallets/werkzeug/commit/249527ff981e7aa22cd714825c5637cc92df7761"><code>249527f</code></a> make cn field a valid single hostname, and use wildcard in SANs field. (<a href="https://redirect.github.com/pallets/werkzeug/issues/2892">#2892</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/793be472c9d145eb9be7d4200672d1806289d84a"><code>793be47</code></a> update adhoc tls dev cert format</li> <li>Additional commits viewable in <a href="https://github.com/pallets/werkzeug/compare/3.0.1...3.0.3">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/data-dot-all/dataall/network/alerts). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 0b49633f Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue May 07 2024 02:24:34 GMT-0400 (Eastern Daylight Time) Bump werkzeug from 3.0.1 to 3.0.3 in /tests_new/integration_tests (#1254) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.1 to 3.0.3. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/releases">werkzeug's releases</a>.</em></p> <blockquote> <h2>3.0.3</h2> <p>This is the Werkzeug 3.0.3 security release, which fixes security issues and bugs but does not otherwise change behavior and should not result in breaking changes.</p> <p>PyPI: <a href="https://pypi.org/project/Werkzeug/3.0.3/">https://pypi.org/project/Werkzeug/3.0.3/</a> Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3</a> Milestone: <a href="https://github.com/pallets/werkzeug/milestone/35?closed=1">https://github.com/pallets/werkzeug/milestone/35?closed=1</a></p> <ul> <li>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. GHSA-2g68-c3qc-8985</li> <li>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2823">#2823</a></li> <li>Better TLS cert format with <code>adhoc</code> dev certs. <a href="https://redirect.github.com/pallets/werkzeug/issues/2891">#2891</a></li> <li>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. <a href="https://redirect.github.com/pallets/werkzeug/issues/2828">#2828</a></li> <li>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2836">#2836</a></li> </ul> <h2>3.0.2</h2> <p>This is a fix release for the 3.0.x feature branch.</p> <ul> <li>Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/blob/main/CHANGES.rst">werkzeug's changelog</a>.</em></p> <blockquote> <h2>Version 3.0.3</h2> <p>Released 2024-05-05</p> <ul> <li> <p>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. :ghsa:<code>2g68-c3qc-8985</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> <li> <p>Better TLS cert format with <code>adhoc</code> dev certs. :pr:<code>2891</code></p> </li> <li> <p>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. :issue:<code>2828</code></p> </li> <li> <p>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. :issue:<code>2836</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> </ul> <h2>Version 3.0.2</h2> <p>Released 2024-04-01</p> <ul> <li>Ensure setting <code>merge_slashes</code> to <code>False</code> results in <code>NotFound</code> for repeated-slash requests against single slash routes. :issue:<code>2834</code></li> <li>Fix handling of <code>TypeError</code> in <code>TypeConversionDict.get()</code> to match <code>ValueError</code>. :issue:<code>2843</code></li> <li>Fix <code>response_wrapper</code> type check in test client. :issue:<code>2831</code></li> <li>Make the return type of <code>MultiPartParser.parse</code> more precise. :issue:<code>2840</code></li> <li>Raise an error if converter arguments cannot be parsed. :issue:<code>2822</code></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pallets/werkzeug/commit/f9995e967979eb694d6b31536cc65314fd7e9c8c"><code>f9995e9</code></a> release version 3.0.3</li> <li><a href="https://github.com/pallets/werkzeug/commit/3386395b24c7371db11a5b8eaac0c91da5362692"><code>3386395</code></a> Merge pull request from GHSA-2g68-c3qc-8985</li> <li><a href="https://github.com/pallets/werkzeug/commit/890b6b62634fa61224222aee31081c61b054ff01"><code>890b6b6</code></a> only require trusted host for evalex</li> <li><a href="https://github.com/pallets/werkzeug/commit/71b69dfb7df3d912e66bab87fbb1f21f83504967"><code>71b69df</code></a> restrict debugger trusted hosts</li> <li><a href="https://github.com/pallets/werkzeug/commit/d2d3869525a4ffb2c41dfb2c0e39d94dab2d870c"><code>d2d3869</code></a> endpoint type is Any (<a href="https://redirect.github.com/pallets/werkzeug/issues/2895">#2895</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/7080b55acd48b68afdda65ee6c7f99e9afafb0ba"><code>7080b55</code></a> endpoint type is Any</li> <li><a href="https://github.com/pallets/werkzeug/commit/7555eff296fbdf12f2e576b6bbb0b506df8417ed"><code>7555eff</code></a> remove iri_to_uri redirect workaround (<a href="https://redirect.github.com/pallets/werkzeug/issues/2894">#2894</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/97fb2f722297ae4e12e36dab024e0acf8477b3c8"><code>97fb2f7</code></a> remove _invalid_iri_to_uri workaround</li> <li><a href="https://github.com/pallets/werkzeug/commit/249527ff981e7aa22cd714825c5637cc92df7761"><code>249527f</code></a> make cn field a valid single hostname, and use wildcard in SANs field. (<a href="https://redirect.github.com/pallets/werkzeug/issues/2892">#2892</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/793be472c9d145eb9be7d4200672d1806289d84a"><code>793be47</code></a> update adhoc tls dev cert format</li> <li>Additional commits viewable in <a href="https://github.com/pallets/werkzeug/compare/3.0.1...3.0.3">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/data-dot-all/dataall/network/alerts). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 08862420 Author: mourya-33 <[email protected]> Date: Tue May 07 2024 02:15:15 GMT-0400 (Eastern Daylight Time) Updated lambda_api.py to add encryption for lambda env vars for custo… (#1255) Feature or Bugfix Bugfix Detail The environment variables for the lambda functions are not encrypted in cdk which are identified by checkov scans. This fix is to enable kms encryption for the lambda environment variables. Relates Security Please answer the questions below briefly where applicable, or write N/A. Based on [OWASP 10](https://owasp.org/Top10/en/). Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? N/A Is the input sanitized? N/A What precautions are you taking before deserializing the data you consume? N/A Is injection prevented by parametrizing queries? N/A Have you ensured no eval or similar functions are used? N/A Does this PR introduce any functionality or component that requires authorization? N/A How have you ensured it respects the existing AuthN/AuthZ mechanisms? N/A Are you logging failed auth attempts? N/A Are you using or adding any cryptographic features? N/A Do you use a standard proven implementations? N/A Are the used keys controlled by the customer? Where are they stored? the KMS keys are generated by cdk and are used to encrypt the environment variables for all lambda functions in the lambda-api stack Are you introducing any new policies/roles/users? - N/A Have you used the least-privilege principle? How? N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit ed7cc3eb Author: Noah Paige <[email protected]> Date: Mon May 06 2024 09:32:30 GMT-0400 (Eastern Daylight Time) Add order_by for paginated queries (#1249) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - This PR aims to solve the following - (1) for particular queries (identified as ones that perform `.outerjoin()` operations and have results paginated with `paginate()` function - sometimes the returned query results is *less than* the limit set by the pageSize of the paginate function even when the total count is greater than the pageSize - Ex 1: 11 envs total, `query_user_environments()` returning 9 envs on 1st page + 2 on 2nd page - Ex 2: 10 envs total, `query_user_environments()` returning 9 envs on 1st page + no 2nd page - Believe this is to be happening due to the way SQLAlchemy is "uniquing" the records resulted from an outerjoin and then returning that result back to the frontend - Adding a `.distinct()` check on the query ensures each distinct record is returned (tested successfully) - (2) Currently we often times do not implement an `.order_by()` condition for the query used in `paginate()` and do not have a stable way of preserving order of the items returned from a query (i.e. when navigating through pages of response) - A generally good practice seems to include an `order_by()` on a column or set of columns - For each query used in `paginate()` this PR adds an `order_by()` condition (full list in comments below) Can read a bit more context from related issue linked below ### Relates - https://github.com/data-dot-all/dataall/issues/1241 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 98e67fa8 Author: Sofia Sazonova <[email protected]> Date: Fri May 03 2024 12:21:57 GMT-0400 (Eastern Daylight Time) fix: DATASET_READ_TABLE read permissions (#1237) ### Feature or Bugfix - Bugfix ### Detail - backfill DATASET_READ_TABLE permissions - delete this permissions, when dataset tables are revoked or deteled - ### Relates - #1173 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <[email protected]> commit 18e2f509 Author: Noah Paige <[email protected]> Date: Fri May 03 2024 10:14:52 GMT-0400 (Eastern Daylight Time) Fix local test groups listing for listGroups query (#1239) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Locally when trying to invite a team to Env or Org we call listGroups and the returned `LOCAL_TEST_GROUPS` is not returning the proper data type expected ### Relates N/A ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit a0be03c4 Author: dlpzx <[email protected]> Date: Fri May 03 2024 10:12:34 GMT-0400 (Eastern Daylight Time) Refactor: uncouple datasets and dataset_sharing modules - part 2-5 FINAL DELETE DATASETS_BASE (#1242) ### Feature or Bugfix - Refactoring ### Detail After all the previous PRs are merged, there should be no circular dependencies between `datasets` and `datasets_sharing`. We can now proceed to: - move `datasets_base` models, repositories, permissions and enums to `datasets` - adjust the `__init__` files to establish the `datasets_sharing` depends on `datasets` - adjust the Module interfaces to ensure that all necessary dataset models... are imported in the interface for sharing Next steps: - share_notifications paramter to dataset_sharing in config.json ### Relates #955 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit b68b40c1 Author: Sofia Sazonova <[email protected]> Date: Fri May 03 2024 10:12:11 GMT-0400 (Eastern Daylight Time) bugfix: EnvironmentGroup can remove other groups (#1234) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Now, if the group can't update other group, it also can not remove them. - ### Relates - #1212 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <[email protected]> commit 264539b5 Author: Noah Paige <[email protected]> Date: Fri May 03 2024 05:23:11 GMT-0400 (Eastern Daylight Time) Fix Alembic Migration: has table checks (#1240) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Fix `has_table()` check to ensure dropping the tables if the exists as part of alembic migration upgrade - Fix `DatasetLock nullable=True` ### Relates - https://github.com/data-dot-all/dataall/issues/1165 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? No - Is the input sanitized? N/A - What precautions are you taking before deserializing the data you consume? N/A - Is injection prevented by parametrizing queries? N/A - Have you ensured no `eval` or similar functions are used? N/A - Does this PR introduce any functionality or component that requires authorization? No - How have you ensured it respects the existing AuthN/AuthZ mechanisms? N/A - Are you logging failed auth attempts? N/A - Are you using or adding any cryptographic features? No - Do you use a standard proven implementations? N/A - Are the used keys controlled by the customer? Where are they stored? N/A - Are you introducing any new policies/roles/users? No - Have you used the least-privilege principle? How? N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 42a5f6bd Author: dlpzx <[email protected]> Date: Fri May 03 2024 02:24:09 GMT-0400 (Eastern Daylight Time) Refactor: uncouple datasets and dataset_sharing modules - part 2-4 (#1214) ### Feature or Bugfix - Refactoring ⚠️ MERGE AFTER https://github.com/data-dot-all/dataall/pull/1213 ### Detail This is needed as explained in full PR [AFTER 2.4] Refactor: uncouple datasets and dataset_sharing modules #1179 - [X] Use interface to resolve dataset roles related to datasets shared and implement logic in the dataset_sharing module - [X] Extend and clean-up stewards share permissions through interface ### Relates - #1179 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 6d3f2d45 Author: Sofia Sazonova <[email protected]> Date: Thu May 02 2024 10:55:00 GMT-0400 (Eastern Daylight Time) [After 2.4]Core Refactoring part5 (#1194) ### Feature or Bugfix - Refactoring ### Detail - focus on core/environments - move logic from resolvers to services - create s3_client in base/aws --> TO BE REFACTORED. Needs to be merged with dataset_sharind/aws/s3_client ### Relates - #741 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <[email protected]> commit 2ea24cbb Author: dlpzx <[email protected]> Date: Thu May 02 2024 08:22:12 GMT-0400 (Eastern Daylight Time) Refactor: uncouple datasets and dataset_sharing modules - part 2-3 (#1213) ### Feature or Bugfix - Refactoring ⚠️ MERGE AFTER https://github.com/data-dot-all/dataall/pull/1187 ### Detail This is needed as explained in full PR [AFTER 2.4] Refactor: uncouple datasets and dataset_sharing modules #1179 - [X] Creates an interface to execute checks and clean-ups of data sharing objects when dataset objects are deleted (initially it was going to be an db interface, but I think it is better in the service) - [X] Move listDatasetShares query to dataset_sharing module in https://github.com/data-dot-all/dataall/pull/1185 ### Relates - #1179 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 750a5ec8 Author: Anushka Singh <[email protected]> Date: Wed May 01 2024 12:28:18 GMT-0400 (Eastern Daylight Time) Feature:1221 - Make visibility of auto-approval toggle configurable based on confidentiality (#1223) ### Feature or Bugfix - Feature ### Detail - Users should be able to disable visibility of auto-approval toggle with code. For example, at our company, we require that shares always go through approval process if their confidentiality classification is Secret. We dont even want to give the option to users to be able to set autoApproval enabled to ensure they dont do so by mistake and end up over sharing. Video demo: https://github.com/data-dot-all/dataall/issues/1221#issuecomment-2077412044 ### Relates - https://github.com/data-dot-all/dataall/issues/1221 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 82044689 Author: dlpzx <[email protected]> Date: Wed May 01 2024 12:26:42 GMT-0400 (Eastern Daylight Time) Refactor: uncouple datasets and dataset_sharing modules - part 2-2 (#1187) ### Feature or Bugfix - Refactoring ⚠️ MERGE AFTER https://github.com/data-dot-all/dataall/pull/1185 ### Detail This is needed as explained in full PR [AFTER 2.4] Refactor: uncouple datasets and dataset_sharing modules #1179 - Split the getDatasetAssumeRole API into 2 APIs, one for dataset owners role (in datasets module) and another one for share requester roles (in datasets_sharing module) ### Relates - #1179 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 5173419f Author: Noah Paige <[email protected]> Date: Wed May 01 2024 12:24:42 GMT-0400 (Eastern Daylight Time) Fix so listValidEnvironments called only once (#1238) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - When request access to a share on data.all the query to `listValidEnvironments` used to be called twice which (depending on how long for query results to return) could cause the environment initially selected to disappear ### Relates - Continuation of https://github.com/data-dot-all/dataall/issues/916 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 7656ea86 Author: dlpzx <[email protected]> Date: Tue Apr 30 2024 07:13:01 GMT-0400 (Eastern Daylight Time) Add integration tests on a real API client and integrate the tests in CICD (#1219) ### Feature or Bugfix - Feature ### Detail Add integration tests that use a real Client to execute different validation actions. - Define the Client and the way API calls are posted to API Gateway in the conftest - Define the Cognito users and the different fixtures needed for all tests - Write tests for the Organization core module as example - Add feature flag in `cdk.json` called `with_approval_tests` that can be defined at the deployment environment level. If set to True, a CodeBuild stage running the tests is created. ### Relates - https://github.com/data-dot-all/dataall/issues/1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit b963fe81 Author: Sofia Sazonova <[email protected]> Date: Mon Apr 29 2024 09:26:36 GMT-0400 (Eastern Daylight Time) Notification link routes to a share request page (#1227) ### Feature or Bugfix <!-- please choose --> - Feature ### Detail - in notification object field `target_uri = 'shareUri|DataSetUri'` - this value is parsed and used to redirect user to a relevant Share Request page ### Relates - #1115 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. Co-authored-by: Sofia Sazonova <[email protected]> commit 6386fe14 Author: dlpzx <[email protected]> Date: Mon Apr 29 2024 07:32:00 GMT-0400 (Eastern Daylight Time) Refactor: uncouple datasets and dataset_sharing modules - part 2 (#1185) ### Feature or Bugfix - Refactoring ### Detail Remove and move logic from dataset to datasets_sharing module. This is needed as explained in full PR [AFTER 2.4] Refactor: uncouple datasets and dataset_sharing modules #1179 - [X] Moves the verify dataset shares mutation to the datasets_sharing module - [X] Move dataset_subscription task to dataset_sharing - [X] Move listDatasetShares query to dataset_sharing module - [X] Remove unused `shares` field from the Dataset graphql type as it was not used in the frontend: listDatasets, listOwnedDatasets, listDatasetsOwnedByEnvGroup, listDatasetsCreatedInEnvironment and getDataset - [x] Move getSharedDatasetTables to data_sharing module and fix reference to DatasetService I am aware that some of the queries and mutations that this PR moves look a bit odd in the dataset_sharing module, but this will be solved once data sharing is divided into dataset_sharing_base and s3_dataset_sharing. ### Relates #1179 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires aut…
noah-paige
added a commit
that referenced
this issue
Jun 25, 2024
commit a06c8cba Author: Noah Paige <[email protected]> Date: Fri May 17 2024 16:37:05 GMT-0400 (Eastern Daylight Time) Merge share logs PR commit aee98cf7 Author: Noah Paige <[email protected]> Date: Fri May 17 2024 16:34:24 GMT-0400 (Eastern Daylight Time) Merge share logs PR commit 5ca55303 Author: Sofia Sazonova <[email protected]> Date: Wed May 15 2024 11:57:41 GMT-0400 (Eastern Daylight Time) remove unused imports commit 8f8bf3dd Author: Sofia Sazonova <[email protected]> Date: Wed May 15 2024 11:56:39 GMT-0400 (Eastern Daylight Time) restrict access to the share logs commit 9137da9b Author: Sofia Sazonova <[email protected]> Date: Wed May 15 2024 11:28:32 GMT-0400 (Eastern Daylight Time) share Logs button is available only for dataset Admins and stewards commit fcb16bd9 Author: Sofia Sazonova <[email protected]> Date: Wed May 15 2024 10:46:25 GMT-0400 (Eastern Daylight Time) getShareLogs query commit 0503a3bb Author: Sofia Sazonova <[email protected]> Date: Wed May 15 2024 10:21:25 GMT-0400 (Eastern Daylight Time) Logs modal in Share View commit bab2f3e6 Author: Sofia Sazonova <[email protected]> Date: Mon May 13 2024 09:09:18 GMT-0400 (Eastern Daylight Time) Add confirmation pop-ups for deletion of team roles and groups (#1231) ### Feature or Bugfix - Feature ### Detail Pop ups added for: - deletion team from environment - deletion of the consumption role - deletion of group from Organization ### Relates - #942 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. Co-authored-by: Sofia Sazonova <[email protected]> commit 93ff7725 Author: Sofia Sazonova <[email protected]> Date: Mon May 13 2024 08:00:38 GMT-0400 (Eastern Daylight Time) Update version.json (#1264) Release info update commit e718d861 Author: Sofia Sazonova <[email protected]> Date: Mon May 13 2024 07:29:27 GMT-0400 (Eastern Daylight Time) fix permission query (#1263) ### Feature or Bugfix - Bugfix ### Detail - The filter -- array of permissions' NAMES, so in order to query policies correctly we need to add join - The filter 'share_type' and 'share_item_status' must be string - IMPORTANT: in block "finally" the param session was used, but session was defined only in "try" block. So, the lock failed to be released. ### Relates - <URL or Ticket> ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <[email protected]> commit 479b8f3f Author: mourya-33 <[email protected]> Date: Wed May 08 2024 10:29:36 GMT-0400 (Eastern Daylight Time) Add encryption and tag immutability to ECR repository (#1224) ### Feature or Bugfix - Bugfix ### Detail - Currently the ecr repository created do not have encryption and tag immutability enabled which is identified by checkov scans. This fix is to enable both. ### Relates [- <URL or Ticket>](https://github.com/data-dot-all/dataall/issues/1200) ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes N/A - Is the input sanitized? N/A - What precautions are you taking before deserializing the data you consume? N/A - Is injection prevented by parametrizing queries? N/A - Have you ensured no `eval` or similar functions are used? N/A - Does this PR introduce any functionality or component that requires authorization? N/A - How have you ensured it respects the existing AuthN/AuthZ mechanisms? N/A - Are you logging failed auth attempts? N/A - Are you using or adding any cryptographic features? N/A - Do you use a standard proven implementations? N/A - Are the used keys controlled by the customer? Where are they stored? No. This is with default encryption - Are you introducing any new policies/roles/users? N/A - Have you used the least-privilege principle? How? N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 2f885773 Author: Sofia Sazonova <[email protected]> Date: Wed May 08 2024 09:22:40 GMT-0400 (Eastern Daylight Time) Multiple permission roots (#1259) ### Feature or Bugfix - Bugfix ### Detail - GET_DATASET_TABLE (FOLDER) permissions are granted to the group only if they are not granted already - these permissions are removed if group is not admin|steward and there are no other shares of this item. ### Relates - #1174 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <[email protected]> commit c4cc07ee Author: Petros Kalos <[email protected]> Date: Wed May 08 2024 08:54:02 GMT-0400 (Eastern Daylight Time) explicitly specify dataset_client s3 endpoint_url (#1260) * AWS requires that the endpoint_url should be explicitly specified for some regions * Remove misleading CORS error message, the upload step can fail for many reason ### Feature or Bugfix - Bugfix ### Detail Resolves #778 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 40defe8e Author: dlpzx <[email protected]> Date: Tue May 07 2024 11:52:17 GMT-0400 (Eastern Daylight Time) Generic dataset module and specific s3_datasets module - part 1 (Rename datasets as s3_datasets) (#1250) ### Feature or Bugfix - Refactoring ### Detail - Rename `datasets` module to `s3_datasets` module This PR is the first step to extract a generic datasets_base module that implements the undifferentiated concepts of Dataset in data.all. s3_datasets will use this base module to implement the specific implementation for S3 datatasets. ### Relates - #1123 - #955 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 74a303cb Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue May 07 2024 02:26:09 GMT-0400 (Eastern Daylight Time) Bump werkzeug from 3.0.1 to 3.0.3 in /tests (#1253) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.1 to 3.0.3. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/releases">werkzeug's releases</a>.</em></p> <blockquote> <h2>3.0.3</h2> <p>This is the Werkzeug 3.0.3 security release, which fixes security issues and bugs but does not otherwise change behavior and should not result in breaking changes.</p> <p>PyPI: <a href="https://pypi.org/project/Werkzeug/3.0.3/">https://pypi.org/project/Werkzeug/3.0.3/</a> Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3</a> Milestone: <a href="https://github.com/pallets/werkzeug/milestone/35?closed=1">https://github.com/pallets/werkzeug/milestone/35?closed=1</a></p> <ul> <li>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. GHSA-2g68-c3qc-8985</li> <li>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2823">#2823</a></li> <li>Better TLS cert format with <code>adhoc</code> dev certs. <a href="https://redirect.github.com/pallets/werkzeug/issues/2891">#2891</a></li> <li>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. <a href="https://redirect.github.com/pallets/werkzeug/issues/2828">#2828</a></li> <li>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2836">#2836</a></li> </ul> <h2>3.0.2</h2> <p>This is a fix release for the 3.0.x feature branch.</p> <ul> <li>Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/blob/main/CHANGES.rst">werkzeug's changelog</a>.</em></p> <blockquote> <h2>Version 3.0.3</h2> <p>Released 2024-05-05</p> <ul> <li> <p>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. :ghsa:<code>2g68-c3qc-8985</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> <li> <p>Better TLS cert format with <code>adhoc</code> dev certs. :pr:<code>2891</code></p> </li> <li> <p>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. :issue:<code>2828</code></p> </li> <li> <p>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. :issue:<code>2836</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> </ul> <h2>Version 3.0.2</h2> <p>Released 2024-04-01</p> <ul> <li>Ensure setting <code>merge_slashes</code> to <code>False</code> results in <code>NotFound</code> for repeated-slash requests against single slash routes. :issue:<code>2834</code></li> <li>Fix handling of <code>TypeError</code> in <code>TypeConversionDict.get()</code> to match <code>ValueError</code>. :issue:<code>2843</code></li> <li>Fix <code>response_wrapper</code> type check in test client. :issue:<code>2831</code></li> <li>Make the return type of <code>MultiPartParser.parse</code> more precise. :issue:<code>2840</code></li> <li>Raise an error if converter arguments cannot be parsed. :issue:<code>2822</code></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pallets/werkzeug/commit/f9995e967979eb694d6b31536cc65314fd7e9c8c"><code>f9995e9</code></a> release version 3.0.3</li> <li><a href="https://github.com/pallets/werkzeug/commit/3386395b24c7371db11a5b8eaac0c91da5362692"><code>3386395</code></a> Merge pull request from GHSA-2g68-c3qc-8985</li> <li><a href="https://github.com/pallets/werkzeug/commit/890b6b62634fa61224222aee31081c61b054ff01"><code>890b6b6</code></a> only require trusted host for evalex</li> <li><a href="https://github.com/pallets/werkzeug/commit/71b69dfb7df3d912e66bab87fbb1f21f83504967"><code>71b69df</code></a> restrict debugger trusted hosts</li> <li><a href="https://github.com/pallets/werkzeug/commit/d2d3869525a4ffb2c41dfb2c0e39d94dab2d870c"><code>d2d3869</code></a> endpoint type is Any (<a href="https://redirect.github.com/pallets/werkzeug/issues/2895">#2895</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/7080b55acd48b68afdda65ee6c7f99e9afafb0ba"><code>7080b55</code></a> endpoint type is Any</li> <li><a href="https://github.com/pallets/werkzeug/commit/7555eff296fbdf12f2e576b6bbb0b506df8417ed"><code>7555eff</code></a> remove iri_to_uri redirect workaround (<a href="https://redirect.github.com/pallets/werkzeug/issues/2894">#2894</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/97fb2f722297ae4e12e36dab024e0acf8477b3c8"><code>97fb2f7</code></a> remove _invalid_iri_to_uri workaround</li> <li><a href="https://github.com/pallets/werkzeug/commit/249527ff981e7aa22cd714825c5637cc92df7761"><code>249527f</code></a> make cn field a valid single hostname, and use wildcard in SANs field. (<a href="https://redirect.github.com/pallets/werkzeug/issues/2892">#2892</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/793be472c9d145eb9be7d4200672d1806289d84a"><code>793be47</code></a> update adhoc tls dev cert format</li> <li>Additional commits viewable in <a href="https://github.com/pallets/werkzeug/compare/3.0.1...3.0.3">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/data-dot-all/dataall/network/alerts). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 2f33320c Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue May 07 2024 02:25:03 GMT-0400 (Eastern Daylight Time) Bump werkzeug from 3.0.1 to 3.0.3 in /backend/dataall/base/cdkproxy (#1252) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.1 to 3.0.3. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/releases">werkzeug's releases</a>.</em></p> <blockquote> <h2>3.0.3</h2> <p>This is the Werkzeug 3.0.3 security release, which fixes security issues and bugs but does not otherwise change behavior and should not result in breaking changes.</p> <p>PyPI: <a href="https://pypi.org/project/Werkzeug/3.0.3/">https://pypi.org/project/Werkzeug/3.0.3/</a> Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3</a> Milestone: <a href="https://github.com/pallets/werkzeug/milestone/35?closed=1">https://github.com/pallets/werkzeug/milestone/35?closed=1</a></p> <ul> <li>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. GHSA-2g68-c3qc-8985</li> <li>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2823">#2823</a></li> <li>Better TLS cert format with <code>adhoc</code> dev certs. <a href="https://redirect.github.com/pallets/werkzeug/issues/2891">#2891</a></li> <li>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. <a href="https://redirect.github.com/pallets/werkzeug/issues/2828">#2828</a></li> <li>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2836">#2836</a></li> </ul> <h2>3.0.2</h2> <p>This is a fix release for the 3.0.x feature branch.</p> <ul> <li>Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/blob/main/CHANGES.rst">werkzeug's changelog</a>.</em></p> <blockquote> <h2>Version 3.0.3</h2> <p>Released 2024-05-05</p> <ul> <li> <p>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. :ghsa:<code>2g68-c3qc-8985</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> <li> <p>Better TLS cert format with <code>adhoc</code> dev certs. :pr:<code>2891</code></p> </li> <li> <p>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. :issue:<code>2828</code></p> </li> <li> <p>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. :issue:<code>2836</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> </ul> <h2>Version 3.0.2</h2> <p>Released 2024-04-01</p> <ul> <li>Ensure setting <code>merge_slashes</code> to <code>False</code> results in <code>NotFound</code> for repeated-slash requests against single slash routes. :issue:<code>2834</code></li> <li>Fix handling of <code>TypeError</code> in <code>TypeConversionDict.get()</code> to match <code>ValueError</code>. :issue:<code>2843</code></li> <li>Fix <code>response_wrapper</code> type check in test client. :issue:<code>2831</code></li> <li>Make the return type of <code>MultiPartParser.parse</code> more precise. :issue:<code>2840</code></li> <li>Raise an error if converter arguments cannot be parsed. :issue:<code>2822</code></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pallets/werkzeug/commit/f9995e967979eb694d6b31536cc65314fd7e9c8c"><code>f9995e9</code></a> release version 3.0.3</li> <li><a href="https://github.com/pallets/werkzeug/commit/3386395b24c7371db11a5b8eaac0c91da5362692"><code>3386395</code></a> Merge pull request from GHSA-2g68-c3qc-8985</li> <li><a href="https://github.com/pallets/werkzeug/commit/890b6b62634fa61224222aee31081c61b054ff01"><code>890b6b6</code></a> only require trusted host for evalex</li> <li><a href="https://github.com/pallets/werkzeug/commit/71b69dfb7df3d912e66bab87fbb1f21f83504967"><code>71b69df</code></a> restrict debugger trusted hosts</li> <li><a href="https://github.com/pallets/werkzeug/commit/d2d3869525a4ffb2c41dfb2c0e39d94dab2d870c"><code>d2d3869</code></a> endpoint type is Any (<a href="https://redirect.github.com/pallets/werkzeug/issues/2895">#2895</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/7080b55acd48b68afdda65ee6c7f99e9afafb0ba"><code>7080b55</code></a> endpoint type is Any</li> <li><a href="https://github.com/pallets/werkzeug/commit/7555eff296fbdf12f2e576b6bbb0b506df8417ed"><code>7555eff</code></a> remove iri_to_uri redirect workaround (<a href="https://redirect.github.com/pallets/werkzeug/issues/2894">#2894</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/97fb2f722297ae4e12e36dab024e0acf8477b3c8"><code>97fb2f7</code></a> remove _invalid_iri_to_uri workaround</li> <li><a href="https://github.com/pallets/werkzeug/commit/249527ff981e7aa22cd714825c5637cc92df7761"><code>249527f</code></a> make cn field a valid single hostname, and use wildcard in SANs field. (<a href="https://redirect.github.com/pallets/werkzeug/issues/2892">#2892</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/793be472c9d145eb9be7d4200672d1806289d84a"><code>793be47</code></a> update adhoc tls dev cert format</li> <li>Additional commits viewable in <a href="https://github.com/pallets/werkzeug/compare/3.0.1...3.0.3">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/data-dot-all/dataall/network/alerts). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 0b49633f Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue May 07 2024 02:24:34 GMT-0400 (Eastern Daylight Time) Bump werkzeug from 3.0.1 to 3.0.3 in /tests_new/integration_tests (#1254) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.1 to 3.0.3. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/releases">werkzeug's releases</a>.</em></p> <blockquote> <h2>3.0.3</h2> <p>This is the Werkzeug 3.0.3 security release, which fixes security issues and bugs but does not otherwise change behavior and should not result in breaking changes.</p> <p>PyPI: <a href="https://pypi.org/project/Werkzeug/3.0.3/">https://pypi.org/project/Werkzeug/3.0.3/</a> Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3</a> Milestone: <a href="https://github.com/pallets/werkzeug/milestone/35?closed=1">https://github.com/pallets/werkzeug/milestone/35?closed=1</a></p> <ul> <li>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. GHSA-2g68-c3qc-8985</li> <li>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2823">#2823</a></li> <li>Better TLS cert format with <code>adhoc</code> dev certs. <a href="https://redirect.github.com/pallets/werkzeug/issues/2891">#2891</a></li> <li>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. <a href="https://redirect.github.com/pallets/werkzeug/issues/2828">#2828</a></li> <li>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2836">#2836</a></li> </ul> <h2>3.0.2</h2> <p>This is a fix release for the 3.0.x feature branch.</p> <ul> <li>Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/blob/main/CHANGES.rst">werkzeug's changelog</a>.</em></p> <blockquote> <h2>Version 3.0.3</h2> <p>Released 2024-05-05</p> <ul> <li> <p>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. :ghsa:<code>2g68-c3qc-8985</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> <li> <p>Better TLS cert format with <code>adhoc</code> dev certs. :pr:<code>2891</code></p> </li> <li> <p>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. :issue:<code>2828</code></p> </li> <li> <p>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. :issue:<code>2836</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> </ul> <h2>Version 3.0.2</h2> <p>Released 2024-04-01</p> <ul> <li>Ensure setting <code>merge_slashes</code> to <code>False</code> results in <code>NotFound</code> for repeated-slash requests against single slash routes. :issue:<code>2834</code></li> <li>Fix handling of <code>TypeError</code> in <code>TypeConversionDict.get()</code> to match <code>ValueError</code>. :issue:<code>2843</code></li> <li>Fix <code>response_wrapper</code> type check in test client. :issue:<code>2831</code></li> <li>Make the return type of <code>MultiPartParser.parse</code> more precise. :issue:<code>2840</code></li> <li>Raise an error if converter arguments cannot be parsed. :issue:<code>2822</code></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pallets/werkzeug/commit/f9995e967979eb694d6b31536cc65314fd7e9c8c"><code>f9995e9</code></a> release version 3.0.3</li> <li><a href="https://github.com/pallets/werkzeug/commit/3386395b24c7371db11a5b8eaac0c91da5362692"><code>3386395</code></a> Merge pull request from GHSA-2g68-c3qc-8985</li> <li><a href="https://github.com/pallets/werkzeug/commit/890b6b62634fa61224222aee31081c61b054ff01"><code>890b6b6</code></a> only require trusted host for evalex</li> <li><a href="https://github.com/pallets/werkzeug/commit/71b69dfb7df3d912e66bab87fbb1f21f83504967"><code>71b69df</code></a> restrict debugger trusted hosts</li> <li><a href="https://github.com/pallets/werkzeug/commit/d2d3869525a4ffb2c41dfb2c0e39d94dab2d870c"><code>d2d3869</code></a> endpoint type is Any (<a href="https://redirect.github.com/pallets/werkzeug/issues/2895">#2895</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/7080b55acd48b68afdda65ee6c7f99e9afafb0ba"><code>7080b55</code></a> endpoint type is Any</li> <li><a href="https://github.com/pallets/werkzeug/commit/7555eff296fbdf12f2e576b6bbb0b506df8417ed"><code>7555eff</code></a> remove iri_to_uri redirect workaround (<a href="https://redirect.github.com/pallets/werkzeug/issues/2894">#2894</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/97fb2f722297ae4e12e36dab024e0acf8477b3c8"><code>97fb2f7</code></a> remove _invalid_iri_to_uri workaround</li> <li><a href="https://github.com/pallets/werkzeug/commit/249527ff981e7aa22cd714825c5637cc92df7761"><code>249527f</code></a> make cn field a valid single hostname, and use wildcard in SANs field. (<a href="https://redirect.github.com/pallets/werkzeug/issues/2892">#2892</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/793be472c9d145eb9be7d4200672d1806289d84a"><code>793be47</code></a> update adhoc tls dev cert format</li> <li>Additional commits viewable in <a href="https://github.com/pallets/werkzeug/compare/3.0.1...3.0.3">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/data-dot-all/dataall/network/alerts). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 08862420 Author: mourya-33 <[email protected]> Date: Tue May 07 2024 02:15:15 GMT-0400 (Eastern Daylight Time) Updated lambda_api.py to add encryption for lambda env vars for custo… (#1255) Feature or Bugfix Bugfix Detail The environment variables for the lambda functions are not encrypted in cdk which are identified by checkov scans. This fix is to enable kms encryption for the lambda environment variables. Relates Security Please answer the questions below briefly where applicable, or write N/A. Based on [OWASP 10](https://owasp.org/Top10/en/). Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? N/A Is the input sanitized? N/A What precautions are you taking before deserializing the data you consume? N/A Is injection prevented by parametrizing queries? N/A Have you ensured no eval or similar functions are used? N/A Does this PR introduce any functionality or component that requires authorization? N/A How have you ensured it respects the existing AuthN/AuthZ mechanisms? N/A Are you logging failed auth attempts? N/A Are you using or adding any cryptographic features? N/A Do you use a standard proven implementations? N/A Are the used keys controlled by the customer? Where are they stored? the KMS keys are generated by cdk and are used to encrypt the environment variables for all lambda functions in the lambda-api stack Are you introducing any new policies/roles/users? - N/A Have you used the least-privilege principle? How? N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit ed7cc3eb Author: Noah Paige <[email protected]> Date: Mon May 06 2024 09:32:30 GMT-0400 (Eastern Daylight Time) Add order_by for paginated queries (#1249) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - This PR aims to solve the following - (1) for particular queries (identified as ones that perform `.outerjoin()` operations and have results paginated with `paginate()` function - sometimes the returned query results is *less than* the limit set by the pageSize of the paginate function even when the total count is greater than the pageSize - Ex 1: 11 envs total, `query_user_environments()` returning 9 envs on 1st page + 2 on 2nd page - Ex 2: 10 envs total, `query_user_environments()` returning 9 envs on 1st page + no 2nd page - Believe this is to be happening due to the way SQLAlchemy is "uniquing" the records resulted from an outerjoin and then returning that result back to the frontend - Adding a `.distinct()` check on the query ensures each distinct record is returned (tested successfully) - (2) Currently we often times do not implement an `.order_by()` condition for the query used in `paginate()` and do not have a stable way of preserving order of the items returned from a query (i.e. when navigating through pages of response) - A generally good practice seems to include an `order_by()` on a column or set of columns - For each query used in `paginate()` this PR adds an `order_by()` condition (full list in comments below) Can read a bit more context from related issue linked below ### Relates - https://github.com/data-dot-all/dataall/issues/1241 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 98e67fa8 Author: Sofia Sazonova <[email protected]> Date: Fri May 03 2024 12:21:57 GMT-0400 (Eastern Daylight Time) fix: DATASET_READ_TABLE read permissions (#1237) ### Feature or Bugfix - Bugfix ### Detail - backfill DATASET_READ_TABLE permissions - delete this permissions, when dataset tables are revoked or deteled - ### Relates - #1173 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <[email protected]> commit 18e2f509 Author: Noah Paige <[email protected]> Date: Fri May 03 2024 10:14:52 GMT-0400 (Eastern Daylight Time) Fix local test groups listing for listGroups query (#1239) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Locally when trying to invite a team to Env or Org we call listGroups and the returned `LOCAL_TEST_GROUPS` is not returning the proper data type expected ### Relates N/A ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit a0be03c4 Author: dlpzx <[email protected]> Date: Fri May 03 2024 10:12:34 GMT-0400 (Eastern Daylight Time) Refactor: uncouple datasets and dataset_sharing modules - part 2-5 FINAL DELETE DATASETS_BASE (#1242) ### Feature or Bugfix - Refactoring ### Detail After all the previous PRs are merged, there should be no circular dependencies between `datasets` and `datasets_sharing`. We can now proceed to: - move `datasets_base` models, repositories, permissions and enums to `datasets` - adjust the `__init__` files to establish the `datasets_sharing` depends on `datasets` - adjust the Module interfaces to ensure that all necessary dataset models... are imported in the interface for sharing Next steps: - share_notifications paramter to dataset_sharing in config.json ### Relates #955 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit b68b40c1 Author: Sofia Sazonova <[email protected]> Date: Fri May 03 2024 10:12:11 GMT-0400 (Eastern Daylight Time) bugfix: EnvironmentGroup can remove other groups (#1234) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Now, if the group can't update other group, it also can not remove them. - ### Relates - #1212 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <[email protected]> commit 264539b5 Author: Noah Paige <[email protected]> Date: Fri May 03 2024 05:23:11 GMT-0400 (Eastern Daylight Time) Fix Alembic Migration: has table checks (#1240) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Fix `has_table()` check to ensure dropping the tables if the exists as part of alembic migration upgrade - Fix `DatasetLock nullable=True` ### Relates - https://github.com/data-dot-all/dataall/issues/1165 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? No - Is the input sanitized? N/A - What precautions are you taking before deserializing the data you consume? N/A - Is injection prevented by parametrizing queries? N/A - Have you ensured no `eval` or similar functions are used? N/A - Does this PR introduce any functionality or component that requires authorization? No - How have you ensured it respects the existing AuthN/AuthZ mechanisms? N/A - Are you logging failed auth attempts? N/A - Are you using or adding any cryptographic features? No - Do you use a standard proven implementations? N/A - Are the used keys controlled by the customer? Where are they stored? N/A - Are you introducing any new policies/roles/users? No - Have you used the least-privilege principle? How? N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 42a5f6bd Author: dlpzx <[email protected]> Date: Fri May 03 2024 02:24:09 GMT-0400 (Eastern Daylight Time) Refactor: uncouple datasets and dataset_sharing modules - part 2-4 (#1214) ### Feature or Bugfix - Refactoring ⚠️ MERGE AFTER https://github.com/data-dot-all/dataall/pull/1213 ### Detail This is needed as explained in full PR [AFTER 2.4] Refactor: uncouple datasets and dataset_sharing modules #1179 - [X] Use interface to resolve dataset roles related to datasets shared and implement logic in the dataset_sharing module - [X] Extend and clean-up stewards share permissions through interface ### Relates - #1179 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 6d3f2d45 Author: Sofia Sazonova <[email protected]> Date: Thu May 02 2024 10:55:00 GMT-0400 (Eastern Daylight Time) [After 2.4]Core Refactoring part5 (#1194) ### Feature or Bugfix - Refactoring ### Detail - focus on core/environments - move logic from resolvers to services - create s3_client in base/aws --> TO BE REFACTORED. Needs to be merged with dataset_sharind/aws/s3_client ### Relates - #741 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <[email protected]> commit 2ea24cbb Author: dlpzx <[email protected]> Date: Thu May 02 2024 08:22:12 GMT-0400 (Eastern Daylight Time) Refactor: uncouple datasets and dataset_sharing modules - part 2-3 (#1213) ### Feature or Bugfix - Refactoring ⚠️ MERGE AFTER https://github.com/data-dot-all/dataall/pull/1187 ### Detail This is needed as explained in full PR [AFTER 2.4] Refactor: uncouple datasets and dataset_sharing modules #1179 - [X] Creates an interface to execute checks and clean-ups of data sharing objects when dataset objects are deleted (initially it was going to be an db interface, but I think it is better in the service) - [X] Move listDatasetShares query to dataset_sharing module in https://github.com/data-dot-all/dataall/pull/1185 ### Relates - #1179 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 750a5ec8 Author: Anushka Singh <[email protected]> Date: Wed May 01 2024 12:28:18 GMT-0400 (Eastern Daylight Time) Feature:1221 - Make visibility of auto-approval toggle configurable based on confidentiality (#1223) ### Feature or Bugfix - Feature ### Detail - Users should be able to disable visibility of auto-approval toggle with code. For example, at our company, we require that shares always go through approval process if their confidentiality classification is Secret. We dont even want to give the option to users to be able to set autoApproval enabled to ensure they dont do so by mistake and end up over sharing. Video demo: https://github.com/data-dot-all/dataall/issues/1221#issuecomment-2077412044 ### Relates - https://github.com/data-dot-all/dataall/issues/1221 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 82044689 Author: dlpzx <[email protected]> Date: Wed May 01 2024 12:26:42 GMT-0400 (Eastern Daylight Time) Refactor: uncouple datasets and dataset_sharing modules - part 2-2 (#1187) ### Feature or Bugfix - Refactoring ⚠️ MERGE AFTER https://github.com/data-dot-all/dataall/pull/1185 ### Detail This is needed as explained in full PR [AFTER 2.4] Refactor: uncouple datasets and dataset_sharing modules #1179 - Split the getDatasetAssumeRole API into 2 APIs, one for dataset owners role (in datasets module) and another one for share requester roles (in datasets_sharing module) ### Relates - #1179 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 5173419f Author: Noah Paige <[email protected]> Date: Wed May 01 2024 12:24:42 GMT-0400 (Eastern Daylight Time) Fix so listValidEnvironments called only once (#1238) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - When request access to a share on data.all the query to `listValidEnvironments` used to be called twice which (depending on how long for query results to return) could cause the environment initially selected to disappear ### Relates - Continuation of https://github.com/data-dot-all/dataall/issues/916 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 7656ea86 Author: dlpzx <[email protected]> Date: Tue Apr 30 2024 07:13:01 GMT-0400 (Eastern Daylight Time) Add integration tests on a real API client and integrate the tests in CICD (#1219) ### Feature or Bugfix - Feature ### Detail Add integration tests that use a real Client to execute different validation actions. - Define the Client and the way API calls are posted to API Gateway in the conftest - Define the Cognito users and the different fixtures needed for all tests - Write tests for the Organization core module as example - Add feature flag in `cdk.json` called `with_approval_tests` that can be defined at the deployment environment level. If set to True, a CodeBuild stage running the tests is created. ### Relates - https://github.com/data-dot-all/dataall/issues/1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit b963fe81 Author: Sofia Sazonova <[email protected]> Date: Mon Apr 29 2024 09:26:36 GMT-0400 (Eastern Daylight Time) Notification link routes to a share request page (#1227) ### Feature or Bugfix <!-- please choose --> - Feature ### Detail - in notification object field `target_uri = 'shareUri|DataSetUri'` - this value is parsed and used to redirect user to a relevant Share Request page ### Relates - #1115 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the da…
noah-paige
added a commit
that referenced
this issue
Jun 25, 2024
commit d8497c55 Author: Noah Paige <[email protected]> Date: Mon May 20 2024 15:19:35 GMT-0400 (Eastern Daylight Time) fix commit 199ab505 Author: Admin/noahpaig-Isengard <Admin/noahpaig-Isengard> Date: Mon May 20 2024 15:16:14 GMT-0400 (Eastern Daylight Time) Conflicts resolved in the console. commit ad415575 Author: Noah Paige <[email protected]> Date: Mon May 20 2024 15:13:06 GMT-0400 (Eastern Daylight Time) Merge branch 'chatbot-test' into noah-main-2 commit 2893efc7 Author: Noah Paige <[email protected]> Date: Mon May 20 2024 15:10:20 GMT-0400 (Eastern Daylight Time) Merge branch 'chatbot-test' into noah-main-2 commit caad12e1 Author: Noah Paige <[email protected]> Date: Mon May 20 2024 15:12:47 GMT-0400 (Eastern Daylight Time) ruff commit a73b7110 Author: Noah Paige <[email protected]> Date: Mon May 20 2024 15:09:19 GMT-0400 (Eastern Daylight Time) Remove hardcoding commit 7b32a8f7 Author: Noah Paige <[email protected]> Date: Mon May 20 2024 15:07:29 GMT-0400 (Eastern Daylight Time) Chatbot POC commit e25e5815 Author: Noah Paige <[email protected]> Date: Mon May 20 2024 15:08:32 GMT-0400 (Eastern Daylight Time) Merge branch '1215-share-logs' into noah-main-2 commit a06c8cba Author: Noah Paige <[email protected]> Date: Fri May 17 2024 16:37:05 GMT-0400 (Eastern Daylight Time) Merge share logs PR commit a5626670 Author: Sofia Sazonova <[email protected]> Date: Fri May 17 2024 17:19:25 GMT-0400 (Eastern Daylight Time) make ruff happy commit aee98cf7 Author: Noah Paige <[email protected]> Date: Fri May 17 2024 16:34:24 GMT-0400 (Eastern Daylight Time) Merge share logs PR commit c3ee2f7c Author: Sofia Sazonova <[email protected]> Date: Fri May 17 2024 17:11:00 GMT-0400 (Eastern Daylight Time) PR comments commit 5ca55303 Author: Sofia Sazonova <[email protected]> Date: Wed May 15 2024 11:57:41 GMT-0400 (Eastern Daylight Time) remove unused imports commit 8f8bf3dd Author: Sofia Sazonova <[email protected]> Date: Wed May 15 2024 11:56:39 GMT-0400 (Eastern Daylight Time) restrict access to the share logs commit 9137da9b Author: Sofia Sazonova <[email protected]> Date: Wed May 15 2024 11:28:32 GMT-0400 (Eastern Daylight Time) share Logs button is available only for dataset Admins and stewards commit fcb16bd9 Author: Sofia Sazonova <[email protected]> Date: Wed May 15 2024 10:46:25 GMT-0400 (Eastern Daylight Time) getShareLogs query commit 0503a3bb Author: Sofia Sazonova <[email protected]> Date: Wed May 15 2024 10:21:25 GMT-0400 (Eastern Daylight Time) Logs modal in Share View commit bab2f3e6 Author: Sofia Sazonova <[email protected]> Date: Mon May 13 2024 09:09:18 GMT-0400 (Eastern Daylight Time) Add confirmation pop-ups for deletion of team roles and groups (#1231) ### Feature or Bugfix - Feature ### Detail Pop ups added for: - deletion team from environment - deletion of the consumption role - deletion of group from Organization ### Relates - #942 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. Co-authored-by: Sofia Sazonova <[email protected]> commit 93ff7725 Author: Sofia Sazonova <[email protected]> Date: Mon May 13 2024 08:00:38 GMT-0400 (Eastern Daylight Time) Update version.json (#1264) Release info update commit e718d861 Author: Sofia Sazonova <[email protected]> Date: Mon May 13 2024 07:29:27 GMT-0400 (Eastern Daylight Time) fix permission query (#1263) ### Feature or Bugfix - Bugfix ### Detail - The filter -- array of permissions' NAMES, so in order to query policies correctly we need to add join - The filter 'share_type' and 'share_item_status' must be string - IMPORTANT: in block "finally" the param session was used, but session was defined only in "try" block. So, the lock failed to be released. ### Relates - <URL or Ticket> ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <[email protected]> commit 479b8f3f Author: mourya-33 <[email protected]> Date: Wed May 08 2024 10:29:36 GMT-0400 (Eastern Daylight Time) Add encryption and tag immutability to ECR repository (#1224) ### Feature or Bugfix - Bugfix ### Detail - Currently the ecr repository created do not have encryption and tag immutability enabled which is identified by checkov scans. This fix is to enable both. ### Relates [- <URL or Ticket>](https://github.com/data-dot-all/dataall/issues/1200) ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes N/A - Is the input sanitized? N/A - What precautions are you taking before deserializing the data you consume? N/A - Is injection prevented by parametrizing queries? N/A - Have you ensured no `eval` or similar functions are used? N/A - Does this PR introduce any functionality or component that requires authorization? N/A - How have you ensured it respects the existing AuthN/AuthZ mechanisms? N/A - Are you logging failed auth attempts? N/A - Are you using or adding any cryptographic features? N/A - Do you use a standard proven implementations? N/A - Are the used keys controlled by the customer? Where are they stored? No. This is with default encryption - Are you introducing any new policies/roles/users? N/A - Have you used the least-privilege principle? How? N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 2f885773 Author: Sofia Sazonova <[email protected]> Date: Wed May 08 2024 09:22:40 GMT-0400 (Eastern Daylight Time) Multiple permission roots (#1259) ### Feature or Bugfix - Bugfix ### Detail - GET_DATASET_TABLE (FOLDER) permissions are granted to the group only if they are not granted already - these permissions are removed if group is not admin|steward and there are no other shares of this item. ### Relates - #1174 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <[email protected]> commit c4cc07ee Author: Petros Kalos <[email protected]> Date: Wed May 08 2024 08:54:02 GMT-0400 (Eastern Daylight Time) explicitly specify dataset_client s3 endpoint_url (#1260) * AWS requires that the endpoint_url should be explicitly specified for some regions * Remove misleading CORS error message, the upload step can fail for many reason ### Feature or Bugfix - Bugfix ### Detail Resolves #778 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 40defe8e Author: dlpzx <[email protected]> Date: Tue May 07 2024 11:52:17 GMT-0400 (Eastern Daylight Time) Generic dataset module and specific s3_datasets module - part 1 (Rename datasets as s3_datasets) (#1250) ### Feature or Bugfix - Refactoring ### Detail - Rename `datasets` module to `s3_datasets` module This PR is the first step to extract a generic datasets_base module that implements the undifferentiated concepts of Dataset in data.all. s3_datasets will use this base module to implement the specific implementation for S3 datatasets. ### Relates - #1123 - #955 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 74a303cb Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue May 07 2024 02:26:09 GMT-0400 (Eastern Daylight Time) Bump werkzeug from 3.0.1 to 3.0.3 in /tests (#1253) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.1 to 3.0.3. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/releases">werkzeug's releases</a>.</em></p> <blockquote> <h2>3.0.3</h2> <p>This is the Werkzeug 3.0.3 security release, which fixes security issues and bugs but does not otherwise change behavior and should not result in breaking changes.</p> <p>PyPI: <a href="https://pypi.org/project/Werkzeug/3.0.3/">https://pypi.org/project/Werkzeug/3.0.3/</a> Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3</a> Milestone: <a href="https://github.com/pallets/werkzeug/milestone/35?closed=1">https://github.com/pallets/werkzeug/milestone/35?closed=1</a></p> <ul> <li>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. GHSA-2g68-c3qc-8985</li> <li>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2823">#2823</a></li> <li>Better TLS cert format with <code>adhoc</code> dev certs. <a href="https://redirect.github.com/pallets/werkzeug/issues/2891">#2891</a></li> <li>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. <a href="https://redirect.github.com/pallets/werkzeug/issues/2828">#2828</a></li> <li>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2836">#2836</a></li> </ul> <h2>3.0.2</h2> <p>This is a fix release for the 3.0.x feature branch.</p> <ul> <li>Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/blob/main/CHANGES.rst">werkzeug's changelog</a>.</em></p> <blockquote> <h2>Version 3.0.3</h2> <p>Released 2024-05-05</p> <ul> <li> <p>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. :ghsa:<code>2g68-c3qc-8985</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> <li> <p>Better TLS cert format with <code>adhoc</code> dev certs. :pr:<code>2891</code></p> </li> <li> <p>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. :issue:<code>2828</code></p> </li> <li> <p>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. :issue:<code>2836</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> </ul> <h2>Version 3.0.2</h2> <p>Released 2024-04-01</p> <ul> <li>Ensure setting <code>merge_slashes</code> to <code>False</code> results in <code>NotFound</code> for repeated-slash requests against single slash routes. :issue:<code>2834</code></li> <li>Fix handling of <code>TypeError</code> in <code>TypeConversionDict.get()</code> to match <code>ValueError</code>. :issue:<code>2843</code></li> <li>Fix <code>response_wrapper</code> type check in test client. :issue:<code>2831</code></li> <li>Make the return type of <code>MultiPartParser.parse</code> more precise. :issue:<code>2840</code></li> <li>Raise an error if converter arguments cannot be parsed. :issue:<code>2822</code></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pallets/werkzeug/commit/f9995e967979eb694d6b31536cc65314fd7e9c8c"><code>f9995e9</code></a> release version 3.0.3</li> <li><a href="https://github.com/pallets/werkzeug/commit/3386395b24c7371db11a5b8eaac0c91da5362692"><code>3386395</code></a> Merge pull request from GHSA-2g68-c3qc-8985</li> <li><a href="https://github.com/pallets/werkzeug/commit/890b6b62634fa61224222aee31081c61b054ff01"><code>890b6b6</code></a> only require trusted host for evalex</li> <li><a href="https://github.com/pallets/werkzeug/commit/71b69dfb7df3d912e66bab87fbb1f21f83504967"><code>71b69df</code></a> restrict debugger trusted hosts</li> <li><a href="https://github.com/pallets/werkzeug/commit/d2d3869525a4ffb2c41dfb2c0e39d94dab2d870c"><code>d2d3869</code></a> endpoint type is Any (<a href="https://redirect.github.com/pallets/werkzeug/issues/2895">#2895</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/7080b55acd48b68afdda65ee6c7f99e9afafb0ba"><code>7080b55</code></a> endpoint type is Any</li> <li><a href="https://github.com/pallets/werkzeug/commit/7555eff296fbdf12f2e576b6bbb0b506df8417ed"><code>7555eff</code></a> remove iri_to_uri redirect workaround (<a href="https://redirect.github.com/pallets/werkzeug/issues/2894">#2894</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/97fb2f722297ae4e12e36dab024e0acf8477b3c8"><code>97fb2f7</code></a> remove _invalid_iri_to_uri workaround</li> <li><a href="https://github.com/pallets/werkzeug/commit/249527ff981e7aa22cd714825c5637cc92df7761"><code>249527f</code></a> make cn field a valid single hostname, and use wildcard in SANs field. (<a href="https://redirect.github.com/pallets/werkzeug/issues/2892">#2892</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/793be472c9d145eb9be7d4200672d1806289d84a"><code>793be47</code></a> update adhoc tls dev cert format</li> <li>Additional commits viewable in <a href="https://github.com/pallets/werkzeug/compare/3.0.1...3.0.3">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/data-dot-all/dataall/network/alerts). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 2f33320c Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue May 07 2024 02:25:03 GMT-0400 (Eastern Daylight Time) Bump werkzeug from 3.0.1 to 3.0.3 in /backend/dataall/base/cdkproxy (#1252) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.1 to 3.0.3. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/releases">werkzeug's releases</a>.</em></p> <blockquote> <h2>3.0.3</h2> <p>This is the Werkzeug 3.0.3 security release, which fixes security issues and bugs but does not otherwise change behavior and should not result in breaking changes.</p> <p>PyPI: <a href="https://pypi.org/project/Werkzeug/3.0.3/">https://pypi.org/project/Werkzeug/3.0.3/</a> Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3</a> Milestone: <a href="https://github.com/pallets/werkzeug/milestone/35?closed=1">https://github.com/pallets/werkzeug/milestone/35?closed=1</a></p> <ul> <li>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. GHSA-2g68-c3qc-8985</li> <li>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2823">#2823</a></li> <li>Better TLS cert format with <code>adhoc</code> dev certs. <a href="https://redirect.github.com/pallets/werkzeug/issues/2891">#2891</a></li> <li>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. <a href="https://redirect.github.com/pallets/werkzeug/issues/2828">#2828</a></li> <li>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2836">#2836</a></li> </ul> <h2>3.0.2</h2> <p>This is a fix release for the 3.0.x feature branch.</p> <ul> <li>Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/blob/main/CHANGES.rst">werkzeug's changelog</a>.</em></p> <blockquote> <h2>Version 3.0.3</h2> <p>Released 2024-05-05</p> <ul> <li> <p>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. :ghsa:<code>2g68-c3qc-8985</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> <li> <p>Better TLS cert format with <code>adhoc</code> dev certs. :pr:<code>2891</code></p> </li> <li> <p>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. :issue:<code>2828</code></p> </li> <li> <p>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. :issue:<code>2836</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> </ul> <h2>Version 3.0.2</h2> <p>Released 2024-04-01</p> <ul> <li>Ensure setting <code>merge_slashes</code> to <code>False</code> results in <code>NotFound</code> for repeated-slash requests against single slash routes. :issue:<code>2834</code></li> <li>Fix handling of <code>TypeError</code> in <code>TypeConversionDict.get()</code> to match <code>ValueError</code>. :issue:<code>2843</code></li> <li>Fix <code>response_wrapper</code> type check in test client. :issue:<code>2831</code></li> <li>Make the return type of <code>MultiPartParser.parse</code> more precise. :issue:<code>2840</code></li> <li>Raise an error if converter arguments cannot be parsed. :issue:<code>2822</code></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pallets/werkzeug/commit/f9995e967979eb694d6b31536cc65314fd7e9c8c"><code>f9995e9</code></a> release version 3.0.3</li> <li><a href="https://github.com/pallets/werkzeug/commit/3386395b24c7371db11a5b8eaac0c91da5362692"><code>3386395</code></a> Merge pull request from GHSA-2g68-c3qc-8985</li> <li><a href="https://github.com/pallets/werkzeug/commit/890b6b62634fa61224222aee31081c61b054ff01"><code>890b6b6</code></a> only require trusted host for evalex</li> <li><a href="https://github.com/pallets/werkzeug/commit/71b69dfb7df3d912e66bab87fbb1f21f83504967"><code>71b69df</code></a> restrict debugger trusted hosts</li> <li><a href="https://github.com/pallets/werkzeug/commit/d2d3869525a4ffb2c41dfb2c0e39d94dab2d870c"><code>d2d3869</code></a> endpoint type is Any (<a href="https://redirect.github.com/pallets/werkzeug/issues/2895">#2895</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/7080b55acd48b68afdda65ee6c7f99e9afafb0ba"><code>7080b55</code></a> endpoint type is Any</li> <li><a href="https://github.com/pallets/werkzeug/commit/7555eff296fbdf12f2e576b6bbb0b506df8417ed"><code>7555eff</code></a> remove iri_to_uri redirect workaround (<a href="https://redirect.github.com/pallets/werkzeug/issues/2894">#2894</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/97fb2f722297ae4e12e36dab024e0acf8477b3c8"><code>97fb2f7</code></a> remove _invalid_iri_to_uri workaround</li> <li><a href="https://github.com/pallets/werkzeug/commit/249527ff981e7aa22cd714825c5637cc92df7761"><code>249527f</code></a> make cn field a valid single hostname, and use wildcard in SANs field. (<a href="https://redirect.github.com/pallets/werkzeug/issues/2892">#2892</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/793be472c9d145eb9be7d4200672d1806289d84a"><code>793be47</code></a> update adhoc tls dev cert format</li> <li>Additional commits viewable in <a href="https://github.com/pallets/werkzeug/compare/3.0.1...3.0.3">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/data-dot-all/dataall/network/alerts). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 0b49633f Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue May 07 2024 02:24:34 GMT-0400 (Eastern Daylight Time) Bump werkzeug from 3.0.1 to 3.0.3 in /tests_new/integration_tests (#1254) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.1 to 3.0.3. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/releases">werkzeug's releases</a>.</em></p> <blockquote> <h2>3.0.3</h2> <p>This is the Werkzeug 3.0.3 security release, which fixes security issues and bugs but does not otherwise change behavior and should not result in breaking changes.</p> <p>PyPI: <a href="https://pypi.org/project/Werkzeug/3.0.3/">https://pypi.org/project/Werkzeug/3.0.3/</a> Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3</a> Milestone: <a href="https://github.com/pallets/werkzeug/milestone/35?closed=1">https://github.com/pallets/werkzeug/milestone/35?closed=1</a></p> <ul> <li>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. GHSA-2g68-c3qc-8985</li> <li>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2823">#2823</a></li> <li>Better TLS cert format with <code>adhoc</code> dev certs. <a href="https://redirect.github.com/pallets/werkzeug/issues/2891">#2891</a></li> <li>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. <a href="https://redirect.github.com/pallets/werkzeug/issues/2828">#2828</a></li> <li>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2836">#2836</a></li> </ul> <h2>3.0.2</h2> <p>This is a fix release for the 3.0.x feature branch.</p> <ul> <li>Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/blob/main/CHANGES.rst">werkzeug's changelog</a>.</em></p> <blockquote> <h2>Version 3.0.3</h2> <p>Released 2024-05-05</p> <ul> <li> <p>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. :ghsa:<code>2g68-c3qc-8985</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> <li> <p>Better TLS cert format with <code>adhoc</code> dev certs. :pr:<code>2891</code></p> </li> <li> <p>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. :issue:<code>2828</code></p> </li> <li> <p>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. :issue:<code>2836</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> </ul> <h2>Version 3.0.2</h2> <p>Released 2024-04-01</p> <ul> <li>Ensure setting <code>merge_slashes</code> to <code>False</code> results in <code>NotFound</code> for repeated-slash requests against single slash routes. :issue:<code>2834</code></li> <li>Fix handling of <code>TypeError</code> in <code>TypeConversionDict.get()</code> to match <code>ValueError</code>. :issue:<code>2843</code></li> <li>Fix <code>response_wrapper</code> type check in test client. :issue:<code>2831</code></li> <li>Make the return type of <code>MultiPartParser.parse</code> more precise. :issue:<code>2840</code></li> <li>Raise an error if converter arguments cannot be parsed. :issue:<code>2822</code></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pallets/werkzeug/commit/f9995e967979eb694d6b31536cc65314fd7e9c8c"><code>f9995e9</code></a> release version 3.0.3</li> <li><a href="https://github.com/pallets/werkzeug/commit/3386395b24c7371db11a5b8eaac0c91da5362692"><code>3386395</code></a> Merge pull request from GHSA-2g68-c3qc-8985</li> <li><a href="https://github.com/pallets/werkzeug/commit/890b6b62634fa61224222aee31081c61b054ff01"><code>890b6b6</code></a> only require trusted host for evalex</li> <li><a href="https://github.com/pallets/werkzeug/commit/71b69dfb7df3d912e66bab87fbb1f21f83504967"><code>71b69df</code></a> restrict debugger trusted hosts</li> <li><a href="https://github.com/pallets/werkzeug/commit/d2d3869525a4ffb2c41dfb2c0e39d94dab2d870c"><code>d2d3869</code></a> endpoint type is Any (<a href="https://redirect.github.com/pallets/werkzeug/issues/2895">#2895</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/7080b55acd48b68afdda65ee6c7f99e9afafb0ba"><code>7080b55</code></a> endpoint type is Any</li> <li><a href="https://github.com/pallets/werkzeug/commit/7555eff296fbdf12f2e576b6bbb0b506df8417ed"><code>7555eff</code></a> remove iri_to_uri redirect workaround (<a href="https://redirect.github.com/pallets/werkzeug/issues/2894">#2894</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/97fb2f722297ae4e12e36dab024e0acf8477b3c8"><code>97fb2f7</code></a> remove _invalid_iri_to_uri workaround</li> <li><a href="https://github.com/pallets/werkzeug/commit/249527ff981e7aa22cd714825c5637cc92df7761"><code>249527f</code></a> make cn field a valid single hostname, and use wildcard in SANs field. (<a href="https://redirect.github.com/pallets/werkzeug/issues/2892">#2892</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/793be472c9d145eb9be7d4200672d1806289d84a"><code>793be47</code></a> update adhoc tls dev cert format</li> <li>Additional commits viewable in <a href="https://github.com/pallets/werkzeug/compare/3.0.1...3.0.3">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/data-dot-all/dataall/network/alerts). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 08862420 Author: mourya-33 <[email protected]> Date: Tue May 07 2024 02:15:15 GMT-0400 (Eastern Daylight Time) Updated lambda_api.py to add encryption for lambda env vars for custo… (#1255) Feature or Bugfix Bugfix Detail The environment variables for the lambda functions are not encrypted in cdk which are identified by checkov scans. This fix is to enable kms encryption for the lambda environment variables. Relates Security Please answer the questions below briefly where applicable, or write N/A. Based on [OWASP 10](https://owasp.org/Top10/en/). Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? N/A Is the input sanitized? N/A What precautions are you taking before deserializing the data you consume? N/A Is injection prevented by parametrizing queries? N/A Have you ensured no eval or similar functions are used? N/A Does this PR introduce any functionality or component that requires authorization? N/A How have you ensured it respects the existing AuthN/AuthZ mechanisms? N/A Are you logging failed auth attempts? N/A Are you using or adding any cryptographic features? N/A Do you use a standard proven implementations? N/A Are the used keys controlled by the customer? Where are they stored? the KMS keys are generated by cdk and are used to encrypt the environment variables for all lambda functions in the lambda-api stack Are you introducing any new policies/roles/users? - N/A Have you used the least-privilege principle? How? N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit ed7cc3eb Author: Noah Paige <[email protected]> Date: Mon May 06 2024 09:32:30 GMT-0400 (Eastern Daylight Time) Add order_by for paginated queries (#1249) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - This PR aims to solve the following - (1) for particular queries (identified as ones that perform `.outerjoin()` operations and have results paginated with `paginate()` function - sometimes the returned query results is *less than* the limit set by the pageSize of the paginate function even when the total count is greater than the pageSize - Ex 1: 11 envs total, `query_user_environments()` returning 9 envs on 1st page + 2 on 2nd page - Ex 2: 10 envs total, `query_user_environments()` returning 9 envs on 1st page + no 2nd page - Believe this is to be happening due to the way SQLAlchemy is "uniquing" the records resulted from an outerjoin and then returning that result back to the frontend - Adding a `.distinct()` check on the query ensures each distinct record is returned (tested successfully) - (2) Currently we often times do not implement an `.order_by()` condition for the query used in `paginate()` and do not have a stable way of preserving order of the items returned from a query (i.e. when navigating through pages of response) - A generally good practice seems to include an `order_by()` on a column or set of columns - For each query used in `paginate()` this PR adds an `order_by()` condition (full list in comments below) Can read a bit more context from related issue linked below ### Relates - https://github.com/data-dot-all/dataall/issues/1241 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 98e67fa8 Author: Sofia Sazonova <[email protected]> Date: Fri May 03 2024 12:21:57 GMT-0400 (Eastern Daylight Time) fix: DATASET_READ_TABLE read permissions (#1237) ### Feature or Bugfix - Bugfix ### Detail - backfill DATASET_READ_TABLE permissions - delete this permissions, when dataset tables are revoked or deteled - ### Relates - #1173 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <[email protected]> commit 18e2f509 Author: Noah Paige <[email protected]> Date: Fri May 03 2024 10:14:52 GMT-0400 (Eastern Daylight Time) Fix local test groups listing for listGroups query (#1239) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Locally when trying to invite a team to Env or Org we call listGroups and the returned `LOCAL_TEST_GROUPS` is not returning the proper data type expected ### Relates N/A ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit a0be03c4 Author: dlpzx <[email protected]> Date: Fri May 03 2024 10:12:34 GMT-0400 (Eastern Daylight Time) Refactor: uncouple datasets and dataset_sharing modules - part 2-5 FINAL DELETE DATASETS_BASE (#1242) ### Feature or Bugfix - Refactoring ### Detail After all the previous PRs are merged, there should be no circular dependencies between `datasets` and `datasets_sharing`. We can now proceed to: - move `datasets_base` models, repositories, permissions and enums to `datasets` - adjust the `__init__` files to establish the `datasets_sharing` depends on `datasets` - adjust the Module interfaces to ensure that all necessary dataset models... are imported in the interface for sharing Next steps: - share_notifications paramter to dataset_sharing in config.json ### Relates #955 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit b68b40c1 Author: Sofia Sazonova <[email protected]> Date: Fri May 03 2024 10:12:11 GMT-0400 (Eastern Daylight Time) bugfix: EnvironmentGroup can remove other groups (#1234) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Now, if the group can't update other group, it also can not remove them. - ### Relates - #1212 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <[email protected]> commit 264539b5 Author: Noah Paige <[email protected]> Date: Fri May 03 2024 05:23:11 GMT-0400 (Eastern Daylight Time) Fix Alembic Migration: has table checks (#1240) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Fix `has_table()` check to ensure dropping the tables if the exists as part of alembic migration upgrade - Fix `DatasetLock nullable=True` ### Relates - https://github.com/data-dot-all/dataall/issues/1165 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? No - Is the input sanitized? N/A - What precautions are you taking before deserializing the data you consume? N/A - Is injection prevented by parametrizing queries? N/A - Have you ensured no `eval` or similar functions are used? N/A - Does this PR introduce any functionality or component that requires authorization? No - How have you ensured it respects the existing AuthN/AuthZ mechanisms? N/A - Are you logging failed auth attempts? N/A - Are you using or adding any cryptographic features? No - Do you use a standard proven implementations? N/A - Are the used keys controlled by the customer? Where are they stored? N/A - Are you introducing any new policies/roles/users? No - Have you used the least-privilege principle? How? N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 42a5f6bd Author: dlpzx <[email protected]> Date: Fri May 03 2024 02:24:09 GMT-0400 (Eastern Daylight Time) Refactor: uncouple datasets and dataset_sharing modules - part 2-4 (#1214) ### Feature or Bugfix - Refactoring ⚠️ MERGE AFTER https://github.com/data-dot-all/dataall/pull/1213 ### Detail This is needed as explained in full PR [AFTER 2.4] Refactor: uncouple datasets and dataset_sharing modules #1179 - [X] Use interface to resolve dataset roles related to datasets shared and implement logic in the dataset_sharing module - [X] Extend and clean-up stewards share permissions through interface ### Relates - #1179 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 6d3f2d45 Author: Sofia Sazonova <[email protected]> Date: Thu May 02 2024 10:55:00 GMT-0400 (Eastern Daylight Time) [After 2.4]Core Refactoring part5 (#1194) ### Feature or Bugfix - Refactoring ### Detail - focus on core/environments - move logic from resolvers to services - create s3_client in base/aws --> TO BE REFACTORED. Needs to be merged with dataset_sharind/aws/s3_client ### Relates - #741 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <[email protected]> commit 2ea24cbb Author: dlpzx <[email protected]> Date: Thu May 02 2024 08:22:12 GMT-0400 (Eastern Daylight Time) Refactor: uncouple datasets and dataset_sharing modules - part 2-3 (#1213) ### Feature or Bugfix - Refactoring ⚠️ MERGE AFTER https://github.com/data-dot-all/dataall/pull/1187 ### Detail This is needed as explained in full PR [AFTER 2.4] Refactor: uncouple datasets and dataset_sharing modules #1179 - [X] Creates an interface to execute checks and clean-ups of data sharing objects when dataset objects are deleted (initially it was going to be an db interface, but I think it is better in the service) - [X] Move listDatasetShares query to dataset_sharing module in https://github.com/data-dot-all/dataall/pull/1185 ### Relates - #1179 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 750a5ec8 Author: Anushka Singh <[email protected]> Date: Wed May 01 2024 12:28:18 GMT-0400 (Eastern Daylight Time) Feature:1221 - Make visibility of auto-approval toggle configurable based on confidentiality (#1223) ### Feature or Bugfix - Feature ### Detail - Users should be able to disable visibility of auto-approval toggle with code. For example, at our company, we require that shares always go through approval process if their confidentiality classification is Secret. We dont even want to give the option to users to be able to set autoApproval enabled to ensure they dont do so by mistake and end up over sharing. Video demo: https://github.com/data-dot-all/dataall/issues/1221#issuecomment-2077412044 ### Relates - https://github.com/data-dot-all/dataall/issues/1221 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 82044689 Author: dlpzx <[email protected]> Date: Wed May 01 2024 12:26:42 GMT-0400 (Eastern Daylight Time) Refactor: uncouple datasets and dataset_sharing modules - part 2-2 (#1187) ### Feature or Bugfix - Refactoring ⚠️ MERGE AFTER https://github.com/data-dot-all/dataall/pull/1185 ### Detail This is needed as explained in full PR [AFTER 2.4] Refactor: uncouple datasets and dataset_sharing modules #1179 - Split the getDatasetAssumeRole API into 2 APIs, one for dataset owners role (in datasets module) and another one for share requester roles (in datasets_sharing module) ### Relates - #1179 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 5173419f Author: Noah Paige <[email protected]> Date: Wed May 01 2024 12:24:42 GMT-0400 (Eastern Daylight Time) Fix so listValidEnvironments called only once (#1238) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - When request access to a share on data.all the query to `listValidEnvironments` used to be called twice which (depending on how long for query results to return) could cause the environment initially selected to disappear ### Relates - Continuation of https://github.com/data-dot-all/dataall/issues/916 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 7656ea86 Author: dlpzx <[email protected]> Date: Tue Apr 30 2024 07:13:01 GMT-0400 (Eastern Daylight Time) Add integration tests on a real API client and integrate the tests in CICD (#1219) ### Feature or Bugfix - Feature ### Detail Add integration tests that use a real Client to execute different validation actions. - Define the Client and the way API calls are posted to API Gateway in the conftest - Define the Cognito users and the different fixtures needed for all tests - Write tests for the Organization core module as example - Add feature flag in `cdk.json` called `with_approval_tests` that can be defined at the deployment environment level. If set to True, a CodeBuild stage running the tests is created. ### Relates - https://github.com/data-dot-all/dataall/issues/1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you c…
noah-paige
added a commit
that referenced
this issue
Jun 25, 2024
commit 1617953c Author: Noah Paige <[email protected]> Date: Mon May 20 2024 16:48:17 GMT-0400 (Eastern Daylight Time) Add open dependency matrix commit d8497c55 Author: Noah Paige <[email protected]> Date: Mon May 20 2024 15:19:35 GMT-0400 (Eastern Daylight Time) fix commit 199ab505 Author: Admin/noahpaig-Isengard <Admin/noahpaig-Isengard> Date: Mon May 20 2024 15:16:14 GMT-0400 (Eastern Daylight Time) Conflicts resolved in the console. commit ad415575 Author: Noah Paige <[email protected]> Date: Mon May 20 2024 15:13:06 GMT-0400 (Eastern Daylight Time) Merge branch 'chatbot-test' into noah-main-2 commit 2893efc7 Author: Noah Paige <[email protected]> Date: Mon May 20 2024 15:10:20 GMT-0400 (Eastern Daylight Time) Merge branch 'chatbot-test' into noah-main-2 commit caad12e1 Author: Noah Paige <[email protected]> Date: Mon May 20 2024 15:12:47 GMT-0400 (Eastern Daylight Time) ruff commit a73b7110 Author: Noah Paige <[email protected]> Date: Mon May 20 2024 15:09:19 GMT-0400 (Eastern Daylight Time) Remove hardcoding commit 7b32a8f7 Author: Noah Paige <[email protected]> Date: Mon May 20 2024 15:07:29 GMT-0400 (Eastern Daylight Time) Chatbot POC commit e25e5815 Author: Noah Paige <[email protected]> Date: Mon May 20 2024 15:08:32 GMT-0400 (Eastern Daylight Time) Merge branch '1215-share-logs' into noah-main-2 commit a06c8cba Author: Noah Paige <[email protected]> Date: Fri May 17 2024 16:37:05 GMT-0400 (Eastern Daylight Time) Merge share logs PR commit a5626670 Author: Sofia Sazonova <[email protected]> Date: Fri May 17 2024 17:19:25 GMT-0400 (Eastern Daylight Time) make ruff happy commit aee98cf7 Author: Noah Paige <[email protected]> Date: Fri May 17 2024 16:34:24 GMT-0400 (Eastern Daylight Time) Merge share logs PR commit c3ee2f7c Author: Sofia Sazonova <[email protected]> Date: Fri May 17 2024 17:11:00 GMT-0400 (Eastern Daylight Time) PR comments commit 5ca55303 Author: Sofia Sazonova <[email protected]> Date: Wed May 15 2024 11:57:41 GMT-0400 (Eastern Daylight Time) remove unused imports commit 8f8bf3dd Author: Sofia Sazonova <[email protected]> Date: Wed May 15 2024 11:56:39 GMT-0400 (Eastern Daylight Time) restrict access to the share logs commit 9137da9b Author: Sofia Sazonova <[email protected]> Date: Wed May 15 2024 11:28:32 GMT-0400 (Eastern Daylight Time) share Logs button is available only for dataset Admins and stewards commit fcb16bd9 Author: Sofia Sazonova <[email protected]> Date: Wed May 15 2024 10:46:25 GMT-0400 (Eastern Daylight Time) getShareLogs query commit 0503a3bb Author: Sofia Sazonova <[email protected]> Date: Wed May 15 2024 10:21:25 GMT-0400 (Eastern Daylight Time) Logs modal in Share View commit bab2f3e6 Author: Sofia Sazonova <[email protected]> Date: Mon May 13 2024 09:09:18 GMT-0400 (Eastern Daylight Time) Add confirmation pop-ups for deletion of team roles and groups (#1231) ### Feature or Bugfix - Feature ### Detail Pop ups added for: - deletion team from environment - deletion of the consumption role - deletion of group from Organization ### Relates - #942 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. Co-authored-by: Sofia Sazonova <[email protected]> commit 93ff7725 Author: Sofia Sazonova <[email protected]> Date: Mon May 13 2024 08:00:38 GMT-0400 (Eastern Daylight Time) Update version.json (#1264) Release info update commit e718d861 Author: Sofia Sazonova <[email protected]> Date: Mon May 13 2024 07:29:27 GMT-0400 (Eastern Daylight Time) fix permission query (#1263) ### Feature or Bugfix - Bugfix ### Detail - The filter -- array of permissions' NAMES, so in order to query policies correctly we need to add join - The filter 'share_type' and 'share_item_status' must be string - IMPORTANT: in block "finally" the param session was used, but session was defined only in "try" block. So, the lock failed to be released. ### Relates - <URL or Ticket> ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <[email protected]> commit 479b8f3f Author: mourya-33 <[email protected]> Date: Wed May 08 2024 10:29:36 GMT-0400 (Eastern Daylight Time) Add encryption and tag immutability to ECR repository (#1224) ### Feature or Bugfix - Bugfix ### Detail - Currently the ecr repository created do not have encryption and tag immutability enabled which is identified by checkov scans. This fix is to enable both. ### Relates [- <URL or Ticket>](https://github.com/data-dot-all/dataall/issues/1200) ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes N/A - Is the input sanitized? N/A - What precautions are you taking before deserializing the data you consume? N/A - Is injection prevented by parametrizing queries? N/A - Have you ensured no `eval` or similar functions are used? N/A - Does this PR introduce any functionality or component that requires authorization? N/A - How have you ensured it respects the existing AuthN/AuthZ mechanisms? N/A - Are you logging failed auth attempts? N/A - Are you using or adding any cryptographic features? N/A - Do you use a standard proven implementations? N/A - Are the used keys controlled by the customer? Where are they stored? No. This is with default encryption - Are you introducing any new policies/roles/users? N/A - Have you used the least-privilege principle? How? N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 2f885773 Author: Sofia Sazonova <[email protected]> Date: Wed May 08 2024 09:22:40 GMT-0400 (Eastern Daylight Time) Multiple permission roots (#1259) ### Feature or Bugfix - Bugfix ### Detail - GET_DATASET_TABLE (FOLDER) permissions are granted to the group only if they are not granted already - these permissions are removed if group is not admin|steward and there are no other shares of this item. ### Relates - #1174 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <[email protected]> commit c4cc07ee Author: Petros Kalos <[email protected]> Date: Wed May 08 2024 08:54:02 GMT-0400 (Eastern Daylight Time) explicitly specify dataset_client s3 endpoint_url (#1260) * AWS requires that the endpoint_url should be explicitly specified for some regions * Remove misleading CORS error message, the upload step can fail for many reason ### Feature or Bugfix - Bugfix ### Detail Resolves #778 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 40defe8e Author: dlpzx <[email protected]> Date: Tue May 07 2024 11:52:17 GMT-0400 (Eastern Daylight Time) Generic dataset module and specific s3_datasets module - part 1 (Rename datasets as s3_datasets) (#1250) ### Feature or Bugfix - Refactoring ### Detail - Rename `datasets` module to `s3_datasets` module This PR is the first step to extract a generic datasets_base module that implements the undifferentiated concepts of Dataset in data.all. s3_datasets will use this base module to implement the specific implementation for S3 datatasets. ### Relates - #1123 - #955 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 74a303cb Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue May 07 2024 02:26:09 GMT-0400 (Eastern Daylight Time) Bump werkzeug from 3.0.1 to 3.0.3 in /tests (#1253) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.1 to 3.0.3. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/releases">werkzeug's releases</a>.</em></p> <blockquote> <h2>3.0.3</h2> <p>This is the Werkzeug 3.0.3 security release, which fixes security issues and bugs but does not otherwise change behavior and should not result in breaking changes.</p> <p>PyPI: <a href="https://pypi.org/project/Werkzeug/3.0.3/">https://pypi.org/project/Werkzeug/3.0.3/</a> Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3</a> Milestone: <a href="https://github.com/pallets/werkzeug/milestone/35?closed=1">https://github.com/pallets/werkzeug/milestone/35?closed=1</a></p> <ul> <li>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. GHSA-2g68-c3qc-8985</li> <li>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2823">#2823</a></li> <li>Better TLS cert format with <code>adhoc</code> dev certs. <a href="https://redirect.github.com/pallets/werkzeug/issues/2891">#2891</a></li> <li>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. <a href="https://redirect.github.com/pallets/werkzeug/issues/2828">#2828</a></li> <li>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2836">#2836</a></li> </ul> <h2>3.0.2</h2> <p>This is a fix release for the 3.0.x feature branch.</p> <ul> <li>Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/blob/main/CHANGES.rst">werkzeug's changelog</a>.</em></p> <blockquote> <h2>Version 3.0.3</h2> <p>Released 2024-05-05</p> <ul> <li> <p>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. :ghsa:<code>2g68-c3qc-8985</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> <li> <p>Better TLS cert format with <code>adhoc</code> dev certs. :pr:<code>2891</code></p> </li> <li> <p>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. :issue:<code>2828</code></p> </li> <li> <p>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. :issue:<code>2836</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> </ul> <h2>Version 3.0.2</h2> <p>Released 2024-04-01</p> <ul> <li>Ensure setting <code>merge_slashes</code> to <code>False</code> results in <code>NotFound</code> for repeated-slash requests against single slash routes. :issue:<code>2834</code></li> <li>Fix handling of <code>TypeError</code> in <code>TypeConversionDict.get()</code> to match <code>ValueError</code>. :issue:<code>2843</code></li> <li>Fix <code>response_wrapper</code> type check in test client. :issue:<code>2831</code></li> <li>Make the return type of <code>MultiPartParser.parse</code> more precise. :issue:<code>2840</code></li> <li>Raise an error if converter arguments cannot be parsed. :issue:<code>2822</code></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pallets/werkzeug/commit/f9995e967979eb694d6b31536cc65314fd7e9c8c"><code>f9995e9</code></a> release version 3.0.3</li> <li><a href="https://github.com/pallets/werkzeug/commit/3386395b24c7371db11a5b8eaac0c91da5362692"><code>3386395</code></a> Merge pull request from GHSA-2g68-c3qc-8985</li> <li><a href="https://github.com/pallets/werkzeug/commit/890b6b62634fa61224222aee31081c61b054ff01"><code>890b6b6</code></a> only require trusted host for evalex</li> <li><a href="https://github.com/pallets/werkzeug/commit/71b69dfb7df3d912e66bab87fbb1f21f83504967"><code>71b69df</code></a> restrict debugger trusted hosts</li> <li><a href="https://github.com/pallets/werkzeug/commit/d2d3869525a4ffb2c41dfb2c0e39d94dab2d870c"><code>d2d3869</code></a> endpoint type is Any (<a href="https://redirect.github.com/pallets/werkzeug/issues/2895">#2895</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/7080b55acd48b68afdda65ee6c7f99e9afafb0ba"><code>7080b55</code></a> endpoint type is Any</li> <li><a href="https://github.com/pallets/werkzeug/commit/7555eff296fbdf12f2e576b6bbb0b506df8417ed"><code>7555eff</code></a> remove iri_to_uri redirect workaround (<a href="https://redirect.github.com/pallets/werkzeug/issues/2894">#2894</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/97fb2f722297ae4e12e36dab024e0acf8477b3c8"><code>97fb2f7</code></a> remove _invalid_iri_to_uri workaround</li> <li><a href="https://github.com/pallets/werkzeug/commit/249527ff981e7aa22cd714825c5637cc92df7761"><code>249527f</code></a> make cn field a valid single hostname, and use wildcard in SANs field. (<a href="https://redirect.github.com/pallets/werkzeug/issues/2892">#2892</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/793be472c9d145eb9be7d4200672d1806289d84a"><code>793be47</code></a> update adhoc tls dev cert format</li> <li>Additional commits viewable in <a href="https://github.com/pallets/werkzeug/compare/3.0.1...3.0.3">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/data-dot-all/dataall/network/alerts). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 2f33320c Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue May 07 2024 02:25:03 GMT-0400 (Eastern Daylight Time) Bump werkzeug from 3.0.1 to 3.0.3 in /backend/dataall/base/cdkproxy (#1252) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.1 to 3.0.3. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/releases">werkzeug's releases</a>.</em></p> <blockquote> <h2>3.0.3</h2> <p>This is the Werkzeug 3.0.3 security release, which fixes security issues and bugs but does not otherwise change behavior and should not result in breaking changes.</p> <p>PyPI: <a href="https://pypi.org/project/Werkzeug/3.0.3/">https://pypi.org/project/Werkzeug/3.0.3/</a> Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3</a> Milestone: <a href="https://github.com/pallets/werkzeug/milestone/35?closed=1">https://github.com/pallets/werkzeug/milestone/35?closed=1</a></p> <ul> <li>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. GHSA-2g68-c3qc-8985</li> <li>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2823">#2823</a></li> <li>Better TLS cert format with <code>adhoc</code> dev certs. <a href="https://redirect.github.com/pallets/werkzeug/issues/2891">#2891</a></li> <li>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. <a href="https://redirect.github.com/pallets/werkzeug/issues/2828">#2828</a></li> <li>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2836">#2836</a></li> </ul> <h2>3.0.2</h2> <p>This is a fix release for the 3.0.x feature branch.</p> <ul> <li>Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/blob/main/CHANGES.rst">werkzeug's changelog</a>.</em></p> <blockquote> <h2>Version 3.0.3</h2> <p>Released 2024-05-05</p> <ul> <li> <p>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. :ghsa:<code>2g68-c3qc-8985</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> <li> <p>Better TLS cert format with <code>adhoc</code> dev certs. :pr:<code>2891</code></p> </li> <li> <p>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. :issue:<code>2828</code></p> </li> <li> <p>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. :issue:<code>2836</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> </ul> <h2>Version 3.0.2</h2> <p>Released 2024-04-01</p> <ul> <li>Ensure setting <code>merge_slashes</code> to <code>False</code> results in <code>NotFound</code> for repeated-slash requests against single slash routes. :issue:<code>2834</code></li> <li>Fix handling of <code>TypeError</code> in <code>TypeConversionDict.get()</code> to match <code>ValueError</code>. :issue:<code>2843</code></li> <li>Fix <code>response_wrapper</code> type check in test client. :issue:<code>2831</code></li> <li>Make the return type of <code>MultiPartParser.parse</code> more precise. :issue:<code>2840</code></li> <li>Raise an error if converter arguments cannot be parsed. :issue:<code>2822</code></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pallets/werkzeug/commit/f9995e967979eb694d6b31536cc65314fd7e9c8c"><code>f9995e9</code></a> release version 3.0.3</li> <li><a href="https://github.com/pallets/werkzeug/commit/3386395b24c7371db11a5b8eaac0c91da5362692"><code>3386395</code></a> Merge pull request from GHSA-2g68-c3qc-8985</li> <li><a href="https://github.com/pallets/werkzeug/commit/890b6b62634fa61224222aee31081c61b054ff01"><code>890b6b6</code></a> only require trusted host for evalex</li> <li><a href="https://github.com/pallets/werkzeug/commit/71b69dfb7df3d912e66bab87fbb1f21f83504967"><code>71b69df</code></a> restrict debugger trusted hosts</li> <li><a href="https://github.com/pallets/werkzeug/commit/d2d3869525a4ffb2c41dfb2c0e39d94dab2d870c"><code>d2d3869</code></a> endpoint type is Any (<a href="https://redirect.github.com/pallets/werkzeug/issues/2895">#2895</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/7080b55acd48b68afdda65ee6c7f99e9afafb0ba"><code>7080b55</code></a> endpoint type is Any</li> <li><a href="https://github.com/pallets/werkzeug/commit/7555eff296fbdf12f2e576b6bbb0b506df8417ed"><code>7555eff</code></a> remove iri_to_uri redirect workaround (<a href="https://redirect.github.com/pallets/werkzeug/issues/2894">#2894</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/97fb2f722297ae4e12e36dab024e0acf8477b3c8"><code>97fb2f7</code></a> remove _invalid_iri_to_uri workaround</li> <li><a href="https://github.com/pallets/werkzeug/commit/249527ff981e7aa22cd714825c5637cc92df7761"><code>249527f</code></a> make cn field a valid single hostname, and use wildcard in SANs field. (<a href="https://redirect.github.com/pallets/werkzeug/issues/2892">#2892</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/793be472c9d145eb9be7d4200672d1806289d84a"><code>793be47</code></a> update adhoc tls dev cert format</li> <li>Additional commits viewable in <a href="https://github.com/pallets/werkzeug/compare/3.0.1...3.0.3">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/data-dot-all/dataall/network/alerts). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 0b49633f Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue May 07 2024 02:24:34 GMT-0400 (Eastern Daylight Time) Bump werkzeug from 3.0.1 to 3.0.3 in /tests_new/integration_tests (#1254) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.1 to 3.0.3. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/releases">werkzeug's releases</a>.</em></p> <blockquote> <h2>3.0.3</h2> <p>This is the Werkzeug 3.0.3 security release, which fixes security issues and bugs but does not otherwise change behavior and should not result in breaking changes.</p> <p>PyPI: <a href="https://pypi.org/project/Werkzeug/3.0.3/">https://pypi.org/project/Werkzeug/3.0.3/</a> Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-3</a> Milestone: <a href="https://github.com/pallets/werkzeug/milestone/35?closed=1">https://github.com/pallets/werkzeug/milestone/35?closed=1</a></p> <ul> <li>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. GHSA-2g68-c3qc-8985</li> <li>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2823">#2823</a></li> <li>Better TLS cert format with <code>adhoc</code> dev certs. <a href="https://redirect.github.com/pallets/werkzeug/issues/2891">#2891</a></li> <li>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. <a href="https://redirect.github.com/pallets/werkzeug/issues/2828">#2828</a></li> <li>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. <a href="https://redirect.github.com/pallets/werkzeug/issues/2836">#2836</a></li> </ul> <h2>3.0.2</h2> <p>This is a fix release for the 3.0.x feature branch.</p> <ul> <li>Changes: <a href="https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2">https://werkzeug.palletsprojects.com/en/3.0.x/changes/#version-3-0-2</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pallets/werkzeug/blob/main/CHANGES.rst">werkzeug's changelog</a>.</em></p> <blockquote> <h2>Version 3.0.3</h2> <p>Released 2024-05-05</p> <ul> <li> <p>Only allow <code>localhost</code>, <code>.localhost</code>, <code>127.0.0.1</code>, or the specified hostname when running the dev server, to make debugger requests. Additional hosts can be added by using the debugger middleware directly. The debugger UI makes requests using the full URL rather than only the path. :ghsa:<code>2g68-c3qc-8985</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> <li> <p>Better TLS cert format with <code>adhoc</code> dev certs. :pr:<code>2891</code></p> </li> <li> <p>Inform Python < 3.12 how to handle <code>itms-services</code> URIs correctly, rather than using an overly-broad workaround in Werkzeug that caused some redirect URIs to be passed on without encoding. :issue:<code>2828</code></p> </li> <li> <p>Type annotation for <code>Rule.endpoint</code> and other uses of <code>endpoint</code> is <code>Any</code>. :issue:<code>2836</code></p> </li> <li> <p>Make reloader more robust when <code>""</code> is in <code>sys.path</code>. :pr:<code>2823</code></p> </li> </ul> <h2>Version 3.0.2</h2> <p>Released 2024-04-01</p> <ul> <li>Ensure setting <code>merge_slashes</code> to <code>False</code> results in <code>NotFound</code> for repeated-slash requests against single slash routes. :issue:<code>2834</code></li> <li>Fix handling of <code>TypeError</code> in <code>TypeConversionDict.get()</code> to match <code>ValueError</code>. :issue:<code>2843</code></li> <li>Fix <code>response_wrapper</code> type check in test client. :issue:<code>2831</code></li> <li>Make the return type of <code>MultiPartParser.parse</code> more precise. :issue:<code>2840</code></li> <li>Raise an error if converter arguments cannot be parsed. :issue:<code>2822</code></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pallets/werkzeug/commit/f9995e967979eb694d6b31536cc65314fd7e9c8c"><code>f9995e9</code></a> release version 3.0.3</li> <li><a href="https://github.com/pallets/werkzeug/commit/3386395b24c7371db11a5b8eaac0c91da5362692"><code>3386395</code></a> Merge pull request from GHSA-2g68-c3qc-8985</li> <li><a href="https://github.com/pallets/werkzeug/commit/890b6b62634fa61224222aee31081c61b054ff01"><code>890b6b6</code></a> only require trusted host for evalex</li> <li><a href="https://github.com/pallets/werkzeug/commit/71b69dfb7df3d912e66bab87fbb1f21f83504967"><code>71b69df</code></a> restrict debugger trusted hosts</li> <li><a href="https://github.com/pallets/werkzeug/commit/d2d3869525a4ffb2c41dfb2c0e39d94dab2d870c"><code>d2d3869</code></a> endpoint type is Any (<a href="https://redirect.github.com/pallets/werkzeug/issues/2895">#2895</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/7080b55acd48b68afdda65ee6c7f99e9afafb0ba"><code>7080b55</code></a> endpoint type is Any</li> <li><a href="https://github.com/pallets/werkzeug/commit/7555eff296fbdf12f2e576b6bbb0b506df8417ed"><code>7555eff</code></a> remove iri_to_uri redirect workaround (<a href="https://redirect.github.com/pallets/werkzeug/issues/2894">#2894</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/97fb2f722297ae4e12e36dab024e0acf8477b3c8"><code>97fb2f7</code></a> remove _invalid_iri_to_uri workaround</li> <li><a href="https://github.com/pallets/werkzeug/commit/249527ff981e7aa22cd714825c5637cc92df7761"><code>249527f</code></a> make cn field a valid single hostname, and use wildcard in SANs field. (<a href="https://redirect.github.com/pallets/werkzeug/issues/2892">#2892</a>)</li> <li><a href="https://github.com/pallets/werkzeug/commit/793be472c9d145eb9be7d4200672d1806289d84a"><code>793be47</code></a> update adhoc tls dev cert format</li> <li>Additional commits viewable in <a href="https://github.com/pallets/werkzeug/compare/3.0.1...3.0.3">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/data-dot-all/dataall/network/alerts). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 08862420 Author: mourya-33 <[email protected]> Date: Tue May 07 2024 02:15:15 GMT-0400 (Eastern Daylight Time) Updated lambda_api.py to add encryption for lambda env vars for custo… (#1255) Feature or Bugfix Bugfix Detail The environment variables for the lambda functions are not encrypted in cdk which are identified by checkov scans. This fix is to enable kms encryption for the lambda environment variables. Relates Security Please answer the questions below briefly where applicable, or write N/A. Based on [OWASP 10](https://owasp.org/Top10/en/). Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? N/A Is the input sanitized? N/A What precautions are you taking before deserializing the data you consume? N/A Is injection prevented by parametrizing queries? N/A Have you ensured no eval or similar functions are used? N/A Does this PR introduce any functionality or component that requires authorization? N/A How have you ensured it respects the existing AuthN/AuthZ mechanisms? N/A Are you logging failed auth attempts? N/A Are you using or adding any cryptographic features? N/A Do you use a standard proven implementations? N/A Are the used keys controlled by the customer? Where are they stored? the KMS keys are generated by cdk and are used to encrypt the environment variables for all lambda functions in the lambda-api stack Are you introducing any new policies/roles/users? - N/A Have you used the least-privilege principle? How? N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit ed7cc3eb Author: Noah Paige <[email protected]> Date: Mon May 06 2024 09:32:30 GMT-0400 (Eastern Daylight Time) Add order_by for paginated queries (#1249) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - This PR aims to solve the following - (1) for particular queries (identified as ones that perform `.outerjoin()` operations and have results paginated with `paginate()` function - sometimes the returned query results is *less than* the limit set by the pageSize of the paginate function even when the total count is greater than the pageSize - Ex 1: 11 envs total, `query_user_environments()` returning 9 envs on 1st page + 2 on 2nd page - Ex 2: 10 envs total, `query_user_environments()` returning 9 envs on 1st page + no 2nd page - Believe this is to be happening due to the way SQLAlchemy is "uniquing" the records resulted from an outerjoin and then returning that result back to the frontend - Adding a `.distinct()` check on the query ensures each distinct record is returned (tested successfully) - (2) Currently we often times do not implement an `.order_by()` condition for the query used in `paginate()` and do not have a stable way of preserving order of the items returned from a query (i.e. when navigating through pages of response) - A generally good practice seems to include an `order_by()` on a column or set of columns - For each query used in `paginate()` this PR adds an `order_by()` condition (full list in comments below) Can read a bit more context from related issue linked below ### Relates - https://github.com/data-dot-all/dataall/issues/1241 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 98e67fa8 Author: Sofia Sazonova <[email protected]> Date: Fri May 03 2024 12:21:57 GMT-0400 (Eastern Daylight Time) fix: DATASET_READ_TABLE read permissions (#1237) ### Feature or Bugfix - Bugfix ### Detail - backfill DATASET_READ_TABLE permissions - delete this permissions, when dataset tables are revoked or deteled - ### Relates - #1173 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <[email protected]> commit 18e2f509 Author: Noah Paige <[email protected]> Date: Fri May 03 2024 10:14:52 GMT-0400 (Eastern Daylight Time) Fix local test groups listing for listGroups query (#1239) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Locally when trying to invite a team to Env or Org we call listGroups and the returned `LOCAL_TEST_GROUPS` is not returning the proper data type expected ### Relates N/A ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit a0be03c4 Author: dlpzx <[email protected]> Date: Fri May 03 2024 10:12:34 GMT-0400 (Eastern Daylight Time) Refactor: uncouple datasets and dataset_sharing modules - part 2-5 FINAL DELETE DATASETS_BASE (#1242) ### Feature or Bugfix - Refactoring ### Detail After all the previous PRs are merged, there should be no circular dependencies between `datasets` and `datasets_sharing`. We can now proceed to: - move `datasets_base` models, repositories, permissions and enums to `datasets` - adjust the `__init__` files to establish the `datasets_sharing` depends on `datasets` - adjust the Module interfaces to ensure that all necessary dataset models... are imported in the interface for sharing Next steps: - share_notifications paramter to dataset_sharing in config.json ### Relates #955 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit b68b40c1 Author: Sofia Sazonova <[email protected]> Date: Fri May 03 2024 10:12:11 GMT-0400 (Eastern Daylight Time) bugfix: EnvironmentGroup can remove other groups (#1234) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Now, if the group can't update other group, it also can not remove them. - ### Relates - #1212 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <[email protected]> commit 264539b5 Author: Noah Paige <[email protected]> Date: Fri May 03 2024 05:23:11 GMT-0400 (Eastern Daylight Time) Fix Alembic Migration: has table checks (#1240) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Fix `has_table()` check to ensure dropping the tables if the exists as part of alembic migration upgrade - Fix `DatasetLock nullable=True` ### Relates - https://github.com/data-dot-all/dataall/issues/1165 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? No - Is the input sanitized? N/A - What precautions are you taking before deserializing the data you consume? N/A - Is injection prevented by parametrizing queries? N/A - Have you ensured no `eval` or similar functions are used? N/A - Does this PR introduce any functionality or component that requires authorization? No - How have you ensured it respects the existing AuthN/AuthZ mechanisms? N/A - Are you logging failed auth attempts? N/A - Are you using or adding any cryptographic features? No - Do you use a standard proven implementations? N/A - Are the used keys controlled by the customer? Where are they stored? N/A - Are you introducing any new policies/roles/users? No - Have you used the least-privilege principle? How? N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 42a5f6bd Author: dlpzx <[email protected]> Date: Fri May 03 2024 02:24:09 GMT-0400 (Eastern Daylight Time) Refactor: uncouple datasets and dataset_sharing modules - part 2-4 (#1214) ### Feature or Bugfix - Refactoring ⚠️ MERGE AFTER https://github.com/data-dot-all/dataall/pull/1213 ### Detail This is needed as explained in full PR [AFTER 2.4] Refactor: uncouple datasets and dataset_sharing modules #1179 - [X] Use interface to resolve dataset roles related to datasets shared and implement logic in the dataset_sharing module - [X] Extend and clean-up stewards share permissions through interface ### Relates - #1179 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 6d3f2d45 Author: Sofia Sazonova <[email protected]> Date: Thu May 02 2024 10:55:00 GMT-0400 (Eastern Daylight Time) [After 2.4]Core Refactoring part5 (#1194) ### Feature or Bugfix - Refactoring ### Detail - focus on core/environments - move logic from resolvers to services - create s3_client in base/aws --> TO BE REFACTORED. Needs to be merged with dataset_sharind/aws/s3_client ### Relates - #741 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <[email protected]> commit 2ea24cbb Author: dlpzx <[email protected]> Date: Thu May 02 2024 08:22:12 GMT-0400 (Eastern Daylight Time) Refactor: uncouple datasets and dataset_sharing modules - part 2-3 (#1213) ### Feature or Bugfix - Refactoring ⚠️ MERGE AFTER https://github.com/data-dot-all/dataall/pull/1187 ### Detail This is needed as explained in full PR [AFTER 2.4] Refactor: uncouple datasets and dataset_sharing modules #1179 - [X] Creates an interface to execute checks and clean-ups of data sharing objects when dataset objects are deleted (initially it was going to be an db interface, but I think it is better in the service) - [X] Move listDatasetShares query to dataset_sharing module in https://github.com/data-dot-all/dataall/pull/1185 ### Relates - #1179 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 750a5ec8 Author: Anushka Singh <[email protected]> Date: Wed May 01 2024 12:28:18 GMT-0400 (Eastern Daylight Time) Feature:1221 - Make visibility of auto-approval toggle configurable based on confidentiality (#1223) ### Feature or Bugfix - Feature ### Detail - Users should be able to disable visibility of auto-approval toggle with code. For example, at our company, we require that shares always go through approval process if their confidentiality classification is Secret. We dont even want to give the option to users to be able to set autoApproval enabled to ensure they dont do so by mistake and end up over sharing. Video demo: https://github.com/data-dot-all/dataall/issues/1221#issuecomment-2077412044 ### Relates - https://github.com/data-dot-all/dataall/issues/1221 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 82044689 Author: dlpzx <[email protected]> Date: Wed May 01 2024 12:26:42 GMT-0400 (Eastern Daylight Time) Refactor: uncouple datasets and dataset_sharing modules - part 2-2 (#1187) ### Feature or Bugfix - Refactoring ⚠️ MERGE AFTER https://github.com/data-dot-all/dataall/pull/1185 ### Detail This is needed as explained in full PR [AFTER 2.4] Refactor: uncouple datasets and dataset_sharing modules #1179 - Split the getDatasetAssumeRole API into 2 APIs, one for dataset owners role (in datasets module) and another one for share requester roles (in datasets_sharing module) ### Relates - #1179 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 5173419f Author: Noah Paige <[email protected]> Date: Wed May 01 2024 12:24:42 GMT-0400 (Eastern Daylight Time) Fix so listValidEnvironments called only once (#1238) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - When request access to a share on data.all the query to `listValidEnvironments` used to be called twice which (depending on how long for query results to return) could cause the environment initially selected to disappear ### Relates - Continuation of https://github.com/data-dot-all/dataall/issues/916 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 7656ea86 Author: dlpzx <[email protected]> Date: Tue Apr 30 2024 07:13:01 GMT-0400 (Eastern Daylight Time) Add integration tests on a real API client and integrate the tests in CICD (#1219) ### Feature or Bugfix - Feature ### Detail Add integration tests that use a real Client to execute different validation actions. - Define the Client and the way API calls are posted to API Gateway in the conftest - Define the Cognito users and the different fixtures needed for all tests - Write tests for the Organization core module as example - Add feature flag in `cdk.json` called `with_approval_tests` that can be defined at the deployment environment level. If set to True, a CodeBuild stage running the tests is created. ### Relates - https://github.com/data-dot-all/dataall/issues/1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage …
Merged
petrkalos
added a commit
that referenced
this issue
Jun 28, 2024
### Feature or Bugfix Feature ### Detail * add list_environment tests * add test for updating an environment (via update_stack) * generalise the polling functions for stacks ### Relates #1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
dlpzx
added a commit
that referenced
this issue
Jul 9, 2024
### Feature or Bugfix - Feature - tests ### Detail Integration tests for all API calls except for run athena sql - the draft is commented out in the second commit; but to unblock this PR I removed them <img width="995" alt="image" src="https://github.com/data-dot-all/dataall/assets/71252798/943769b9-1e9f-4bce-80e3-ce0f7e7c9a94"> ### Relates #1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
noah-paige
added a commit
that referenced
this issue
Aug 30, 2024
commit 22a6f6ef Author: Noah Paige <[email protected]> Date: Mon Jul 08 2024 11:28:07 GMT-0400 (Eastern Daylight Time) Add integ tests commit 4fb7d653 Author: Noah Paige <[email protected]> Date: Mon Jul 08 2024 11:26:36 GMT-0400 (Eastern Daylight Time) Merge env test changes commit 4cf42e8 Author: Petros Kalos <[email protected]> Date: Fri Jul 05 2024 08:19:34 GMT-0400 (Eastern Daylight Time) improve docs commit 65f930a Author: Petros Kalos <[email protected]> Date: Fri Jul 05 2024 08:10:56 GMT-0400 (Eastern Daylight Time) fix failures commit 170b7ce Author: Petros Kalos <[email protected]> Date: Wed Jul 03 2024 10:52:20 GMT-0400 (Eastern Daylight Time) add group/consumption_role invite/remove tests commit ba77d69 Author: dlpzx <[email protected]> Date: Wed Jul 03 2024 06:51:47 GMT-0400 (Eastern Daylight Time) Rename alias for env_vars kms key in cognito lambdas FE and BE (#1385) ### Feature or Bugfix - Bugfix ### Detail For the case in which we deploy FE and BE in us-east-1 the new lambda env_key alias is the same one for TriggerFunctionCognitoUrlsConfig in FE and for TriggerFunctionCognitoConfig in BE, which results in a failure of the CICD in the FE stack because the alias already exists. This PR changes the name of both aliases to avoid this conflict. It also adds envname to avoid issues with other deployment environments/tooling account in the future ### Relates - <URL or Ticket> ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit e5923a9 Author: dlpzx <[email protected]> Date: Wed Jul 03 2024 04:27:11 GMT-0400 (Eastern Daylight Time) Fix lambda_env_key out of scope for vpc-facing cognito setup (#1384) ### Feature or Bugfix - Bugfix ### Detail The KMS key for the Lambda environment variables in the Cognito IdP stack was defined inside an if-clause for internet facing frontend. Outside of that if, for vpc-facing architecture the kms key does not exist and the CICD pipeline fails. This PRs move the creation of the KMS key outside of the if. ### Relates ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 3ccacfc Author: Noah Paige <[email protected]> Date: Mon Jul 01 2024 13:56:58 GMT-0400 (Eastern Daylight Time) Add delete docs not found when re indexing in catalog task (#1365) ### Feature or Bugfix <!-- please choose --> - Feature ### Detail - Add logic to Catalog Indexer Task to Delete Docs No Longer in RDS - TODO: Add Ability to Re-index Catalog Items via Dataall Admin UI ### Relates - #1078 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit e2817a1 Author: Noah Paige <[email protected]> Date: Mon Jul 01 2024 05:14:07 GMT-0400 (Eastern Daylight Time) Fix/glossary status (#1373) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Add back `status` to Glossary GQL Object for GQL Operations (getGlossary, listGlossaries) - Fix `listOrganizationGroupPermissions` enforce non null on FE ### Relates ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit c3c58bd Author: Petros Kalos <[email protected]> Date: Fri Jun 28 2024 06:55:42 GMT-0400 (Eastern Daylight Time) add environment tests (#1371) ### Feature or Bugfix Feature ### Detail * add list_environment tests * add test for updating an environment (via update_stack) * generalise the polling functions for stacks ### Relates #1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit e913d48 Author: dlpzx <[email protected]> Date: Fri Jun 28 2024 04:15:49 GMT-0400 (Eastern Daylight Time) Add search (Autocomplete) in miscellaneous dropdowns (#1367) ### Feature or Bugfix - Feature ### Detail Autocomplete for environments and teams in the following frontend views as requested in #1012. In this case the views required custom dropdowns. ❗ I used `noOptionsText` whenever it was necessary instead of checking groupOptions lenght >0 - [x] DatasetEditForm.js -> ❗ I kept the stewards field as `freesolo` - what that means is that users CAN specify options that are not on the list. I would like the reviewer to confirm this is what we want. At the end stewardship is a delegation of permissions, it makes sense that delegation happens to other teams. Also changed DatasetCreateForm - [X] RequestDashboardAccessModal.js - already implemented, minor changes - [X] EnvironmentTeamInviteForm.js - already implemented, minor changes. -> Kept `freesolo` because invited teams might not be the user teams. Same reason why there is no check for groupOptions == 0, if there are no options there is still the free text option. - [X] EnvironmentRoleAddForm.js - [X] NetworkCreateModal.js ### Relates - #1012 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit ee71d7b Author: Tejas Rajopadhye <[email protected]> Date: Thu Jun 27 2024 14:08:27 GMT-0400 (Eastern Daylight Time) [Gh 1301] Enhancement Feature - Bulk share reapply on dataset (#1363) ### Feature or Bugfix - Feature ### Detail - Adds feature to reapply shares in bulk for a dataset. - Also contains bugfix for AWS worker lambda errors ### Relates - #1301 - #1364 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? N/A - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? N/A - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? N/A - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? N/A - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: trajopadhye <[email protected]> commit 27f1ad7 Author: Noah Paige <[email protected]> Date: Thu Jun 27 2024 13:18:32 GMT-0400 (Eastern Daylight Time) Convert Dataset Lock Mechanism to Generic Resource Lock (#1338) ### Feature or Bugfix <!-- please choose --> - Feature - Bugfix - Refactoring ### Detail - Convert Dataset Lock Mechanism to Generic Resource Lock - Extend locking to Share principals (i.e. EnvironmentGroup and Consumption Roles) - Making locking a generic component not tied to datasets ### Relates - #1093 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: dlpzx <[email protected]> commit e3b8658 Author: Petros Kalos <[email protected]> Date: Thu Jun 27 2024 12:50:59 GMT-0400 (Eastern Daylight Time) ignore ruff change in blame (#1372) ### Feature or Bugfix <!-- please choose --> - Feature - Bugfix - Refactoring ### Detail - <feature1 or bug1> - <feature2 or bug2> ### Relates - <URL or Ticket> ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 2e80de4 Author: dlpzx <[email protected]> Date: Thu Jun 27 2024 10:59:18 GMT-0400 (Eastern Daylight Time) Generic shares_base module and specific s3_datasets_shares module - part 11 (renaming and cleaning up s3_shares) (#1359) ### Feature or Bugfix - Refactoring ### Detail As explained in the design for #1123 and #1283 we are trying to implement generic `datasets_base` and `shares_base` modules that can be used by any type of datasets and by any type of shareable object in a generic way. This is one of the last PRs focused on renaming files and cleaning-up the s3_datasets_shares module. The first step is a consolidation of the file and classes names in the services to clearly refer to s3_shares: - `services.managed_share_policy_service.SharePolicyService` ---> `services.s3_share_managed_policy_service.S3SharePolicyService` - `services.dataset_sharing_alarm_service.DatasetSharingAlarmService` --> `services.s3_share_alarm_service.S3ShareAlarmService` - `services.managed_share_policy_service.SharePolicyService` --> `services.s3_share_managed_policy_service.S3SharePolicyService` 👀 The main refactoring happens in what is used to be `services.dataset_sharing_service`. - The part that implements the `DatasetServiceInterface` has been moved to `services/s3_share_dataset_service.py` as the `S3ShareDatasetService` - The part used in the resolvers and by other methods has been renamed as `services.s3_share_service.py` and the methods for the folder/table permissions are also added to the S3ShareService (from share_item_service) Lastly, there is one method previously in share_item_service that has been moved to the GlueClient directly as `get_glue_database_from_catalog`. ### Relates - #1283 - #1123 - #955 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 1c09015 Author: Noah Paige <[email protected]> Date: Thu Jun 27 2024 04:16:14 GMT-0400 (Eastern Daylight Time) fix listOrganizationGroupPermissions (#1369) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Fix listOrganizationGroupPermissions ### Relates - <URL or Ticket> ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 976ec6b Author: dlpzx <[email protected]> Date: Thu Jun 27 2024 04:13:14 GMT-0400 (Eastern Daylight Time) Add search (Autocomplete) in create pipelines (#1368) ### Feature or Bugfix - Feature ### Detail Autocomplete for environments and teams in the following frontend views as requested in #1012. This PR implements it for createPipelines ### Relates - #1012 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 6c909a3 Author: Noah Paige <[email protected]> Date: Wed Jun 26 2024 11:18:04 GMT-0400 (Eastern Daylight Time) fix migration to not rely on OrganizationService or RequestContext (#1361) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Ensure migration script does not need RequestContext - otherwise fails in migration trigger lambda as context info not set / available ### Relates - #1306 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 90835fb Author: Anushka Singh <[email protected]> Date: Wed Jun 26 2024 11:17:22 GMT-0400 (Eastern Daylight Time) Issue1248: Persistent Email Reminders (#1354) ### Feature or Bugfix - Feature ### Detail - When a share request is initiated and remains pending for an extended period, dataset producers will receive automated email reminders at predefined intervals. These reminders will prompt producers to either approve or extend the share request, thereby preventing delays in accessing datasets. Attaching screenshots for emails: <img width="1336" alt="Screenshot 2024-06-20 at 5 34 31 PM" src="https://github.com/data-dot-all/dataall/assets/26413731/d7be28c3-5c98-4146-92b1-295e136137a3"> <img width="1322" alt="Screenshot 2024-06-20 at 5 34 52 PM" src="https://github.com/data-dot-all/dataall/assets/26413731/047556e8-59ee-4ebf-b8a7-c0a6684e2a63"> - Email will be sent every Monday at 9am UTC. Schedule can be changed in cron expression in container.py ### Relates - #1248 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Anushka Singh <[email protected]> Co-authored-by: trajopadhye <[email protected]> Co-authored-by: Mohit Arora <[email protected]> Co-authored-by: rbernota <[email protected]> Co-authored-by: Rick Bernotas <[email protected]> Co-authored-by: Raj Chopde <[email protected]> Co-authored-by: Noah Paige <[email protected]> Co-authored-by: dlpzx <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <[email protected]> Co-authored-by: dlpzx <[email protected]> Co-authored-by: mourya-33 <[email protected]> Co-authored-by: nikpodsh <[email protected]> Co-authored-by: MK <[email protected]> Co-authored-by: Manjula <[email protected]> Co-authored-by: Zilvinas Saltys <[email protected]> Co-authored-by: Zilvinas Saltys <[email protected]> Co-authored-by: Daniel Lorch <[email protected]> Co-authored-by: Tejas Rajopadhye <[email protected]> Co-authored-by: Zilvinas Saltys <[email protected]> Co-authored-by: Sofia Sazonova <[email protected]> Co-authored-by: Sofia Sazonova <[email protected]> commit e477bdf Author: Noah Paige <[email protected]> Date: Wed Jun 26 2024 10:39:09 GMT-0400 (Eastern Daylight Time) Enforce non null on GQL query string if non null defined (#1362) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Add `String!` to ensure non null input argument on FE if defined as such on backend GQL operation for `listS3DatasetsSharedWithEnvGroup` ### Relates ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit d6b59b3 Author: Noah Paige <[email protected]> Date: Wed Jun 26 2024 08:48:52 GMT-0400 (Eastern Daylight Time) Fix Init Share Base (#1360) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Need to register processors in init for s3 dataset shares API module ### Relates ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit bd3698c Author: Petros Kalos <[email protected]> Date: Wed Jun 26 2024 05:19:14 GMT-0400 (Eastern Daylight Time) split cognito urls setup and cognito user creation (#1366) ### Feature or Bugfix - Bugfix ### Details For more details about the issue read #1353 In this PR we are solving the problem by splitting the configuration of Cognito in 2. * First part (cognito_users_config.py) is setting up the required groups and users and runs after UserPool deployment * Second part (cognito_urls_config.py) is setting up Cognito's callback/logout urls and runs after the CloudFront deployment We chose to split the functionality because we need to have the users/groups setup for the integration tests which are run after the backend deployment. The other althernative is to keep the config functionality as one but make the integ tests run after CloudFront stage. ### Relates - Solves #1353 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
noah-paige
added a commit
that referenced
this issue
Aug 30, 2024
commit 4425e756 Author: Noah Paige <[email protected]> Date: Mon Jul 08 2024 11:57:31 GMT-0400 (Eastern Daylight Time) Fix commit 4cd2bf77 Author: Noah Paige <[email protected]> Date: Mon Jul 08 2024 11:56:38 GMT-0400 (Eastern Daylight Time) Fix commit 22a6f6ef Author: Noah Paige <[email protected]> Date: Mon Jul 08 2024 11:28:07 GMT-0400 (Eastern Daylight Time) Add integ tests commit 4fb7d653 Author: Noah Paige <[email protected]> Date: Mon Jul 08 2024 11:26:36 GMT-0400 (Eastern Daylight Time) Merge env test changes commit 4cf42e8 Author: Petros Kalos <[email protected]> Date: Fri Jul 05 2024 08:19:34 GMT-0400 (Eastern Daylight Time) improve docs commit 65f930a Author: Petros Kalos <[email protected]> Date: Fri Jul 05 2024 08:10:56 GMT-0400 (Eastern Daylight Time) fix failures commit 170b7ce Author: Petros Kalos <[email protected]> Date: Wed Jul 03 2024 10:52:20 GMT-0400 (Eastern Daylight Time) add group/consumption_role invite/remove tests commit ba77d69 Author: dlpzx <[email protected]> Date: Wed Jul 03 2024 06:51:47 GMT-0400 (Eastern Daylight Time) Rename alias for env_vars kms key in cognito lambdas FE and BE (#1385) ### Feature or Bugfix - Bugfix ### Detail For the case in which we deploy FE and BE in us-east-1 the new lambda env_key alias is the same one for TriggerFunctionCognitoUrlsConfig in FE and for TriggerFunctionCognitoConfig in BE, which results in a failure of the CICD in the FE stack because the alias already exists. This PR changes the name of both aliases to avoid this conflict. It also adds envname to avoid issues with other deployment environments/tooling account in the future ### Relates - <URL or Ticket> ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit e5923a9 Author: dlpzx <[email protected]> Date: Wed Jul 03 2024 04:27:11 GMT-0400 (Eastern Daylight Time) Fix lambda_env_key out of scope for vpc-facing cognito setup (#1384) ### Feature or Bugfix - Bugfix ### Detail The KMS key for the Lambda environment variables in the Cognito IdP stack was defined inside an if-clause for internet facing frontend. Outside of that if, for vpc-facing architecture the kms key does not exist and the CICD pipeline fails. This PRs move the creation of the KMS key outside of the if. ### Relates ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 3ccacfc Author: Noah Paige <[email protected]> Date: Mon Jul 01 2024 13:56:58 GMT-0400 (Eastern Daylight Time) Add delete docs not found when re indexing in catalog task (#1365) ### Feature or Bugfix <!-- please choose --> - Feature ### Detail - Add logic to Catalog Indexer Task to Delete Docs No Longer in RDS - TODO: Add Ability to Re-index Catalog Items via Dataall Admin UI ### Relates - #1078 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit e2817a1 Author: Noah Paige <[email protected]> Date: Mon Jul 01 2024 05:14:07 GMT-0400 (Eastern Daylight Time) Fix/glossary status (#1373) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Add back `status` to Glossary GQL Object for GQL Operations (getGlossary, listGlossaries) - Fix `listOrganizationGroupPermissions` enforce non null on FE ### Relates ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit c3c58bd Author: Petros Kalos <[email protected]> Date: Fri Jun 28 2024 06:55:42 GMT-0400 (Eastern Daylight Time) add environment tests (#1371) ### Feature or Bugfix Feature ### Detail * add list_environment tests * add test for updating an environment (via update_stack) * generalise the polling functions for stacks ### Relates #1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit e913d48 Author: dlpzx <[email protected]> Date: Fri Jun 28 2024 04:15:49 GMT-0400 (Eastern Daylight Time) Add search (Autocomplete) in miscellaneous dropdowns (#1367) ### Feature or Bugfix - Feature ### Detail Autocomplete for environments and teams in the following frontend views as requested in #1012. In this case the views required custom dropdowns. ❗ I used `noOptionsText` whenever it was necessary instead of checking groupOptions lenght >0 - [x] DatasetEditForm.js -> ❗ I kept the stewards field as `freesolo` - what that means is that users CAN specify options that are not on the list. I would like the reviewer to confirm this is what we want. At the end stewardship is a delegation of permissions, it makes sense that delegation happens to other teams. Also changed DatasetCreateForm - [X] RequestDashboardAccessModal.js - already implemented, minor changes - [X] EnvironmentTeamInviteForm.js - already implemented, minor changes. -> Kept `freesolo` because invited teams might not be the user teams. Same reason why there is no check for groupOptions == 0, if there are no options there is still the free text option. - [X] EnvironmentRoleAddForm.js - [X] NetworkCreateModal.js ### Relates - #1012 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit ee71d7b Author: Tejas Rajopadhye <[email protected]> Date: Thu Jun 27 2024 14:08:27 GMT-0400 (Eastern Daylight Time) [Gh 1301] Enhancement Feature - Bulk share reapply on dataset (#1363) ### Feature or Bugfix - Feature ### Detail - Adds feature to reapply shares in bulk for a dataset. - Also contains bugfix for AWS worker lambda errors ### Relates - #1301 - #1364 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? N/A - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? N/A - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? N/A - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? N/A - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: trajopadhye <[email protected]> commit 27f1ad7 Author: Noah Paige <[email protected]> Date: Thu Jun 27 2024 13:18:32 GMT-0400 (Eastern Daylight Time) Convert Dataset Lock Mechanism to Generic Resource Lock (#1338) ### Feature or Bugfix <!-- please choose --> - Feature - Bugfix - Refactoring ### Detail - Convert Dataset Lock Mechanism to Generic Resource Lock - Extend locking to Share principals (i.e. EnvironmentGroup and Consumption Roles) - Making locking a generic component not tied to datasets ### Relates - #1093 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: dlpzx <[email protected]> commit e3b8658 Author: Petros Kalos <[email protected]> Date: Thu Jun 27 2024 12:50:59 GMT-0400 (Eastern Daylight Time) ignore ruff change in blame (#1372) ### Feature or Bugfix <!-- please choose --> - Feature - Bugfix - Refactoring ### Detail - <feature1 or bug1> - <feature2 or bug2> ### Relates - <URL or Ticket> ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 2e80de4 Author: dlpzx <[email protected]> Date: Thu Jun 27 2024 10:59:18 GMT-0400 (Eastern Daylight Time) Generic shares_base module and specific s3_datasets_shares module - part 11 (renaming and cleaning up s3_shares) (#1359) ### Feature or Bugfix - Refactoring ### Detail As explained in the design for #1123 and #1283 we are trying to implement generic `datasets_base` and `shares_base` modules that can be used by any type of datasets and by any type of shareable object in a generic way. This is one of the last PRs focused on renaming files and cleaning-up the s3_datasets_shares module. The first step is a consolidation of the file and classes names in the services to clearly refer to s3_shares: - `services.managed_share_policy_service.SharePolicyService` ---> `services.s3_share_managed_policy_service.S3SharePolicyService` - `services.dataset_sharing_alarm_service.DatasetSharingAlarmService` --> `services.s3_share_alarm_service.S3ShareAlarmService` - `services.managed_share_policy_service.SharePolicyService` --> `services.s3_share_managed_policy_service.S3SharePolicyService` 👀 The main refactoring happens in what is used to be `services.dataset_sharing_service`. - The part that implements the `DatasetServiceInterface` has been moved to `services/s3_share_dataset_service.py` as the `S3ShareDatasetService` - The part used in the resolvers and by other methods has been renamed as `services.s3_share_service.py` and the methods for the folder/table permissions are also added to the S3ShareService (from share_item_service) Lastly, there is one method previously in share_item_service that has been moved to the GlueClient directly as `get_glue_database_from_catalog`. ### Relates - #1283 - #1123 - #955 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 1c09015 Author: Noah Paige <[email protected]> Date: Thu Jun 27 2024 04:16:14 GMT-0400 (Eastern Daylight Time) fix listOrganizationGroupPermissions (#1369) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Fix listOrganizationGroupPermissions ### Relates - <URL or Ticket> ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 976ec6b Author: dlpzx <[email protected]> Date: Thu Jun 27 2024 04:13:14 GMT-0400 (Eastern Daylight Time) Add search (Autocomplete) in create pipelines (#1368) ### Feature or Bugfix - Feature ### Detail Autocomplete for environments and teams in the following frontend views as requested in #1012. This PR implements it for createPipelines ### Relates - #1012 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 6c909a3 Author: Noah Paige <[email protected]> Date: Wed Jun 26 2024 11:18:04 GMT-0400 (Eastern Daylight Time) fix migration to not rely on OrganizationService or RequestContext (#1361) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Ensure migration script does not need RequestContext - otherwise fails in migration trigger lambda as context info not set / available ### Relates - #1306 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit 90835fb Author: Anushka Singh <[email protected]> Date: Wed Jun 26 2024 11:17:22 GMT-0400 (Eastern Daylight Time) Issue1248: Persistent Email Reminders (#1354) ### Feature or Bugfix - Feature ### Detail - When a share request is initiated and remains pending for an extended period, dataset producers will receive automated email reminders at predefined intervals. These reminders will prompt producers to either approve or extend the share request, thereby preventing delays in accessing datasets. Attaching screenshots for emails: <img width="1336" alt="Screenshot 2024-06-20 at 5 34 31 PM" src="https://github.com/data-dot-all/dataall/assets/26413731/d7be28c3-5c98-4146-92b1-295e136137a3"> <img width="1322" alt="Screenshot 2024-06-20 at 5 34 52 PM" src="https://github.com/data-dot-all/dataall/assets/26413731/047556e8-59ee-4ebf-b8a7-c0a6684e2a63"> - Email will be sent every Monday at 9am UTC. Schedule can be changed in cron expression in container.py ### Relates - #1248 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Anushka Singh <[email protected]> Co-authored-by: trajopadhye <[email protected]> Co-authored-by: Mohit Arora <[email protected]> Co-authored-by: rbernota <[email protected]> Co-authored-by: Rick Bernotas <[email protected]> Co-authored-by: Raj Chopde <[email protected]> Co-authored-by: Noah Paige <[email protected]> Co-authored-by: dlpzx <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <[email protected]> Co-authored-by: dlpzx <[email protected]> Co-authored-by: mourya-33 <[email protected]> Co-authored-by: nikpodsh <[email protected]> Co-authored-by: MK <[email protected]> Co-authored-by: Manjula <[email protected]> Co-authored-by: Zilvinas Saltys <[email protected]> Co-authored-by: Zilvinas Saltys <[email protected]> Co-authored-by: Daniel Lorch <[email protected]> Co-authored-by: Tejas Rajopadhye <[email protected]> Co-authored-by: Zilvinas Saltys <[email protected]> Co-authored-by: Sofia Sazonova <[email protected]> Co-authored-by: Sofia Sazonova <[email protected]> commit e477bdf Author: Noah Paige <[email protected]> Date: Wed Jun 26 2024 10:39:09 GMT-0400 (Eastern Daylight Time) Enforce non null on GQL query string if non null defined (#1362) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Add `String!` to ensure non null input argument on FE if defined as such on backend GQL operation for `listS3DatasetsSharedWithEnvGroup` ### Relates ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit d6b59b3 Author: Noah Paige <[email protected]> Date: Wed Jun 26 2024 08:48:52 GMT-0400 (Eastern Daylight Time) Fix Init Share Base (#1360) ### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Need to register processors in init for s3 dataset shares API module ### Relates ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. commit bd3698c Author: Petros Kalos <[email protected]> Date: Wed Jun 26 2024 05:19:14 GMT-0400 (Eastern Daylight Time) split cognito urls setup and cognito user creation (#1366) ### Feature or Bugfix - Bugfix ### Details For more details about the issue read #1353 In this PR we are solving the problem by splitting the configuration of Cognito in 2. * First part (cognito_users_config.py) is setting up the required groups and users and runs after UserPool deployment * Second part (cognito_urls_config.py) is setting up Cognito's callback/logout urls and runs after the CloudFront deployment We chose to split the functionality because we need to have the users/groups setup for the integration tests which are run after the backend deployment. The other althernative is to keep the config functionality as one but make the integ tests run after CloudFront stage. ### Relates - Solves #1353 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
petrkalos
pushed a commit
that referenced
this issue
Sep 11, 2024
### Feature or Bugfix - Feature ### Detail We are adding more and more integration tests to be executed in a CodeBuild stage of the CICD pipeline. There are cases in which the test execution might take longer than the CodeBuild default time of 1h. This PR increases the timeout period to 36hours ### Relates - #1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
This was referenced Sep 11, 2024
petrkalos
added a commit
that referenced
this issue
Sep 13, 2024
### Feature or Bugfix Feature ### Detail Adding integration tests for ML Studio PENDING TESTS PASSING IN DEV AWS ENV ### Relates related to #1220 and resolves #1534 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
noah-paige
added a commit
that referenced
this issue
Sep 16, 2024
### Feature or Bugfix <!-- please choose --> - Feature ### Detail - Adding integration tests for Dataset Table Data Filters - PENDING TESTS PASSING IN DEV AWS ENV - Merge after #1391 ### Relates - related to #1220 and #1358 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: dlpzx <[email protected]>
This was referenced Sep 17, 2024
dlpzx
added a commit
that referenced
this issue
Sep 18, 2024
### Feature or Bugfix - Feature: testing ### Detail Implement tests for Permissions api calls (inside core/permissions) as part of #1220 !Excludes updateSSMParameter mutation - I think it is unused ### Relates - #1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
dlpzx
added a commit
that referenced
this issue
Sep 19, 2024
### Feature or Bugfix Implement tests for Stacks and KeyValueTags api calls (inside core/stacks) as part of #1220 ### Detail ### Relates - #1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
dlpzx
added a commit
that referenced
this issue
Sep 20, 2024
### Feature or Bugfix - Feature: testing - Bugfix ### Detail Implement tests for Networksapi calls (inside core/vpc + listEnvironmentNetworks) as part of #1220 ++ It also fixes a small bug on networks - tags were not correctly saved in the database! ### Relates - #1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
dlpzx
added a commit
that referenced
this issue
Sep 20, 2024
### Feature or Bugfix - Feature:Testing ### Detail Implement tests for Glossaries/Catalog as part of #1220⚠️ To test glossary associations, we need resources that use the glossary terms (associate a term to a resource). Possible resources include: s3_datasets, s3_tables, s3_folders, dashboards, redshift_datasets and redshift_tables. All of which are part of other modules that might be enabled or disabled. In this PR we assume that the s3_datasets module is enabled! If it was not enabled, then the glossary tests would fail! During the implementation several enhancement ideas came up and are collected in #1557 Tested in local with connection to real AWS deployment:  ### Relates - #1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
This was referenced Sep 25, 2024
noah-paige
added a commit
that referenced
this issue
Oct 1, 2024
### Feature or Bugfix <!-- please choose --> - Feature ### Detail - Add integration tests for `feed` module ### Relates - #1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
noah-paige
added a commit
that referenced
this issue
Oct 1, 2024
### Feature or Bugfix <!-- please choose --> - Feature ### Detail - Add integration tests for `vote` module - Fix Vote creation by username ### Relates - #1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
This was referenced Oct 2, 2024
Closed
SofiaSazonova
pushed a commit
that referenced
this issue
Oct 8, 2024
### Feature or Bugfix - Feature ### Detail Environment and Dataset stacks leave S3 Buckets to be deleted even when the CloudFormation stack is deleted. This PR deletes the S3 Buckets when the session and temp fixtures for datasets and environments are deleted. ### Testing Tested that environment and dataset buckets are all deleted - in real AWS CICD pipeline with NOT-empty buckets. Tested that dataset buckets with some manually created access points succeed. ### Relates - #1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
SofiaSazonova
pushed a commit
that referenced
this issue
Oct 8, 2024
### Feature or Bugfix - Feature ### Detail The new integration tests take quite some time (>1h) to get executed. For this reason the AWS clients passed as fixtures to perform different actions (e.g. create S3 Buckets) have expired STS tokens when we reach the teardown operations (e.g. delete S3 Buckets). This PR implements some logic similar to the one proposed in https://github.com/benkehoe/aws-assume-role-lib which was referenced by the boto3 maintainers in the [GitHub issue](boto/boto3#443) for this topic. ### Relates - #1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
This was referenced Oct 10, 2024
Merged
dlpzx
added a commit
that referenced
this issue
Oct 15, 2024
… Redshift Connections (#1628)⚠️ NEEDS SOME CHANGES AFTER #1638 is merged ### Feature or Bugfix - Feature: Testing ### Detail Add integration tests for Redshift connections It also includes the global conftest fixtures that will be used for redshift-datasets and redshift dataset sharing At the moment it assumes Redshift infrastructure is already provided; we might want to implement the deployment of this pre-required infra, but that can be done on a separate PR. +++ Additional fixes found during testing and fixed in this PR - Fixed Pivot role permissions in CDK when no workgroups are defined (it handled None wrong) - Add update environment stack on Connection delete to ensure permissions to pivot role are removed ### Testing - Fixes deployed in CICD pipeline in AWS - Tests executed locally (in this case we can be sure it will work as in AWS as there are no AWS Clients, just API calls)  ### Relates - #1619 - #1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
dlpzx
added a commit
that referenced
this issue
Oct 16, 2024
… Redshift Datasets (#1636)⚠️ MERGE AFTER #1628 ### Feature or Bugfix - Feature: Testing ### Detail Add integration tests for Redshift datasets It also includes the global conftest fixtures that will be used for redshift dataset sharing At the moment it assumes Redshift infrastructure is already provided; we might want to implement the deployment of this pre-required infra, but that can be done on a separate PR. ## Testing Tested locally:  ### Relates - #1619 - #1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
dlpzx
added a commit
that referenced
this issue
Oct 22, 2024
… Redshift Shares (#1643)⚠️ MERGE AFTER #1636 ### Feature or Bugfix - Feature: Testing ### Detail Add integration tests for Redshift shares. Implements #1620 - Implemented inside the shares modules in a subdirectory so that each share type can have its own conftest but still re-use common methods from shares queries - This PR is focused on testing the Redshift shares functionality, it does not include all tests that test the workflow of the share (e.g. submit, reject...) - It does not validate if after a share the user has access to data. We could implement it using the Redshift Data API, but I left it as optional for a separate PR ### Tested Locally:  ### Relates - #1620 - #1619 - #1220 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
I would like data.all to be tested "in reality" once it is deployed. That allows us to verify code changes on real infrastructure. It speeds up development because it increases the confidence that developers and maintainers have that a feature does not introduce bugs.
Describe the solution you'd like
A way to execute locally and in a CICD pipeline tests that run against a real API of data.all.
Describe alternatives you've considered
Defining the tests using pytest. Once #950 is complete we will be able to use the SDK directly to configure the test clients and the API call definitions.
Additional context
Add any other context or screenshots about the feature request here.
P.S. Please Don't attach files. Add code snippets directly in the message body instead.
The text was updated successfully, but these errors were encountered: