Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

clearlydefined requests need a rate limit or backoff #218

Open
kpreid opened this issue Feb 14, 2023 · 0 comments
Open

clearlydefined requests need a rate limit or backoff #218

kpreid opened this issue Feb 14, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@kpreid
Copy link
Contributor

kpreid commented Feb 14, 2023

Today while running my build which includes a cargo about generate I got a flood of warnings like:

2023-02-14 16:24:17.131075 +00:00:00 [WARN] failed to request license information from clearly defined: HTTP status: 429 Too Many Requests
2023-02-14 16:24:17.134912 +00:00:00 [WARN] failed to request license information from clearly defined: HTTP status: 429 Too Many Requests
2023-02-14 16:24:17.138537 +00:00:00 [WARN] failed to request license information from clearly defined: HTTP status: 429 Too Many Requests
2023-02-14 16:24:17.138705 +00:00:00 [WARN] failed to request license information from clearly defined: HTTP status: 429 Too Many Requests
2023-02-14 16:24:17.147567 +00:00:00 [WARN] failed to request license information from clearly defined: HTTP status: 429 Too Many Requests
2023-02-14 16:24:17.153689 +00:00:00 [WARN] failed to request license information from clearly defined: HTTP status: 429 Too Many Requests
2023-02-14 16:24:17.154867 +00:00:00 [WARN] failed to request license information from clearly defined: HTTP status: 429 Too Many Requests
2023-02-14 16:24:17.155505 +00:00:00 [WARN] failed to request license information from clearly defined: HTTP status: 429 Too Many Requests
2023-02-14 16:24:17.156357 +00:00:00 [WARN] failed to request license information from clearly defined: HTTP status: 429 Too Many Requests
2023-02-14 16:24:17.159223 +00:00:00 [WARN] failed to request license information from clearly defined: HTTP status: 429 Too Many Requests
2023-02-14 16:24:17.159581 +00:00:00 [WARN] failed to request license information from clearly defined: HTTP status: 429 Too Many Requests
2023-02-14 16:24:17.160625 +00:00:00 [WARN] failed to request license information from clearly defined: HTTP status: 429 Too Many Requests
...

It seems that cargo-about is making requests too rapidly and needs to implement a rate limit or backoff/retry policy.

It would also be nice if cargo-about documented that it was using a network service by default — I wasn't aware until now that it was.
@kpreid kpreid added the bug Something isn't working label Feb 14, 2023
kpreid added a commit to kpreid/all-is-cubes that referenced this issue Mar 9, 2023
@kpreid
Disable cargo about talking to clearlydefined.io.
cc2f5ec 
This is a workaround for a lack of rate limiting causing failures
(<EmbarkStudios/cargo-about#218>), but also,
it does not seem to be necessary for my use case (no unknown licenses
are reported), so we might as well *not* talk to an external service.

Also add `--fail` option to fail the build if there ever is a problem
license.
apljungquist added a commit to AxisCommunications/acap-rs that referenced this issue Oct 15, 2024
@apljungquist
chore: Don't use clearly defined for license info
de6570b 
I recently struggled with merging a change because the checksums seemed
to keep changing. One potential explanation is that the LICENSE file
generation is not reproducible; when I test locally only 2/10 runs
produce identical license files for all apps. After this change 10/10
runs produce identical license files for all apps.

The problems may coincide with warnings about 429 and 502 responses
from clearly defined, but I have not confirmed if there is actually a
correlation.

There are a couple of issues vaguely related to the problems I see, but
it does not seem like a priority:
- EmbarkStudios/cargo-about#218
- EmbarkStudios/cargo-about#246

I compared the generated license for the `hello_world` app and the
differences do not seem catastrophic:
- Some licences have been reordered. These all seem to have had an
  instantiation of the license template replaced with the template
  itself.
- Notices about `aho-corasick` and `memchr` being dual licensed have
  been removed; The MIT license of both remain.
- A comment about `winapi` being dual licensed have been removed.

`Makefile`:
- Set `--fail` for good measure; It didn't cause any failures for me
  either with or without `no-clearly-defined` meaning I could not
  observe any improvements. But in theory it should be easier to detect
  that it was unnecessarily set than the other way around.

`about.toml`:
- Set `no-clearly-defined` because this stops `cargo-about` from
  consulting clearly defined, which seems to make the builds
  reproducible. Other potential benefits from this include lower risk
  of supply chain attacks and faster builds.
@apljungquist apljungquist mentioned this issue Oct 15, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kpreid