Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Delay between same-domain requests based on time when download was finished #179

Open
aecio opened this issue Apr 18, 2019 · 0 comments · May be fixed by #180
Open

Delay between same-domain requests based on time when download was finished #179

aecio opened this issue Apr 18, 2019 · 0 comments · May be fixed by #180
Labels
enhancement

Comments

@aecio
Copy link
Member

aecio commented Apr 18, 2019

Currently, the delay between requests to the same domain (host) considers only the time when the URLs were scheduled to be downloaded, not the time when the download was finished. This can cause problems when downloading from slow servers or when large files that take minutes to download. ACHE should record the time when the last download from a host was finished, and compute the next visit based on this time.
aecio added a commit that referenced this issue Apr 22, 2019
@aecio
- Modified PolitenessScheduler to compute the delay between
31894a5 
  same-domain requests based on the time when the download
  finished
- Refactoring of FethcedResultHandler to simply notify the
  LinkStorage that the download finished and to delegate data
  processing to other handlers for the appropriate link type
- Fixed unit and integration tests to consider the new scheduler
  workflow that requires notification when the download was
  finished

Fixes #179.
aecio added a commit that referenced this issue Apr 22, 2019
@aecio
Minimum delay between based on download finished time
5a0f441 
- Modified PolitenessScheduler to compute the delay between
same-domain requests based on the time when the download
finished
- Refactoring of FethcedResultHandler to simply notify the
LinkStorage that the download finished and to delegate data
processing to other handlers for the appropriate link type
- Fixed unit and integration tests to consider the new scheduler
workflow that requires notification when the download was
finished

Fixes #179.
@aecio aecio linked a pull request Apr 22, 2019 that will close this issue
Open
aecio added a commit that referenced this issue Jan 18, 2020
@aecio
Minimum delay between based on download finished time
18c18d2 
- Modified PolitenessScheduler to compute the delay between
same-domain requests based on the time when the download
finished
- Refactoring of FethcedResultHandler to simply notify the
LinkStorage that the download finished and to delegate data
processing to other handlers for the appropriate link type
- Fixed unit and integration tests to consider the new scheduler
workflow that requires notification when the download was
finished

Fixes #179.
aecio added a commit that referenced this issue Dec 1, 2020
@aecio
Minimum delay between based on download finished time
0eaeef0 
- Modified PolitenessScheduler to compute the delay between
same-domain requests based on the time when the download
finished
- Refactoring of FethcedResultHandler to simply notify the
LinkStorage that the download finished and to delegate data
processing to other handlers for the appropriate link type
- Fixed unit and integration tests to consider the new scheduler
workflow that requires notification when the download was
finished

Fixes #179.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Minimum delay based on download finish time VIDA-NYU/ache
1 participant
@aecio