Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error "Attempted to use detached Frame" #446

Open
vitaly-zdanevich opened this issue Dec 20, 2024 · 5 comments
Open

Error "Attempted to use detached Frame" #446

vitaly-zdanevich opened this issue Dec 20, 2024 · 5 comments

Comments

@vitaly-zdanevich
Copy link

vitaly-zdanevich commented Dec 20, 2024 •
edited
Loading

My command:

docker run -v .:/output --name urbandictionary-com-2024-december --rm ghcr.io/openzim/zimit:latest zimit --url "https://urbandictionary.com" --name urbandictionary-com-2024-december --workers 350 --diskUtilization 0

Yes 350 workers - I have a powerful PC with 64 GB of DDR5 RAM, top cpu AMD 7950x

Part of logs:

{"timestamp":"2024-12-20T09:04:24.213Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://www.urbandictionary.com/define.php?term=upsidedown","workerid":239}}
{"timestamp":"2024-12-20T09:04:24.213Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://www.urbandictionary.com/author.php?author=gekyumey","workerid":281}}
{"timestamp":"2024-12-20T09:04:24.213Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://www.urbandictionary.com/author.php?author=lessashamed","workerid":29
6}}
{"timestamp":"2024-12-20T09:04:24.213Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://www.urbandictionary.com/author.php?author=Wollah","workerid":270}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"warn","context":"behavior","message":"Waiting for custom page load failed","details":{"type":"exception","message":"Attempted to use detached Frame '0EB3DF7DA8931EB
F5C092A872BF4C16C'.","stack":"Error: Attempted to use detached Frame '0EB3DF7DA8931EBF5C092A872BF4C16C'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:2
3)\n    at Crawler.awaitPageLoad (file:///app/dist/crawler.js:1314:25)\n    at Crawler.loadPage (file:///app/dist/crawler.js:1285:20)\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    at listOnTimeou
t (node:internal/timers:545:9)\n    at process.processTimers (node:internal/timers:519:7)\n    at async Crawler.default [as driver] (file:///app/dist/defaultDriver.js:2:5)\n    at async Crawler.crawlPage (file:///app
/dist/crawler.js:586:9)\n    at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n    at async PageWorker.timedCrawlPage (file:///app/dist/util/worker.js:173:13)"}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"warn","context":"links","message":"Link Extraction failed","details":{"type":"exception","message":"Attempted to use detached Frame '0EB3DF7DA8931EBF5C092A872BF4C16
C'.","stack":"Error: Attempted to use detached Frame '0EB3DF7DA8931EBF5C092A872BF4C16C'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n    at file:
///app/dist/crawler.js:1350:26\n    at Array.map (<anonymous>)\n    at Crawler.extractLinks (file:///app/dist/crawler.js:1348:49)\n    at Crawler.loadPage (file:///app/dist/crawler.js:1292:20)\n    at runNextTicks (n
ode:internal/process/task_queues:60:5)\n    at listOnTimeout (node:internal/timers:545:9)\n    at process.processTimers (node:internal/timers:519:7)\n    at async Crawler.default [as driver] (file:///app/dist/default
Driver.js:2:5)\n    at async Crawler.crawlPage (file:///app/dist/crawler.js:586:9)"}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"error","context":"worker","message":"Worker Exception","details":{"type":"exception","message":"Attempted to use detached Frame '0EB3DF7DA8931EBF5C092A872BF4C16C'."
,"stack":"Error: Attempted to use detached Frame '0EB3DF7DA8931EBF5C092A872BF4C16C'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n    at CdpPage.t
itle (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/api/Page.js:1084:43)\n    at Crawler.crawlPage (file:///app/dist/crawler.js:587:42)\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    a
t listOnTimeout (node:internal/timers:545:9)\n    at process.processTimers (node:internal/timers:519:7)\n    at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n    at async PageWorker.timedCrawlP
age (file:///app/dist/util/worker.js:173:13)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)","page":"https://www.urband
ictionary.com/define.php?term=mr%20brown&page=4","workerid":47}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":2,"page":"https://www.urbandictionary.com/define.php?term=mr%20brown&page=4","workerid
":47}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"warn","context":"behavior","message":"Waiting for custom page load failed","details":{"type":"exception","message":"Attempted to use detached Frame '1ECB406F811DCC2
C1FE39432ADCE450A'.","stack":"Error: Attempted to use detached Frame '1ECB406F811DCC2C1FE39432ADCE450A'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:2
3)\n    at Crawler.awaitPageLoad (file:///app/dist/crawler.js:1314:25)\n    at Crawler.loadPage (file:///app/dist/crawler.js:1285:20)\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    at listOnTimeou
t (node:internal/timers:545:9)\n    at process.processTimers (node:internal/timers:519:7)\n    at async Crawler.default [as driver] (file:///app/dist/defaultDriver.js:2:5)\n    at async Crawler.crawlPage (file:///app
/dist/crawler.js:586:9)\n    at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n    at async PageWorker.timedCrawlPage (file:///app/dist/util/worker.js:173:13)"}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"warn","context":"links","message":"Link Extraction failed","details":{"type":"exception","message":"Attempted to use detached Frame '1ECB406F811DCC2C1FE39432ADCE450
A'.","stack":"Error: Attempted to use detached Frame '1ECB406F811DCC2C1FE39432ADCE450A'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n    at file:
///app/dist/crawler.js:1350:26\n    at Array.map (<anonymous>)\n    at Crawler.extractLinks (file:///app/dist/crawler.js:1348:49)\n    at Crawler.loadPage (file:///app/dist/crawler.js:1292:20)\n    at runNextTicks (n
ode:internal/process/task_queues:60:5)\n    at listOnTimeout (node:internal/timers:545:9)\n    at process.processTimers (node:internal/timers:519:7)\n    at async Crawler.default [as driver] (file:///app/dist/default
Driver.js:2:5)\n    at async Crawler.crawlPage (file:///app/dist/crawler.js:586:9)"}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"error","context":"worker","message":"Worker Exception","details":{"type":"exception","message":"Attempted to use detached Frame '1ECB406F811DCC2C1FE39432ADCE450A'."
,"stack":"Error: Attempted to use detached Frame '1ECB406F811DCC2C1FE39432ADCE450A'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n    at CdpPage.t
itle (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/api/Page.js:1084:43)\n    at Crawler.crawlPage (file:///app/dist/crawler.js:587:42)\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    a
t listOnTimeout (node:internal/timers:545:9)\n    at process.processTimers (node:internal/timers:519:7)\n    at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n    at async PageWorker.timedCrawlP
age (file:///app/dist/util/worker.js:173:13)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)","page":"https://www.urband
ictionary.com/define.php?term=mr%20brown&page=2","workerid":155}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":2,"page":"https://www.urbandictionary.com/define.php?term=mr%20brown&page=2","workerid
":155}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"warn","context":"behavior","message":"Waiting for custom page load failed","details":{"type":"exception","message":"Attempted to use detached Frame '53B2C9B5AC0A846
8F4CF19756A7F3A97'.","stack":"Error: Attempted to use detached Frame '53B2C9B5AC0A8468F4CF19756A7F3A97'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:2
3)\n    at Crawler.awaitPageLoad (file:///app/dist/crawler.js:1314:25)\n    at Crawler.loadPage (file:///app/dist/crawler.js:1285:20)\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    at listOnTimeou
t (node:internal/timers:545:9)\n    at process.processTimers (node:internal/timers:519:7)\n    at async Crawler.default [as driver] (file:///app/dist/defaultDriver.js:2:5)\n    at async Crawler.crawlPage (file:///app
/dist/crawler.js:586:9)\n    at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n    at async PageWorker.timedCrawlPage (file:///app/dist/util/worker.js:173:13)"}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"warn","context":"links","message":"Link Extraction failed","details":{"type":"exception","message":"Attempted to use detached Frame '53B2C9B5AC0A8468F4CF19756A7F3A9
7'.","stack":"Error: Attempted to use detached Frame '53B2C9B5AC0A8468F4CF19756A7F3A97'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n    at file:
///app/dist/crawler.js:1350:26\n    at Array.map (<anonymous>)\n    at Crawler.extractLinks (file:///app/dist/crawler.js:1348:49)\n    at Crawler.loadPage (file:///app/dist/crawler.js:1292:20)\n    at runNextTicks (n
ode:internal/process/task_queues:60:5)\n    at listOnTimeout (node:internal/timers:545:9)\n    at process.processTimers (node:internal/timers:519:7)\n    at async Crawler.default [as driver] (file:///app/dist/default
Driver.js:2:5)\n    at async Crawler.crawlPage (file:///app/dist/crawler.js:586:9)"}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"error","context":"worker","message":"Worker Exception","details":{"type":"exception","message":"Attempted to use detached Frame '53B2C9B5AC0A8468F4CF19756A7F3A97'."
,"stack":"Error: Attempted to use detached Frame '53B2C9B5AC0A8468F4CF19756A7F3A97'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n    at CdpPage.t
itle (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/api/Page.js:1084:43)\n    at Crawler.crawlPage (file:///app/dist/crawler.js:587:42)\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    a
t listOnTimeout (node:internal/timers:545:9)\n    at process.processTimers (node:internal/timers:519:7)\n    at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n    at async PageWorker.timedCrawlP
age (file:///app/dist/util/worker.js:173:13)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)","page":"https://www.urband
ictionary.com/define.php?term=songwriter%27s","workerid":226}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":2,"page":"https://www.urbandictionary.com/define.php?term=songwriter%27s","workerid":2
26}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"warn","context":"behavior","message":"Waiting for custom page load failed","details":{"type":"exception","message":"Attempted to use detached Frame '0CD38CC9AF9210E
5A31258F6F1956C98'.","stack":"Error: Attempted to use detached Frame '0CD38CC9AF9210E5A31258F6F1956C98'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:2
3)\n    at Crawler.awaitPageLoad (file:///app/dist/crawler.js:1314:25)\n    at Crawler.loadPage (file:///app/dist/crawler.js:1285:20)\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    at listOnTimeou
t (node:internal/timers:545:9)\n    at process.processTimers (node:internal/timers:519:7)\n    at async Crawler.default [as driver] (file:///app/dist/defaultDriver.js:2:5)\n    at async Crawler.crawlPage (file:///app
/dist/crawler.js:586:9)\n    at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n    at async PageWorker.timedCrawlPage (file:///app/dist/util/worker.js:173:13)"}}
{"timestamp":"2024-12-20T09:04:24.216Z","logLevel":"warn","context":"links","message":"Link Extraction failed","details":{"type":"exception","message":"Attempted to use detached Frame '0CD38CC9AF9210E5A31258F6F1956C9
8'.","stack":"Error: Attempted to use detached Frame '0CD38CC9AF9210E5A31258F6F1956C98'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n    at file:
///app/dist/crawler.js:1350:26\n    at Array.map (<anonymous>)\n    at Crawler.extractLinks (file:///app/dist/crawler.js:1348:49)\n    at Crawler.loadPage (file:///app/dist/crawler.js:1292:20)\n    at runNextTicks (n
ode:internal/process/task_queues:60:5)\n    at listOnTimeout (node:internal/timers:545:9)\n    at process.processTimers (node:internal/timers:519:7)\n    at async Crawler.default [as driver] (file:///app/dist/default
Driver.js:2:5)\n    at async Crawler.crawlPage (file:///app/dist/crawler.js:586:9)"}}
{"timestamp":"2024-12-20T09:04:24.216Z","logLevel":"error","context":"worker","message":"Worker Exception","details":{"type":"exception","message":"Attempted to use detached Frame '0CD38CC9AF9210E5A31258F6F1956C98'."
,"stack":"Error: Attempted to use detached Frame '0CD38CC9AF9210E5A31258F6F1956C98'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n    at CdpPage.t
itle (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/api/Page.js:1084:43)\n    at Crawler.crawlPage (file:///app/dist/crawler.js:587:42)\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    a
t listOnTimeout (node:internal/timers:545:9)\n    at process.processTimers (node:internal/timers:519:7)\n    at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n    at async PageWorker.timedCrawlP
age (file:///app/dist/util/worker.js:173:13)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)","page":"https://www.urband
ictionary.com/define.php?term=mr%20brown&page=3","workerid":182}}
{"timestamp":"2024-12-20T09:04:24.216Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":2,"page":"https://www.urbandictionary.com/define.php?term=mr%20brown&page=3","workerid
":182}}
{"timestamp":"2024-12-20T09:04:24.216Z","logLevel":"warn","context":"behavior","message":"Waiting for custom page load failed","details":{"type":"exception","message":"Attempted to use detached Frame '79D619504161484
ED626C47808D05CED'.","stack":"Error: Attempted to use detached Frame '79D619504161484ED626C47808D05CED'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:2
3)\n    at Crawler.awaitPageLoad (file:///app/dist/crawler.js:1314:25)\n    at Crawler.loadPage (file:///app/dist/crawler.js:1285:20)\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    at listOnTimeou
t (node:internal/timers:545:9)\n    at process.processTimers (node:internal/timers:519:7)\n    at async Crawler.default [as driver] (file:///app/dist/defaultDriver.js:2:5)\n    at async Crawler.crawlPage (file:///app
/dist/crawler.js:586:9)\n    at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n    at async PageWorker.timedCrawlPage (file:///app/dist/util/worker.js:173:13)"}}
{"timestamp":"2024-12-20T09:04:24.216Z","logLevel":"warn","context":"links","message":"Link Extraction failed","details":{"type":"exception","message":"Attempted to use detached Frame '79D619504161484ED626C47808D05CE
D'.","stack":"Error: Attempted to use detached Frame '79D619504161484ED626C47808D05CED'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n    at file:
///app/dist/crawler.js:1350:26\n    at Array.map (<anonymous>)\n    at Crawler.extractLinks (file:///app/dist/crawler.js:1348:49)\n    at Crawler.loadPage (file:///app/dist/crawler.js:1292:20)\n    at runNextTicks (n
ode:internal/process/task_queues:60:5)\n    at listOnTimeout (node:internal/timers:545:9)\n    at process.processTimers (node:internal/timers:519:7)\n    at async Crawler.default [as driver] (file:///app/dist/default
Driver.js:2:5)\n    at async Crawler.crawlPage (file:///app/dist/crawler.js:586:9)"}}
{"timestamp":"2024-12-20T09:04:24.216Z","logLevel":"error","context":"worker","message":"Worker Exception","details":{"type":"exception","message":"Attempted to use detached Frame '79D619504161484ED626C47808D05CED'."
,"stack":"Error: Attempted to use detached Frame '79D619504161484ED626C47808D05CED'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n    at CdpPage.t
itle (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/api/Page.js:1084:43)\n    at Crawler.crawlPage (file:///app/dist/crawler.js:587:42)\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    a
t listOnTimeout (node:internal/timers:545:9)\n    at process.processTimers (node:internal/timers:519:7)\n    at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n    at async PageWorker.timedCrawlP
age (file:///app/dist/util/worker.js:173:13)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)","page":"https://www.urband
ictionary.com/define.php?term=Mrs%20Brown","workerid":137}}
{"timestamp":"2024-12-20T09:04:24.217Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":2,"page":"https://www.urbandictionary.com/define.php?term=Mrs%20Brown","workerid":137}
}
{"timestamp":"2024-12-20T09:04:24.217Z","logLevel":"warn","context":"behavior","message":"Waiting for custom page load failed","details":{"type":"exception","message":"Attempted to use detached Frame '7A94E3CBE62EA1D
AEF3711B8D454102C'.","stack":"Error: Attempted to use detached Frame '7A94E3CBE62EA1DAEF3711B8D454102C'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:2
3)\n    at Crawler.awaitPageLoad (file:///app/dist/crawler.js:1314:25)\n    at Crawler.loadPage (file:///app/dist/crawler.js:1285:20)\n    at async Crawler.default [as driver] (file:///app/dist/defaultDriver.js:2:5)\
n    at async Crawler.crawlPage (file:///app/dist/crawler.js:586:9)\n    at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n    at async PageWorker.timedCrawlPage (file:///app/dist/util/worker.js
:173:13)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)\n    at async Promise.allSettled (index 88)"}}
{"timestamp":"2024-12-20T09:04:24.217Z","logLevel":"warn","context":"links","message":"Link Extraction failed","details":{"type":"exception","message":"Attempted to use detached Frame '7A94E3CBE62EA1DAEF3711B8D454102
C'.","stack":"Error: Attempted to use detached Frame '7A94E3CBE62EA1DAEF3711B8D454102C'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n    at file:
///app/dist/crawler.js:1350:26\n    at Array.map (<anonymous>)\n    at Crawler.extractLinks (file:///app/dist/crawler.js:1348:49)\n    at Crawler.loadPage (file:///app/dist/crawler.js:1292:20)\n    at async Crawler.d
efault [as driver] (file:///app/dist/defaultDriver.js:2:5)\n    at async Crawler.crawlPage (file:///app/dist/crawler.js:586:9)\n    at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n    at async
 PageWorker.timedCrawlPage (file:///app/dist/util/worker.js:173:13)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)"}}
{"timestamp":"2024-12-20T09:04:24.217Z","logLevel":"error","context":"worker","message":"Worker Exception","details":{"type":"exception","message":"Attempted to use detached Frame '7A94E3CBE62EA1DAEF3711B8D454102C'."
,"stack":"Error: Attempted to use detached Frame '7A94E3CBE62EA1DAEF3711B8D454102C'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n    at CdpPage.t
itle (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/api/Page.js:1084:43)\n    at Crawler.crawlPage (file:///app/dist/crawler.js:587:42)\n    at async PageWorker.crawlPage (file:///app/dist/util/worker.js:
157:21)\n    at async PageWorker.timedCrawlPage (file:///app/dist/util/worker.js:173:13)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)\n    at async PageWorker.run (file:///app/dist/util/w
orker.js:199:13)\n    at async Promise.allSettled (index 88)\n    at async runWorkers (file:///app/dist/util/worker.js:276:5)\n    at async Crawler.crawl (file:///app/dist/crawler.js:926:9)","page":"https://www.urban
dictionary.com/define.php?term=mrs+brown","workerid":88}}
{"timestamp":"2024-12-20T09:04:24.217Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":2,"page":"https://www.urbandictionary.com/define.php?term=mrs+brown","workerid":88}}
{"timestamp":"2024-12-20T09:04:24.547Z","logLevel":"warn","context":"behavior","message":"Waiting for custom page load failed","details":{"type":"exception","message":"Attempted to use detached Frame '1D92D7BD0F88345
24B3BA723CD369208'.","stack":"Error: Attempted to use detached Frame '1D92D7BD0F8834524B3BA723CD369208'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:2
3)\n    at Crawler.awaitPageLoad (file:///app/dist/crawler.js:1314:25)\n    at Crawler.loadPage (file:///app/dist/crawler.js:1285:20)\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    at listOnTimeou
t (node:internal/timers:545:9)\n    at process.processTimers (node:internal/timers:519:7)\n    at async Crawler.default [as driver] (file:///app/dist/defaultDriver.js:2:5)\n    at async Crawler.crawlPage (file:///app
/dist/crawler.js:586:9)\n    at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n    at async PageWorker.timedCrawlPage (file:///app/dist/util/worker.js:173:13)"}}
{"timestamp":"2024-12-20T09:04:24.547Z","logLevel":"error","context":"worker","message":"Worker Exception","details":{"type":"exception","message":"Attempted to use detached Frame '1D92D7BD0F8834524B3BA723CD369208'."
,"stack":"Error: Attempted to use detached Frame '1D92D7BD0F8834524B3BA723CD369208'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n    at CdpPage.t
itle (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/api/Page.js:1084:43)\n    at Crawler.crawlPage (file:///app/dist/crawler.js:587:42)\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    a
t listOnTimeout (node:internal/timers:545:9)\n    at process.processTimers (node:internal/timers:519:7)\n    at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n    at async PageWorker.timedCrawlP
age (file:///app/dist/util/worker.js:173:13)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)","page":"https://www.urband
ictionary.com/remove.form.php?reconsider%5Bdefid_to_remove%5D=14272067","workerid":265}}
{"timestamp":"2024-12-20T09:04:24.548Z","logLevel":"warn","context":"behavior","message":"Waiting for custom page load failed","details":{"type":"exception","message":"Attempted to use detached Frame 'C4D6925CAE2A432
C554DC8AE58B1D361'.","stack":"Error: Attempted to use detached Frame 'C4D6925CAE2A432C554DC8AE58B1D361'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:2
3)\n    at Crawler.awaitPageLoad (file:///app/dist/crawler.js:1314:25)\n    at Crawler.loadPage (file:///app/dist/crawler.js:1285:20)\n    at async Crawler.default [as driver] (file:///app/dist/defaultDriver.js:2:5)\
n    at async Crawler.crawlPage (file:///app/dist/crawler.js:586:9)\n    at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n    at async PageWorker.timedCrawlPage (file:///app/dist/util/worker.js
:173:13)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)\n    at async Promise.allSettled (index 191)"}}
{"timestamp":"2024-12-20T09:04:24.548Z","logLevel":"error","context":"worker","message":"Worker Exception","details":{"type":"exception","message":"Attempted to use detached Frame 'C4D6925CAE2A432C554DC8AE58B1D361'."
,"stack":"Error: Attempted to use detached Frame 'C4D6925CAE2A432C554DC8AE58B1D361'.\n    at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n    at CdpPage.t
itle (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/api/Page.js:1084:43)\n    at Crawler.crawlPage (file:///app/dist/crawler.js:587:42)\n    at async PageWorker.crawlPage (file:///app/dist/util/worker.js:
157:21)\n    at async PageWorker.timedCrawlPage (file:///app/dist/util/worker.js:173:13)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)\n    at async PageWorker.run (file:///app/dist/util/w
orker.js:199:13)\n    at async Promise.allSettled (index 191)\n    at async runWorkers (file:///app/dist/util/worker.js:276:5)\n    at async Crawler.crawl (file:///app/dist/crawler.js:926:9)","page":"https://www.urba
ndictionary.com/remove.form.php?reconsider%5Bdefid_to_remove%5D=13230748","workerid":191}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":171}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":286}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":241}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":17}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":52}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":15}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":211}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":242}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":43}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":183}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":122}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":81}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":25}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":99}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":314}}
...
...
{"timestamp":"2024-12-20T09:04:24.721Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":182}}
{"timestamp":"2024-12-20T09:04:24.725Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":191}}
{"timestamp":"2024-12-20T09:04:24.725Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":265}}
{"timestamp":"2024-12-20T09:07:50.218Z","logLevel":"info","context":"general","message":"Saving crawl state to: /output/.tmpyirbj5uu/collections/crawl-20241219210812241/crawls/crawl-20241220090424-7dc588181026.yaml",
"details":{}}
{"timestamp":"2024-12-20T09:07:50.598Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":261062,"total":1901133,"pending":22,"failed":140,"limit":{"max":0,"hit":false},"pend
ingPages":["{\"seedId\":1,\"started\":\"2024-12-20T09:04:18.715Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/define.php?term=Parramatta%20Eels\",\"added\":\"2024-12-19T21:56:36.833Z\",\"depth\":
4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:18.594Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/define.php?term=bogey&page=3\",\"added\":\"2024-12-19T21:56:36.617Z\",\"depth\":4}","{\"seed
Id\":1,\"started\":\"2024-12-20T09:04:18.045Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/define.php?term=vegitation\",\"added\":\"2024-12-19T21:56:36.608Z\",\"depth\":4}","{\"seedId\":1,\"start
ed\":\"2024-12-20T09:04:17.769Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/define.php?term=Penny+candy\",\"added\":\"2024-12-19T21:56:35.140Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-1
2-20T09:04:17.993Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/author.php?author=goldendeath1134\",\"added\":\"2024-12-19T21:56:35.667Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09
:04:17.928Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/remove.form.php?reconsider%5Bdefid_to_remove%5D=1087053\",\"added\":\"2024-12-19T21:56:35.141Z\",\"depth\":4}","{\"seedId\":1,\"started\":
\"2024-12-20T09:04:21.190Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/define.php?term=K%20Thanks\",\"added\":\"2024-12-19T21:56:37.044Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T0
9:04:17.750Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/define.php?term=predecessor\",\"added\":\"2024-12-19T21:56:35.139Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:17.994Z\
",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/define.php?term=sup%2C+yo%3F\",\"added\":\"2024-12-19T21:56:35.669Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:19.909Z\",\"extraHop
s\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/author.php?author=MJTS\",\"added\":\"2024-12-19T21:56:36.834Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:21.228Z\",\"extraHops\":0,\"url\":\"ht
tps:\\/\\/www.urbandictionary.com\\/define.php?term=k%20thanks%20bye\",\"added\":\"2024-12-19T21:56:37.044Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:18.028Z\",\"extraHops\":0,\"url\":\"https:\\/\
\/www.urbandictionary.com\\/remove.form.php?reconsider%5Bdefid_to_remove%5D=13607153\",\"added\":\"2024-12-19T21:56:36.607Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:18.028Z\",\"extraHops\":0,\"ur
l\":\"https:\\/\\/www.urbandictionary.com\\/remove.form.php?reconsider%5Bdefid_to_remove%5D=3326420\",\"added\":\"2024-12-19T21:56:36.606Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:21.169Z\",\"ext
raHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/remove.form.php?reconsider%5Bdefid_to_remove%5D=4999156\",\"added\":\"2024-12-19T21:56:37.043Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:
19.732Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/remove.form.php?reconsider%5Bdefid_to_remove%5D=12136914\",\"added\":\"2024-12-19T21:56:36.833Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2
024-12-20T09:04:21.188Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/author.php?author=allvanishing\",\"added\":\"2024-12-19T21:56:37.044Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T
09:04:18.117Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/define.php?term=One%20over\",\"added\":\"2024-12-19T21:56:36.613Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:18.028Z\
",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/define.php?term=Vertical\",\"added\":\"2024-12-19T21:56:36.607Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:19.324Z\",\"extraHops\":
0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/define.php?term=scungy\",\"added\":\"2024-12-19T21:56:36.833Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:17.993Z\",\"extraHops\":0,\"url\":\"https:
\\/\\/www.urbandictionary.com\\/author.php?author=Get%20real%2C%20Ya%20All%21%21%21\",\"added\":\"2024-12-19T21:56:35.668Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:18.000Z\",\"extraHops\":0,\"url
\":\"https:\\/\\/www.urbandictionary.com\\/remove.form.php?reconsider%5Bdefid_to_remove%5D=1127164\",\"added\":\"2024-12-19T21:56:35.804Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:19.732Z\",\"extr
aHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/remove.form.php?reconsider%5Bdefid_to_remove%5D=17806886\",\"added\":\"2024-12-19T21:56:36.833Z\",\"depth\":4}"]}}
{"timestamp":"2024-12-20T09:07:50.598Z","logLevel":"info","context":"general","message":"Crawling done","details":{}}
{"timestamp":"2024-12-20T09:07:50.598Z","logLevel":"info","context":"general","message":"Exiting, Crawl status: interrupted","details":{}}
Traceback (most recent call last):
  File "/usr/bin/zimit", line 8, in <module>
    sys.exit(zimit.zimit())
             ^^^^^^^^^^^^^
  File "/app/zimit/lib/python3.12/site-packages/zimit/zimit.py", line 688, in zimit
    run(sys.argv[1:])
  File "/app/zimit/lib/python3.12/site-packages/zimit/zimit.py", line 574, in run
    raise subprocess.CalledProcessError(crawl.returncode, cmd_args)
subprocess.CalledProcessError: Command '['crawl', '--failOnFailedSeed', '--workers', '350', '--waitUntil', 'load', '--depth', '-1', '--timeout', '90', '--behaviors', 'autoplay,autofetch,siteSpecific', '--behaviorTime
out', '90', '--diskUtilization', '0', '--url', 'https://urbandictionary.com', '--userAgentSuffix', '+Zimit', '--mobileDevice', 'Pixel 2', '--cwd', '/output/.tmpyirbj5uu']' returned non-zero exit status 10.
[zimit::2024-12-20 09:07:50,690] INFO:
[zimit::2024-12-20 09:07:50,690] INFO:----------
[zimit::2024-12-20 09:07:50,690] INFO:Cleanup, removing temp dir: /output/.tmpyirbj5uu

Current version of docker image.
@benoit74
Copy link
Collaborator

This is beyond usual usage of the crawler, thanks for reporting, always good to know the limits, but I doubt it might be solved soon

@vitaly-zdanevich
Copy link
Author

vitaly-zdanevich commented Dec 20, 2024 •
edited
Loading

Limit you mean 350 workers?

But what this error about? What is detached Frame?

@benoit74
Copy link
Collaborator

The Frame is a puppeteer Frame: https://pptr.dev/api/puppeteer.frame

@vitaly-zdanevich
Copy link
Author

And what detached Frame mean?

@benoit74
Copy link
Collaborator

benoit74 commented Jan 6, 2025

I don't know

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vitaly-zdanevich @benoit74