TheKitchn website scraper still does not work in Mealie latest with latest scraper

[calm108]

Pre-filing checks

• [ YES] I have searched for open issues that report the same problem
• [ YES] I have checked that the bug affects the latest version of the library

The URL of the recipe(s) that are not being scraped correctly

• https://www.thekitchn.com/beef-bourguignon-recipe-23603515#post-recipe-575913341
  -https://www.thekitchn.com/beef-tips-recipe-23578525

...

The results you expect to see

...the recipe

The results (including any Python error messages) that you are seeing

...
Cannot find anything error from Mealie.

[calm108]

I ahve added this bug before and it was closed. I get no 403 error and can visti the URL no problem. Tested other recipes sites (allecipes and it works fine) EVERY recipe I try on TheKitchn fails.

[jayaddison]

Ok, thanks @calm108 - let's take another look at this. Based on what you've explained so far, could you let me know if the following is accurate:

• You're able to open the mentioned recipe(s) in the browser absolutely fine.
• The recipes cannot be loaded from Mealie.
  • Previously this appeared as some kind of HTTP 403.
  • Now it is... ? (but either way, it works when opening a normal web browser).

If so: first of all, I can replicate those locally, so I believe you. Secondly: it certainly seems to indicate that something is blocking particular HTTP clients.

[calm108]

You have it exactly correct 👍 I have tried multiple recipe URL’s on thekitchn and all with the same result.

[jayaddison]

Ok; I should also note that when I replicated the problem, that was using recipe-scrapers directly, without the involvement of the Mealie application.

I'm going to add the 'bots protection' label to this bug, since it seems to be a case where a recipe site is rejecting some network traffic.