1
0
Fork 0
mirror of https://github.com/mealie-recipes/mealie.git synced 2025-08-03 12:35:22 +02:00

Convert scraper to use async (#1915)

* add httpx depedency for async http requests

* rework scraper strategies to download recipe html asynchronously

* rework recipe_data_service to download recipe images asynchronously

* fix recipe_parser test, so it can use async results

* fix bulk import so that it also works with async scraper

* fix broken recipe_parser tests

* Fix issues found by scanners

* Add additional checks for ingredient and instruction count in test_create_by_url

* Revert changes in test recipe_data
Since we are checking ingredients and instructions in test_create_url now, these would fail with the stored html of recipe data

* Add explicit type annotation in recipe_data_service.largest_content_len

* Fix typo in annotation
This commit is contained in:
Sören 2023-01-29 01:43:27 +01:00 committed by GitHub
parent 7275dd2696
commit 3415a9c310
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
11 changed files with 129 additions and 115 deletions

View file

@ -45,7 +45,7 @@ def get_recipe_test_cases():
html="jam-roly-poly-with-custard.html",
html_file=test_data.html_jam_roly_poly_with_custard,
expected_slug="jam-roly-poly-with-custard",
num_ingredients=13,
num_ingredients=11,
num_steps=9,
),
RecipeSiteTestCase(
@ -54,7 +54,7 @@ def get_recipe_test_cases():
html_file=test_data.html_sous_vide_shrimp,
expected_slug="sous-vide-shrimp",
num_ingredients=5,
num_steps=0,
num_steps=1,
),
RecipeSiteTestCase(
url="https://www.bonappetit.com/recipe/detroit-style-pepperoni-pizza",