Docker/mealie
Docker/mealie
1
0
Fork 0
mirror of https://github.com/mealie-recipes/mealie.git synced 2025-08-02 20:15:24 +02:00
Code Issues Releases Activity

Convert scraper to use async (#1915)

Browse source 
* add httpx depedency for async http requests

* rework scraper strategies to download recipe html asynchronously

* rework recipe_data_service to download recipe images asynchronously

* fix recipe_parser test, so it can use async results

* fix bulk import so that it also works with async scraper

* fix broken recipe_parser tests

* Fix issues found by scanners

* Add additional checks for ingredient and instruction count in test_create_by_url

* Revert changes in test recipe_data
Since we are checking ingredients and instructions in test_create_url now, these would fail with the stored html of recipe data

* Add explicit type annotation in recipe_data_service.largest_content_len

* Fix typo in annotation
This commit is contained in:
Sören 2023-01-29 01:43:27 +01:00 • committed by GitHub
parent 7275dd2696
commit 3415a9c310
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
11 changed files with 129 additions and 115 deletions
Download patch file Download diff file Expand all files Collapse all files

5
tests/unit_tests/test_recipe_parser.py
View file

@ -16,8 +16,9 @@ and then use this test case by removing the `@pytest.mark.skip` and than testing
@pytest.mark.skipif(True, reason="Long Running API Test - manually run when updating the parser")
@pytest.mark.parametrize("recipe_test_data", test_cases)
def test_recipe_parser(recipe_test_data: RecipeSiteTestCase):
recipe, _ = scraper.create_from_url(recipe_test_data.url)
@pytest.mark.asyncio
async def test_recipe_parser(recipe_test_data: RecipeSiteTestCase):
recipe, _ = await scraper.create_from_url(recipe_test_data.url)
assert recipe.slug == recipe_test_data.expected_slug
assert len(recipe.recipe_instructions) == recipe_test_data.num_steps