python - 阅读具有并行请求的网站?

我想并行阅读网站的 html 内容并尝试使用以下代码 - 通常工作正常 -

resultText = {}
start = timeit.default_timer()  
async def main():
    loop = asyncio.get_event_loop()
    futures = [
        loop.run_in_executor(
            None, 
            requests.get, 
            websites[i]
        )
        for i in range(22)
    ]
    for i, response in enumerate(await asyncio.gather(*futures)):
        # resultText.append(response.text)
        resultText[websites[i]] = response.text

loop = asyncio.get_event_loop()
loop.run_until_complete(main())
stop = timeit.default_timer()
print(f"Time for whole process: {round((stop-start)/60,2)} min")
for k,v in resultText.items():
  print(k,len(v))
print(len(resultText))

但它似乎只适用于 22 个站点。 (当我将 for 循环从 22 更改为例如 23 时,它会因以下问题而停止)

Traceback (most recent call last):
  File "C:\DEV\Fiverr\TRY\robalf\checkPages2.py", line 67, in <module>
    loop.run_until_complete(main())
  File "C:\Users\WRSPOL\AppData\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 642, in run_until_complete
    return future.result()
  File "C:\DEV\Fiverr\TRY\robalf\checkPages2.py", line 62, in main
    for i, response in enumerate(await asyncio.gather(*futures)):
  File "C:\Users\WRSPOL\AppData\Local\Programs\Python\Python39\lib\concurrent\futures\thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "C:\Users\WRSPOL\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\WRSPOL\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\WRSPOL\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\WRSPOL\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\WRSPOL\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'Eine vorhandene Verbindung wurde vom Remotehost geschlossen', None, 10054, None))

我怎样才能阅读超过 22 个网站? (没有必要并行读取所有 x 站点 - 对我来说,并行运行前 22 个站点就足够了 - 然后是接下来的 22 个站点,依此类推......)但我尝试围绕异步工作流程进行循环看来我也得到了上述错误。

回答1

您可以改用 httpx:

import httpx

async def get_stock_price_data(stock):
  client = httpx.AsyncClient() 
  stock_page = await client.get( 'https://finance.yahoo.com/quote/TSLA')

这里有一篇完整的文章详细描述了 await/async:https://pythonhowtoprogram.com/python-await-async-tutorial-with-real-examples-and-simple-explanations/

相似文章

opengl - OpenGL pipeline 具体是如何执行的?

我正在寻找关于openglpipeline中订单执行的说明。我有两种情况。哪一个是正确的?或者,如果两者都不是?标准openglpipeline:顶点着色器->原始组件->几何着色器(可选)->光栅化...

最新文章