Python

requests异常:requests.exceptions.ConnectionError

使用 requests抓取360搜索结果页的时候,由于要抓取上万个网页,遇到以下异常:

requests.exceptions.ConnectionError: HTTPSConnectionPool Max retries exceeded

  • 在网上找了下原因,主要有以下三点
  1. .http的连接数超过最大限制,默认的情况下连接是Keep-alive的,所以这就导致了服务器保持了太多连接而不能再新建连接。
  2. ip被封
  3. 请求过快

 

 

解决方案如下:

1.在header中不使用持久连接,而是使用‘Connection’: ‘close’requests.adapters.DEFAULT_RETRIES = 5

2.若是请求过快,可设置time.sleep

3.使用代理ip

4.Requests请求时有时会请求不到页面,或是请求到空白的页面,超时要重试几次,使用try…except语句

因为已经使用了代理。所以只是将headers中的 Connectionsh属性改为    ‘Connection’: ‘close’。并且在循环中,请求一次之后休眠0.1s

 

 

异常代码

Traceback (most recent call last):
  File "C:\Python 3.82\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "C:\Python 3.82\lib\site-packages\urllib3\connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "C:\Python 3.82\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Python 3.82\lib\http\client.py", line 1322, in getresponse
    response.begin()
  File "C:\Python 3.82\lib\http\client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "C:\Python 3.82\lib\http\client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python 3.82\lib\site-packages\requests\adapters.py", line 439, in send
    resp = conn.urlopen(
  File "C:\Python 3.82\lib\site-packages\urllib3\connectionpool.py", line 724, in urlopen
    retries = retries.increment(
  File "C:\Python 3.82\lib\site-packages\urllib3\util\retry.py", line 403, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Python 3.82\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "C:\Python 3.82\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "C:\Python 3.82\lib\site-packages\urllib3\connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "C:\Python 3.82\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Python 3.82\lib\http\client.py", line 1322, in getresponse
    response.begin()
  File "C:\Python 3.82\lib\http\client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "C:\Python 3.82\lib\http\client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:/OneDrive/python_code/search_enigine_indexable-2/test.py", line 13, in <module>
    indexable.query_haosou_with_proxy(txt_addr)
  File "D:\OneDrive\python_code\search_enigine_indexable-2\run\indexable.py", line 30, in query_haosou_with_proxy
    results_list =hsindex.query_many_urls_with_proxy(urls_list)
  File "D:\OneDrive\python_code\search_enigine_indexable-2\haosou\indexable.py", line 191, in query_many_urls_with_proxy
    proxy = proxies.get_one_http_proxy()
  File "D:\OneDrive\python_code\search_enigine_indexable-2\proxies\proxies.py", line 14, in get_one_http_proxy
    return  zhima_proxy.get_one_http_proxy()
  File "D:\OneDrive\python_code\search_enigine_indexable-2\proxies\zhima_proxy.py", line 6, in get_one_http_proxy
    res=requests.get(url)
  File "C:\Python 3.82\lib\site-packages\requests\api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Python 3.82\lib\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Python 3.82\lib\site-packages\requests\sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python 3.82\lib\site-packages\requests\sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python 3.82\lib\site-packages\requests\adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

 

 

(0)

本文由 SilenceLee的学习笔记 作者:silencelee 发表,转载请注明来源!

关键词:

热评文章

发表评论

解决 : *
14 + 1 =