Bug #3414

AssertionError in Astakos gunicorn

Added by Vangelis Koukis about 11 years ago. Updated about 11 years ago.

Status:Closed Start date:03/08/2013
Priority:High Due date:
Assignee:Stratos Psomadakis % Done:

0%

Category:Astakos Spent time: -
Target version:0.13.0

Description

As seen in production:

2013-03-08 12:58:28,175 [18951] astakos.im.api.backends.lib.django __init__ [ERROR] This event is already used by another greenlet: (<Greenlet at 0x23cf9e0: <bound method GeventWorker.handle of <gunicorn.workers.ggevent.GeventWorker object at 0x11a0390>>(<socket at 0x2b8e7d0 fileno=24>, ('2001:648:2ffc:106::82', 46750, 0, 0))>, SSLError('The read operation timed out',))
Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/astakos/im/api/backends/lib/django/__init__.py", line 71, in wrapper
    data = func(self, *args, **kwargs) or ()
  File "/usr/lib/python2.6/dist-packages/astakos/im/api/backends/lib/django/__init__.py", line 241, in get_resource_usage
    data = get_quota([user])
  File "/usr/lib/python2.6/dist-packages/astakos/im/functions.py", line 393, in get_quota
    return qh_get_quota(users, resources)
  File "/usr/lib/python2.6/dist-packages/astakos/im/endpoints/qh.py", line 154, in qh_get_quota
    result = c.get_quota(context={}, clientkey=clientkey, get_quota=payload)
  File "/usr/lib/python2.6/dist-packages/kamaki/clients/commissioning/callpoint.py", line 84, in call_func
    return self.make_call(local_call_name, data)
  File "/usr/lib/python2.6/dist-packages/kamaki/clients/commissioning/callpoint.py", line 161, in make_call
    data = self.do_make_call(call_name, data)
  File "/usr/lib/python2.6/dist-packages/kamaki/clients/commissioning_client.py", line 59, in do_make_call
    resp = method(path, data=json_data, success=(200, 450, 500))
  File "/usr/lib/python2.6/dist-packages/kamaki/clients/__init__.py", line 248, in get
    return self.request('get', path, **kwargs)
  File "/usr/lib/python2.6/dist-packages/kamaki/clients/__init__.py", line 216, in request
    recvlog.info('%d %s', r.status_code, r.status)
  File "/usr/lib/python2.6/dist-packages/kamaki/clients/connection/__init__.py", line 108, in status_code
    self._get_response()
  File "/usr/lib/python2.6/dist-packages/kamaki/clients/connection/kamakicon.py", line 58, in _get_response
    r = self.request.getresponse()
  File "/usr/lib/python2.6/httplib.py", line 990, in getresponse
    response.begin()
  File "/usr/lib/python2.6/httplib.py", line 391, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.6/httplib.py", line 349, in _read_status
    line = self.fp.readline()
  File "/usr/lib/python2.6/socket.py", line 427, in readline
    data = recv(1)
  File "/usr/lib/pymodules/python2.6/gevent/ssl.py", line 232, in recv
    return self.read(buflen)
  File "/usr/lib/pymodules/python2.6/gevent/ssl.py", line 116, in read
    wait_read(self.fileno(), timeout=self.timeout, timeout_exc=_SSLErrorReadTimeout, event=self._read_event)
  File "/usr/lib/pymodules/python2.6/gevent/socket.py", line 165, in wait_read
    assert event.arg is None, 'This event is already used by another greenlet: %r' % (event.arg, )
AssertionError: This event is already used by another greenlet: (<Greenlet at 0x23cf9e0: <bound method GeventWorker.handle of <gunicorn.workers.ggevent.GeventWorker object at 0x11a0390>>(<socket at 0x2b8e7d0 fileno=24>, ('2001:648:2ffc:106::82', 46750, 0, 0))>, SSLError('The read operation timed out',))

Could it be an interaction between astakos and qholder?
Astakos/kamaki/qholder using non-monkey-patched socket functionality, giving gevent trouble?

This is a major problem for production, it brings down Astakos.


Related issues

related to Synnefo - Bug #3410: "This event is already used by another greenlet:" Error Closed 03/08/2013
related to Synnefo - Bug #3492: get_http_connection() patches .close() which gets called ... Closed 03/22/2013

History

#1 Updated by Georgios Tsoukalas about 11 years ago

  • Status changed from Assigned to Closed

Also available in: Atom PDF