Bug #1017
"Broken pipe" exception thrown when trying to post message to AMQP
| Status: | Closed | Start date: | 08/23/2011 | |
|---|---|---|---|---|
| Priority: | High | Due date: | ||
| Assignee: | Giorgos Gousios | % Done: | 100% |
|
| Category: | logic | Spent time: | - | |
| Target version: | v0.5.5 |
Description
The code handling feedback submission tries to post a message to the AMQP queue
and throws a "Broken pipe" exception:
On Tue, Aug 23, 2011 at 01:33:17pm -0000, ~okeanos wrote:
Traceback (most recent call last):
File "/usr/lib/pymodules/python2.6/django/core/handlers/base.py", line 100, in get_response
response = callback(request, *callback_args, **callback_kwargs)File "/srv/okeanos/synnefo/ui/views.py", line 244, in feedback_submit
body = mail_contentFile "/srv/okeanos/synnefo/logic/email_send.py", line 56, in send_async
amqp_connection.send(json.dumps(msg), settings.EXCHANGE_API, routekey)File "/srv/okeanos/synnefo/logic/amqp_connection.py", line 83, in send
_connect()File "/srv/okeanos/synnefo/logic/amqp_connection.py", line 50, in _connect
_chan = _conn.channel()File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/connection.py", line 230, in channel
return Channel(self, channel_id)File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/channel.py", line 82, in init
self._x_open()File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/channel.py", line 469, in _x_open
self._send_method((20, 10), args)File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/abstract_channel.py", line 70, in _send_method
method_sig, args, content)File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/method_framing.py", line 233, in write_method
self.dest.write_frame(1, channel, payload)File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/transport.py", line 125, in write_frame
frame_type, channel, size, payload, 0xce))File "<string>", line 1, in sendall
error: [Errno 32] Broken pipe
This could be similar to the problem described here:
https://groups.google.com/group/celery-users/browse_thread/thread/1e886caee1fef58b/107501de95f8471e
https://github.com/ask/celery/issues/354
triggered by a restart of the RabbitMQ server.
Associated revisions
Force the connection obj to re-initialize on error
When an error occurs to the AMQP connection, the code will automatically
attempt to re-establish the connection. Before this fix, it used the
same (stale) connection object. The fix deletes the object explicitely.
(Hopefully) Fixes: #1017
credit to: vkoukis :-)
History
#1 Updated by Giorgos Gousios over 12 years ago
I don't think that the problems reported in the two links you have posted bear any resemblance to this, because:
-They are talking for a completely different setup (not using the amqp lib) and different RabbitMQ version
-If the problem was similar, it would have affected all the dispatcher handlers (since they all connect to RabbitMQ) and not just the email one.
My hypothesis involves the fact that we do not know how wsgi handles object persistence between requests and the fact that we are using persistent AMQP connections, but I am not sure yet.
#2 Updated by Faidon Liambotis over 12 years ago
JFYI, celery uses kombu and kombu can use amqplib and in fact it does in those specific examples. If you look more closely in those backtraces, you'll see that the exception originates at the exact same file and line within amqplib.
Not claiming that they are related but they're not that different as you suggested.
#3 Updated by Giorgos Gousios over 12 years ago
JFYI, celery uses kombu and kombu can use amqplib and in fact it does in those specific examples. If you look more closely in those backtraces, you'll see that the exception originates at the exact same file and line within amqplib.
True, my bad! I should get used to read exception traces top down rather than bottom up :-)
#4 Updated by Vangelis Koukis over 12 years ago
Celery ticket 354 mentions this has been fixed on their master.
I could find the following relevant commit:
https://github.com/ask/celery/commit/ab301c20040366db05090d26f94211f6e11fbbf7
They seem to re-establish the connection explicitly once it's lost.
#5 Updated by Giorgos Gousios over 12 years ago
- Status changed from Assigned to Closed
- % Done changed from 0 to 100
Applied in changeset 29053267cf43bca97081b94b4d6ac82d1c33b5cf.