Bug #3527

Some kamaki requests return "404 Not Found"

Added by Ilias Tsitsimpis about 11 years ago. Updated almost 11 years ago.

Status:Closed Start date:04/02/2013
Priority:High Due date:
Assignee:Stavros Sachtouris % Done:

100%

Category:kamaki Spent time: -
Target version:v0.7

Description

Some kamaki requests return "404 Not Found"

When using kamaki to download an image from Pithos+ sometimes the user
may encourage a 404 response.

How to reproduce:
-----------------
Run `kamaki store download --account=25ecced9-bf53-4145-91ee-cf47377e9fb2 images:debian_base-6.0-10-x86_64.diskdump`
a few times. One out of ten requests usually fails.

How to debug:
Add the following line to kamakicon.py, in function prerform_request, just before the conn.request function call.

print "Connection: method=%s, url=%s, headers=%s, body=%s" % (method.upper(), self.path, http_headers, data)

When kamaki fails with a 404 response one should see:
Connection: method=GET, url=/25ecced9-bf53-4145-91ee-cf47377e9fb2/images/debian_base-6.0-10-x86_64.diskdump, headers={'Range': 'bytes=125829120-130023423', 'X-Auth-Token': 'KzY7zdX8DeRqfOpilPWSBQ=='}, body=None
whereas the url should be "/v1/25ecced9-bf53-4145-91ee-cf47377e9fb2/images/debian_base-6.0-10-x86_64.diskdump?format=json"

Therefore it's definitely a kamaki bug.

out2.txt Magnifier (47.4 kB) Ilias Tsitsimpis, 04/08/2013 02:26 pm


Related issues

related to ./kamaki - Support #3526: Support the new objpool HTTP connection-response model an... Closed 04/02/2013 04/10/2013

History

#1 Updated by Stavros Sachtouris about 11 years ago

  • Status changed from New to Resolved
  • % Done changed from 0 to 100

The issue is caused by threads using the same instance of kamaki.clients.connection.kamakicon.KamakiHTTPRequest. The KamakiTTHPRequest.path variable is accessed by multiple methods. This is fixed with hotfix 0.7.8, where the path is copied to a new KamakiHTTPResponse object before it becomes "exposed" to other threads, and it is not used any further after that.

In 0.8 a redesign of the connection module will prevent bugs like this to happen.

#2 Updated by Stavros Sachtouris about 11 years ago

  • Status changed from Resolved to Closed

Shared variables are to blame for this error, it is fixed with hotfix 0.7.8

#3 Updated by Ilias Tsitsimpis about 11 years ago

It seems that the bug still exists in version 0.7.8

Attached is the output of kamaki store download --account=25ecced9-bf53-4145-91ee-cf47377e9fb2 images:debian_base-6.0-10-x86_64.diskdump, where I have add a print debug message just before the conn.request function call.
In line 990 kamaki tries to connect to url "/25ecced9-bf53-4145-91ee-cf47377e9fb2/images/debian_base-6.0-10-x86_64.diskdump" whereas the correct is "/v1/25ecced9-bf53-4145-91ee-cf47377e9fb2/images/debian_base-6.0-10-x86_64.diskdump?format=json".

Use kamaki from debian repos, version 0.7.8-2.

#4 Updated by Stavros Sachtouris about 11 years ago

  • % Done changed from 100 to 50

#5 Updated by Vangelis Koukis about 11 years ago

  • Status changed from Closed to Assigned

Re-opening, after discussion with [iliastsi].

#6 Updated by Stavros Sachtouris about 11 years ago

  • Status changed from Assigned to Resolved

Resolved: it's a limitation of obj pool:

https://github.com/grnet/objpool/blob/e4c7df4a17e964243c1a052d5ce4d66412dc2363/objpool/__init__.py#L113

The error apears after forking a process wich contains a kamaki.clients.Client instance and performing parallel calls with the Client instance. Unfortunately, Objpool does not support forked connections (yet).

#7 Updated by Stavros Sachtouris about 11 years ago

  • Status changed from Resolved to Assigned
  • % Done changed from 50 to 0

#8 Updated by Stavros Sachtouris about 11 years ago

  • Status changed from Assigned to Resolved
  • % Done changed from 0 to 100

The problem is caused by a race condition on the url member variable in a kamaki.clients.Client instance, during the http connection setup. The problem only apears in threaded operations (upload or download) and it is very rare.

In kamaki 0.8 each thread passes request parameters by value and creates new, own-thread-only instances of RequestManager and ResponseManager kamaki.clients classes, so there are no race conditions in the http connection setup.

#9 Updated by Stavros Sachtouris almost 11 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF