Bug #1964

glance upload does not interleave read()s with POSTs to remote

Added by Vangelis Koukis about 12 years ago. Updated about 11 years ago.

Status:Closed Start date:01/30/2012
Priority:High Due date:05/30/2012
Assignee:Giorgos Verigakis % Done:

100%

Category:- Spent time: -
Target version:-

Description

It seems kamaki glance upload does not interleave reading from an image file with POSTing blocks to the remote Pithos+ API.

This means 24GB of RAM are needed to upload a 24GB image.

History

#1 Updated by Giorgos Verigakis about 12 years ago

This is not the case. kamaki has to read the whole file before uploading, but it only keeps the hashes of its blocks in memory.

A 24GB file will have 6144 blocks, so kamaki will have to keep 192KB of hashes in memory (not including Python's overhead).

#2 Updated by Vangelis Koukis about 12 years ago

  • Target version deleted (v0.3)

Writing the main points of our extensive offline discussion:

./kamaki only stores hashes in memory, then has to re-read the parts that have to be uploaded to Pithos. This means re-reading the whole file, potentially.

This is an architectural decision made at the Pithos side, to reduce the load on the Pithos server: instead of flooding it with requests, one per block, the client will make a single request with all hashes at once, then re-read the parts that need to be uploaded.

This is relevant to Pithos ticket #1887 (Provide an interface to query blocks/hashmaps).

This ticket can be re-examined based on our experience with using kamaki for uploads of VM images.

#3 Updated by Stavros Sachtouris about 11 years ago

  • Status changed from Assigned to Resolved
  • % Done changed from 0 to 100

Looks like this bug has long been fixed

#4 Updated by Stavros Sachtouris about 11 years ago

  • Due date set to 05/30/2012

#5 Updated by Stavros Sachtouris about 11 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF