Container-level block upload documentation and integration with tools.

Fixes #1371

diff --git a/docs/source/devguide.rst b/docs/source/devguide.rst
index 94fd5fb..c69387f 100644 (file)
--- a/docs/source/devguide.rst
+++ b/docs/source/devguide.rst
@@ -25,11 +25,12 @@ Document Revisions
 =========================  ================================
 Revision                   Description
 =========================  ================================
-0.7 (Sept 30, 2011)        Suggest upload/download methods using hashmaps.
+0.7 (Oct 4, 2011)          Suggest upload/download methods using hashmaps.
 \                          Propose syncing algorithm.
 \                          Support cross-account object copy and move.
 \                          Pass token as a request parameter when using ``POST`` via an HTML form.
 \                          Optionally use source account to update object from another object.
+\                          Use container ``POST`` to upload missing blocks of data.
 0.6 (Sept 13, 2011)        Reply with Merkle hash as the ETag when updating objects.
 \                          Include version id in object replace/change replies.
 \                          Change conflict (409) replies format to text.
@@ -499,6 +500,9 @@ POST
 ====================  ================================
 Request Header Name   Value
 ====================  ================================
+Content-Length        The size of the supplied data (optional, to upload)
+Content-Type          The MIME content type of the supplied data (optional, to upload)
+Transfer-Encoding     Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)
 X-Container-Policy-*  Container behavior and limits
 X-Container-Meta-*    Optional user defined metadata
 ====================  ================================
@@ -511,11 +515,13 @@ Request Parameter Name  Value
 update                  Do not replace metadata/policy (no value parameter)
 ======================  ============================================
 
-No reply content/headers.
+No reply content/headers, except when uploading data, where the reply consists of a list of hashes for the blocks received (in a simple text format, with one hash per line).
 
 The operation will overwrite all user defined metadata, except if ``update`` is defined.
 To change policy, include an ``X-Container-Policy-*`` header with the name in the key. If no ``X-Container-Policy-*`` header is present, no changes will be applied to policy. The ``update`` parameter also applies to policy - deleted values will revert to defaults. To delete/revert a specific policy directive, use ``update`` and an empty header value. See container ``PUT`` for a reference of policy directives.
 
+To upload blocks of data to the container, set ``Content-Type`` to ``application/octet-stream`` and ``Content-Length`` to a valid value (except if using ``chunked`` as the ``Transfer-Encoding``).
+
 ================  ===============================
 Return Code       Description
 ================  ===============================
@@ -1055,9 +1061,9 @@ In the case of an upload, only the missing blocks will be submitted to the serve
   * Server responds with status ``409`` (Conflict):
 
     * Server's response body contains the hashes of the blocks that do not exist on the server.
-    * For each one of the hash values in the server's response:
+    * For each hash value in the server's response (or all hashes together):
 
-      * Send a ``PUT`` request to the server with the corresponding data block. Individual blocks are uploaded to a file named ``.upload``.
+      * Send a ``POST`` request to the destination container with the corresponding data.
 
 * Repeat hashmap ``PUT``. Fail if the server's response is not ``201``.
 

diff --git a/tools/lib/client.py b/tools/lib/client.py
index 1f99c68..e2ed963 100644 (file)
--- a/tools/lib/client.py
+++ b/tools/lib/client.py
@@ -711,6 +711,16 @@ class Pithos_Client(OOS_Client):
             headers['x-container-policy-%s' % key] = val
         return self.post(path, headers=headers)
     
+    def update_container_data(self, container, f=stdin):
+        """adds blocks of data to the container"""
+        account = self.account
+        path = '/%s/%s' % (account, container)
+        params = {'update': None}
+        headers = {'content_type': 'application/octet-stream'}
+        data = f.read() if f else None
+        headers['content_length'] = len(data)
+        return self.post(path, data, headers=headers, params=params)
+    
     def delete_container(self, container, until=None, account=None):
         """deletes a container or the container history until the date provided"""
         account = account or self.account

diff --git a/tools/lib/transfer.py b/tools/lib/transfer.py
index 3456fe1..2d10b5d 100644 (file)
--- a/tools/lib/transfer.py
+++ b/tools/lib/transfer.py
@@ -74,7 +74,7 @@ def upload(client, file, container, prefix, name=None):
             offset = hashes.index(unhexlify(hash)) * blocksize
             fp.seek(offset)
             block = fp.read(blocksize)
-            client.create_object(container, '.upload', StringIO(block))
+            client.update_container_data(container, StringIO(block))
     
     client.create_object_by_hashmap(container, object, map)