Revision 3ab38c43 docs/source/devguide.rst
| b/docs/source/devguide.rst | ||
|---|---|---|
| 6 | 6 |
|
| 7 | 7 |
Pithos is a storage service implemented by GRNET (http://www.grnet.gr). Data is stored as objects, organized in containers, belonging to an account. This hierarchy of storage layers has been inspired by the OpenStack Object Storage (OOS) API and similar CloudFiles API by Rackspace. The Pithos API follows the OOS API as closely as possible. One of the design requirements has been to be able to use Pithos with clients built for the OOS, without changes. |
| 8 | 8 |
|
| 9 |
However, to be able to take full advantage of the Pithos infrastructure, client software should be aware of the extensions that differentiate Pithos from OOS. Pithos objects can be updated, or appended to. Automatic version management, allows taking account and container listings back in time, as well as reading previous instances of objects. |
|
| 9 |
However, to be able to take full advantage of the Pithos infrastructure, client software should be aware of the extensions that differentiate Pithos from OOS. Pithos objects can be updated, or appended to. Pithos will store sharing permissions per object and enforce corresponding authorization policies. Automatic version management, allows taking account and container listings back in time, as well as reading previous instances of objects.
|
|
| 10 | 10 |
|
| 11 | 11 |
The storage backend of Pithos is block oriented, permitting efficient, deduplicated data placement. The block structure of objects is exposed at the API layer, in order to encourage external software to implement advanced data management operations. |
| 12 | 12 |
|
| ... | ... | |
| 25 | 25 |
========================= ================================ |
| 26 | 26 |
Revision Description |
| 27 | 27 |
========================= ================================ |
| 28 |
0.4 (June 22, 2011) Support updating/deleting individual metadata with ``POST``. |
|
| 28 |
0.4 (June 30, 2011) Object permissions and account groups. |
|
| 29 |
\ Control versioning behavior and container quotas with container policy directives. |
|
| 30 |
\ Support updating/deleting individual metadata with ``POST``. |
|
| 29 | 31 |
0.3 (June 14, 2011) Large object support with ``X-Object-Manifest``. |
| 30 | 32 |
\ Allow for publicly available objects via ``https://hostname/public``. |
| 31 | 33 |
\ Support time-variant account/container listings. |
| ... | ... | |
| 118 | 120 |
X-Account-Bytes-Remaining The total number of bytes remaining (**TBD**) |
| 119 | 121 |
X-Account-Last-Login The last login (**TBD**) |
| 120 | 122 |
X-Account-Until-Timestamp The last account modification date until the timestamp provided |
| 123 |
X-Account-Group-* Optional user defined groups |
|
| 121 | 124 |
X-Account-Meta-* Optional user defined metadata |
| 122 | 125 |
Last-Modified The last account modification date (regardless of ``until``) |
| 123 | 126 |
========================== ===================== |
| ... | ... | |
| 164 | 167 |
bytes The total size of the objects inside the container |
| 165 | 168 |
last_modified The last container modification date (regardless of ``until``) |
| 166 | 169 |
x_container_until_timestamp The last container modification date until the timestamp provided |
| 170 |
x_container_policy_* Container behavior and limits |
|
| 167 | 171 |
x_container_meta_* Optional user defined metadata |
| 168 | 172 |
=========================== ============================ |
| 169 | 173 |
|
| ... | ... | |
| 187 | 191 |
====================== ============================================ |
| 188 | 192 |
Request Parameter Name Value |
| 189 | 193 |
====================== ============================================ |
| 190 |
update Do not replace metadata (no value parameter) |
|
| 194 |
update Do not replace metadata/groups (no value parameter)
|
|
| 191 | 195 |
====================== ============================================ |
| 192 | 196 |
|
| 193 | 197 |
| |
| ... | ... | |
| 195 | 199 |
==================== =========================== |
| 196 | 200 |
Request Header Name Value |
| 197 | 201 |
==================== =========================== |
| 202 |
X-Account-Group-* Optional user defined groups |
|
| 198 | 203 |
X-Account-Meta-* Optional user defined metadata |
| 199 | 204 |
==================== =========================== |
| 200 | 205 |
|
| 201 | 206 |
No reply content/headers. |
| 202 | 207 |
|
| 203 | 208 |
The operation will overwrite all user defined metadata, except if ``update`` is defined. |
| 209 |
To create a group, include an ``X-Account-Group-*`` header with the name in the key and a comma separated list of user names in the value. If no ``X-Account-Group-*`` header is present, no changes will be applied to groups. The ``update`` parameter also applies to groups. To delete a specific group, use ``update`` and an empty header value. |
|
| 204 | 210 |
|
| 205 | 211 |
================ =============================== |
| 206 | 212 |
Return Code Description |
| ... | ... | |
| 245 | 251 |
X-Container-Block-Hash The hash algorithm used for block identifiers in object hashmaps |
| 246 | 252 |
X-Container-Until-Timestamp The last container modification date until the timestamp provided |
| 247 | 253 |
X-Container-Object-Meta A list with all meta keys used by objects |
| 254 |
X-Container-Policy-* Container behavior and limits |
|
| 248 | 255 |
X-Container-Meta-* Optional user defined metadata |
| 249 | 256 |
Last-Modified The last container modification date (regardless of ``until``) |
| 250 | 257 |
=========================== =============================== |
| 251 | 258 |
|
| 252 |
The keys returned in ``X-Container-Object-Meta`` are all the unique strings after the ``X-Object-Meta-`` prefix. |
|
| 259 |
The keys returned in ``X-Container-Object-Meta`` are all the unique strings after the ``X-Object-Meta-`` prefix. See container ``PUT`` for a reference of policy directives.
|
|
| 253 | 260 |
|
| 254 | 261 |
================ =============================== |
| 255 | 262 |
Return Code Description |
| ... | ... | |
| 305 | 312 |
x_object_version_timestamp The object's version timestamp |
| 306 | 313 |
x_object_modified_by The user that committed the object's version |
| 307 | 314 |
x_object_manifest Object parts prefix in ``<container>/<object>`` form (optional) |
| 308 |
x_object_public Object is publicly accessible (optional) (**TBD**) |
|
| 315 |
x_object_sharing Object permissions (optional) |
|
| 316 |
x_object_shared_by Object inheriting permissions (optional) |
|
| 317 |
x_object_public Object's publicly accessible URI (optional) |
|
| 309 | 318 |
x_object_meta_* Optional user defined metadata |
| 310 | 319 |
========================== ====================================== |
| 311 | 320 |
|
| ... | ... | |
| 334 | 343 |
==================== ================================ |
| 335 | 344 |
Request Header Name Value |
| 336 | 345 |
==================== ================================ |
| 346 |
X-Container-Policy-* Container behavior and limits |
|
| 337 | 347 |
X-Container-Meta-* Optional user defined metadata |
| 338 | 348 |
==================== ================================ |
| 339 | 349 |
|
| 340 | 350 |
No reply content/headers. |
| 351 |
|
|
| 352 |
If no policy is defined, the container will be created with the default values. |
|
| 353 |
Available policy directives: |
|
| 354 |
|
|
| 355 |
* ``versioning``: Set to ``auto``, ``manual`` or ``none`` (default is ``manual``) |
|
| 356 |
* ``quota``: Size limit in KB (default is ``0`` - unlimited) |
|
| 341 | 357 |
|
| 342 | 358 |
================ =============================== |
| 343 | 359 |
Return Code Description |
| ... | ... | |
| 353 | 369 |
====================== ============================================ |
| 354 | 370 |
Request Parameter Name Value |
| 355 | 371 |
====================== ============================================ |
| 356 |
update Do not replace metadata (no value parameter) |
|
| 372 |
update Do not replace metadata/policy (no value parameter)
|
|
| 357 | 373 |
====================== ============================================ |
| 358 | 374 |
|
| 359 | 375 |
| |
| ... | ... | |
| 361 | 377 |
==================== ================================ |
| 362 | 378 |
Request Header Name Value |
| 363 | 379 |
==================== ================================ |
| 380 |
X-Container-Policy-* Container behavior and limits |
|
| 364 | 381 |
X-Container-Meta-* Optional user defined metadata |
| 365 | 382 |
==================== ================================ |
| 366 | 383 |
|
| 367 | 384 |
No reply content/headers. |
| 368 | 385 |
|
| 369 | 386 |
The operation will overwrite all user defined metadata, except if ``update`` is defined. |
| 387 |
To change policy, include an ``X-Container-Policy-*`` header with the name in the key. If no ``X-Container-Policy-*`` header is present, no changes will be applied to policy. The ``update`` parameter also applies to policy - deleted values will revert to defaults. To delete/revert a specific policy directive, use ``update`` and an empty header value. See container ``PUT`` for a reference of policy directives. |
|
| 370 | 388 |
|
| 371 | 389 |
================ =============================== |
| 372 | 390 |
Return Code Description |
| ... | ... | |
| 432 | 450 |
X-Object-Version-Timestamp The object's version timestamp |
| 433 | 451 |
X-Object-Modified-By The user that comitted the object's version |
| 434 | 452 |
X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional) |
| 435 |
X-Object-Public Object is publicly accessible (optional) (**TBD**) |
|
| 453 |
X-Object-Sharing Object permissions (optional) |
|
| 454 |
X-Object-Shared-By Object inheriting permissions (optional) |
|
| 455 |
X-Object-Public Object's publicly accessible URI (optional) |
|
| 436 | 456 |
X-Object-Meta-* Optional user defined metadata |
| 437 | 457 |
========================== =============================== |
| 438 | 458 |
|
| ... | ... | |
| 522 | 542 |
X-Object-Version-Timestamp The object's version timestamp |
| 523 | 543 |
X-Object-Modified-By The user that comitted the object's version |
| 524 | 544 |
X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional) |
| 525 |
X-Object-Public Object is publicly accessible (optional) (**TBD**) |
|
| 545 |
X-Object-Sharing Object permissions (optional) |
|
| 546 |
X-Object-Shared-By Object inheriting permissions (optional) |
|
| 547 |
X-Object-Public Object's publicly accessible URI (optional) |
|
| 526 | 548 |
X-Object-Meta-* Optional user defined metadata |
| 527 | 549 |
========================== =============================== |
| 528 | 550 |
|
| ... | ... | |
| 555 | 577 |
Content-Encoding The encoding of the object (optional) |
| 556 | 578 |
Content-Disposition The presentation style of the object (optional) |
| 557 | 579 |
X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional) |
| 558 |
X-Object-Public Object is publicly accessible (optional) (**TBD**) |
|
| 580 |
X-Object-Sharing Object permissions (optional) |
|
| 581 |
X-Object-Public Object is publicly accessible (optional) |
|
| 559 | 582 |
X-Object-Meta-* Optional user defined metadata |
| 560 | 583 |
==================== ================================ |
| 561 | 584 |
|
| ... | ... | |
| 567 | 590 |
ETag The MD5 hash of the object (on create) |
| 568 | 591 |
========================== =============================== |
| 569 | 592 |
|
| 570 |
|
|
|
| 593 |
The ``X-Object-Sharing`` header may include either a ``read=...`` comma-separated user/group list, or a ``write=...`` comma-separated user/group list, or both separated by a semicolon (``;``). To publish the object, set ``X-Object-Public`` to ``true``. To unpublish, set to ``false``, or use an empty header value.
|
|
| 571 | 594 |
|
| 572 | 595 |
=========================== ============================== |
| 573 | 596 |
Return Code Description |
| ... | ... | |
| 590 | 613 |
Content-Disposition The presentation style of the object (optional) |
| 591 | 614 |
X-Source-Version The source version to copy from |
| 592 | 615 |
X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional) |
| 593 |
X-Object-Public Object is publicly accessible (optional) (**TBD**) |
|
| 616 |
X-Object-Sharing Object permissions (optional) |
|
| 617 |
X-Object-Public Object is publicly accessible (optional) |
|
| 594 | 618 |
X-Object-Meta-* Optional user defined metadata |
| 595 | 619 |
==================== ================================ |
| 596 | 620 |
|
| 597 |
Refer to ``POST`` for a description of request headers. Metadata is also copied, updated with any values defined.
|
|
| 621 |
Refer to ``PUT``/``POST`` for a description of request headers. Metadata is also copied, updated with any values defined. Sharing/publishing options are not copied.
|
|
| 598 | 622 |
|
| 599 | 623 |
No reply content/headers. |
| 600 | 624 |
|
| ... | ... | |
| 632 | 656 |
Content-Encoding The encoding of the object (optional) |
| 633 | 657 |
Content-Disposition The presentation style of the object (optional) |
| 634 | 658 |
X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional) |
| 635 |
X-Object-Public Object is publicly accessible (optional) (**TBD**) |
|
| 659 |
X-Object-Sharing Object permissions (optional) |
|
| 660 |
X-Object-Public Object is publicly accessible (optional) |
|
| 636 | 661 |
X-Object-Meta-* Optional user defined metadata |
| 637 | 662 |
==================== ================================ |
| 638 | 663 |
|
| 639 |
The ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``, ``X-Object-Public`` (**TBD**) and ``X-Object-Meta-*`` headers are considered to be user defined metadata. An operation without the ``update`` parameter will overwrite all previous values and remove any keys not supplied. When using ``update`` any metadata with an empty value will be deleted. |
|
| 664 |
The ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest`` and ``X-Object-Meta-*`` headers are considered to be user defined metadata. An operation without the ``update`` parameter will overwrite all previous values and remove any keys not supplied. When using ``update`` any metadata with an empty value will be deleted. |
|
| 665 |
|
|
| 666 |
To change permissions, include an ``X-Object-Sharing`` header (as defined in ``PUT``). To publish, include an ``X-Object-Public`` header, with a value of ``true``. If no such headers are defined, no changes will be applied to sharing/public. Use empty values to remove permissions/unpublish (unpublishing also works with ``false`` as a header value). Sharing options are applied to the object - not its versions. |
|
| 640 | 667 |
|
| 641 | 668 |
To update an object's data: |
| 642 | 669 |
|
| ... | ... | |
| 683 | 710 |
204 (No Content) The request succeeded |
| 684 | 711 |
=========================== ============================== |
| 685 | 712 |
|
| 686 |
Public Objects |
|
| 687 |
^^^^^^^^^^^^^^ |
|
| 713 |
Sharing and Public Objects |
|
| 714 |
^^^^^^^^^^^^^^^^^^^^^^^^^^ |
|
| 715 |
|
|
| 716 |
Read and write control in Pithos is managed by setting appropriate permissions with the ``X-Object-Sharing`` header. The permissions are applied using prefix-based inheritance. Thus, each set of authorization directives is applied to all objects sharing the same prefix with the object where the corresponding ``X-Object-Sharing`` header is defined. For simplicity, nested/overlapping permissions are not allowed. Setting ``X-Object-Sharing`` will fail, if the object is already "covered", or another object with a longer common-prefix name already has permissions. When retrieving an object, the ``X-Object-Shared-By`` header reports where it gets its permissions from. If not present, the object is the actual source of authorization directives. |
|
| 688 | 717 |
|
| 689 |
Objects that are marked as public, via the ``X-Object-Public`` meta (**TBD**), are also available at the corresponding URI ``https://hostname/public/<account>/<container>/<object>`` for ``HEAD`` or ``GET``. Requests for public objects do not need to include an ``X-Auth-Token``. Pithos will ignore request parameters and only include the following headers in the reply (all ``X-Object-*`` meta is hidden).
|
|
| 718 |
Objects that are marked as public, via the ``X-Object-Public`` meta, are also available at the corresponding URI returned for ``HEAD`` or ``GET``. Requests for public objects do not need to include an ``X-Auth-Token``. Pithos will ignore request parameters and only include the following headers in the reply (all ``X-Object-*`` meta is hidden).
|
|
| 690 | 719 |
|
| 691 | 720 |
========================== =============================== |
| 692 | 721 |
Reply Header Name Value |
| ... | ... | |
| 706 | 735 |
List of differences from the OOS API: |
| 707 | 736 |
|
| 708 | 737 |
* Support for ``X-Account-Meta-*`` style headers at the account level. Use ``POST`` to update. |
| 709 |
* Support for ``X-Container-Meta-*`` style headers at the account level. Can be set when creating via ``PUT``. Use ``POST`` to update.
|
|
| 738 |
* Support for ``X-Container-Meta-*`` style headers at the container level. Can be set when creating via ``PUT``. Use ``POST`` to update.
|
|
| 710 | 739 |
* Header ``X-Container-Object-Meta`` at the container level and parameter ``meta`` in container listings. |
| 740 |
* Container policies to manage behavior and limits. |
|
| 711 | 741 |
* Headers ``X-Container-Block-*`` at the container level, exposing the underlying storage characteristics. |
| 712 | 742 |
* All metadata replies, at all levels, include latest modification information. |
| 713 | 743 |
* At all levels, a ``GET`` request may use ``If-Modified-Since`` and ``If-Unmodified-Since`` headers. |
| 714 | 744 |
* Container/object lists include all associated metadata if the reply is of type json/xml. Some names are kept to their OOS API equivalents for compatibility. |
| 715 |
* Object metadata allowed, in addition to ``X-Object-Meta-*``: ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``, ``X-Object-Public`` (**TBD**). These are all replaced with every update operation, except if using the ``update`` parameter (in which case individual keys can also be deleted). Deleting meta by providing empty values also works when copying/moving an object.
|
|
| 745 |
* Object metadata allowed, in addition to ``X-Object-Meta-*``: ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``. These are all replaced with every update operation, except if using the ``update`` parameter (in which case individual keys can also be deleted). Deleting meta by providing empty values also works when copying/moving an object. |
|
| 716 | 746 |
* Multi-range object GET support as outlined in RFC2616. |
| 717 | 747 |
* Object hashmap retrieval through GET and the ``format`` parameter. |
| 718 | 748 |
* Partial object updates through POST, using the ``Content-Length``, ``Content-Type``, ``Content-Range`` and ``Transfer-Encoding`` headers. |
| 719 | 749 |
* Object ``MOVE`` support. |
| 720 | 750 |
* Time-variant account/container listings via the ``until`` parameter. |
| 721 | 751 |
* Object versions - parameter ``version`` in HEAD/GET (list versions with GET), ``X-Object-Version-*`` meta in replies, ``X-Source-Version`` in PUT/COPY. |
| 722 |
* Publicly accessible objects via ``https://hostname/public``. Control with ``X-Object-Public`` (**TBD**). |
|
| 752 |
* Sharing/publishing with ``X-Object-Sharing``, ``X-Object-Public`` at the object level. Permissions may include groups defined with ``X-Account-Group-*`` at the account level. These apply to the object - not its versions. |
|
| 753 |
* Support for prefix-based inheritance when enforcing permissions. Parent object carrying the authorization directives is reported in ``X-Object-Shared-By``. |
|
| 723 | 754 |
* Large object support with ``X-Object-Manifest``. |
| 724 | 755 |
* Trace the user that created/modified an object with ``X-Object-Modified-By``. |
| 725 | 756 |
|
Also available in: Unified diff