root / docs / dev-guide.rst @ 382ca20a
History | View | Annotate | Download (8.2 kB)
1 | bc055d09 | Constantinos Venetsanopoulos | .. _dev-guide: |
---|---|---|---|
2 | bc055d09 | Constantinos Venetsanopoulos | |
3 | bc055d09 | Constantinos Venetsanopoulos | Synnefo Developer's Guide |
4 | bc055d09 | Constantinos Venetsanopoulos | ^^^^^^^^^^^^^^^^^^^^^^^^^ |
5 | bc055d09 | Constantinos Venetsanopoulos | |
6 | bc055d09 | Constantinos Venetsanopoulos | This is the complete Synnefo Developer's Guide |
7 | bc055d09 | Constantinos Venetsanopoulos | |
8 | bc055d09 | Constantinos Venetsanopoulos | Tying it all up with kamaki |
9 | bc055d09 | Constantinos Venetsanopoulos | =========================== |
10 | bc055d09 | Constantinos Venetsanopoulos | |
11 | bc055d09 | Constantinos Venetsanopoulos | kamaki |
12 | bc055d09 | Constantinos Venetsanopoulos | ------ |
13 | bc055d09 | Constantinos Venetsanopoulos | |
14 | 8f9976c6 | Constantinos Venetsanopoulos | IM API (Astakos) |
15 | 8f9976c6 | Constantinos Venetsanopoulos | ================ |
16 | 8f9976c6 | Constantinos Venetsanopoulos | |
17 | 8f9976c6 | Constantinos Venetsanopoulos | This is the Identity Management API: |
18 | 8f9976c6 | Constantinos Venetsanopoulos | |
19 | 8f9976c6 | Constantinos Venetsanopoulos | .. toctree:: |
20 | 8f9976c6 | Constantinos Venetsanopoulos | :maxdepth: 2 |
21 | 8f9976c6 | Constantinos Venetsanopoulos | |
22 | 8f9976c6 | Constantinos Venetsanopoulos | IM API <astakos-api-guide> |
23 | 8f9976c6 | Constantinos Venetsanopoulos | |
24 | bc055d09 | Constantinos Venetsanopoulos | Compute API (Cyclades) |
25 | bc055d09 | Constantinos Venetsanopoulos | ====================== |
26 | bc055d09 | Constantinos Venetsanopoulos | |
27 | bc055d09 | Constantinos Venetsanopoulos | This is the Cyclades Compute API: |
28 | bc055d09 | Constantinos Venetsanopoulos | |
29 | bc055d09 | Constantinos Venetsanopoulos | .. toctree:: |
30 | bc055d09 | Constantinos Venetsanopoulos | :maxdepth: 2 |
31 | bc055d09 | Constantinos Venetsanopoulos | |
32 | bc055d09 | Constantinos Venetsanopoulos | Compute API <cyclades-api-guide> |
33 | bc055d09 | Constantinos Venetsanopoulos | |
34 | bc055d09 | Constantinos Venetsanopoulos | Network API (Cyclades) |
35 | bc055d09 | Constantinos Venetsanopoulos | ====================== |
36 | bc055d09 | Constantinos Venetsanopoulos | |
37 | 1837aa54 | Giorgos Verigakis | The Network API is currently implemented inside Cyclades. Please consult the |
38 | 1837aa54 | Giorgos Verigakis | :ref:`Compute API <cyclades-api-guide>` for more details. |
39 | 1837aa54 | Giorgos Verigakis | |
40 | bc055d09 | Constantinos Venetsanopoulos | |
41 | bc055d09 | Constantinos Venetsanopoulos | Images API (Plankton) |
42 | bc055d09 | Constantinos Venetsanopoulos | ===================== |
43 | bc055d09 | Constantinos Venetsanopoulos | |
44 | 1837aa54 | Giorgos Verigakis | This is the Plankton Image API: |
45 | 1837aa54 | Giorgos Verigakis | |
46 | 1837aa54 | Giorgos Verigakis | .. toctree:: |
47 | 1837aa54 | Giorgos Verigakis | :maxdepth: 2 |
48 | 1837aa54 | Giorgos Verigakis | |
49 | 1837aa54 | Giorgos Verigakis | Image API <plankton-api-guide> |
50 | 1837aa54 | Giorgos Verigakis | |
51 | bc055d09 | Constantinos Venetsanopoulos | |
52 | bc055d09 | Constantinos Venetsanopoulos | Storage API (Pithos+) |
53 | bc055d09 | Constantinos Venetsanopoulos | ===================== |
54 | bc055d09 | Constantinos Venetsanopoulos | |
55 | dad708b4 | Antony Chazapis | This is the Pithos+ Object Storage API: |
56 | 8f9976c6 | Constantinos Venetsanopoulos | |
57 | 8f9976c6 | Constantinos Venetsanopoulos | .. toctree:: |
58 | 8f9976c6 | Constantinos Venetsanopoulos | :maxdepth: 2 |
59 | 8f9976c6 | Constantinos Venetsanopoulos | |
60 | dad708b4 | Antony Chazapis | Object Storage API <pithos-api-guide> |
61 | 8f9976c6 | Constantinos Venetsanopoulos | |
62 | 8f9976c6 | Constantinos Venetsanopoulos | Implementing new clients |
63 | 8f9976c6 | Constantinos Venetsanopoulos | ======================== |
64 | 8f9976c6 | Constantinos Venetsanopoulos | |
65 | 8f9976c6 | Constantinos Venetsanopoulos | In this section we discuss implementation guidelines, that a developer should |
66 | 8f9976c6 | Constantinos Venetsanopoulos | take into account before writing his own client for the above APIs. Before, |
67 | 8f9976c6 | Constantinos Venetsanopoulos | starting your client implementation, make sure you have thoroughly read the |
68 | 8f9976c6 | Constantinos Venetsanopoulos | corresponding Synnefo API. |
69 | 8f9976c6 | Constantinos Venetsanopoulos | |
70 | 8f9976c6 | Constantinos Venetsanopoulos | Pithos+ clients |
71 | 8f9976c6 | Constantinos Venetsanopoulos | --------------- |
72 | 8f9976c6 | Constantinos Venetsanopoulos | |
73 | 8f9976c6 | Constantinos Venetsanopoulos | User Experience |
74 | 8f9976c6 | Constantinos Venetsanopoulos | ~~~~~~~~~~~~~~~ |
75 | 8f9976c6 | Constantinos Venetsanopoulos | |
76 | 8f9976c6 | Constantinos Venetsanopoulos | Hopefully this API will allow for a multitude of client implementations, each |
77 | 8f9976c6 | Constantinos Venetsanopoulos | supporting a different device or operating system. All clients will be able to |
78 | 8f9976c6 | Constantinos Venetsanopoulos | manipulate containers and objects - even software only designed for OOS API |
79 | 8f9976c6 | Constantinos Venetsanopoulos | compatibility. But a Pithos interface should not be only about showing |
80 | 8f9976c6 | Constantinos Venetsanopoulos | containers and folders. There are some extra user interface elements and |
81 | 8f9976c6 | Constantinos Venetsanopoulos | functionalities that should be common to all implementations. |
82 | 8f9976c6 | Constantinos Venetsanopoulos | |
83 | 8f9976c6 | Constantinos Venetsanopoulos | Upon entrance to the service, a user is presented with the following elements - |
84 | 8f9976c6 | Constantinos Venetsanopoulos | which can be represented as folders or with other related icons: |
85 | 8f9976c6 | Constantinos Venetsanopoulos | |
86 | 8f9976c6 | Constantinos Venetsanopoulos | * The ``home`` element, which is used as the default entry point to the user's |
87 | 8f9976c6 | Constantinos Venetsanopoulos | "files". Objects under ``home`` are represented in the usual hierarchical |
88 | 8f9976c6 | Constantinos Venetsanopoulos | organization of folders and files. |
89 | 8f9976c6 | Constantinos Venetsanopoulos | * The ``trash`` element, which contains files that have been marked for |
90 | 8f9976c6 | Constantinos Venetsanopoulos | deletion, but can still be recovered. |
91 | 8f9976c6 | Constantinos Venetsanopoulos | * The ``shared`` element, which contains all objects shared by the user to |
92 | 8f9976c6 | Constantinos Venetsanopoulos | other users of the system. |
93 | 8f9976c6 | Constantinos Venetsanopoulos | * The ``others`` element, which contains all objects that other users share |
94 | 8f9976c6 | Constantinos Venetsanopoulos | with the user. |
95 | 8f9976c6 | Constantinos Venetsanopoulos | * The ``groups`` element, which contains the names of groups the user has |
96 | 8f9976c6 | Constantinos Venetsanopoulos | defined. Each group consists of a user list. Group creation, deletion, and |
97 | 8f9976c6 | Constantinos Venetsanopoulos | manipulation is carried out by actions originating here. |
98 | 8f9976c6 | Constantinos Venetsanopoulos | * The ``history`` element, which allows browsing past instances of ``home`` |
99 | 8f9976c6 | Constantinos Venetsanopoulos | and - optionally - ``trash``. |
100 | 8f9976c6 | Constantinos Venetsanopoulos | |
101 | 8f9976c6 | Constantinos Venetsanopoulos | Objects in Pithos+ can be: |
102 | 8f9976c6 | Constantinos Venetsanopoulos | |
103 | 8f9976c6 | Constantinos Venetsanopoulos | * Moved to trash and then deleted. |
104 | 8f9976c6 | Constantinos Venetsanopoulos | * Shared with specific permissions. |
105 | 8f9976c6 | Constantinos Venetsanopoulos | * Made public (shared with non-Pithos+ users). |
106 | 8f9976c6 | Constantinos Venetsanopoulos | * Restored from previous versions. |
107 | 8f9976c6 | Constantinos Venetsanopoulos | |
108 | 8f9976c6 | Constantinos Venetsanopoulos | Some of these functions are performed by the client software and some by the |
109 | 8f9976c6 | Constantinos Venetsanopoulos | Pithos+ server. |
110 | 8f9976c6 | Constantinos Venetsanopoulos | |
111 | 8f9976c6 | Constantinos Venetsanopoulos | In the first version of Pithos, objects could also be assigned custom tags. |
112 | 8f9976c6 | Constantinos Venetsanopoulos | This is no longer supported. Existing deployments can migrate tags into a |
113 | 8f9976c6 | Constantinos Venetsanopoulos | specific metadata value, i.e. ``X-Object-Meta-Tags``. |
114 | 8f9976c6 | Constantinos Venetsanopoulos | |
115 | 8f9976c6 | Constantinos Venetsanopoulos | Implementation Guidelines |
116 | 8f9976c6 | Constantinos Venetsanopoulos | ~~~~~~~~~~~~~~~~~~~~~~~~~ |
117 | 8f9976c6 | Constantinos Venetsanopoulos | |
118 | 8f9976c6 | Constantinos Venetsanopoulos | Pithos+ clients should use the ``pithos`` and ``trash`` containers for active |
119 | 8f9976c6 | Constantinos Venetsanopoulos | and inactive objects respectively. If any of these containers is not found, the |
120 | 8f9976c6 | Constantinos Venetsanopoulos | client software should create it, without interrupting the user's workflow. The |
121 | 8f9976c6 | Constantinos Venetsanopoulos | ``home`` element corresponds to ``pithos`` and the ``trash`` element to |
122 | 8f9976c6 | Constantinos Venetsanopoulos | ``trash``. Use ``PUT`` with the ``X-Move-From`` header, or ``MOVE`` to transfer |
123 | 8f9976c6 | Constantinos Venetsanopoulos | objects from one container to the other. Use ``DELETE`` to remove from |
124 | 8f9976c6 | Constantinos Venetsanopoulos | ``pithos`` without trashing, or to remove from ``trash``. When moving objects, |
125 | 8f9976c6 | Constantinos Venetsanopoulos | detect naming conflicts with the ``If-Match`` or ``If-None-Match`` headers. |
126 | 8f9976c6 | Constantinos Venetsanopoulos | Such conflicts should be resolved by the user. |
127 | 8f9976c6 | Constantinos Venetsanopoulos | |
128 | 8f9976c6 | Constantinos Venetsanopoulos | Object names should use the ``/`` delimiter to impose a hierarchy of folders |
129 | 8f9976c6 | Constantinos Venetsanopoulos | and files. |
130 | 8f9976c6 | Constantinos Venetsanopoulos | |
131 | 8f9976c6 | Constantinos Venetsanopoulos | The ``shared`` element should be implemented as a read-only view of the |
132 | 8f9976c6 | Constantinos Venetsanopoulos | ``pithos`` container, using the ``shared`` parameter when listing objects. The |
133 | 8f9976c6 | Constantinos Venetsanopoulos | ``others`` element, should start with a top-level ``GET`` to retrieve the list |
134 | 8f9976c6 | Constantinos Venetsanopoulos | of accounts accessible to the user. It is suggested that the client software |
135 | 8f9976c6 | Constantinos Venetsanopoulos | hides the next step of navigation - the container - if it only includes |
136 | 8f9976c6 | Constantinos Venetsanopoulos | ``pithos`` and forwards the user directly to the objects. |
137 | 8f9976c6 | Constantinos Venetsanopoulos | |
138 | 8f9976c6 | Constantinos Venetsanopoulos | Public objects are not included in ``shared`` and ``others`` listings. It is |
139 | 8f9976c6 | Constantinos Venetsanopoulos | suggested that they are marked in a visually distinctive way in ``pithos`` |
140 | 8f9976c6 | Constantinos Venetsanopoulos | listings (for example using an icon overlay). |
141 | 8f9976c6 | Constantinos Venetsanopoulos | |
142 | 8f9976c6 | Constantinos Venetsanopoulos | A special application menu, or a section in application preferences, should be |
143 | 8f9976c6 | Constantinos Venetsanopoulos | devoted to managing groups (the ``groups`` element). All group-related actions |
144 | 8f9976c6 | Constantinos Venetsanopoulos | are implemented at the account level. |
145 | 8f9976c6 | Constantinos Venetsanopoulos | |
146 | 8f9976c6 | Constantinos Venetsanopoulos | Browsing past versions of objects should be available both at the object and |
147 | 8f9976c6 | Constantinos Venetsanopoulos | the container level. At the object level, a list of past versions can be |
148 | 8f9976c6 | Constantinos Venetsanopoulos | included in the screen showing details or more information on the object |
149 | 8f9976c6 | Constantinos Venetsanopoulos | (metadata, permissions, etc.). At the container level, it is suggested that |
150 | 8f9976c6 | Constantinos Venetsanopoulos | clients use a ``history`` element, which presents to the user a read-only, |
151 | 8f9976c6 | Constantinos Venetsanopoulos | time-variable view of ``pithos`` contents. This can be accomplished via the |
152 | 8f9976c6 | Constantinos Venetsanopoulos | ``until`` parameter in listings. Optionally, ``history`` may include ``trash``. |
153 | 8f9976c6 | Constantinos Venetsanopoulos | |
154 | 8f9976c6 | Constantinos Venetsanopoulos | Uploading and downloading data |
155 | 8f9976c6 | Constantinos Venetsanopoulos | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
156 | 8f9976c6 | Constantinos Venetsanopoulos | |
157 | 8f9976c6 | Constantinos Venetsanopoulos | By using hashmaps to upload and download objects the corresponding operations |
158 | 8f9976c6 | Constantinos Venetsanopoulos | can complete much faster. |
159 | 8f9976c6 | Constantinos Venetsanopoulos | |
160 | 8f9976c6 | Constantinos Venetsanopoulos | In the case of an upload, only the missing blocks will be submitted to the |
161 | 8f9976c6 | Constantinos Venetsanopoulos | server: |
162 | 8f9976c6 | Constantinos Venetsanopoulos | |
163 | 8f9976c6 | Constantinos Venetsanopoulos | * Calculate the hash value for each block of the object to be uploaded. Use |
164 | 8f9976c6 | Constantinos Venetsanopoulos | the hash algorithm and block size of the destination container. |
165 | 8f9976c6 | Constantinos Venetsanopoulos | * Send a hashmap ``PUT`` request for the object. |
166 | 8f9976c6 | Constantinos Venetsanopoulos | |
167 | 8f9976c6 | Constantinos Venetsanopoulos | * Server responds with status ``201`` (Created): |
168 | 8f9976c6 | Constantinos Venetsanopoulos | |
169 | 8f9976c6 | Constantinos Venetsanopoulos | * Blocks are already on the server. The object has been created. Done. |
170 | 8f9976c6 | Constantinos Venetsanopoulos | |
171 | 8f9976c6 | Constantinos Venetsanopoulos | * Server responds with status ``409`` (Conflict): |
172 | 8f9976c6 | Constantinos Venetsanopoulos | |
173 | 8f9976c6 | Constantinos Venetsanopoulos | * Server's response body contains the hashes of the blocks that do not |
174 | 8f9976c6 | Constantinos Venetsanopoulos | exist on the server. |
175 | 8f9976c6 | Constantinos Venetsanopoulos | * For each hash value in the server's response (or all hashes together): |
176 | 8f9976c6 | Constantinos Venetsanopoulos | |
177 | 8f9976c6 | Constantinos Venetsanopoulos | * Send a ``POST`` request to the destination container with the |
178 | 8f9976c6 | Constantinos Venetsanopoulos | corresponding data. |
179 | 8f9976c6 | Constantinos Venetsanopoulos | |
180 | 8f9976c6 | Constantinos Venetsanopoulos | * Repeat hashmap ``PUT``. Fail if the server's response is not ``201``. |
181 | 8f9976c6 | Constantinos Venetsanopoulos | |
182 | 8f9976c6 | Constantinos Venetsanopoulos | Consulting hashmaps when downloading allows for resuming partially transferred |
183 | 8f9976c6 | Constantinos Venetsanopoulos | objects. The client should retrieve the hashmap from the server and compare it |
184 | 8f9976c6 | Constantinos Venetsanopoulos | with the hashmap computed from the respective local file. Any missing parts can |
185 | 8f9976c6 | Constantinos Venetsanopoulos | be downloaded with ``GET`` requests with the additional ``Range`` header. |
186 | 8f9976c6 | Constantinos Venetsanopoulos | |
187 | 8f9976c6 | Constantinos Venetsanopoulos | Syncing |
188 | 8f9976c6 | Constantinos Venetsanopoulos | ~~~~~~~ |
189 | 8f9976c6 | Constantinos Venetsanopoulos | |
190 | 8f9976c6 | Constantinos Venetsanopoulos | Consider the following algorithm for synchronizing a local folder with the |
191 | 8f9976c6 | Constantinos Venetsanopoulos | server. The "state" is the complete object listing, with the corresponding |
192 | 8f9976c6 | Constantinos Venetsanopoulos | attributes. |
193 | 8f9976c6 | Constantinos Venetsanopoulos | |
194 | 30aae6bf | Giorgos Verigakis | .. code-block:: python |
195 | 30aae6bf | Giorgos Verigakis | |
196 | 30aae6bf | Giorgos Verigakis | # L: Local State, the last synced state of the object. |
197 | 30aae6bf | Giorgos Verigakis | # Stored locally (e.g. in an SQLite database) |
198 | 30aae6bf | Giorgos Verigakis | |
199 | 30aae6bf | Giorgos Verigakis | # C: Current State, the current local state of the object |
200 | 30aae6bf | Giorgos Verigakis | # Returned by the filesystem |
201 | 30aae6bf | Giorgos Verigakis | |
202 | 30aae6bf | Giorgos Verigakis | # S: Server State, the current server state of the object |
203 | 30aae6bf | Giorgos Verigakis | # Returned by the server (HTTP request) |
204 | 30aae6bf | Giorgos Verigakis | |
205 | 30aae6bf | Giorgos Verigakis | def sync(path): |
206 | 30aae6bf | Giorgos Verigakis | L = get_local_state(path) # Database action |
207 | 30aae6bf | Giorgos Verigakis | C = get_current_state(path) # Filesystem action |
208 | 30aae6bf | Giorgos Verigakis | S = get_server_state(path) # Network action |
209 | 30aae6bf | Giorgos Verigakis | |
210 | 30aae6bf | Giorgos Verigakis | if C == L: |
211 | 30aae6bf | Giorgos Verigakis | # No local changes |
212 | 30aae6bf | Giorgos Verigakis | if S == L: |
213 | 30aae6bf | Giorgos Verigakis | # No remote changes, nothing to do |
214 | 30aae6bf | Giorgos Verigakis | return |
215 | 30aae6bf | Giorgos Verigakis | else: |
216 | 30aae6bf | Giorgos Verigakis | # Update local state to match that of the server |
217 | 30aae6bf | Giorgos Verigakis | download(path) |
218 | 30aae6bf | Giorgos Verigakis | update_local_state(path, S) |
219 | 8f9976c6 | Constantinos Venetsanopoulos | else: |
220 | 30aae6bf | Giorgos Verigakis | # Local changes exist |
221 | 30aae6bf | Giorgos Verigakis | if S == L: |
222 | 30aae6bf | Giorgos Verigakis | # No remote changes, update the server and the local state |
223 | 30aae6bf | Giorgos Verigakis | upload(path) |
224 | 30aae6bf | Giorgos Verigakis | update_local_state(path, C) |
225 | 8f9976c6 | Constantinos Venetsanopoulos | else: |
226 | 30aae6bf | Giorgos Verigakis | # Both local and server changes exist |
227 | 30aae6bf | Giorgos Verigakis | if C == S: |
228 | 30aae6bf | Giorgos Verigakis | # We were lucky, both did the same |
229 | 30aae6bf | Giorgos Verigakis | update_local_state(path, C) |
230 | 30aae6bf | Giorgos Verigakis | else: |
231 | 30aae6bf | Giorgos Verigakis | # Conflicting changes exist |
232 | 30aae6bf | Giorgos Verigakis | conflict() |
233 | 30aae6bf | Giorgos Verigakis | |
234 | 8f9976c6 | Constantinos Venetsanopoulos | |
235 | 8f9976c6 | Constantinos Venetsanopoulos | Notes: |
236 | 8f9976c6 | Constantinos Venetsanopoulos | |
237 | 8f9976c6 | Constantinos Venetsanopoulos | * States represent file hashes (it is suggested to use Merkle). Deleted or |
238 | 8f9976c6 | Constantinos Venetsanopoulos | non-existing files are assumed to have a magic hash (e.g. empty string). |