root / docs / dev-guide.rst @ 980190ae
History | View | Annotate | Download (8.2 kB)
1 | bc055d09 | Constantinos Venetsanopoulos | .. _dev-guide: |
---|---|---|---|
2 | bc055d09 | Constantinos Venetsanopoulos | |
3 | bc055d09 | Constantinos Venetsanopoulos | Synnefo Developer's Guide |
4 | bc055d09 | Constantinos Venetsanopoulos | ^^^^^^^^^^^^^^^^^^^^^^^^^ |
5 | bc055d09 | Constantinos Venetsanopoulos | |
6 | 301294a9 | Constantinos Venetsanopoulos | This is the complete Synnefo Developer's Guide. Here we document all Synnefo APIs |
7 | 301294a9 | Constantinos Venetsanopoulos | to allow external developers write independent tools that interact with Synnefo. |
8 | bc055d09 | Constantinos Venetsanopoulos | |
9 | 8f9976c6 | Constantinos Venetsanopoulos | IM API (Astakos) |
10 | 8f9976c6 | Constantinos Venetsanopoulos | ================ |
11 | 8f9976c6 | Constantinos Venetsanopoulos | |
12 | 8f9976c6 | Constantinos Venetsanopoulos | This is the Identity Management API: |
13 | 8f9976c6 | Constantinos Venetsanopoulos | |
14 | 8f9976c6 | Constantinos Venetsanopoulos | .. toctree:: |
15 | 8f9976c6 | Constantinos Venetsanopoulos | :maxdepth: 2 |
16 | 8f9976c6 | Constantinos Venetsanopoulos | |
17 | 8f9976c6 | Constantinos Venetsanopoulos | IM API <astakos-api-guide> |
18 | 8f9976c6 | Constantinos Venetsanopoulos | |
19 | bc055d09 | Constantinos Venetsanopoulos | Compute API (Cyclades) |
20 | bc055d09 | Constantinos Venetsanopoulos | ====================== |
21 | bc055d09 | Constantinos Venetsanopoulos | |
22 | bc055d09 | Constantinos Venetsanopoulos | This is the Cyclades Compute API: |
23 | bc055d09 | Constantinos Venetsanopoulos | |
24 | bc055d09 | Constantinos Venetsanopoulos | .. toctree:: |
25 | bc055d09 | Constantinos Venetsanopoulos | :maxdepth: 2 |
26 | bc055d09 | Constantinos Venetsanopoulos | |
27 | bc055d09 | Constantinos Venetsanopoulos | Compute API <cyclades-api-guide> |
28 | bc055d09 | Constantinos Venetsanopoulos | |
29 | bc055d09 | Constantinos Venetsanopoulos | Network API (Cyclades) |
30 | bc055d09 | Constantinos Venetsanopoulos | ====================== |
31 | bc055d09 | Constantinos Venetsanopoulos | |
32 | 1837aa54 | Giorgos Verigakis | The Network API is currently implemented inside Cyclades. Please consult the |
33 | 1837aa54 | Giorgos Verigakis | :ref:`Compute API <cyclades-api-guide>` for more details. |
34 | 1837aa54 | Giorgos Verigakis | |
35 | bc055d09 | Constantinos Venetsanopoulos | |
36 | bc055d09 | Constantinos Venetsanopoulos | Images API (Plankton) |
37 | bc055d09 | Constantinos Venetsanopoulos | ===================== |
38 | bc055d09 | Constantinos Venetsanopoulos | |
39 | 1837aa54 | Giorgos Verigakis | This is the Plankton Image API: |
40 | 1837aa54 | Giorgos Verigakis | |
41 | 1837aa54 | Giorgos Verigakis | .. toctree:: |
42 | 1837aa54 | Giorgos Verigakis | :maxdepth: 2 |
43 | 1837aa54 | Giorgos Verigakis | |
44 | 1837aa54 | Giorgos Verigakis | Image API <plankton-api-guide> |
45 | 1837aa54 | Giorgos Verigakis | |
46 | bc055d09 | Constantinos Venetsanopoulos | |
47 | 301294a9 | Constantinos Venetsanopoulos | Storage API (Pithos) |
48 | bc055d09 | Constantinos Venetsanopoulos | ===================== |
49 | bc055d09 | Constantinos Venetsanopoulos | |
50 | 301294a9 | Constantinos Venetsanopoulos | This is the Pithos Object Storage API: |
51 | 8f9976c6 | Constantinos Venetsanopoulos | |
52 | 8f9976c6 | Constantinos Venetsanopoulos | .. toctree:: |
53 | 8f9976c6 | Constantinos Venetsanopoulos | :maxdepth: 2 |
54 | 8f9976c6 | Constantinos Venetsanopoulos | |
55 | dad708b4 | Antony Chazapis | Object Storage API <pithos-api-guide> |
56 | 8f9976c6 | Constantinos Venetsanopoulos | |
57 | 8f9976c6 | Constantinos Venetsanopoulos | Implementing new clients |
58 | 8f9976c6 | Constantinos Venetsanopoulos | ======================== |
59 | 8f9976c6 | Constantinos Venetsanopoulos | |
60 | 8f9976c6 | Constantinos Venetsanopoulos | In this section we discuss implementation guidelines, that a developer should |
61 | 8f9976c6 | Constantinos Venetsanopoulos | take into account before writing his own client for the above APIs. Before, |
62 | 8f9976c6 | Constantinos Venetsanopoulos | starting your client implementation, make sure you have thoroughly read the |
63 | 8f9976c6 | Constantinos Venetsanopoulos | corresponding Synnefo API. |
64 | 8f9976c6 | Constantinos Venetsanopoulos | |
65 | 301294a9 | Constantinos Venetsanopoulos | Pithos clients |
66 | 301294a9 | Constantinos Venetsanopoulos | -------------- |
67 | 8f9976c6 | Constantinos Venetsanopoulos | |
68 | 8f9976c6 | Constantinos Venetsanopoulos | User Experience |
69 | 8f9976c6 | Constantinos Venetsanopoulos | ~~~~~~~~~~~~~~~ |
70 | 8f9976c6 | Constantinos Venetsanopoulos | |
71 | 8f9976c6 | Constantinos Venetsanopoulos | Hopefully this API will allow for a multitude of client implementations, each |
72 | 8f9976c6 | Constantinos Venetsanopoulos | supporting a different device or operating system. All clients will be able to |
73 | 8f9976c6 | Constantinos Venetsanopoulos | manipulate containers and objects - even software only designed for OOS API |
74 | 8f9976c6 | Constantinos Venetsanopoulos | compatibility. But a Pithos interface should not be only about showing |
75 | 8f9976c6 | Constantinos Venetsanopoulos | containers and folders. There are some extra user interface elements and |
76 | 8f9976c6 | Constantinos Venetsanopoulos | functionalities that should be common to all implementations. |
77 | 8f9976c6 | Constantinos Venetsanopoulos | |
78 | 8f9976c6 | Constantinos Venetsanopoulos | Upon entrance to the service, a user is presented with the following elements - |
79 | 8f9976c6 | Constantinos Venetsanopoulos | which can be represented as folders or with other related icons: |
80 | 8f9976c6 | Constantinos Venetsanopoulos | |
81 | 8f9976c6 | Constantinos Venetsanopoulos | * The ``home`` element, which is used as the default entry point to the user's |
82 | 8f9976c6 | Constantinos Venetsanopoulos | "files". Objects under ``home`` are represented in the usual hierarchical |
83 | 8f9976c6 | Constantinos Venetsanopoulos | organization of folders and files. |
84 | 8f9976c6 | Constantinos Venetsanopoulos | * The ``trash`` element, which contains files that have been marked for |
85 | 8f9976c6 | Constantinos Venetsanopoulos | deletion, but can still be recovered. |
86 | 8f9976c6 | Constantinos Venetsanopoulos | * The ``shared`` element, which contains all objects shared by the user to |
87 | 8f9976c6 | Constantinos Venetsanopoulos | other users of the system. |
88 | 8f9976c6 | Constantinos Venetsanopoulos | * The ``others`` element, which contains all objects that other users share |
89 | 8f9976c6 | Constantinos Venetsanopoulos | with the user. |
90 | 8f9976c6 | Constantinos Venetsanopoulos | * The ``groups`` element, which contains the names of groups the user has |
91 | 8f9976c6 | Constantinos Venetsanopoulos | defined. Each group consists of a user list. Group creation, deletion, and |
92 | 8f9976c6 | Constantinos Venetsanopoulos | manipulation is carried out by actions originating here. |
93 | 8f9976c6 | Constantinos Venetsanopoulos | * The ``history`` element, which allows browsing past instances of ``home`` |
94 | 8f9976c6 | Constantinos Venetsanopoulos | and - optionally - ``trash``. |
95 | 8f9976c6 | Constantinos Venetsanopoulos | |
96 | 8f9976c6 | Constantinos Venetsanopoulos | Objects in Pithos+ can be: |
97 | 8f9976c6 | Constantinos Venetsanopoulos | |
98 | 8f9976c6 | Constantinos Venetsanopoulos | * Moved to trash and then deleted. |
99 | 8f9976c6 | Constantinos Venetsanopoulos | * Shared with specific permissions. |
100 | 8f9976c6 | Constantinos Venetsanopoulos | * Made public (shared with non-Pithos+ users). |
101 | 8f9976c6 | Constantinos Venetsanopoulos | * Restored from previous versions. |
102 | 8f9976c6 | Constantinos Venetsanopoulos | |
103 | 8f9976c6 | Constantinos Venetsanopoulos | Some of these functions are performed by the client software and some by the |
104 | 8f9976c6 | Constantinos Venetsanopoulos | Pithos+ server. |
105 | 8f9976c6 | Constantinos Venetsanopoulos | |
106 | 8f9976c6 | Constantinos Venetsanopoulos | In the first version of Pithos, objects could also be assigned custom tags. |
107 | 8f9976c6 | Constantinos Venetsanopoulos | This is no longer supported. Existing deployments can migrate tags into a |
108 | 8f9976c6 | Constantinos Venetsanopoulos | specific metadata value, i.e. ``X-Object-Meta-Tags``. |
109 | 8f9976c6 | Constantinos Venetsanopoulos | |
110 | 8f9976c6 | Constantinos Venetsanopoulos | Implementation Guidelines |
111 | 8f9976c6 | Constantinos Venetsanopoulos | ~~~~~~~~~~~~~~~~~~~~~~~~~ |
112 | 8f9976c6 | Constantinos Venetsanopoulos | |
113 | 8f9976c6 | Constantinos Venetsanopoulos | Pithos+ clients should use the ``pithos`` and ``trash`` containers for active |
114 | 8f9976c6 | Constantinos Venetsanopoulos | and inactive objects respectively. If any of these containers is not found, the |
115 | 8f9976c6 | Constantinos Venetsanopoulos | client software should create it, without interrupting the user's workflow. The |
116 | 8f9976c6 | Constantinos Venetsanopoulos | ``home`` element corresponds to ``pithos`` and the ``trash`` element to |
117 | 8f9976c6 | Constantinos Venetsanopoulos | ``trash``. Use ``PUT`` with the ``X-Move-From`` header, or ``MOVE`` to transfer |
118 | 8f9976c6 | Constantinos Venetsanopoulos | objects from one container to the other. Use ``DELETE`` to remove from |
119 | 8f9976c6 | Constantinos Venetsanopoulos | ``pithos`` without trashing, or to remove from ``trash``. When moving objects, |
120 | 8f9976c6 | Constantinos Venetsanopoulos | detect naming conflicts with the ``If-Match`` or ``If-None-Match`` headers. |
121 | 8f9976c6 | Constantinos Venetsanopoulos | Such conflicts should be resolved by the user. |
122 | 8f9976c6 | Constantinos Venetsanopoulos | |
123 | 8f9976c6 | Constantinos Venetsanopoulos | Object names should use the ``/`` delimiter to impose a hierarchy of folders |
124 | 8f9976c6 | Constantinos Venetsanopoulos | and files. |
125 | 8f9976c6 | Constantinos Venetsanopoulos | |
126 | 8f9976c6 | Constantinos Venetsanopoulos | The ``shared`` element should be implemented as a read-only view of the |
127 | 8f9976c6 | Constantinos Venetsanopoulos | ``pithos`` container, using the ``shared`` parameter when listing objects. The |
128 | 8f9976c6 | Constantinos Venetsanopoulos | ``others`` element, should start with a top-level ``GET`` to retrieve the list |
129 | 8f9976c6 | Constantinos Venetsanopoulos | of accounts accessible to the user. It is suggested that the client software |
130 | 8f9976c6 | Constantinos Venetsanopoulos | hides the next step of navigation - the container - if it only includes |
131 | 8f9976c6 | Constantinos Venetsanopoulos | ``pithos`` and forwards the user directly to the objects. |
132 | 8f9976c6 | Constantinos Venetsanopoulos | |
133 | 8f9976c6 | Constantinos Venetsanopoulos | Public objects are not included in ``shared`` and ``others`` listings. It is |
134 | 8f9976c6 | Constantinos Venetsanopoulos | suggested that they are marked in a visually distinctive way in ``pithos`` |
135 | 8f9976c6 | Constantinos Venetsanopoulos | listings (for example using an icon overlay). |
136 | 8f9976c6 | Constantinos Venetsanopoulos | |
137 | 8f9976c6 | Constantinos Venetsanopoulos | A special application menu, or a section in application preferences, should be |
138 | 8f9976c6 | Constantinos Venetsanopoulos | devoted to managing groups (the ``groups`` element). All group-related actions |
139 | 8f9976c6 | Constantinos Venetsanopoulos | are implemented at the account level. |
140 | 8f9976c6 | Constantinos Venetsanopoulos | |
141 | 8f9976c6 | Constantinos Venetsanopoulos | Browsing past versions of objects should be available both at the object and |
142 | 8f9976c6 | Constantinos Venetsanopoulos | the container level. At the object level, a list of past versions can be |
143 | 8f9976c6 | Constantinos Venetsanopoulos | included in the screen showing details or more information on the object |
144 | 8f9976c6 | Constantinos Venetsanopoulos | (metadata, permissions, etc.). At the container level, it is suggested that |
145 | 8f9976c6 | Constantinos Venetsanopoulos | clients use a ``history`` element, which presents to the user a read-only, |
146 | 8f9976c6 | Constantinos Venetsanopoulos | time-variable view of ``pithos`` contents. This can be accomplished via the |
147 | 8f9976c6 | Constantinos Venetsanopoulos | ``until`` parameter in listings. Optionally, ``history`` may include ``trash``. |
148 | 8f9976c6 | Constantinos Venetsanopoulos | |
149 | 8f9976c6 | Constantinos Venetsanopoulos | Uploading and downloading data |
150 | 8f9976c6 | Constantinos Venetsanopoulos | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
151 | 8f9976c6 | Constantinos Venetsanopoulos | |
152 | 8f9976c6 | Constantinos Venetsanopoulos | By using hashmaps to upload and download objects the corresponding operations |
153 | 8f9976c6 | Constantinos Venetsanopoulos | can complete much faster. |
154 | 8f9976c6 | Constantinos Venetsanopoulos | |
155 | 8f9976c6 | Constantinos Venetsanopoulos | In the case of an upload, only the missing blocks will be submitted to the |
156 | 8f9976c6 | Constantinos Venetsanopoulos | server: |
157 | 8f9976c6 | Constantinos Venetsanopoulos | |
158 | 8f9976c6 | Constantinos Venetsanopoulos | * Calculate the hash value for each block of the object to be uploaded. Use |
159 | 8f9976c6 | Constantinos Venetsanopoulos | the hash algorithm and block size of the destination container. |
160 | 8f9976c6 | Constantinos Venetsanopoulos | * Send a hashmap ``PUT`` request for the object. |
161 | 8f9976c6 | Constantinos Venetsanopoulos | |
162 | 8f9976c6 | Constantinos Venetsanopoulos | * Server responds with status ``201`` (Created): |
163 | 8f9976c6 | Constantinos Venetsanopoulos | |
164 | 8f9976c6 | Constantinos Venetsanopoulos | * Blocks are already on the server. The object has been created. Done. |
165 | 8f9976c6 | Constantinos Venetsanopoulos | |
166 | 8f9976c6 | Constantinos Venetsanopoulos | * Server responds with status ``409`` (Conflict): |
167 | 8f9976c6 | Constantinos Venetsanopoulos | |
168 | 8f9976c6 | Constantinos Venetsanopoulos | * Server's response body contains the hashes of the blocks that do not |
169 | 8f9976c6 | Constantinos Venetsanopoulos | exist on the server. |
170 | 8f9976c6 | Constantinos Venetsanopoulos | * For each hash value in the server's response (or all hashes together): |
171 | 8f9976c6 | Constantinos Venetsanopoulos | |
172 | 8f9976c6 | Constantinos Venetsanopoulos | * Send a ``POST`` request to the destination container with the |
173 | 8f9976c6 | Constantinos Venetsanopoulos | corresponding data. |
174 | 8f9976c6 | Constantinos Venetsanopoulos | |
175 | 8f9976c6 | Constantinos Venetsanopoulos | * Repeat hashmap ``PUT``. Fail if the server's response is not ``201``. |
176 | 8f9976c6 | Constantinos Venetsanopoulos | |
177 | 8f9976c6 | Constantinos Venetsanopoulos | Consulting hashmaps when downloading allows for resuming partially transferred |
178 | 8f9976c6 | Constantinos Venetsanopoulos | objects. The client should retrieve the hashmap from the server and compare it |
179 | 8f9976c6 | Constantinos Venetsanopoulos | with the hashmap computed from the respective local file. Any missing parts can |
180 | 8f9976c6 | Constantinos Venetsanopoulos | be downloaded with ``GET`` requests with the additional ``Range`` header. |
181 | 8f9976c6 | Constantinos Venetsanopoulos | |
182 | 8f9976c6 | Constantinos Venetsanopoulos | Syncing |
183 | 8f9976c6 | Constantinos Venetsanopoulos | ~~~~~~~ |
184 | 8f9976c6 | Constantinos Venetsanopoulos | |
185 | 8f9976c6 | Constantinos Venetsanopoulos | Consider the following algorithm for synchronizing a local folder with the |
186 | 8f9976c6 | Constantinos Venetsanopoulos | server. The "state" is the complete object listing, with the corresponding |
187 | 8f9976c6 | Constantinos Venetsanopoulos | attributes. |
188 | 8f9976c6 | Constantinos Venetsanopoulos | |
189 | 30aae6bf | Giorgos Verigakis | .. code-block:: python |
190 | 30aae6bf | Giorgos Verigakis | |
191 | 30aae6bf | Giorgos Verigakis | # L: Local State, the last synced state of the object. |
192 | 30aae6bf | Giorgos Verigakis | # Stored locally (e.g. in an SQLite database) |
193 | 30aae6bf | Giorgos Verigakis | |
194 | 30aae6bf | Giorgos Verigakis | # C: Current State, the current local state of the object |
195 | 30aae6bf | Giorgos Verigakis | # Returned by the filesystem |
196 | 30aae6bf | Giorgos Verigakis | |
197 | 30aae6bf | Giorgos Verigakis | # S: Server State, the current server state of the object |
198 | 30aae6bf | Giorgos Verigakis | # Returned by the server (HTTP request) |
199 | 30aae6bf | Giorgos Verigakis | |
200 | 30aae6bf | Giorgos Verigakis | def sync(path): |
201 | 30aae6bf | Giorgos Verigakis | L = get_local_state(path) # Database action |
202 | 30aae6bf | Giorgos Verigakis | C = get_current_state(path) # Filesystem action |
203 | 30aae6bf | Giorgos Verigakis | S = get_server_state(path) # Network action |
204 | 30aae6bf | Giorgos Verigakis | |
205 | 30aae6bf | Giorgos Verigakis | if C == L: |
206 | 30aae6bf | Giorgos Verigakis | # No local changes |
207 | 30aae6bf | Giorgos Verigakis | if S == L: |
208 | 30aae6bf | Giorgos Verigakis | # No remote changes, nothing to do |
209 | 30aae6bf | Giorgos Verigakis | return |
210 | 30aae6bf | Giorgos Verigakis | else: |
211 | 30aae6bf | Giorgos Verigakis | # Update local state to match that of the server |
212 | 30aae6bf | Giorgos Verigakis | download(path) |
213 | 30aae6bf | Giorgos Verigakis | update_local_state(path, S) |
214 | 8f9976c6 | Constantinos Venetsanopoulos | else: |
215 | 30aae6bf | Giorgos Verigakis | # Local changes exist |
216 | 30aae6bf | Giorgos Verigakis | if S == L: |
217 | 30aae6bf | Giorgos Verigakis | # No remote changes, update the server and the local state |
218 | 30aae6bf | Giorgos Verigakis | upload(path) |
219 | 30aae6bf | Giorgos Verigakis | update_local_state(path, C) |
220 | 8f9976c6 | Constantinos Venetsanopoulos | else: |
221 | 30aae6bf | Giorgos Verigakis | # Both local and server changes exist |
222 | 30aae6bf | Giorgos Verigakis | if C == S: |
223 | 30aae6bf | Giorgos Verigakis | # We were lucky, both did the same |
224 | 30aae6bf | Giorgos Verigakis | update_local_state(path, C) |
225 | 30aae6bf | Giorgos Verigakis | else: |
226 | 30aae6bf | Giorgos Verigakis | # Conflicting changes exist |
227 | 30aae6bf | Giorgos Verigakis | conflict() |
228 | 30aae6bf | Giorgos Verigakis | |
229 | 8f9976c6 | Constantinos Venetsanopoulos | |
230 | 8f9976c6 | Constantinos Venetsanopoulos | Notes: |
231 | 8f9976c6 | Constantinos Venetsanopoulos | |
232 | 8f9976c6 | Constantinos Venetsanopoulos | * States represent file hashes (it is suggested to use Merkle). Deleted or |
233 | 8f9976c6 | Constantinos Venetsanopoulos | non-existing files are assumed to have a magic hash (e.g. empty string). |