Revision 3df48a3f docs/dev-guide.rst
b/docs/dev-guide.rst | ||
---|---|---|
25 | 25 |
|
26 | 26 |
.. code-block:: console |
27 | 27 |
|
28 |
# pip install gitpython
|
|
28 |
# pip install gitpython
|
|
29 | 29 |
|
30 | 30 |
**4. Install devflow** |
31 | 31 |
|
... | ... | |
34 | 34 |
|
35 | 35 |
.. code-block:: console |
36 | 36 |
|
37 |
# pip install devflow
|
|
37 |
# pip install devflow
|
|
38 | 38 |
|
39 | 39 |
**5. Get Synnefo code** |
40 | 40 |
|
... | ... | |
42 | 42 |
|
43 | 43 |
.. code-block:: console |
44 | 44 |
|
45 |
# apt-get install git
|
|
45 |
# apt-get install git
|
|
46 | 46 |
|
47 | 47 |
And now get the Synnefo code from the official Synnefo repository |
48 | 48 |
|
49 | 49 |
.. code-block:: console |
50 | 50 |
|
51 |
# su some_regular_user
|
|
52 |
$ git clone https://code.grnet.gr/git/synnefo
|
|
51 |
# su some_regular_user |
|
52 |
$ git clone https://code.grnet.gr/git/synnefo
|
|
53 | 53 |
|
54 | 54 |
Make sure you clone the repository as a regular user. Otherwise you will |
55 | 55 |
have problems with file permissions when deploying. |
... | ... | |
60 | 60 |
|
61 | 61 |
.. code-block:: console |
62 | 62 |
|
63 |
$ devflow-update-version
|
|
63 |
$ devflow-update-version
|
|
64 | 64 |
|
65 | 65 |
2. Code |
66 | 66 |
3. In every component you change, run as root |
67 | 67 |
|
68 | 68 |
.. code-block:: console |
69 | 69 |
|
70 |
# python setup.py develop -N
|
|
70 |
# python setup.py develop -N
|
|
71 | 71 |
|
72 | 72 |
This does not automatically install dependencies, in order to avoid |
73 | 73 |
confusion with synnefo packages installed by ``snf-deploy``. External |
... | ... | |
78 | 78 |
|
79 | 79 |
.. code-block:: console |
80 | 80 |
|
81 |
# service gunicorn restart
|
|
81 |
# service gunicorn restart |
|
82 | 82 |
|
83 | 83 |
5. If your changes affected ``snf-dispatcher`` (from package |
84 | 84 |
``snf-cyclades-app``) or ``snf-ganeti-eventd`` (from |
... | ... | |
89 | 89 |
|
90 | 90 |
.. code-block:: console |
91 | 91 |
|
92 |
# ln -sf /usr/local/bin/snf-dispatcher /usr/bin/snf-dispatcher
|
|
92 |
# ln -sf /usr/local/bin/snf-dispatcher /usr/bin/snf-dispatcher |
|
93 | 93 |
|
94 | 94 |
and then restart the daemons |
95 | 95 |
|
96 | 96 |
.. code-block:: console |
97 | 97 |
|
98 |
# service snf-dispatcher restart
|
|
99 |
# service snf-ganeti-eventd restart
|
|
98 |
# service snf-dispatcher restart |
|
99 |
# service snf-ganeti-eventd restart |
|
100 | 100 |
|
101 | 101 |
6. Refresh the web page and see your changes |
102 |
|
|
103 |
Synnefo REST APIs |
|
104 |
================= |
|
105 |
|
|
106 |
Here, we document all Synnefo |
|
107 |
REST APIs, to allow external developers write independent tools that interact |
|
108 |
with Synnefo. |
|
109 |
|
|
110 |
Synnefo exposes the OpenStack APIs for all its operations. Also, extensions |
|
111 |
have been written for advanced operations wherever needed, and minor changes |
|
112 |
for things that were missing or change frequently. |
|
113 |
|
|
114 |
Most Synnefo services have a corresponding OpenStack API: |
|
115 |
|
|
116 |
| Cyclades/Compute Service -> OpenStack Compute API |
|
117 |
| Cyclades/Network Service -> OpenStack Compute/Network API (not Quantum yet) |
|
118 |
| Cyclades/Image Service -> OpenStack Glance API |
|
119 |
| Pithos/Storage Service -> OpenStack Object Store API |
|
120 |
| Astakos/Identity Service -> Proprietary, moving to OpenStack Keystone API |
|
121 |
| Astakos/Quota Service -> Proprietary API |
|
122 |
| Astakos/Resource Service -> Proprietary API |
|
123 |
|
|
124 |
Below, we will describe all Synnefo APIs with conjuction to the OpenStack APIs. |
|
125 |
|
|
126 |
|
|
127 |
Identity Service API (Astakos) |
|
128 |
============================== |
|
129 |
|
|
130 |
Currently, Astakos which is the Identity Management Service of Synnefo, has a |
|
131 |
proprietary API, but we are moving to the OpenStack Keystone API. |
|
132 |
|
|
133 |
The current Identity Management API is: |
|
134 |
|
|
135 |
.. toctree:: |
|
136 |
:maxdepth: 2 |
|
137 |
|
|
138 |
Identity API <astakos-api-guide> |
|
139 |
|
|
140 |
|
|
141 |
Resource and Quota Service API (Astakos) |
|
142 |
======================================== |
|
143 |
|
|
144 |
.. toctree:: |
|
145 |
:maxdepth: 2 |
|
146 |
|
|
147 |
Resource and Quota API <quota-api-guide.rst> |
|
148 |
|
|
149 |
Project API |
|
150 |
=========== |
|
151 |
|
|
152 |
.. toctree:: |
|
153 |
:maxdepth: 2 |
|
154 |
|
|
155 |
Project API <project-api-guide.rst> |
|
156 |
|
|
157 |
Compute Service API (Cyclades) |
|
158 |
============================== |
|
159 |
|
|
160 |
The Compute part of Cyclades exposes the OpenStack Compute API with minor |
|
161 |
changes wherever needed. |
|
162 |
|
|
163 |
This is the Cyclades/Compute API: |
|
164 |
|
|
165 |
.. toctree:: |
|
166 |
:maxdepth: 2 |
|
167 |
|
|
168 |
Compute API <cyclades-api-guide> |
|
169 |
|
|
170 |
|
|
171 |
Network Service API (Cyclades) |
|
172 |
============================== |
|
173 |
|
|
174 |
The Network Service is implemented inside Cyclades. It exposes the part of the |
|
175 |
OpenStack Compute API that has to do with Networks. The OpenStack Quantum API |
|
176 |
is not implemented yet. |
|
177 |
|
|
178 |
Please consult the :ref:`Cyclades/Network API <cyclades-api-guide>` for more |
|
179 |
details. |
|
180 |
|
|
181 |
|
|
182 |
Image Service API (Cyclades) |
|
183 |
============================ |
|
184 |
|
|
185 |
The Image Service is implemented inside Cyclades. It exposes the OpenStack |
|
186 |
Glance API with minor changes wherever needed. |
|
187 |
|
|
188 |
This is the Cyclades/Image API: |
|
189 |
|
|
190 |
.. toctree:: |
|
191 |
:maxdepth: 2 |
|
192 |
|
|
193 |
Image API <plankton-api-guide> |
|
194 |
|
|
195 |
|
|
196 |
Storage Service API (Pithos) |
|
197 |
============================ |
|
198 |
|
|
199 |
Pithos is the Storage Service of Synnefo and it exposes the OpenStack Object |
|
200 |
Storage API with extensions for advanced operations, e.g., syncing. |
|
201 |
|
|
202 |
This is the Pithos Object Storage API: |
|
203 |
|
|
204 |
.. toctree:: |
|
205 |
:maxdepth: 2 |
|
206 |
|
|
207 |
Object Storage API <pithos-api-guide> |
|
208 |
|
|
209 |
|
|
210 |
Implementing new clients |
|
211 |
======================== |
|
212 |
|
|
213 |
In this section we discuss implementation guidelines, that a developer should |
|
214 |
take into account before writing his own client for the above APIs. Before, |
|
215 |
starting your client implementation, make sure you have thoroughly read the |
|
216 |
corresponding Synnefo API. |
|
217 |
|
|
218 |
Pithos clients |
|
219 |
-------------- |
|
220 |
|
|
221 |
User Experience |
|
222 |
~~~~~~~~~~~~~~~ |
|
223 |
|
|
224 |
Hopefully this API will allow for a multitude of client implementations, each |
|
225 |
supporting a different device or operating system. All clients will be able to |
|
226 |
manipulate containers and objects - even software only designed for OOS API |
|
227 |
compatibility. But a Pithos interface should not be only about showing |
|
228 |
containers and folders. There are some extra user interface elements and |
|
229 |
functionalities that should be common to all implementations. |
|
230 |
|
|
231 |
Upon entrance to the service, a user is presented with the following elements - |
|
232 |
which can be represented as folders or with other related icons: |
|
233 |
|
|
234 |
* The ``home`` element, which is used as the default entry point to the user's |
|
235 |
"files". Objects under ``home`` are represented in the usual hierarchical |
|
236 |
organization of folders and files. |
|
237 |
* The ``trash`` element, which contains files that have been marked for |
|
238 |
deletion, but can still be recovered. |
|
239 |
* The ``shared`` element, which contains all objects shared by the user to |
|
240 |
other users of the system. |
|
241 |
* The ``others`` element, which contains all objects that other users share |
|
242 |
with the user. |
|
243 |
* The ``groups`` element, which contains the names of groups the user has |
|
244 |
defined. Each group consists of a user list. Group creation, deletion, and |
|
245 |
manipulation is carried out by actions originating here. |
|
246 |
* The ``history`` element, which allows browsing past instances of ``home`` |
|
247 |
and - optionally - ``trash``. |
|
248 |
|
|
249 |
Objects in Pithos can be: |
|
250 |
|
|
251 |
* Moved to trash and then deleted. |
|
252 |
* Shared with specific permissions. |
|
253 |
* Made public (shared with non-Pithos users). |
|
254 |
* Restored from previous versions. |
|
255 |
|
|
256 |
Some of these functions are performed by the client software and some by the |
|
257 |
Pithos server. |
|
258 |
|
|
259 |
In the first version of Pithos, objects could also be assigned custom tags. |
|
260 |
This is no longer supported. Existing deployments can migrate tags into a |
|
261 |
specific metadata value, i.e. ``X-Object-Meta-Tags``. |
|
262 |
|
|
263 |
Implementation Guidelines |
|
264 |
~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
265 |
|
|
266 |
Pithos clients should use the ``pithos`` and ``trash`` containers for active |
|
267 |
and inactive objects respectively. If any of these containers is not found, the |
|
268 |
client software should create it, without interrupting the user's workflow. The |
|
269 |
``home`` element corresponds to ``pithos`` and the ``trash`` element to |
|
270 |
``trash``. Use ``PUT`` with the ``X-Move-From`` header, or ``MOVE`` to transfer |
|
271 |
objects from one container to the other. Use ``DELETE`` to remove from |
|
272 |
``pithos`` without trashing, or to remove from ``trash``. When moving objects, |
|
273 |
detect naming conflicts with the ``If-Match`` or ``If-None-Match`` headers. |
|
274 |
Such conflicts should be resolved by the user. |
|
275 |
|
|
276 |
Object names should use the ``/`` delimiter to impose a hierarchy of folders |
|
277 |
and files. |
|
278 |
|
|
279 |
The ``shared`` element should be implemented as a read-only view of the |
|
280 |
``pithos`` container, using the ``shared`` parameter when listing objects. The |
|
281 |
``others`` element, should start with a top-level ``GET`` to retrieve the list |
|
282 |
of accounts accessible to the user. It is suggested that the client software |
|
283 |
hides the next step of navigation - the container - if it only includes |
|
284 |
``pithos`` and forwards the user directly to the objects. |
|
285 |
|
|
286 |
Public objects are not included in ``shared`` and ``others`` listings. It is |
|
287 |
suggested that they are marked in a visually distinctive way in ``pithos`` |
|
288 |
listings (for example using an icon overlay). |
|
289 |
|
|
290 |
A special application menu, or a section in application preferences, should be |
|
291 |
devoted to managing groups (the ``groups`` element). All group-related actions |
|
292 |
are implemented at the account level. |
|
293 |
|
|
294 |
Browsing past versions of objects should be available both at the object and |
|
295 |
the container level. At the object level, a list of past versions can be |
|
296 |
included in the screen showing details or more information on the object |
|
297 |
(metadata, permissions, etc.). At the container level, it is suggested that |
|
298 |
clients use a ``history`` element, which presents to the user a read-only, |
|
299 |
time-variable view of ``pithos`` contents. This can be accomplished via the |
|
300 |
``until`` parameter in listings. Optionally, ``history`` may include ``trash``. |
|
301 |
|
|
302 |
Uploading and downloading data |
|
303 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
304 |
|
|
305 |
By using hashmaps to upload and download objects the corresponding operations |
|
306 |
can complete much faster. |
|
307 |
|
|
308 |
In the case of an upload, only the missing blocks will be submitted to the |
|
309 |
server: |
|
310 |
|
|
311 |
* Calculate the hash value for each block of the object to be uploaded. Use |
|
312 |
the hash algorithm and block size of the destination container. |
|
313 |
* Send a hashmap ``PUT`` request for the object. |
|
314 |
|
|
315 |
* Server responds with status ``201`` (Created): |
|
316 |
|
|
317 |
* Blocks are already on the server. The object has been created. Done. |
|
318 |
|
|
319 |
* Server responds with status ``409`` (Conflict): |
|
320 |
|
|
321 |
* Server's response body contains the hashes of the blocks that do not |
|
322 |
exist on the server. |
|
323 |
* For each hash value in the server's response (or all hashes together): |
|
324 |
|
|
325 |
* Send a ``POST`` request to the destination container with the |
|
326 |
corresponding data. |
|
327 |
|
|
328 |
* Repeat hashmap ``PUT``. Fail if the server's response is not ``201``. |
|
329 |
|
|
330 |
Consulting hashmaps when downloading allows for resuming partially transferred |
|
331 |
objects. The client should retrieve the hashmap from the server and compare it |
|
332 |
with the hashmap computed from the respective local file. Any missing parts can |
|
333 |
be downloaded with ``GET`` requests with the additional ``Range`` header. |
|
334 |
|
|
335 |
Syncing |
|
336 |
~~~~~~~ |
|
337 |
|
|
338 |
Consider the following algorithm for synchronizing a local folder with the |
|
339 |
server. The "state" is the complete object listing, with the corresponding |
|
340 |
attributes. |
|
341 |
|
|
342 |
.. code-block:: python |
|
343 |
|
|
344 |
# L: Local State, the last synced state of the object. |
|
345 |
# Stored locally (e.g. in an SQLite database) |
|
346 |
|
|
347 |
# C: Current State, the current local state of the object |
|
348 |
# Returned by the filesystem |
|
349 |
|
|
350 |
# S: Server State, the current server state of the object |
|
351 |
# Returned by the server (HTTP request) |
|
352 |
|
|
353 |
def sync(path): |
|
354 |
L = get_local_state(path) # Database action |
|
355 |
C = get_current_state(path) # Filesystem action |
|
356 |
S = get_server_state(path) # Network action |
|
357 |
|
|
358 |
if C == L: |
|
359 |
# No local changes |
|
360 |
if S == L: |
|
361 |
# No remote changes, nothing to do |
|
362 |
return |
|
363 |
else: |
|
364 |
# Update local state to match that of the server |
|
365 |
download(path) |
|
366 |
update_local_state(path, S) |
|
367 |
else: |
|
368 |
# Local changes exist |
|
369 |
if S == L: |
|
370 |
# No remote changes, update the server and the local state |
|
371 |
upload(path) |
|
372 |
update_local_state(path, C) |
|
373 |
else: |
|
374 |
# Both local and server changes exist |
|
375 |
if C == S: |
|
376 |
# We were lucky, both did the same |
|
377 |
update_local_state(path, C) |
|
378 |
else: |
|
379 |
# Conflicting changes exist |
|
380 |
conflict() |
|
381 |
|
|
382 |
|
|
383 |
Notes: |
|
384 |
|
|
385 |
* States represent file hashes (it is suggested to use Merkle). Deleted or |
|
386 |
non-existing files are assumed to have a magic hash (e.g. empty string). |
Also available in: Unified diff