Statistics
| Branch: | Tag: | Revision:

root / docs / dev-guide.rst @ 73ebcd68

History | View | Annotate | Download (9.8 kB)

1
.. _dev-guide:
2

    
3
Synnefo Developer's Guide
4
^^^^^^^^^^^^^^^^^^^^^^^^^
5

    
6
This is the complete Synnefo Developer's Guide. Here, we document all Synnefo
7
REST APIs, to allow external developers write independent tools that interact
8
with Synnefo.
9

    
10
Synnefo exposes the OpenStack APIs for all its operations. Also, extensions
11
have been written for advanced operations wherever needed, and minor changes
12
for things that were missing or change frequently.
13

    
14
Most Synnefo services have a corresponding OpenStack API:
15

    
16
| Cyclades/Compute Service -> OpenStack Compute API
17
| Cyclades/Network Service -> OpenStack Compute/Network API (not Quantum yet)
18
| Cyclades/Image Service -> OpenStack Glance API
19
| Pithos/Storage Service -> OpenStack Object Store API
20
| Astakos/Identity Service -> Proprietary, moving to OpenStack Keystone API
21
| Astakos/Quota Service -> Proprietary API
22
| Astakos/Resource Service -> Proprietary API
23

    
24
Below, we will describe all Synnefo APIs with conjuction to the OpenStack APIs.
25

    
26

    
27
Identity Service API (Astakos)
28
==============================
29

    
30
Currently, Astakos which is the Identity Management Service of Synnefo, has a
31
proprietary API, but we are moving to the OpenStack Keystone API.
32

    
33
The current Identity Management API is:
34

    
35
.. toctree::
36
   :maxdepth: 2
37

    
38
    Identity API <astakos-api-guide>
39

    
40

    
41
Resource and Quota Service API (Astakos)
42
========================================
43

    
44
.. toctree::
45
    :maxdepth: 2
46

    
47
    Resource and Quota API <quota-api-guide.rst>
48

    
49

    
50
Compute Service API (Cyclades)
51
==============================
52

    
53
The Compute part of Cyclades exposes the OpenStack Compute API with minor
54
changes wherever needed.
55

    
56
This is the Cyclades/Compute API:
57

    
58
.. toctree::
59
   :maxdepth: 2
60

    
61
   Compute API <cyclades-api-guide>
62

    
63

    
64
Network Service API (Cyclades)
65
==============================
66

    
67
The Network Service is implemented inside Cyclades. It exposes the part of the
68
OpenStack Compute API that has to do with Networks. The OpenStack Quantum API
69
is not implemented yet.
70

    
71
Please consult the :ref:`Cyclades/Network API <cyclades-api-guide>` for more
72
details.
73

    
74

    
75
Image Service API (Cyclades)
76
============================
77

    
78
The Image Service is implemented inside Cyclades. It exposes the OpenStack
79
Glance API with minor changes wherever needed.
80

    
81
This is the Cyclades/Image API:
82

    
83
.. toctree::
84
   :maxdepth: 2
85

    
86
   Image API <plankton-api-guide>
87

    
88

    
89
Storage Service API (Pithos)
90
============================
91

    
92
Pithos is the Storage Service of Synnefo and it exposes the OpenStack Object
93
Storage API with extensions for advanced operations, e.g., syncing.
94

    
95
This is the Pithos Object Storage API:
96

    
97
.. toctree::
98
   :maxdepth: 2
99

    
100
   Object Storage API <pithos-api-guide>
101

    
102

    
103
Implementing new clients
104
========================
105

    
106
In this section we discuss implementation guidelines, that a developer should
107
take into account before writing his own client for the above APIs. Before,
108
starting your client implementation, make sure you have thoroughly read the
109
corresponding Synnefo API.
110

    
111
Pithos clients
112
--------------
113

    
114
User Experience
115
~~~~~~~~~~~~~~~
116

    
117
Hopefully this API will allow for a multitude of client implementations, each
118
supporting a different device or operating system. All clients will be able to
119
manipulate containers and objects - even software only designed for OOS API
120
compatibility. But a Pithos interface should not be only about showing
121
containers and folders. There are some extra user interface elements and
122
functionalities that should be common to all implementations.
123

    
124
Upon entrance to the service, a user is presented with the following elements -
125
which can be represented as folders or with other related icons:
126

    
127
 * The ``home`` element, which is used as the default entry point to the user's
128
   "files". Objects under ``home`` are represented in the usual hierarchical
129
   organization of folders and files.
130
 * The ``trash`` element, which contains files that have been marked for
131
   deletion, but can still be recovered.
132
 * The ``shared`` element, which contains all objects shared by the user to
133
   other users of the system.
134
 * The ``others`` element, which contains all objects that other users share
135
   with the user.
136
 * The ``groups`` element, which contains the names of groups the user has
137
   defined. Each group consists of a user list. Group creation, deletion, and
138
   manipulation is carried out by actions originating here.
139
 * The ``history`` element, which allows browsing past instances of ``home``
140
   and - optionally - ``trash``.
141

    
142
Objects in Pithos can be:
143

    
144
 * Moved to trash and then deleted.
145
 * Shared with specific permissions.
146
 * Made public (shared with non-Pithos users).
147
 * Restored from previous versions.
148

    
149
Some of these functions are performed by the client software and some by the
150
Pithos server.
151

    
152
In the first version of Pithos, objects could also be assigned custom tags.
153
This is no longer supported. Existing deployments can migrate tags into a
154
specific metadata value, i.e. ``X-Object-Meta-Tags``.
155

    
156
Implementation Guidelines
157
~~~~~~~~~~~~~~~~~~~~~~~~~
158

    
159
Pithos clients should use the ``pithos`` and ``trash`` containers for active
160
and inactive objects respectively. If any of these containers is not found, the
161
client software should create it, without interrupting the user's workflow. The
162
``home`` element corresponds to ``pithos`` and the ``trash`` element to
163
``trash``. Use ``PUT`` with the ``X-Move-From`` header, or ``MOVE`` to transfer
164
objects from one container to the other. Use ``DELETE`` to remove from
165
``pithos`` without trashing, or to remove from ``trash``. When moving objects,
166
detect naming conflicts with the ``If-Match`` or ``If-None-Match`` headers.
167
Such conflicts should be resolved by the user.
168

    
169
Object names should use the ``/`` delimiter to impose a hierarchy of folders
170
and files.
171

    
172
The ``shared`` element should be implemented as a read-only view of the
173
``pithos`` container, using the ``shared`` parameter when listing objects. The
174
``others`` element, should start with a top-level ``GET`` to retrieve the list
175
of accounts accessible to the user. It is suggested that the client software
176
hides the next step of navigation - the container - if it only includes
177
``pithos`` and forwards the user directly to the objects.
178

    
179
Public objects are not included in ``shared`` and ``others`` listings. It is
180
suggested that they are marked in a visually distinctive way in ``pithos``
181
listings (for example using an icon overlay).
182

    
183
A special application menu, or a section in application preferences, should be
184
devoted to managing groups (the ``groups`` element). All group-related actions
185
are implemented at the account level.
186

    
187
Browsing past versions of objects should be available both at the object and
188
the container level. At the object level, a list of past versions can be
189
included in the screen showing details or more information on the object
190
(metadata, permissions, etc.). At the container level, it is suggested that
191
clients use a ``history`` element, which presents to the user a read-only,
192
time-variable view of ``pithos`` contents. This can be accomplished via the
193
``until`` parameter in listings. Optionally, ``history`` may include ``trash``.
194

    
195
Uploading and downloading data
196
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
197

    
198
By using hashmaps to upload and download objects the corresponding operations
199
can complete much faster.
200

    
201
In the case of an upload, only the missing blocks will be submitted to the
202
server:
203

    
204
 * Calculate the hash value for each block of the object to be uploaded. Use
205
   the hash algorithm and block size of the destination container.
206
 * Send a hashmap ``PUT`` request for the object.
207

    
208
   * Server responds with status ``201`` (Created):
209

    
210
     * Blocks are already on the server. The object has been created. Done.
211

    
212
   * Server responds with status ``409`` (Conflict):
213

    
214
     * Server's response body contains the hashes of the blocks that do not
215
       exist on the server.
216
     * For each hash value in the server's response (or all hashes together):
217

    
218
       * Send a ``POST`` request to the destination container with the
219
         corresponding data.
220

    
221
 * Repeat hashmap ``PUT``. Fail if the server's response is not ``201``.
222

    
223
Consulting hashmaps when downloading allows for resuming partially transferred
224
objects. The client should retrieve the hashmap from the server and compare it
225
with the hashmap computed from the respective local file. Any missing parts can
226
be downloaded with ``GET`` requests with the additional ``Range`` header.
227

    
228
Syncing
229
~~~~~~~
230

    
231
Consider the following algorithm for synchronizing a local folder with the
232
server. The "state" is the complete object listing, with the corresponding
233
attributes.
234
 
235
.. code-block:: python
236

    
237
  # L: Local State, the last synced state of the object.
238
  # Stored locally (e.g. in an SQLite database)
239
  
240
  # C: Current State, the current local state of the object
241
  # Returned by the filesystem
242
  
243
  # S: Server State, the current server state of the object
244
  # Returned by the server (HTTP request)
245
  
246
  def sync(path):
247
      L = get_local_state(path)   # Database action
248
      C = get_current_state(path) # Filesystem action
249
      S = get_server_state(path)  # Network action
250
  
251
      if C == L:
252
          # No local changes
253
          if S == L:
254
              # No remote changes, nothing to do
255
              return
256
          else:
257
              # Update local state to match that of the server
258
              download(path)
259
              update_local_state(path, S)
260
      else:
261
          # Local changes exist
262
          if S == L:
263
              # No remote changes, update the server and the local state
264
              upload(path)
265
              update_local_state(path, C)
266
          else:
267
              # Both local and server changes exist
268
              if C == S:
269
                  # We were lucky, both did the same
270
                  update_local_state(path, C)
271
              else:
272
                  # Conflicting changes exist
273
                  conflict()
274
  
275

    
276
Notes:
277

    
278
 * States represent file hashes (it is suggested to use Merkle). Deleted or
279
   non-existing files are assumed to have a magic hash (e.g. empty string).