Revision 961e6040

b/docs/cli.rst
1
Command Line Interfaces
1
Command Specifications
2
======================
3

  
4
astakos
5
-------
6

  
7
.. automodule:: kamaki.cli.commands.astakos_cli
8
    :members:
9
    :undoc-members:
10

  
11
cyclades (server, flavor, network)
12
----------------------------------
13

  
14
.. automodule:: kamaki.cli.commands.cyclades_cli
15
    :members:
16
    :undoc-members:
17

  
18
pithos (store)
19
--------------
20

  
21
.. automodule:: kamaki.cli.commands.pithos_cli
22
    :members:
23
    :undoc-members:
24

  
25
image
26
-----
27

  
28
.. automodule:: kamaki.cli.commands.image_cli
29
    :members:
30
    :undoc-members:
31

  
32
Kamaki commands
33
---------------
34

  
35
config
36
^^^^^^
37

  
38
.. automodule:: kamaki.cli.commands.config_cli
39
    :members:
40
    :undoc-members:
41

  
42
history
43
^^^^^^^
44

  
45
.. automodule:: kamaki.cli.commands.history_cli
46
    :members:
47
    :undoc-members:
48

  
49
Command Line Interface Code
2 50
===========================
3 51

  
4 52
argument
b/docs/clients.rst
1
Clients lib
2
===========
1
For Developers
2
==============
3 3

  
4
Creating applications with the clients API
5
------------------------------------------
4
Creating applications with kamaki API
5
-------------------------------------
6 6

  
7
Please do!
7
Kamaki features a clients API for building third-party client applications that communicate with OpenStack and / or Synnefo cloud services. The package is called kamaki.clients and contains a number of 
8

  
9
A good example of an application build on kamaki.clients is kamaki.cli, the command line interface of kamaki. 
10

  
11
Since synnefo services are build as OpenStack extensions, an inheritance approach has been chosen for implementing clients for both. In specific, the *compute*, *storage* and *image* modules are clients of the OS compute, OS storage and Glance APIs, respectively. On the contrary, all the other modules are Synnefo extensions (*cyclades* extents *compute*, *pithos* and *pithos_rest_api* extent *storage*) or novel synnefo services (e.g. *astakos*).
12

  
13
Setup a client instance
14
^^^^^^^^^^^^^^^^^^^^^^^
15

  
16
External applications may instantiate one or more kamaki clients.
17

  
18
.. code-block:: python
19
    :emphasize-lines: 1
20

  
21
    Example 1.1: Instantiate a Cyclades client
22

  
23

  
24
    from kamaki.clients.cyclades import CycladesClient
25
    from kamaki.clients.pithos import PithosClient
26

  
27
    my_cyclades_client = CycladesClient(base_url, token)
28
    my_pithos_client = PithosClient(base_url, token, account, container)
29

  
30
.. note:: *cyclades* and *pithos* clients inherit all methods of *compute* and *storage* clients respectively. Separate compute or storage objects should be used only when implementing applications for strict OS Compute or OS Storage services.
31

  
32
Use client methods
33
^^^^^^^^^^^^^^^^^^
34

  
35
Client methods can now be called. Developers are advised to consult :ref:`the-client-api-ref` for details on the available methods and how to use them.
36

  
37
In the following example, the *cyclades* and *pithos* clients of example 1.1 are used to extract some information, that is then printed to the standard output.
38

  
39

  
40
.. code-block:: python
41
    :emphasize-lines: 1,2
42

  
43
    Example 1.2: Print server name and OS for server with server_id
44
                Print objects in container mycont
45

  
46

  
47
    srv = my_cyclades_client.get_server_info(server_id)
48
    print("Server Name: %s (with OS %s" % (srv['name'], srv['os']))
49

  
50
    obj_list = my_pithos_client.list_objects(mycont)
51
    for obj in obj_list:
52
        print('  %s of %s bytes' % (obj['name'], obj['bytes']))
53

  
54
.. code-block:: console
55
    :emphasize-lines: 1
56

  
57
    Run of examples 1.1 + 1.2
58

  
59

  
60
    $ python test_script.py
61
    Server Name: A Debian Server (with OS Debian Base)
62
      lala.txt of 34 bytes
63
      test.txt of 1232 bytes
64
      testDir/ of 0 bytes
65
    $ 
66

  
67
Error handling
68
^^^^^^^^^^^^^^
69

  
70
The kamaki.clients standard error is ClientError. A ClientError is raised for any kind of kamaki.clients errors (errors reported by servers, type errors in arguments, etc.).
71

  
72
A ClientError contains::
73

  
74
    message     The error message.
75
    status      An optional error code, e.g. after a server error.
76
    details     Optional list of messages with error details.
77

  
78
The following example concatenates examples 1.1 and 1.2 plus error handling
79

  
80
.. code-block:: python
81

  
82
    Example 1.3: Error handling
83

  
84

  
85
    from kamaki.clients.cyclades import CycladesClient
86
    from kamaki.clients.pithos import PithosClient
87

  
88
    try:
89
        my_cyclades_client = CycladesClient(base_url, token)
90
    except ClientError:
91
        print('Failed to initialize Cyclades client')
92

  
93
    try:
94
        my_pithos_client = PithosClient(base_url, token, account, container)
95
    except ClientError:
96
        print('Failed to initialize Pithos+ client')
97

  
98
    try:
99
        srv = my_cyclades_client.get_server_info(server_id)
100
        print("Server Name: %s (with OS %s" % (srv['name'], srv['os']))
101

  
102
        obj_list = my_pithos_client.list_objects(mycont)
103
        for obj in obj_list:
104
            print('  %s of %s bytes' % (obj['name'], obj['bytes']))
105
    except ClientError as e:
106
        print('Error: %s' % e)
107
        if e.status:
108
            print('- error code: %s' % e.status)
109
        if e.details:
110
            for detail in e.details:
111
                print('- %s' % detail)
112

  
113
Extending kamaki.clients
114
------------------------
115

  
116
By default, kamaki clients are REST clients (they manage HTTP requests and responses to communicate with services). This is achieved by importing the connection module, which is an httplib rapper.
117

  
118
Connection
119
^^^^^^^^^^
120

  
121
The connection module features an error handling and logging system, a lazy response mechanism, a pooling mechanism as well as concurrency control for thread-demanding client functions (e.g. store upload).
122

  
123
How to build a client
124
^^^^^^^^^^^^^^^^^^^^^
125

  
126
All service clients consist of a subclass of the Client class and implement separate client functionalities as member methods. There is also an error class to raise exceptions that can be handled by kamaki interfaces.
127

  
128
.. code-block:: python
129
    
130
    #  ${KAMAKI_PATH}/kamaki/clients/mynewclient.py
131

  
132
    from kamaki.clients import Client, ClientError
133

  
134
    class MyNewClient(Client):
135
        """MyNewClient Description Here"""
136

  
137
        def my_first_method(self, **args):
138
            """Method description"""
139
            try:
140
                ...
141
                method code
142
                ...
143
            except SomeKnownException as e1:
144
                raise ClientError('MyError: %s' % e1)
145
            except SomeOtherException as e2:
146
                raise ClientError('MyError: %s' % e2)
147

  
148
        def my_second_method(self, **params):
149
            """Method description"""
150
            ...
151

  
152
Custom clients can use a set of convenience methods for easy HTTP requests
153

  
154
.. code-block:: python
155

  
156
    def get(self, path, **kwargs)
157
    def head(self, path, **kwargs)
158
    def post(self, path, **kwargs)
159
    def put(self, path, **kwargs)
160
    def delete(self, path, **kwargs)
161
    def copy(self, path, **kwargs)
162
    def move(self, path, **kwargs)
163

  
164
How to use your client
165
^^^^^^^^^^^^^^^^^^^^^^
166

  
167
External applications must instantiate a MyNewClient object.
168

  
169
.. code-block:: python
170

  
171
    from kamaki.clients import ClientError
172
    from kamaki.clients.mynewclient import MyNewClient
173

  
174
    ...
175
    try:
176
        cl = MyNewClient(args)
177
        cl.my_first_method(other_args)
178
    except ClientError as cle:
179
        print('Client Error: %s' % cle)
180
    ...
181

  
182
Concurrency control
183
^^^^^^^^^^^^^^^^^^^
184

  
185
Kamaki clients may handle multiple requests at once, using threads. In that case, users might implement their own thread handling mechanism, use an external solution or take advantage of the mechanism featured in kamaki.clients
186

  
187
.. code-block:: python
188

  
189
    from threading import enumerate
190
    from kamaki.clients import SilentEvent
191
    ...
192

  
193
    class MyNewClient(Client):
194
        ...
195

  
196
        def _single_threaded_method(self, **args):
197
            ...
198
            request code
199
            ...
200

  
201
        def multithread_method(self):
202
            thread_list = []
203
            self._init_thread_limit()
204
            while some_condition or thread_list:
205
                ...
206
                event = SilentEvent(self._single_threaded_method, **args)
207
                event.start()
208
                thread_list.append(event)
209
                thread_list = self._watch_thread_limit(thread_list)
210

  
211
The CLI API
212
-----------
213

  
214
.. toctree::
215

  
216
    cli
217

  
218
.. _the-client-api-ref:
8 219

  
9 220
The clients API
10 221
---------------
11 222

  
12
The clients API is based on:
223
Imports
224
^^^^^^^
13 225

  
14 226
.. toctree::
15 227
    connection
16 228

  
17
It contains the following modules:
229
Modules list
230
^^^^^^^^^^^^
18 231

  
19 232
compute
20 233
^^^^^^^
b/docs/commands.rst
1
Commands
2
========
3

  
4 1
List of commands
5
----------------
2
================
3

  
4
The commands described bellow are grouped by service. The examples showcase a sample set of group commands. The kamaki interactive shell (check `Usage section <usage.html#interactive-shell>`_ for details) is chosen as the execution environment.
6 5

  
7
The commands described bellow are grouped by service. The examples showcase a sample set of group commands. The kamaki interactive shell (check `Usage section <usage.html#interactive-shell>`_ for details) is chosen as the execution environment:
8 6

  
9 7
astakos (Identity Manager)
10
^^^^^^^^^^^^^^^^^^^^^^^^^^
8
--------------------------
11 9

  
12 10
.. code-block:: text
13 11

  
14 12
    authenticate:  Authenticate a user
15 13

  
16
Showcase: get user information, provided the token was set
14
Showcase: get user information
15
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
16

  
17
In the following, the token has been set in a previous step (see `setup section <setup.html>`_ or the `quick setup guide <usage.html#quick-setup>`_)
17 18

  
18 19
.. code-block:: console
19 20
    :emphasize-lines: 1,4
......
34 35
    username          :  4215th3b357num9323v32
35 36

  
36 37
flavor (Compute/Cyclades)
37
^^^^^^^^^^^^^^^^^^^^^^^^^
38
-------------------------
38 39

  
39 40
.. code-block:: text
40 41

  
......
42 43
    list:  List flavors
43 44

  
44 45
Showcase: show details for flavor with id 43
46
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
45 47

  
46 48
.. code-block:: console
47 49
    :emphasize-lines: 1,4
......
59 61
    ram              :  2048
60 62

  
61 63
image (Compute/Cyclades + Glance)
62
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
64
---------------------------------
63 65

  
64 66
.. code-block:: text
65 67

  
......
80 82
    shared     :  List shared images
81 83

  
82 84
Showcase: Pick an image and list the properties
85
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
83 86

  
84 87
.. code-block:: console
85 88
    :emphasize-lines: 1,4,18
......
113 116
    users         :  root
114 117

  
115 118
server (Compute/Cyclades)
116
^^^^^^^^^^^^^^^^^^^^^^^^^
119
-------------------------
117 120

  
118 121
.. code-block:: text
119 122

  
......
135 138
    stats   :  Get server statistics
136 139
    wait    :  Wait for server to finish
137 140

  
138
Showcase: Create a server.
141
Showcase: Create a server
142
^^^^^^^^^^^^^^^^^^^^^^^^^
139 143

  
140 144
.. code-block:: console
141 145
    :emphasize-lines: 1,4,21,35,44,62
......
209 213
.. Note:: In kamaki shell, / is used to access top-level command groups while working in command group contexts
210 214

  
211 215
network (Compute/Cyclades)
212
^^^^^^^^^^^^^^^^^^^^^^^^^^
216
--------------------------
213 217

  
214 218
.. code-block:: text
215 219

  
......
222 226
    rename    :  Update network name
223 227

  
224 228
Showcase: Connect a network to a VM
229
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
225 230

  
226 231
.. code-block:: console
227 232
    :emphasize-lines: 1,4,9,24,27,44
......
286 291
.. Note:: In kamaki shell, / is used to access top-level command groups while working in command group contexts
287 292

  
288 293
store (Storage/Pithos+)
289
^^^^^^^^^^^^^^^^^^^^^^^
294
-----------------------
290 295

  
291 296
.. code-block:: text
292 297

  
......
324 329
    versioning    :  Get  versioning for account [or container ]
325 330
    versions      :  Get the version list of an object
326 331

  
327
Showcase: Upload and download a file.
332
Showcase: Upload and download a file
333
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
328 334

  
329 335
.. code-block:: console
330 336
    :emphasize-lines: 1,7,11,16,21,29,33,37,41,44,51,55,60,64
......
399 405
    [store]:!diff rndm_local.file rndm_remote.file
400 406

  
401 407
.. Note:: In kamaki shell, ! is used to execute OS shell commands (bash in the above)
402

  
403

  
404
Command specifications
405
----------------------
406

  
407
cyclades_cli
408
^^^^^^^^^^^^
409

  
410
.. automodule:: kamaki.cli.commands.cyclades_cli
411
    :members:
412
    :undoc-members:
413

  
414

  
415
pithos_cli
416
^^^^^^^^^^
417

  
418
.. automodule:: kamaki.cli.commands.pithos_cli
419
    :members:
420
    :show-inheritance:
421
    :undoc-members:
422

  
423
image_cli
424
^^^^^^^^^
425

  
426
.. automodule:: kamaki.cli.commands.image_cli
427
    :members:
428
    :show-inheritance:
429
    :undoc-members:
430

  
431

  
432
astakos_cli
433
^^^^^^^^^^^
434

  
435
.. automodule:: kamaki.cli.commands.astakos_cli
436
    :members:
437
    :undoc-members:
438

  
439
config_cli
440
^^^^^^^^^^
441

  
442
.. automodule:: kamaki.cli.commands.config_cli
443
    :members:
444
    :undoc-members:
445

  
446
history_cli
447
^^^^^^^^^^^
448

  
449
.. automodule:: kamaki.cli.commands.history_cli
450
    :members:
451
    :undoc-members:
b/docs/index.rst
21 21
*********
22 22

  
23 23
.. toctree::
24
   :maxdepth: 3
24
   :maxdepth: 2
25 25
   
26 26
   overview
27 27
   installation
b/docs/installation.rst
16 16
Installing from source (git repos.)
17 17
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
18 18

  
19
Requirements
20
""""""""""""
21

  
22
Essential:
23

  
24
 * Python 2.6 or better [http://www.python.org]
25
 * Python setuptools [http://pypi.python.org/pypi/setuptools]
26

  
27
Optional:
28

  
29
 * VirtualEnv (python-virtualenv) [http://www.virtualenv.org]
30

  
19 31
1. Setup a virtual enviroment (optional)
20 32
""""""""""""""""""""""""""""""""""""""""
21 33

  
......
27 39

  
28 40
    $ source kamaki-env/bin/activate
29 41

  
30
.. hint:: More about virtualenv: `<http://www.virtualenv.org>`_
42
A more detailed example of using virtual env can be found at the `snf-image-creator setup guide <http://docs.dev.grnet.gr/snf-image-creator/latest/install.html#python-virtual-environment>`_
31 43

  
32
2. Install snf-common from synnefo project (required since v0.6.0)
33
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
44
2. Install snf-common
45
"""""""""""""""""""""
34 46

  
35
Package snf-common is required since kamaki 0.6.0.
47
Package snf-common is part of the synnefo project and is a kamaki dependency since version 0.6.0.
36 48

  
37 49
.. code-block:: console
38 50

  
b/docs/setup.rst
6 6
Requirements
7 7
------------
8 8

  
9
* python 2.6 or better
9
Essential:
10

  
11
 * Python 2.6 or better [http://www.python.org/]
12
 * snf-common 0.10 or better [https://code.grnet.gr/projects/synnefo]
13

  
14
Optional:
15

  
16
 * ANSI colors for Python [http://pypi.python.org/pypi/ansicolors]
17
 * progress [http://pypi.python.org/pypi/progress]
10 18

  
11
* snf-common 0.10 or better
12 19

  
13 20
The snf-common package is part of the Synnefo project of the Greek Research and Development Network and is available from the same official sources as Kamaki (e.g. http://apt.dev.grnet.gr ).
14 21

  
15 22
Optional features
16 23
-----------------
17 24

  
25
For installing any all of the following, consult the `kamaki installation guide <installation.html#install-progress-and-or-ansicolors-optional>`_
26

  
18 27
* ansicolors
19 28
    * Make command line / console user interface responses prettier with text formating (colors, bold, etc.)
20 29
    * Can be switched on/off in kamaki configuration file: colors=on/off
21
    * Installation: pip install ansicolors
22 30

  
23 31
* progress 
24 32
    * Attach progress bars to various kamaki commands (e.g. kamaki store upload)
25
    * Installation: pip install progressbar
26 33
    * Since version 0.6.1 kamaki "requires" progress version 1.0.2 or better
27 34

  
28 35
Any of the above features can be installed at any time before or after kamaki installation.
b/docs/usage.rst
5 5

  
6 6
What's more, kamaki offers a clients API that allows the development of external applications for synnefo. The clients API is listed in the `Clients lib <clients.html>`_ section. The recommended method of utilizing this API is explained in the present.
7 7

  
8
Setup
9
-----
8
Quick Setup
9
-----------
10 10

  
11 11
Kamaki interfaces rely on a list of configuration options. In the initial state, kamaki is configured to communicate with the Okeanos IaaS. A detailed guide for setting up kamaki can be found in the `Setup <setup.html>`_ section.
12 12

  
13
Quick guide
14
^^^^^^^^^^^
15

  
16
It is essential for users to get a configuration token (to get in Okeanos.grnet.gr log `here <https://accounts.okeanos.grnet.gr/im/>`_) and provide it to kamaki:
13
It is essential for users to get a configuration token (okeanos.grnet.gr users go `here <https://accounts.okeanos.grnet.gr/im/>`_) and provide it to kamaki:
17 14

  
18 15
.. code-block:: console
19 16
    :emphasize-lines: 1
20 17

  
21
    Example 1.1.1: Set user token to myt0k3n==
18
    Example 1.1: Set user token to myt0k3n==
22 19

  
23 20
    $ kamaki set token myt0k3n==
24 21

  
......
27 24
.. code-block:: console
28 25
    :emphasize-lines: 1
29 26

  
30
    Example 1.1.2: Set user name to user@domain.com
27
    Example 1.2: Set user name to user@domain.com
31 28

  
32 29
    $ kamaki set account user@domain.com
33 30

  
b/kamaki/clients/pithos.py
206 206
        assert offset == size
207 207

  
208 208
    def _upload_missing_blocks(self, missing, hmap, fileobj, upload_cb=None):
209
        """upload missing blocks asynchronously. 
209
        """upload missing blocks asynchronously.
210 210
        """
211 211
        if upload_cb:
212 212
            upload_gen = upload_cb(len(missing))

Also available in: Unified diff