Statistics
| Branch: | Tag: | Revision:

root / docs / developers / extending-clients-api.rst @ ef04bdeb

History | View | Annotate | Download (8.9 kB)

1 9e4508df Stavros Sachtouris
Extending kamaki.clients
2 9e4508df Stavros Sachtouris
========================
3 9e4508df Stavros Sachtouris
4 fa382f9e Stavros Sachtouris
By default, kamaki clients are REST clients (they manage HTTP requests and
5 fa382f9e Stavros Sachtouris
responses to communicate with services).
6 9e4508df Stavros Sachtouris
7 9e4508df Stavros Sachtouris
How to build a client
8 9e4508df Stavros Sachtouris
---------------------
9 9e4508df Stavros Sachtouris
10 fa382f9e Stavros Sachtouris
All service clients consist of a subclass of the Client class and implement
11 fa382f9e Stavros Sachtouris
separate client functionalities as member methods. There is also an error class
12 fa382f9e Stavros Sachtouris
to raise exceptions that can be handled by kamaki interfaces.
13 9e4508df Stavros Sachtouris
14 9e4508df Stavros Sachtouris
.. code-block:: python
15 9e4508df Stavros Sachtouris
    
16 9e4508df Stavros Sachtouris
    #  ${KAMAKI_PATH}/kamaki/clients/mynewclient.py
17 9e4508df Stavros Sachtouris
18 9e4508df Stavros Sachtouris
    from kamaki.clients import Client, ClientError
19 9e4508df Stavros Sachtouris
20 9e4508df Stavros Sachtouris
    class MyNewClient(Client):
21 9e4508df Stavros Sachtouris
        """MyNewClient Description Here"""
22 9e4508df Stavros Sachtouris
23 9e4508df Stavros Sachtouris
        def my_first_method(self, **args):
24 9e4508df Stavros Sachtouris
            """Method description"""
25 9e4508df Stavros Sachtouris
            try:
26 9e4508df Stavros Sachtouris
                ...
27 9e4508df Stavros Sachtouris
                method code
28 9e4508df Stavros Sachtouris
                ...
29 9e4508df Stavros Sachtouris
            except SomeKnownException as e1:
30 9e4508df Stavros Sachtouris
                raise ClientError('MyError: %s' % e1)
31 9e4508df Stavros Sachtouris
            except SomeOtherException as e2:
32 9e4508df Stavros Sachtouris
                raise ClientError('MyError: %s' % e2)
33 9e4508df Stavros Sachtouris
34 9e4508df Stavros Sachtouris
        def my_second_method(self, **params):
35 9e4508df Stavros Sachtouris
            """Method description"""
36 9e4508df Stavros Sachtouris
            ...
37 9e4508df Stavros Sachtouris
38 9e4508df Stavros Sachtouris
Custom clients can use a set of convenience methods for easy HTTP requests
39 9e4508df Stavros Sachtouris
40 9e4508df Stavros Sachtouris
.. code-block:: python
41 9e4508df Stavros Sachtouris
42 9e4508df Stavros Sachtouris
    def get(self, path, **kwargs)
43 9e4508df Stavros Sachtouris
    def head(self, path, **kwargs)
44 9e4508df Stavros Sachtouris
    def post(self, path, **kwargs)
45 9e4508df Stavros Sachtouris
    def put(self, path, **kwargs)
46 9e4508df Stavros Sachtouris
    def delete(self, path, **kwargs)
47 9e4508df Stavros Sachtouris
    def copy(self, path, **kwargs)
48 9e4508df Stavros Sachtouris
    def move(self, path, **kwargs)
49 9e4508df Stavros Sachtouris
50 9e4508df Stavros Sachtouris
How to use your client
51 9e4508df Stavros Sachtouris
----------------------
52 9e4508df Stavros Sachtouris
53 9e4508df Stavros Sachtouris
External applications must instantiate a MyNewClient object.
54 9e4508df Stavros Sachtouris
55 9e4508df Stavros Sachtouris
.. code-block:: python
56 9e4508df Stavros Sachtouris
57 9e4508df Stavros Sachtouris
    from kamaki.clients import ClientError
58 9e4508df Stavros Sachtouris
    from kamaki.clients.mynewclient import MyNewClient
59 9e4508df Stavros Sachtouris
60 9e4508df Stavros Sachtouris
    ...
61 9e4508df Stavros Sachtouris
    try:
62 9e4508df Stavros Sachtouris
        cl = MyNewClient(args)
63 9e4508df Stavros Sachtouris
        cl.my_first_method(other_args)
64 9e4508df Stavros Sachtouris
    except ClientError as cle:
65 9e4508df Stavros Sachtouris
        print('Client Error: %s' % cle)
66 9e4508df Stavros Sachtouris
    ...
67 9e4508df Stavros Sachtouris
68 9e4508df Stavros Sachtouris
Concurrency control
69 9e4508df Stavros Sachtouris
-------------------
70 9e4508df Stavros Sachtouris
71 fa382f9e Stavros Sachtouris
Kamaki clients may handle multiple requests at once, using threads. In that
72 fa382f9e Stavros Sachtouris
case, users might implement their own thread handling mechanism, use an
73 fa382f9e Stavros Sachtouris
external solution or take advantage of the mechanism featured in kamaki.clients
74 9e4508df Stavros Sachtouris
75 9e4508df Stavros Sachtouris
.. code-block:: python
76 9e4508df Stavros Sachtouris
77 9e4508df Stavros Sachtouris
    from threading import enumerate
78 9e4508df Stavros Sachtouris
    from kamaki.clients import SilentEvent
79 9e4508df Stavros Sachtouris
    ...
80 9e4508df Stavros Sachtouris
81 9e4508df Stavros Sachtouris
    class MyNewClient(Client):
82 9e4508df Stavros Sachtouris
        ...
83 9e4508df Stavros Sachtouris
84 9e4508df Stavros Sachtouris
        def _single_threaded_method(self, **args):
85 9e4508df Stavros Sachtouris
            ...
86 9e4508df Stavros Sachtouris
            request code
87 9e4508df Stavros Sachtouris
            ...
88 9e4508df Stavros Sachtouris
89 9e4508df Stavros Sachtouris
        def multithread_method(self):
90 9e4508df Stavros Sachtouris
            thread_list = []
91 9e4508df Stavros Sachtouris
            self._init_thread_limit()
92 9e4508df Stavros Sachtouris
            while some_condition or thread_list:
93 9e4508df Stavros Sachtouris
                ...
94 9e4508df Stavros Sachtouris
                event = SilentEvent(self._single_threaded_method, **args)
95 9e4508df Stavros Sachtouris
                event.start()
96 9e4508df Stavros Sachtouris
                thread_list.append(event)
97 4054c46d Stavros Sachtouris
                thread_list = self._watch_thread_limit(thread_list)
98 4054c46d Stavros Sachtouris
99 4054c46d Stavros Sachtouris
Going agile
100 4054c46d Stavros Sachtouris
-----------
101 4054c46d Stavros Sachtouris
102 fa382f9e Stavros Sachtouris
The kamaki.clients package contains a set of fine-grained unit-tests for all
103 fa382f9e Stavros Sachtouris
its packages. 
104 4054c46d Stavros Sachtouris
105 fa382f9e Stavros Sachtouris
.. note:: unit tests require the optional python-mock package, version 1.X or
106 fa382f9e Stavros Sachtouris
    better
107 4054c46d Stavros Sachtouris
108 4054c46d Stavros Sachtouris
Using the tests
109 4054c46d Stavros Sachtouris
^^^^^^^^^^^^^^^
110 4054c46d Stavros Sachtouris
111 4054c46d Stavros Sachtouris
To run the tests, the kamaki source code has to be downloaded.
112 4054c46d Stavros Sachtouris
113 4054c46d Stavros Sachtouris
.. code-block:: console
114 4054c46d Stavros Sachtouris
115 4054c46d Stavros Sachtouris
    $ git clone https://code.grnet.gr/git/kamaki
116 4054c46d Stavros Sachtouris
    $ cd kamaki/kamaki/clients
117 4054c46d Stavros Sachtouris
118 fa382f9e Stavros Sachtouris
In each package under kamaki.clients, there is a test module (test.py) where
119 fa382f9e Stavros Sachtouris
the tests are implemented. To run all tests, run the test.py file from
120 fa382f9e Stavros Sachtouris
kamaki.clients
121 4054c46d Stavros Sachtouris
122 4054c46d Stavros Sachtouris
.. code-block:: console
123 4054c46d Stavros Sachtouris
124 4054c46d Stavros Sachtouris
    $ python test.py
125 4054c46d Stavros Sachtouris
126 fa382f9e Stavros Sachtouris
To test a specific class, add the class name as an argument. E.g. for the
127 fa382f9e Stavros Sachtouris
Client class:
128 4054c46d Stavros Sachtouris
129 4054c46d Stavros Sachtouris
.. code-block:: console
130 4054c46d Stavros Sachtouris
131 4054c46d Stavros Sachtouris
    $ python test.py Client
132 4054c46d Stavros Sachtouris
133 fa382f9e Stavros Sachtouris
To test a specific method in a class, apply an extra argument, e.g. for the
134 fa382f9e Stavros Sachtouris
request method in the Client class:
135 4054c46d Stavros Sachtouris
136 4054c46d Stavros Sachtouris
.. code-block:: console
137 4054c46d Stavros Sachtouris
138 4054c46d Stavros Sachtouris
    $ python test.py Client request
139 4054c46d Stavros Sachtouris
140 fa382f9e Stavros Sachtouris
Each package contains a test module (test.py) which is also runnable from the
141 fa382f9e Stavros Sachtouris
command line. E.g. in the pithos package there is a test module which
142 fa382f9e Stavros Sachtouris
contains, among others, the **download** sub-test:
143 4054c46d Stavros Sachtouris
144 4054c46d Stavros Sachtouris
.. code-block:: console
145 4054c46d Stavros Sachtouris
146 4054c46d Stavros Sachtouris
    $ cd pithos
147 4054c46d Stavros Sachtouris
148 4054c46d Stavros Sachtouris
    # Run all kamaki.clients.pithos tests
149 4054c46d Stavros Sachtouris
    $ python test.py
150 4054c46d Stavros Sachtouris
151 4054c46d Stavros Sachtouris
    # Run all kamaki.clients.pithos.PithoClient tests
152 4054c46d Stavros Sachtouris
    $ python test.py Pithos
153 4054c46d Stavros Sachtouris
154 4054c46d Stavros Sachtouris
    # Test kamaki.clients.pithos.PithosClient.download
155 4054c46d Stavros Sachtouris
    $ python test.py Pithos download
156 4054c46d Stavros Sachtouris
157 fa382f9e Stavros Sachtouris
To fully test a specific package, run test.py from the package location. E.g.
158 fa382f9e Stavros Sachtouris
to test everything in kamaki.clients.pithos package:
159 3c50df2e Stavros Sachtouris
160 3c50df2e Stavros Sachtouris
.. code-block:: console
161 3c50df2e Stavros Sachtouris
162 3c50df2e Stavros Sachtouris
    $ cd pithos
163 3c50df2e Stavros Sachtouris
    $ python test.py
164 3c50df2e Stavros Sachtouris
165 4054c46d Stavros Sachtouris
Mechanism
166 4054c46d Stavros Sachtouris
^^^^^^^^^
167 4054c46d Stavros Sachtouris
168 fa382f9e Stavros Sachtouris
Each folder / package contains a test.py file, that represents the test module
169 fa382f9e Stavros Sachtouris
of this package. All test modules contain a set of classes that extent the
170 fa382f9e Stavros Sachtouris
TestCase class. They also contain a main method to run the tests.
171 4054c46d Stavros Sachtouris
172 fa382f9e Stavros Sachtouris
By convention, testing classes are named as <Tested Class> where <Test Class>
173 fa382f9e Stavros Sachtouris
is the name of the tested class or module. Methods not grouped in classes are
174 fa382f9e Stavros Sachtouris
tested by classes named after their respective module.
175 4054c46d Stavros Sachtouris
176 fa382f9e Stavros Sachtouris
For example, the kamaki.clients.pithos.PithosClient class is tested by the
177 fa382f9e Stavros Sachtouris
kamaki.clients.pithos.test.PithosClient class, while the methods in
178 fa382f9e Stavros Sachtouris
kamaki.clients.utils module are tested by the kamaki.clients.utils.test.Utils
179 fa382f9e Stavros Sachtouris
testing class.
180 4054c46d Stavros Sachtouris
181 4054c46d Stavros Sachtouris
Adding unit tests
182 4054c46d Stavros Sachtouris
^^^^^^^^^^^^^^^^^
183 fa382f9e Stavros Sachtouris
184 fa382f9e Stavros Sachtouris
After modifying or extending kamaki.clients method, classes, modules or
185 fa382f9e Stavros Sachtouris
packages, it is a good practice to also modify or extend the corresponding
186 fa382f9e Stavros Sachtouris
unit tests. What's more, it is recommended to modify or implement the testing
187 fa382f9e Stavros Sachtouris
of new behavior before implementing the behavior itself. The aim for
188 fa382f9e Stavros Sachtouris
kamaki.clients package is an 1 to 1 mapping between methods and their tests.
189 4054c46d Stavros Sachtouris
190 4054c46d Stavros Sachtouris
Modifying an existing method
191 4054c46d Stavros Sachtouris
""""""""""""""""""""""""""""
192 4054c46d Stavros Sachtouris
193 fa382f9e Stavros Sachtouris
In case of an existing method modification, the programmer has to modify the
194 fa382f9e Stavros Sachtouris
corresponding test as well. By convention, the test method is located in the
195 fa382f9e Stavros Sachtouris
test module under the same package, in a TestCase subclass that is named with a
196 fa382f9e Stavros Sachtouris
name similar to the package or class that contains the tested method.
197 4054c46d Stavros Sachtouris
198 fa382f9e Stavros Sachtouris
Example 1: to modify the kamaki.clients.utils.filter_in method, the programmer
199 fa382f9e Stavros Sachtouris
has to also adjust the kamaki.clients.utils.test.Utils.test_filter_in method.
200 4054c46d Stavros Sachtouris
201 fa382f9e Stavros Sachtouris
Example 2: to modify the kamaki.clients.pithos.PithosRestClient.object_get, the
202 fa382f9e Stavros Sachtouris
programmer has to also adjust the
203 fa382f9e Stavros Sachtouris
kamaki.clients.pithos.test.PithosRestClient.test_object_get method.
204 4054c46d Stavros Sachtouris
205 4054c46d Stavros Sachtouris
Adding a new method
206 4054c46d Stavros Sachtouris
"""""""""""""""""""
207 4054c46d Stavros Sachtouris
208 fa382f9e Stavros Sachtouris
Programmers who want to implement a new method in an existing class, are
209 fa382f9e Stavros Sachtouris
encouraged to implement the corresponding unit test first. In order to do that,
210 fa382f9e Stavros Sachtouris
they should find the testing class that is mapped to the class or module they
211 fa382f9e Stavros Sachtouris
need to extend.
212 4054c46d Stavros Sachtouris
213 fa382f9e Stavros Sachtouris
Example 1: To add a **list_special** method to
214 fa382f9e Stavros Sachtouris
kamaki.clients.astakos.AstakosClient, extend the
215 fa382f9e Stavros Sachtouris
kamaki.clients.astakos.test.AstakosClient class, as shown bellow:
216 4054c46d Stavros Sachtouris
217 4054c46d Stavros Sachtouris
.. code-block:: python
218 4054c46d Stavros Sachtouris
219 3c50df2e Stavros Sachtouris
    # file: ${kamaki}/kamaki/clients/astakos/__init__.py
220 3c50df2e Stavros Sachtouris
221 3c50df2e Stavros Sachtouris
    class AstakosClient(TestCase):
222 4054c46d Stavros Sachtouris
        ...
223 4054c46d Stavros Sachtouris
        def test_list_special(self):
224 4054c46d Stavros Sachtouris
            """Test the list_special method"""
225 4054c46d Stavros Sachtouris
            ...
226 4054c46d Stavros Sachtouris
227 fa382f9e Stavros Sachtouris
Example 2: To add a **get_random_int** method in kamaki.clients.utils module,
228 fa382f9e Stavros Sachtouris
extend the kamaki.clients.utils.test.Utils test class, as shown bellow:
229 4054c46d Stavros Sachtouris
230 4054c46d Stavros Sachtouris
.. code-block:: python
231 4054c46d Stavros Sachtouris
232 3c50df2e Stavros Sachtouris
    # file: ${kamaki}/kamaki/clients/utils/__init__.py
233 3c50df2e Stavros Sachtouris
234 4054c46d Stavros Sachtouris
    class Utils(TestCase):
235 4054c46d Stavros Sachtouris
        ...
236 4054c46d Stavros Sachtouris
        def test_get_random_int(self):
237 4054c46d Stavros Sachtouris
            """Test the get_random_int method"""
238 4054c46d Stavros Sachtouris
            ...
239 4054c46d Stavros Sachtouris
240 4054c46d Stavros Sachtouris
Implementing a new class or module
241 4054c46d Stavros Sachtouris
""""""""""""""""""""""""""""""""""
242 4054c46d Stavros Sachtouris
243 fa382f9e Stavros Sachtouris
Each class or module needs a seperate test sub-module. By convention, each
244 fa382f9e Stavros Sachtouris
class or module under the kamaki.clients should be located in a separate
245 fa382f9e Stavros Sachtouris
directory.
246 4054c46d Stavros Sachtouris
247 fa382f9e Stavros Sachtouris
Example 1: To add a NewService class that implements the kamaki.clients.Client
248 fa382f9e Stavros Sachtouris
interface: 
249 4054c46d Stavros Sachtouris
250 4054c46d Stavros Sachtouris
* create a new_service package and implement the unit tests in the kamaki.clients.new_service.test module:
251 4054c46d Stavros Sachtouris
252 4054c46d Stavros Sachtouris
.. code-block:: console
253 4054c46d Stavros Sachtouris
254 4054c46d Stavros Sachtouris
    $ mkdir new_service && touch new_service/test.py
255 4054c46d Stavros Sachtouris
256 4054c46d Stavros Sachtouris
* create the file that will hold the package code and implement the module there:
257 4054c46d Stavros Sachtouris
258 4054c46d Stavros Sachtouris
.. code-block:: console
259 4054c46d Stavros Sachtouris
260 4054c46d Stavros Sachtouris
    $ touch new_service/__init__.py
261 4054c46d Stavros Sachtouris
262 4054c46d Stavros Sachtouris
* Create the test class and methods in kamaki.clients.new_service.test
263 4054c46d Stavros Sachtouris
264 4054c46d Stavros Sachtouris
.. code-block:: python
265 4054c46d Stavros Sachtouris
266 3c50df2e Stavros Sachtouris
    # file: ${kamaki}/kamaki/clients/new_service/test.py
267 4054c46d Stavros Sachtouris
    from unittest import TestCase
268 4054c46d Stavros Sachtouris
269 3c50df2e Stavros Sachtouris
    class NewService(TestCase):
270 4054c46d Stavros Sachtouris
271 4054c46d Stavros Sachtouris
        def test_method1(self):
272 4054c46d Stavros Sachtouris
            ...
273 4054c46d Stavros Sachtouris
274 4054c46d Stavros Sachtouris
* Create the NewService and its actual functionality in kamaki.clients.new_service
275 4054c46d Stavros Sachtouris
276 4054c46d Stavros Sachtouris
.. code-block:: python
277 4054c46d Stavros Sachtouris
278 3c50df2e Stavros Sachtouris
    # file: ${kamaki}/kamaki/clients/new_service/__init__.py
279 4054c46d Stavros Sachtouris
    from kamaki.clients import Client
280 4054c46d Stavros Sachtouris
281 4054c46d Stavros Sachtouris
    class NewService(Client):
282 4054c46d Stavros Sachtouris
283 4054c46d Stavros Sachtouris
        def method1(self, ...):
284 4054c46d Stavros Sachtouris
            ...
285 4054c46d Stavros Sachtouris
286 4054c46d Stavros Sachtouris
* Expose the new tests to top test module, by importing the test class to kamaki.clients.test
287 4054c46d Stavros Sachtouris
288 4054c46d Stavros Sachtouris
..code-block:: python
289 4054c46d Stavros Sachtouris
290 3c50df2e Stavros Sachtouris
    # file: ${kamaki}/kamaki/clients/test.py
291 3c50df2e Stavros Sachtouris
292 4054c46d Stavros Sachtouris
    from kamaki.clients.new_service.test import NewService
293 4054c46d Stavros Sachtouris
294 fa382f9e Stavros Sachtouris
.. note:: If the new class or module is part of an existing sub-package, it is
295 fa382f9e Stavros Sachtouris
    acceptable to append its testing class in the existing test.py file of the
296 fa382f9e Stavros Sachtouris
    sub-package it belongs to. For example, the
297 fa382f9e Stavros Sachtouris
    kamaki.clients.pithos.PithosClient and
298 fa382f9e Stavros Sachtouris
    kamaki.clients.pithos.rest_api.PithosRestClient classes are tested by two
299 fa382f9e Stavros Sachtouris
    different classes (PithosClient and PithosRestClient respectively) in the
300 fa382f9e Stavros Sachtouris
    same module (kamaki.clients.pithos.test).