root / docs / developers / showcase.rst @ e6ce9ae1
History | View | Annotate | Download (27.5 kB)
1 | 6c068db6 | Stavros Sachtouris | |
---|---|---|---|
2 | 6c068db6 | Stavros Sachtouris | Showcase: create a virtual cluster from scratch |
3 | 6c068db6 | Stavros Sachtouris | =============================================== |
4 | 6c068db6 | Stavros Sachtouris | |
5 | 6c068db6 | Stavros Sachtouris | In this section we will create a virtual cluster, from scratch. |
6 | 6c068db6 | Stavros Sachtouris | |
7 | 6c068db6 | Stavros Sachtouris | Requirements: |
8 | 6c068db6 | Stavros Sachtouris | |
9 | 6c068db6 | Stavros Sachtouris | * A `synnefo <http://www.synnefo.org>`_ deployment with functional *Astakos*, |
10 | 6c068db6 | Stavros Sachtouris | *Pithos+*, *Plankton* and *Cyclades* services. |
11 | 6c068db6 | Stavros Sachtouris | |
12 | 6c068db6 | Stavros Sachtouris | * A kamaki setup, configured with a default cloud (see how to do this with |
13 | 6c068db6 | Stavros Sachtouris | kamaki as a |
14 | 6c068db6 | Stavros Sachtouris | `shell command <../examplesdir/configuration.html#multiple-clouds-in-a-single-configuration>`_ , |
15 | 6c068db6 | Stavros Sachtouris | or a |
16 | 6c068db6 | Stavros Sachtouris | `python library <config.html#set-a-new-cloud-name-it-new-cloud-and-set-it-as-default>`_. |
17 | 6c068db6 | Stavros Sachtouris | |
18 | 6c068db6 | Stavros Sachtouris | * An image stored at file *./my_image.diskdump* that can run on a predefined |
19 | 6c068db6 | Stavros Sachtouris | hardware flavor, identifiable by the flavor id *42* (see how to create an |
20 | 6c068db6 | Stavros Sachtouris | image with the |
21 | 6c068db6 | Stavros Sachtouris | `synnefo image creator <http://www.synnefo.org/docs/snf-image-creator/latest/index.html>`_ |
22 | 6c068db6 | Stavros Sachtouris | ). |
23 | 6c068db6 | Stavros Sachtouris | |
24 | 6c068db6 | Stavros Sachtouris | This is the pseudocode: |
25 | 6c068db6 | Stavros Sachtouris | |
26 | 6c068db6 | Stavros Sachtouris | #. Get credentials and service endpoints, with kamaki config and the |
27 | 6c068db6 | Stavros Sachtouris | **Astakos** *identity* and *account* services |
28 | 6c068db6 | Stavros Sachtouris | #. Upload the image file to the **Pithos+** *object-store* service |
29 | 6c068db6 | Stavros Sachtouris | #. Register the image file to the **Plankton** *image* service |
30 | 6c068db6 | Stavros Sachtouris | #. Create a number of virtual servers to the **Cyclades** *compute* service |
31 | 6c068db6 | Stavros Sachtouris | |
32 | 6c068db6 | Stavros Sachtouris | |
33 | 6c068db6 | Stavros Sachtouris | Credentials and endpoints |
34 | 6c068db6 | Stavros Sachtouris | ------------------------- |
35 | 6c068db6 | Stavros Sachtouris | |
36 | 6c068db6 | Stavros Sachtouris | We assume that the kamaki configuration file contains at least one cloud |
37 | 6c068db6 | Stavros Sachtouris | configuration, and this configuration is also set as the default cloud for |
38 | 6c068db6 | Stavros Sachtouris | kamaki. A cloud configuration is basically a name for the cloud, an |
39 | 6c068db6 | Stavros Sachtouris | authentication URL and an authentication TOKEN: the credentials we are looking |
40 | 6c068db6 | Stavros Sachtouris | for! |
41 | 6c068db6 | Stavros Sachtouris | |
42 | 6c068db6 | Stavros Sachtouris | This is the plan: |
43 | 6c068db6 | Stavros Sachtouris | |
44 | 6c068db6 | Stavros Sachtouris | #. Get the credentials from the kamaki configuration |
45 | 6c068db6 | Stavros Sachtouris | #. Initialize an AstakosClient and test the credentials |
46 | 6c068db6 | Stavros Sachtouris | #. Get the endpoints for all services |
47 | 6c068db6 | Stavros Sachtouris | |
48 | 6c068db6 | Stavros Sachtouris | .. code-block:: python |
49 | 6c068db6 | Stavros Sachtouris | |
50 | 6c068db6 | Stavros Sachtouris | from sys import stderr |
51 | 6c068db6 | Stavros Sachtouris | from kamaki.cli.config import Config, CONFIG_PATH |
52 | e6ce9ae1 | Stavros Sachtouris | from kamaki.clients import ClientError |
53 | e6ce9ae1 | Stavros Sachtouris | from kamaki.clients.astakos import AstakosClient |
54 | 6c068db6 | Stavros Sachtouris | |
55 | 6c068db6 | Stavros Sachtouris | # Initialize Config with default values. |
56 | 6c068db6 | Stavros Sachtouris | cnf = Config() |
57 | 6c068db6 | Stavros Sachtouris | |
58 | 6c068db6 | Stavros Sachtouris | # 1. Get the credentials |
59 | e6ce9ae1 | Stavros Sachtouris | # Get default cloud name |
60 | e6ce9ae1 | Stavros Sachtouris | cloud_name = cnf.get('global', 'default_cloud') |
61 | e6ce9ae1 | Stavros Sachtouris | assert cloud_name, 'No default_cloud in file %s\n' % CONFIG_PATH |
62 | 6c068db6 | Stavros Sachtouris | |
63 | 6c068db6 | Stavros Sachtouris | # Get cloud authentication URL and TOKEN |
64 | 6c068db6 | Stavros Sachtouris | try: |
65 | 6c068db6 | Stavros Sachtouris | AUTH_URL = cnf.get_cloud(cloud_name, 'url') |
66 | 6c068db6 | Stavros Sachtouris | except KeyError: |
67 | 6c068db6 | Stavros Sachtouris | stderr.write('No authentication URL in cloud %s\n' % cloud_name) |
68 | 6c068db6 | Stavros Sachtouris | raise |
69 | 6c068db6 | Stavros Sachtouris | try: |
70 | 6c068db6 | Stavros Sachtouris | AUTH_TOKEN = cnf.get_cloud(cloud_name, 'token') |
71 | 6c068db6 | Stavros Sachtouris | except KeyError: |
72 | 6c068db6 | Stavros Sachtouris | stderr.write('No token in cloud %s\n' % cloud_name) |
73 | 6c068db6 | Stavros Sachtouris | raise |
74 | 6c068db6 | Stavros Sachtouris | |
75 | 6c068db6 | Stavros Sachtouris | # 2. Test the credentials |
76 | 6c068db6 | Stavros Sachtouris | # Test authentication credentials |
77 | 6c068db6 | Stavros Sachtouris | try: |
78 | 6c068db6 | Stavros Sachtouris | auth = AstakosClient(AUTH_URL, AUTH_TOKEN) |
79 | 6c068db6 | Stavros Sachtouris | auth.authenticate() |
80 | 6c068db6 | Stavros Sachtouris | except ClientError: |
81 | 6c068db6 | Stavros Sachtouris | stderr.write('Athentication failed with url %s and token %s\n' % ( |
82 | 6c068db6 | Stavros Sachtouris | AUTH_URL, AUTH_TOKEN)) |
83 | 6c068db6 | Stavros Sachtouris | raise |
84 | 6c068db6 | Stavros Sachtouris | |
85 | 6c068db6 | Stavros Sachtouris | # 3. Get the endpoints |
86 | 6c068db6 | Stavros Sachtouris | # Identity, Account --> astakos |
87 | e6ce9ae1 | Stavros Sachtouris | # Compute, Network --> cyclades |
88 | 6c068db6 | Stavros Sachtouris | # Object-store --> pithos |
89 | 6c068db6 | Stavros Sachtouris | # Image --> plankton |
90 | 6c068db6 | Stavros Sachtouris | try: |
91 | 6c068db6 | Stavros Sachtouris | endpoints = dict( |
92 | 6c068db6 | Stavros Sachtouris | astakos=AUTH_URL, |
93 | 6c068db6 | Stavros Sachtouris | cyclades=auth.get_service_endpoints('compute')['publicURL'], |
94 | e6ce9ae1 | Stavros Sachtouris | network=auth.get_service_endpoints('network')['publicURL'], |
95 | 6c068db6 | Stavros Sachtouris | pithos=auth.get_service_endpoints('object-store')['publicURL'], |
96 | 6c068db6 | Stavros Sachtouris | plankton=auth.get_service_endpoints('image')['publicURL'] |
97 | 6c068db6 | Stavros Sachtouris | ) |
98 | e6ce9ae1 | Stavros Sachtouris | user_id = auth.user_info['id'] |
99 | 6c068db6 | Stavros Sachtouris | except ClientError: |
100 | 6c068db6 | Stavros Sachtouris | stderr.write( |
101 | 6c068db6 | Stavros Sachtouris | 'Failed to get user id and endpoints from the identity server\n') |
102 | 6c068db6 | Stavros Sachtouris | raise |
103 | 6c068db6 | Stavros Sachtouris | |
104 | e6ce9ae1 | Stavros Sachtouris | # 4. Pretty print the results |
105 | e6ce9ae1 | Stavros Sachtouris | stderr.write('Endpoints for user with id %s\n' % user_id) |
106 | e6ce9ae1 | Stavros Sachtouris | for k, v in endpoints.items(): |
107 | e6ce9ae1 | Stavros Sachtouris | stderr.write('\t%s:\t%s\n' % (k, v)) |
108 | e6ce9ae1 | Stavros Sachtouris | |
109 | e6ce9ae1 | Stavros Sachtouris | The output of this script should look similar to this:: |
110 | e6ce9ae1 | Stavros Sachtouris | |
111 | e6ce9ae1 | Stavros Sachtouris | Endpoints for user with id my-us3r-1d-asdf-1234-fd324rt |
112 | e6ce9ae1 | Stavros Sachtouris | pithos: https://pithos.example.com/object-store/v1 |
113 | e6ce9ae1 | Stavros Sachtouris | plankton: https://cyclades.example.com/image/v1.0 |
114 | e6ce9ae1 | Stavros Sachtouris | network: https://cyclades.example.com/network/v2.0 |
115 | e6ce9ae1 | Stavros Sachtouris | cyclades: https://cyclades.example.com/compute/v2.0 |
116 | e6ce9ae1 | Stavros Sachtouris | astakos: https://accounts.example.com/identity/v2.0 |
117 | e6ce9ae1 | Stavros Sachtouris | |
118 | e6ce9ae1 | Stavros Sachtouris | |
119 | e6ce9ae1 | Stavros Sachtouris | |
120 | 6c068db6 | Stavros Sachtouris | Upload the image |
121 | 6c068db6 | Stavros Sachtouris | ---------------- |
122 | 6c068db6 | Stavros Sachtouris | |
123 | 6c068db6 | Stavros Sachtouris | We assume there is an image file at the current local directory, at |
124 | 6c068db6 | Stavros Sachtouris | *./my_image.diskdump* and we need to upload it to a Pithos+ container. We also |
125 | 6c068db6 | Stavros Sachtouris | assume the contains does not currently exist. We will name it *images*. |
126 | 6c068db6 | Stavros Sachtouris | |
127 | 6c068db6 | Stavros Sachtouris | This is the plan: |
128 | 6c068db6 | Stavros Sachtouris | |
129 | 6c068db6 | Stavros Sachtouris | #. Initialize a Pithos+ client |
130 | 6c068db6 | Stavros Sachtouris | #. Create the container *images* |
131 | 6c068db6 | Stavros Sachtouris | #. Upload the local file to the container |
132 | 6c068db6 | Stavros Sachtouris | |
133 | 6c068db6 | Stavros Sachtouris | .. code-block:: python |
134 | 6c068db6 | Stavros Sachtouris | |
135 | 6c068db6 | Stavros Sachtouris | from os.path import abspath |
136 | 6c068db6 | Stavros Sachtouris | from kamaki.clients.pithos import PithosClient |
137 | 6c068db6 | Stavros Sachtouris | |
138 | 6c068db6 | Stavros Sachtouris | CONTAINER = 'images' |
139 | 6c068db6 | Stavros Sachtouris | IMAGE_FILE = 'my_image.diskdump' |
140 | 6c068db6 | Stavros Sachtouris | |
141 | e6ce9ae1 | Stavros Sachtouris | |
142 | 6c068db6 | Stavros Sachtouris | # 1. Initialize Pithos+ client and set account to current user |
143 | 6c068db6 | Stavros Sachtouris | try: |
144 | 6c068db6 | Stavros Sachtouris | pithos = PithosClient(endpoints['pithos'], AUTH_TOKEN) |
145 | 6c068db6 | Stavros Sachtouris | except ClientError: |
146 | 6c068db6 | Stavros Sachtouris | stderr.write('Failed to initialize a Pithos+ client\n') |
147 | 6c068db6 | Stavros Sachtouris | raise |
148 | 6c068db6 | Stavros Sachtouris | pithos.account = user_id |
149 | 6c068db6 | Stavros Sachtouris | |
150 | 6c068db6 | Stavros Sachtouris | # 2. Create the container "images" and let pithos client work with that |
151 | 6c068db6 | Stavros Sachtouris | try: |
152 | e6ce9ae1 | Stavros Sachtouris | pithos.create_container(CONTAINER) |
153 | 6c068db6 | Stavros Sachtouris | except ClientError: |
154 | e6ce9ae1 | Stavros Sachtouris | stderr.write('Failed to create container %s\n' % CONTAINER) |
155 | 6c068db6 | Stavros Sachtouris | raise |
156 | 6c068db6 | Stavros Sachtouris | pithos.container = CONTAINER |
157 | 6c068db6 | Stavros Sachtouris | |
158 | 6c068db6 | Stavros Sachtouris | # 3. Upload |
159 | 6c068db6 | Stavros Sachtouris | with open(abspath(IMAGE_FILE)) as f: |
160 | 6c068db6 | Stavros Sachtouris | try: |
161 | e6ce9ae1 | Stavros Sachtouris | stderr.write('This may take a while ...') |
162 | 6c068db6 | Stavros Sachtouris | pithos.upload_object(IMAGE_FILE, f) |
163 | 6c068db6 | Stavros Sachtouris | except ClientError: |
164 | 6c068db6 | Stavros Sachtouris | stderr.write('Failed to upload file %s to container %s\n' % ( |
165 | 6c068db6 | Stavros Sachtouris | IMAGE_FILE, CONTAINER)) |
166 | 6c068db6 | Stavros Sachtouris | raise |
167 | 6c068db6 | Stavros Sachtouris | |
168 | 6c068db6 | Stavros Sachtouris | Register the image |
169 | 6c068db6 | Stavros Sachtouris | ------------------ |
170 | 6c068db6 | Stavros Sachtouris | |
171 | 6c068db6 | Stavros Sachtouris | Now the image is located at *pithos://<user_id>/images/my_image.diskdump* |
172 | 6c068db6 | Stavros Sachtouris | and we want to register it to the Plankton *image* service. |
173 | 6c068db6 | Stavros Sachtouris | |
174 | 6c068db6 | Stavros Sachtouris | .. code-block:: python |
175 | 6c068db6 | Stavros Sachtouris | |
176 | 6c068db6 | Stavros Sachtouris | from kamaki.clients.image import ImageClient |
177 | 6c068db6 | Stavros Sachtouris | |
178 | 6c068db6 | Stavros Sachtouris | IMAGE_NAME = 'My image' |
179 | 6c068db6 | Stavros Sachtouris | IMAGE_LOCATION = (user_id, CONTAINER, IMAGE_FILE) |
180 | 6c068db6 | Stavros Sachtouris | |
181 | 6c068db6 | Stavros Sachtouris | # 3.1 Initialize ImageClient |
182 | 6c068db6 | Stavros Sachtouris | try: |
183 | 6c068db6 | Stavros Sachtouris | plankton = ImageClient(endpoints['plankton'], AUTH_TOKEN) |
184 | 6c068db6 | Stavros Sachtouris | except ClientError: |
185 | 6c068db6 | Stavros Sachtouris | stderr.write('Failed to initialize the Image client client\n') |
186 | 6c068db6 | Stavros Sachtouris | raise |
187 | 6c068db6 | Stavros Sachtouris | |
188 | 6c068db6 | Stavros Sachtouris | # 3.2 Register the image |
189 | 278fae3f | Stavros Sachtouris | properties = dict(osfamily='linux', root_partition='1') |
190 | 6c068db6 | Stavros Sachtouris | try: |
191 | e6ce9ae1 | Stavros Sachtouris | image = plankton.register(IMAGE_NAME, IMAGE_LOCATION) |
192 | 6c068db6 | Stavros Sachtouris | except ClientError: |
193 | 6c068db6 | Stavros Sachtouris | stderr.write('Failed to register image %s\n' % IMAGE_NAME) |
194 | 6c068db6 | Stavros Sachtouris | raise |
195 | 6c068db6 | Stavros Sachtouris | |
196 | 6c068db6 | Stavros Sachtouris | Create the virtual cluster |
197 | 6c068db6 | Stavros Sachtouris | -------------------------- |
198 | 6c068db6 | Stavros Sachtouris | |
199 | 6c068db6 | Stavros Sachtouris | In order to build a virtual cluster, we need some information: |
200 | 6c068db6 | Stavros Sachtouris | |
201 | 6c068db6 | Stavros Sachtouris | * an image id. We can get them from *image['id']* (the id of the image we |
202 | 6c068db6 | Stavros Sachtouris | have just created) |
203 | 6c068db6 | Stavros Sachtouris | * a hardware flavor. Assume we have picked the flavor with id *42* |
204 | 6c068db6 | Stavros Sachtouris | * a set of names for our virtual servers. We will name them *cluster1*, |
205 | 6c068db6 | Stavros Sachtouris | *cluster2*, etc. |
206 | 6c068db6 | Stavros Sachtouris | |
207 | 6c068db6 | Stavros Sachtouris | Here is the plan: |
208 | 6c068db6 | Stavros Sachtouris | |
209 | 6c068db6 | Stavros Sachtouris | #. Initialize a Cyclades/Compute client |
210 | 6c068db6 | Stavros Sachtouris | #. Create a number of virtual servers. Their name should be prefixed as |
211 | 6c068db6 | Stavros Sachtouris | "cluster" |
212 | 6c068db6 | Stavros Sachtouris | |
213 | 6c068db6 | Stavros Sachtouris | .. code-block:: python |
214 | 6c068db6 | Stavros Sachtouris | |
215 | 6c068db6 | Stavros Sachtouris | # 4. Create virtual cluster |
216 | 6c068db6 | Stavros Sachtouris | from kamaki.clients.cyclades import CycladesClient |
217 | 6c068db6 | Stavros Sachtouris | |
218 | 6c068db6 | Stavros Sachtouris | FLAVOR_ID = 42 |
219 | 6c068db6 | Stavros Sachtouris | IMAGE_ID = image['id'] |
220 | 6c068db6 | Stavros Sachtouris | CLUSTER_SIZE = 2 |
221 | 6c068db6 | Stavros Sachtouris | CLUSTER_PREFIX = 'cluster' |
222 | 6c068db6 | Stavros Sachtouris | |
223 | 6c068db6 | Stavros Sachtouris | # 4.1 Initialize a cyclades client |
224 | 6c068db6 | Stavros Sachtouris | try: |
225 | 6c068db6 | Stavros Sachtouris | cyclades = CycladesClient(endpoints['cyclades'], AUTH_TOKEN) |
226 | 6c068db6 | Stavros Sachtouris | except ClientError: |
227 | 6c068db6 | Stavros Sachtouris | stderr.write('Failed to initialize cyclades client\n') |
228 | 6c068db6 | Stavros Sachtouris | raise |
229 | 6c068db6 | Stavros Sachtouris | |
230 | 6c068db6 | Stavros Sachtouris | # 4.2 Create 2 servers prefixed as "cluster" |
231 | 6c068db6 | Stavros Sachtouris | servers = [] |
232 | 6c068db6 | Stavros Sachtouris | for i in range(1, CLUSTER_SIZE + 1): |
233 | 6c068db6 | Stavros Sachtouris | server_name = '%s%s' % (CLUSTER_PREFIX, i) |
234 | 6c068db6 | Stavros Sachtouris | try: |
235 | 6c068db6 | Stavros Sachtouris | servers.append( |
236 | 6c068db6 | Stavros Sachtouris | cyclades.create_server(server_name, FLAVOR_ID, IMAGE_ID)) |
237 | 6c068db6 | Stavros Sachtouris | except ClientError: |
238 | 6c068db6 | Stavros Sachtouris | stderr.write('Failed while creating server %s\n' % server_name) |
239 | 6c068db6 | Stavros Sachtouris | raise |
240 | 6c068db6 | Stavros Sachtouris | |
241 | 6c068db6 | Stavros Sachtouris | Some improvements |
242 | 6c068db6 | Stavros Sachtouris | ----------------- |
243 | 6c068db6 | Stavros Sachtouris | |
244 | 6c068db6 | Stavros Sachtouris | Progress Bars |
245 | 6c068db6 | Stavros Sachtouris | ''''''''''''' |
246 | 6c068db6 | Stavros Sachtouris | |
247 | 6c068db6 | Stavros Sachtouris | Uploading an image might take a while. You can wait patiently, or you can use a |
248 | 6c068db6 | Stavros Sachtouris | progress generator. Even better, combine a generator with the progress bar |
249 | 6c068db6 | Stavros Sachtouris | package that comes with kamaki. The upload_object method accepts two generators |
250 | 6c068db6 | Stavros Sachtouris | as parameters: one for calculating local file hashes and another for uploading |
251 | 6c068db6 | Stavros Sachtouris | |
252 | 6c068db6 | Stavros Sachtouris | .. code-block:: python |
253 | 6c068db6 | Stavros Sachtouris | |
254 | 6c068db6 | Stavros Sachtouris | from progress.bar import Bar |
255 | 6c068db6 | Stavros Sachtouris | |
256 | 6c068db6 | Stavros Sachtouris | def hash_gen(n): |
257 | 6c068db6 | Stavros Sachtouris | bar = Bar('Calculating hashes...') |
258 | 6c068db6 | Stavros Sachtouris | for i in bar.iter(range(int(n))): |
259 | 6c068db6 | Stavros Sachtouris | yield |
260 | 6c068db6 | Stavros Sachtouris | yield |
261 | 6c068db6 | Stavros Sachtouris | |
262 | 6c068db6 | Stavros Sachtouris | def upload_gen(n): |
263 | 6c068db6 | Stavros Sachtouris | bar = Bar('Uploading...') |
264 | 6c068db6 | Stavros Sachtouris | for i in bar.iter(range(int(n))): |
265 | 6c068db6 | Stavros Sachtouris | yield |
266 | 6c068db6 | Stavros Sachtouris | yield |
267 | 6c068db6 | Stavros Sachtouris | |
268 | 6c068db6 | Stavros Sachtouris | ... |
269 | 6c068db6 | Stavros Sachtouris | pithos.upload_object( |
270 | 6c068db6 | Stavros Sachtouris | IMAGE_FILE, f, hash_cb=hash_gen, upload_cb=upload_gen) |
271 | 6c068db6 | Stavros Sachtouris | |
272 | 6c068db6 | Stavros Sachtouris | We can create a method to produce progress bar generators, and use it in other |
273 | 6c068db6 | Stavros Sachtouris | methods as well: |
274 | 6c068db6 | Stavros Sachtouris | |
275 | 6c068db6 | Stavros Sachtouris | .. code-block:: python |
276 | 6c068db6 | Stavros Sachtouris | |
277 | 6c068db6 | Stavros Sachtouris | try: |
278 | 6c068db6 | Stavros Sachtouris | from progress.bar import Bar |
279 | 6c068db6 | Stavros Sachtouris | |
280 | 6c068db6 | Stavros Sachtouris | def create_pb(msg): |
281 | 6c068db6 | Stavros Sachtouris | def generator(n): |
282 | 6c068db6 | Stavros Sachtouris | bar=Bar(msg) |
283 | 6c068db6 | Stavros Sachtouris | for i in bar.iter(range(int(n))): |
284 | 6c068db6 | Stavros Sachtouris | yield |
285 | 6c068db6 | Stavros Sachtouris | yield |
286 | 6c068db6 | Stavros Sachtouris | return generator |
287 | 6c068db6 | Stavros Sachtouris | except ImportError: |
288 | 6c068db6 | Stavros Sachtouris | stderr.write('Suggestion: install python-progress\n') |
289 | 6c068db6 | Stavros Sachtouris | def create_pb(msg): |
290 | 6c068db6 | Stavros Sachtouris | return None |
291 | 6c068db6 | Stavros Sachtouris | |
292 | 6c068db6 | Stavros Sachtouris | ... |
293 | 6c068db6 | Stavros Sachtouris | pithos.upload_object( |
294 | 6c068db6 | Stavros Sachtouris | IMAGE_FILE, f, |
295 | 6c068db6 | Stavros Sachtouris | hash_cb=create_pb('Calculating hashes...'), |
296 | 6c068db6 | Stavros Sachtouris | upload_cb=create_pb('Uploading...')) |
297 | 6c068db6 | Stavros Sachtouris | |
298 | 6c068db6 | Stavros Sachtouris | Wait for servers to built |
299 | 6c068db6 | Stavros Sachtouris | ''''''''''''''''''''''''' |
300 | 6c068db6 | Stavros Sachtouris | |
301 | 6c068db6 | Stavros Sachtouris | When a create_server method is finished successfully, a server is being built. |
302 | 6c068db6 | Stavros Sachtouris | Usually, it takes a while for a server to built. Fortunately, there is a wait |
303 | 6c068db6 | Stavros Sachtouris | method in the kamaki cyclades client. It can use a progress bar too! |
304 | 6c068db6 | Stavros Sachtouris | |
305 | 6c068db6 | Stavros Sachtouris | .. code-block:: python |
306 | 6c068db6 | Stavros Sachtouris | |
307 | 6c068db6 | Stavros Sachtouris | # 4.2 Create 2 servers prefixed as "cluster" |
308 | 6c068db6 | Stavros Sachtouris | ... |
309 | 6c068db6 | Stavros Sachtouris | |
310 | 6c068db6 | Stavros Sachtouris | # 4.3 Wait for servers to built |
311 | 6c068db6 | Stavros Sachtouris | for server in servers: |
312 | 6c068db6 | Stavros Sachtouris | cyclades.wait_server(server['id']) |
313 | 6c068db6 | Stavros Sachtouris | |
314 | 6c068db6 | Stavros Sachtouris | Asynchronous server creation |
315 | 6c068db6 | Stavros Sachtouris | '''''''''''''''''''''''''''' |
316 | 6c068db6 | Stavros Sachtouris | |
317 | 6c068db6 | Stavros Sachtouris | In case of a large virtual cluster, it might be faster to spawn the servers |
318 | 6c068db6 | Stavros Sachtouris | with asynchronous requests. Kamaki clients offer an automated mechanism for |
319 | 6c068db6 | Stavros Sachtouris | asynchronous requests. |
320 | 6c068db6 | Stavros Sachtouris | |
321 | 6c068db6 | Stavros Sachtouris | .. code-block:: python |
322 | 6c068db6 | Stavros Sachtouris | |
323 | 6c068db6 | Stavros Sachtouris | # 4.2 Create 2 servers prefixed as "cluster" |
324 | 6c068db6 | Stavros Sachtouris | create_params = [dict( |
325 | 6c068db6 | Stavros Sachtouris | name='%s%s' % (CLUSTER_PREFIX, i), |
326 | 6c068db6 | Stavros Sachtouris | flavor_id=FLAVOR_ID, |
327 | 6c068db6 | Stavros Sachtouris | image_id=IMAGE_ID) for i in range(1, CLUSTER_SIZE + 1)] |
328 | 6c068db6 | Stavros Sachtouris | try: |
329 | 6c068db6 | Stavros Sachtouris | servers = cyclades.async_run(cyclades.create_server, create_params) |
330 | 6c068db6 | Stavros Sachtouris | except ClientError: |
331 | 6c068db6 | Stavros Sachtouris | stderr.write('Failed while creating servers\n') |
332 | 6c068db6 | Stavros Sachtouris | raise |
333 | 6c068db6 | Stavros Sachtouris | |
334 | 6c068db6 | Stavros Sachtouris | Clean up virtual cluster |
335 | 6c068db6 | Stavros Sachtouris | '''''''''''''''''''''''' |
336 | 6c068db6 | Stavros Sachtouris | |
337 | 6c068db6 | Stavros Sachtouris | We need to clean up Cyclades from servers left from previous cluster creations. |
338 | 6c068db6 | Stavros Sachtouris | This clean up will destroy all servers prefixed with "cluster". It will run |
339 | 6c068db6 | Stavros Sachtouris | before the cluster creation: |
340 | 6c068db6 | Stavros Sachtouris | |
341 | 6c068db6 | Stavros Sachtouris | .. code-block:: python |
342 | 6c068db6 | Stavros Sachtouris | |
343 | 6c068db6 | Stavros Sachtouris | # 4.2 Clean up virtual cluster |
344 | 6c068db6 | Stavros Sachtouris | to_delete = [server for server in cyclades.list_servers(detail=True) if ( |
345 | 6c068db6 | Stavros Sachtouris | server['name'].startswith(CLUSTER_PREFIX))] |
346 | 6c068db6 | Stavros Sachtouris | for server in to_delete: |
347 | 6c068db6 | Stavros Sachtouris | cyclades.delete_server(server['id']) |
348 | 6c068db6 | Stavros Sachtouris | for server in to_delete: |
349 | 6c068db6 | Stavros Sachtouris | cyclades.wait_server( |
350 | 6c068db6 | Stavros Sachtouris | server['id'], server['status'], |
351 | 6c068db6 | Stavros Sachtouris | wait_cb=create_pb('Deleting %s...' % server['name'])) |
352 | 6c068db6 | Stavros Sachtouris | |
353 | 6c068db6 | Stavros Sachtouris | # 4.3 Create 2 servers prefixed as "cluster" |
354 | 6c068db6 | Stavros Sachtouris | ... |
355 | 6c068db6 | Stavros Sachtouris | |
356 | 6c068db6 | Stavros Sachtouris | Inject ssh keys |
357 | 6c068db6 | Stavros Sachtouris | ''''''''''''''' |
358 | 6c068db6 | Stavros Sachtouris | |
359 | 6c068db6 | Stavros Sachtouris | When a server is created, the returned value contains a filed "adminPass". This |
360 | 6c068db6 | Stavros Sachtouris | field can be used to manually log into the server. |
361 | 6c068db6 | Stavros Sachtouris | |
362 | 6c068db6 | Stavros Sachtouris | An easier way is to |
363 | 6c068db6 | Stavros Sachtouris | `inject the ssh keys <../examplesdir/server.html#inject-ssh-keys-to-a-debian-server>`_ |
364 | 6c068db6 | Stavros Sachtouris | of the users who are going to use the virtual servers. |
365 | 6c068db6 | Stavros Sachtouris | |
366 | 6c068db6 | Stavros Sachtouris | Assuming that we have collected the keys in a file named *rsa.pub*, we can |
367 | 6c068db6 | Stavros Sachtouris | inject them into each server, with the personality argument |
368 | 6c068db6 | Stavros Sachtouris | |
369 | 6c068db6 | Stavros Sachtouris | .. code-block:: python |
370 | 6c068db6 | Stavros Sachtouris | |
371 | 6c068db6 | Stavros Sachtouris | SSH_KEYS = 'rsa.pub' |
372 | 6c068db6 | Stavros Sachtouris | |
373 | 6c068db6 | Stavros Sachtouris | ... |
374 | 6c068db6 | Stavros Sachtouris | |
375 | 6c068db6 | Stavros Sachtouris | # 4.3 Create 2 servers prefixed as "cluster" |
376 | 7de6773d | Stavros Sachtouris | personality = [] |
377 | 6c068db6 | Stavros Sachtouris | if SSH_KEYS: |
378 | 7de6773d | Stavros Sachtouris | with open(abspath(SSH_KEYS)) as f: |
379 | 7de6773d | Stavros Sachtouris | personality.append(dict( |
380 | 6c068db6 | Stavros Sachtouris | contents=b64encode(f.read()), |
381 | 6c068db6 | Stavros Sachtouris | path='/root/.ssh/authorized_keys', |
382 | 5cd1aee1 | Stavros Sachtouris | owner='root', group='root', mode=0600) |
383 | 7de6773d | Stavros Sachtouris | personality.append(dict( |
384 | 7de6773d | Stavros Sachtouris | contents=b64encode('StrictHostKeyChecking no'), |
385 | 7de6773d | Stavros Sachtouris | path='/root/.ssh/config', |
386 | 5cd1aee1 | Stavros Sachtouris | owner='root', group='root', mode=0600)) |
387 | 7de6773d | Stavros Sachtouris | |
388 | 6c068db6 | Stavros Sachtouris | create_params = [dict( |
389 | 6c068db6 | Stavros Sachtouris | name='%s%s' % (CLUSTER_PREFIX, i), |
390 | 6c068db6 | Stavros Sachtouris | flavor_id=FLAVOR_ID, |
391 | 6c068db6 | Stavros Sachtouris | image_id=IMAGE_ID, |
392 | 6c068db6 | Stavros Sachtouris | personality=personality) for i in range(1, CLUSTER_SIZE + 1)] |
393 | 6c068db6 | Stavros Sachtouris | ... |
394 | 6c068db6 | Stavros Sachtouris | |
395 | 7de6773d | Stavros Sachtouris | Save server passwords in a file |
396 | 7de6773d | Stavros Sachtouris | ''''''''''''''''''''''''''''''' |
397 | 7de6773d | Stavros Sachtouris | |
398 | 7de6773d | Stavros Sachtouris | A last touch: define a local file to store the created server information, |
399 | 7de6773d | Stavros Sachtouris | including the superuser password. |
400 | 7de6773d | Stavros Sachtouris | |
401 | 7de6773d | Stavros Sachtouris | .. code-block:: python |
402 | 7de6773d | Stavros Sachtouris | |
403 | 7de6773d | Stavros Sachtouris | # 4.4 Store passwords in file |
404 | 7de6773d | Stavros Sachtouris | SERVER_INFO = 'servers.txt' |
405 | 7de6773d | Stavros Sachtouris | with open(abspath(SERVER_INFO), 'w+') as f: |
406 | 7de6773d | Stavros Sachtouris | from json import dump |
407 | 7de6773d | Stavros Sachtouris | dump(servers, f, intend=2) |
408 | 7de6773d | Stavros Sachtouris | |
409 | 7de6773d | Stavros Sachtouris | # 4.5 Wait for 2 servers to built |
410 | 7de6773d | Stavros Sachtouris | ... |
411 | 7de6773d | Stavros Sachtouris | |
412 | 7de6773d | Stavros Sachtouris | Errors and logs |
413 | 7de6773d | Stavros Sachtouris | ''''''''''''''' |
414 | 7de6773d | Stavros Sachtouris | |
415 | 7de6773d | Stavros Sachtouris | Developers may use the kamaki tools for |
416 | 7de6773d | Stavros Sachtouris | `error handling <clients-api.html#error-handling>`_ and |
417 | 7de6773d | Stavros Sachtouris | `logging <logging.html>`_, or implement their own methods. |
418 | 7de6773d | Stavros Sachtouris | |
419 | 7de6773d | Stavros Sachtouris | To demonstrate, we will modify the container creation code to warn users if the |
420 | 7de6773d | Stavros Sachtouris | container already exists. We need a stream logger for the warning and a |
421 | 7de6773d | Stavros Sachtouris | knowledge of the expected return values for the *create_container* method. |
422 | 7de6773d | Stavros Sachtouris | |
423 | 7de6773d | Stavros Sachtouris | First, let's get the logger. |
424 | 7de6773d | Stavros Sachtouris | |
425 | 7de6773d | Stavros Sachtouris | .. code-block:: python |
426 | 7de6773d | Stavros Sachtouris | |
427 | 7de6773d | Stavros Sachtouris | from kamaki.cli.logger import add_stream_logger, get_logger |
428 | 7de6773d | Stavros Sachtouris | |
429 | 7de6773d | Stavros Sachtouris | add_stream_logger(__name__) |
430 | 7de6773d | Stavros Sachtouris | log = get_logger(__name__) |
431 | 7de6773d | Stavros Sachtouris | |
432 | 7de6773d | Stavros Sachtouris | The *create_container* method makes an HTTP request to the pithos server. It |
433 | 7de6773d | Stavros Sachtouris | considers the request succesfull if the status code of the response is 201 |
434 | 7de6773d | Stavros Sachtouris | (created) or 202 (accepted). These status codes mean that the container has |
435 | 7de6773d | Stavros Sachtouris | been created or that it was already there anyway, respectively. |
436 | 7de6773d | Stavros Sachtouris | |
437 | 7de6773d | Stavros Sachtouris | We will force *create_container* to raise an error in case of a 202 response. |
438 | 7de6773d | Stavros Sachtouris | This can be done by instructing *create_container* to accept only 201 as a |
439 | 7de6773d | Stavros Sachtouris | successful status. |
440 | 7de6773d | Stavros Sachtouris | |
441 | 7de6773d | Stavros Sachtouris | .. code-block:: python |
442 | 7de6773d | Stavros Sachtouris | |
443 | 7de6773d | Stavros Sachtouris | try: |
444 | 7de6773d | Stavros Sachtouris | pithos.create_container(CONTAINER, success=(201, )) |
445 | 7de6773d | Stavros Sachtouris | except ClientError as ce: |
446 | 7de6773d | Stavros Sachtouris | if ce.status in (202, ): |
447 | 7de6773d | Stavros Sachtouris | log.warning('Container %s already exists' % CONTAINER') |
448 | 7de6773d | Stavros Sachtouris | else: |
449 | 7de6773d | Stavros Sachtouris | log.debug('Failed to create container %s' % CONTAINER) |
450 | 7de6773d | Stavros Sachtouris | raise |
451 | 7de6773d | Stavros Sachtouris | log.info('Container %s is ready' % CONTAINER) |
452 | 7de6773d | Stavros Sachtouris | |
453 | 7de6773d | Stavros Sachtouris | create a cluster from scratch |
454 | 7de6773d | Stavros Sachtouris | ----------------------------- |
455 | 7de6773d | Stavros Sachtouris | |
456 | 7de6773d | Stavros Sachtouris | We are ready to create a module that uses kamaki to create a cluster from |
457 | 7de6773d | Stavros Sachtouris | scratch. We revised the code by grouping functionality in methods and using |
458 | 7de6773d | Stavros Sachtouris | logging more. We also added some command line interaction candy. |
459 | 7de6773d | Stavros Sachtouris | |
460 | 7de6773d | Stavros Sachtouris | .. code-block:: python |
461 | 7de6773d | Stavros Sachtouris | |
462 | 7de6773d | Stavros Sachtouris | from sys import argv |
463 | 7de6773d | Stavros Sachtouris | from os.path import abspath |
464 | 7de6773d | Stavros Sachtouris | from base64 import b64encode |
465 | 7de6773d | Stavros Sachtouris | from kamaki.clients import ClientError |
466 | 7de6773d | Stavros Sachtouris | from kamaki.cli.logger import get_logger, add_file_logger |
467 | e6ce9ae1 | Stavros Sachtouris | from progress.bar import Bar |
468 | 7de6773d | Stavros Sachtouris | from logging import DEBUG |
469 | 7de6773d | Stavros Sachtouris | |
470 | 7de6773d | Stavros Sachtouris | # Define loggers |
471 | 7de6773d | Stavros Sachtouris | log = get_logger(__name__) |
472 | 7de6773d | Stavros Sachtouris | add_file_logger('kamaki.clients', DEBUG, '%s.log' % __name__) |
473 | 7de6773d | Stavros Sachtouris | add_file_logger(__name__, DEBUG, '%s.log' % __name__) |
474 | 7de6773d | Stavros Sachtouris | |
475 | 7de6773d | Stavros Sachtouris | # Create progress bar generator |
476 | 7de6773d | Stavros Sachtouris | |
477 | e6ce9ae1 | Stavros Sachtouris | |
478 | e6ce9ae1 | Stavros Sachtouris | def create_pb(msg): |
479 | e6ce9ae1 | Stavros Sachtouris | def generator(n): |
480 | e6ce9ae1 | Stavros Sachtouris | bar = Bar(msg) |
481 | e6ce9ae1 | Stavros Sachtouris | for i in bar.iter(range(int(n))): |
482 | 7de6773d | Stavros Sachtouris | yield |
483 | e6ce9ae1 | Stavros Sachtouris | yield |
484 | e6ce9ae1 | Stavros Sachtouris | return generator |
485 | 7de6773d | Stavros Sachtouris | |
486 | 7de6773d | Stavros Sachtouris | |
487 | 7de6773d | Stavros Sachtouris | # kamaki.config |
488 | 7de6773d | Stavros Sachtouris | # Identity,Account / Astakos |
489 | 7de6773d | Stavros Sachtouris | |
490 | 7de6773d | Stavros Sachtouris | def init_astakos(): |
491 | 7de6773d | Stavros Sachtouris | from kamaki.clients.astakos import AstakosClient |
492 | 7de6773d | Stavros Sachtouris | from kamaki.cli.config import Config, CONFIG_PATH |
493 | 7de6773d | Stavros Sachtouris | |
494 | 7de6773d | Stavros Sachtouris | print(' Get the credentials') |
495 | 7de6773d | Stavros Sachtouris | cnf = Config() |
496 | 7de6773d | Stavros Sachtouris | |
497 | 7de6773d | Stavros Sachtouris | # Get default cloud name |
498 | 7de6773d | Stavros Sachtouris | try: |
499 | 7de6773d | Stavros Sachtouris | cloud_name = cnf.get('global', 'default_cloud') |
500 | 7de6773d | Stavros Sachtouris | except KeyError: |
501 | 7de6773d | Stavros Sachtouris | log.debug('No default cloud set in file %' % CONFIG_PATH) |
502 | 7de6773d | Stavros Sachtouris | raise |
503 | 7de6773d | Stavros Sachtouris | |
504 | 7de6773d | Stavros Sachtouris | try: |
505 | 7de6773d | Stavros Sachtouris | AUTH_URL = cnf.get_cloud(cloud_name, 'url') |
506 | 7de6773d | Stavros Sachtouris | except KeyError: |
507 | 7de6773d | Stavros Sachtouris | log.debug('No authentication URL in cloud %s' % cloud_name) |
508 | 7de6773d | Stavros Sachtouris | raise |
509 | 7de6773d | Stavros Sachtouris | try: |
510 | 7de6773d | Stavros Sachtouris | AUTH_TOKEN = cnf.get_cloud(cloud_name, 'token') |
511 | 7de6773d | Stavros Sachtouris | except KeyError: |
512 | 7de6773d | Stavros Sachtouris | log.debug('No token in cloud %s' % cloud_name) |
513 | 7de6773d | Stavros Sachtouris | raise |
514 | 7de6773d | Stavros Sachtouris | |
515 | 7de6773d | Stavros Sachtouris | print(' Test the credentials') |
516 | 7de6773d | Stavros Sachtouris | try: |
517 | 7de6773d | Stavros Sachtouris | auth = AstakosClient(AUTH_URL, AUTH_TOKEN) |
518 | 7de6773d | Stavros Sachtouris | auth.authenticate() |
519 | 7de6773d | Stavros Sachtouris | except ClientError: |
520 | 7de6773d | Stavros Sachtouris | log.debug('Athentication failed with url %s and token %s' % ( |
521 | 7de6773d | Stavros Sachtouris | AUTH_URL, AUTH_TOKEN)) |
522 | 7de6773d | Stavros Sachtouris | raise |
523 | 7de6773d | Stavros Sachtouris | |
524 | 7de6773d | Stavros Sachtouris | return auth, AUTH_TOKEN |
525 | 7de6773d | Stavros Sachtouris | |
526 | 7de6773d | Stavros Sachtouris | |
527 | 7de6773d | Stavros Sachtouris | def endpoints_and_user_id(auth): |
528 | 7de6773d | Stavros Sachtouris | print(' Get the endpoints') |
529 | 7de6773d | Stavros Sachtouris | try: |
530 | 7de6773d | Stavros Sachtouris | endpoints = dict( |
531 | 7de6773d | Stavros Sachtouris | astakos=auth.get_service_endpoints('identity')['publicURL'], |
532 | 7de6773d | Stavros Sachtouris | cyclades=auth.get_service_endpoints('compute')['publicURL'], |
533 | e6ce9ae1 | Stavros Sachtouris | network=auth.get_service_endpoints('network')['publicURL'], |
534 | 7de6773d | Stavros Sachtouris | pithos=auth.get_service_endpoints('object-store')['publicURL'], |
535 | 7de6773d | Stavros Sachtouris | plankton=auth.get_service_endpoints('image')['publicURL'] |
536 | 7de6773d | Stavros Sachtouris | ) |
537 | e6ce9ae1 | Stavros Sachtouris | user_id = auth.user_info['id'] |
538 | 7de6773d | Stavros Sachtouris | except ClientError: |
539 | 7de6773d | Stavros Sachtouris | print('Failed to get endpoints & user_id from identity server') |
540 | 7de6773d | Stavros Sachtouris | raise |
541 | 7de6773d | Stavros Sachtouris | return endpoints, user_id |
542 | 7de6773d | Stavros Sachtouris | |
543 | 7de6773d | Stavros Sachtouris | |
544 | 7de6773d | Stavros Sachtouris | # Object-store / Pithos+ |
545 | 7de6773d | Stavros Sachtouris | |
546 | 7de6773d | Stavros Sachtouris | def init_pithos(endpoint, token, user_id): |
547 | 7de6773d | Stavros Sachtouris | from kamaki.clients.pithos import PithosClient |
548 | 7de6773d | Stavros Sachtouris | |
549 | 7de6773d | Stavros Sachtouris | print(' Initialize Pithos+ client and set account to user uuid') |
550 | 7de6773d | Stavros Sachtouris | try: |
551 | 7de6773d | Stavros Sachtouris | return PithosClient(endpoint, token, user_id) |
552 | 7de6773d | Stavros Sachtouris | except ClientError: |
553 | 7de6773d | Stavros Sachtouris | log.debug('Failed to initialize a Pithos+ client') |
554 | 7de6773d | Stavros Sachtouris | raise |
555 | 7de6773d | Stavros Sachtouris | |
556 | 7de6773d | Stavros Sachtouris | |
557 | 7de6773d | Stavros Sachtouris | def upload_image(pithos, container, image_path): |
558 | 7de6773d | Stavros Sachtouris | |
559 | 7de6773d | Stavros Sachtouris | print(' Create the container "images" and use it') |
560 | 7de6773d | Stavros Sachtouris | try: |
561 | 7de6773d | Stavros Sachtouris | pithos.create_container(container, success=(201, )) |
562 | 7de6773d | Stavros Sachtouris | except ClientError as ce: |
563 | 7de6773d | Stavros Sachtouris | if ce.status in (202, ): |
564 | 7de6773d | Stavros Sachtouris | log.warning('Container %s already exists' % container) |
565 | 7de6773d | Stavros Sachtouris | else: |
566 | 7de6773d | Stavros Sachtouris | log.debug('Failed to create container %s' % container) |
567 | 7de6773d | Stavros Sachtouris | raise |
568 | 7de6773d | Stavros Sachtouris | pithos.container = container |
569 | 7de6773d | Stavros Sachtouris | |
570 | 7de6773d | Stavros Sachtouris | print(' Upload to "images"') |
571 | 7de6773d | Stavros Sachtouris | with open(abspath(image_path)) as f: |
572 | 7de6773d | Stavros Sachtouris | try: |
573 | 7de6773d | Stavros Sachtouris | pithos.upload_object( |
574 | 7de6773d | Stavros Sachtouris | image_path, f, |
575 | 7de6773d | Stavros Sachtouris | hash_cb=create_pb(' Calculating hashes...'), |
576 | 7de6773d | Stavros Sachtouris | upload_cb=create_pb(' Uploading...')) |
577 | 7de6773d | Stavros Sachtouris | except ClientError: |
578 | 7de6773d | Stavros Sachtouris | log.debug('Failed to upload file %s to container %s' % ( |
579 | 7de6773d | Stavros Sachtouris | image_path, container)) |
580 | 7de6773d | Stavros Sachtouris | raise |
581 | 7de6773d | Stavros Sachtouris | |
582 | 7de6773d | Stavros Sachtouris | |
583 | 7de6773d | Stavros Sachtouris | # Image / Plankton |
584 | 7de6773d | Stavros Sachtouris | |
585 | 7de6773d | Stavros Sachtouris | def init_plankton(endpoint, token): |
586 | 7de6773d | Stavros Sachtouris | from kamaki.clients.image import ImageClient |
587 | 7de6773d | Stavros Sachtouris | |
588 | 7de6773d | Stavros Sachtouris | print(' Initialize ImageClient') |
589 | 7de6773d | Stavros Sachtouris | try: |
590 | 7de6773d | Stavros Sachtouris | return ImageClient(endpoint, token) |
591 | 7de6773d | Stavros Sachtouris | except ClientError: |
592 | 7de6773d | Stavros Sachtouris | log.debug('Failed to initialize the Image client') |
593 | 7de6773d | Stavros Sachtouris | raise |
594 | 7de6773d | Stavros Sachtouris | |
595 | 7de6773d | Stavros Sachtouris | |
596 | 278fae3f | Stavros Sachtouris | def register_image(plankton, name, user_id, container, path, properties): |
597 | 7de6773d | Stavros Sachtouris | |
598 | 7de6773d | Stavros Sachtouris | image_location = (user_id, container, path) |
599 | 7de6773d | Stavros Sachtouris | print(' Register the image') |
600 | 7de6773d | Stavros Sachtouris | try: |
601 | e6ce9ae1 | Stavros Sachtouris | return plankton.register(name, image_location, properties) |
602 | 7de6773d | Stavros Sachtouris | except ClientError: |
603 | 7de6773d | Stavros Sachtouris | log.debug('Failed to register image %s' % name) |
604 | 7de6773d | Stavros Sachtouris | raise |
605 | 7de6773d | Stavros Sachtouris | |
606 | 7de6773d | Stavros Sachtouris | |
607 | 7de6773d | Stavros Sachtouris | # Compute / Cyclades |
608 | 7de6773d | Stavros Sachtouris | |
609 | 7de6773d | Stavros Sachtouris | def init_cyclades(endpoint, token): |
610 | 7de6773d | Stavros Sachtouris | from kamaki.clients.cyclades import CycladesClient |
611 | 7de6773d | Stavros Sachtouris | |
612 | 7de6773d | Stavros Sachtouris | print(' Initialize a cyclades client') |
613 | 7de6773d | Stavros Sachtouris | try: |
614 | 7de6773d | Stavros Sachtouris | return CycladesClient(endpoint, token) |
615 | 7de6773d | Stavros Sachtouris | except ClientError: |
616 | 7de6773d | Stavros Sachtouris | log.debug('Failed to initialize cyclades client') |
617 | 7de6773d | Stavros Sachtouris | raise |
618 | 7de6773d | Stavros Sachtouris | |
619 | 7de6773d | Stavros Sachtouris | |
620 | 7de6773d | Stavros Sachtouris | class Cluster(object): |
621 | 7de6773d | Stavros Sachtouris | |
622 | 7de6773d | Stavros Sachtouris | def __init__(self, cyclades, prefix, flavor_id, image_id, size): |
623 | 7de6773d | Stavros Sachtouris | self.client = cyclades |
624 | 7de6773d | Stavros Sachtouris | self.prefix, self.size = prefix, int(size) |
625 | 7de6773d | Stavros Sachtouris | self.flavor_id, self.image_id = flavor_id, image_id |
626 | 7de6773d | Stavros Sachtouris | |
627 | 7de6773d | Stavros Sachtouris | def list(self): |
628 | 7de6773d | Stavros Sachtouris | return [s for s in self.client.list_servers(detail=True) if ( |
629 | 7de6773d | Stavros Sachtouris | s['name'].startswith(self.prefix))] |
630 | 7de6773d | Stavros Sachtouris | |
631 | 7de6773d | Stavros Sachtouris | def clean_up(self): |
632 | 7de6773d | Stavros Sachtouris | to_delete = self.list() |
633 | 7de6773d | Stavros Sachtouris | print(' There are %s servers to clean up' % len(to_delete)) |
634 | 7de6773d | Stavros Sachtouris | for server in to_delete: |
635 | 7de6773d | Stavros Sachtouris | self.client.delete_server(server['id']) |
636 | 7de6773d | Stavros Sachtouris | for server in to_delete: |
637 | 7de6773d | Stavros Sachtouris | self.client.wait_server( |
638 | 7de6773d | Stavros Sachtouris | server['id'], server['status'], |
639 | 7de6773d | Stavros Sachtouris | wait_cb=create_pb(' Deleting %s...' % server['name'])) |
640 | 7de6773d | Stavros Sachtouris | |
641 | 7de6773d | Stavros Sachtouris | def _personality(self, ssh_keys_path='', pub_keys_path=''): |
642 | 7de6773d | Stavros Sachtouris | personality = [] |
643 | 7de6773d | Stavros Sachtouris | if ssh_keys_path: |
644 | 7de6773d | Stavros Sachtouris | with open(abspath(ssh_keys_path)) as f: |
645 | 7de6773d | Stavros Sachtouris | personality.append(dict( |
646 | 7de6773d | Stavros Sachtouris | contents=b64encode(f.read()), |
647 | 7de6773d | Stavros Sachtouris | path='/root/.ssh/id_rsa', |
648 | 5cd1aee1 | Stavros Sachtouris | owner='root', group='root', mode=0600)) |
649 | 7de6773d | Stavros Sachtouris | if pub_keys_path: |
650 | 7de6773d | Stavros Sachtouris | with open(abspath(pub_keys_path)) as f: |
651 | 7de6773d | Stavros Sachtouris | personality.append(dict( |
652 | 7de6773d | Stavros Sachtouris | contents=b64encode(f.read()), |
653 | 7de6773d | Stavros Sachtouris | path='/root/.ssh/authorized_keys', |
654 | 5cd1aee1 | Stavros Sachtouris | owner='root', group='root', mode=0600)) |
655 | 7de6773d | Stavros Sachtouris | if ssh_keys_path or pub_keys_path: |
656 | 7de6773d | Stavros Sachtouris | personality.append(dict( |
657 | 7de6773d | Stavros Sachtouris | contents=b64encode('StrictHostKeyChecking no'), |
658 | 7de6773d | Stavros Sachtouris | path='/root/.ssh/config', |
659 | 5cd1aee1 | Stavros Sachtouris | owner='root', group='root', mode=0600)) |
660 | 7de6773d | Stavros Sachtouris | return personality |
661 | 7de6773d | Stavros Sachtouris | |
662 | 7de6773d | Stavros Sachtouris | def create(self, ssh_k_path='', pub_k_path='', server_log_path=''): |
663 | 7de6773d | Stavros Sachtouris | print('\n Create %s servers prefixed as %s' % ( |
664 | 7de6773d | Stavros Sachtouris | self.size, self.prefix)) |
665 | 7de6773d | Stavros Sachtouris | servers = [] |
666 | 7de6773d | Stavros Sachtouris | for i in range(1, self.size + 1): |
667 | 7de6773d | Stavros Sachtouris | try: |
668 | 7de6773d | Stavros Sachtouris | server_name = '%s%s' % (self.prefix, i) |
669 | e6ce9ae1 | Stavros Sachtouris | |
670 | 7de6773d | Stavros Sachtouris | servers.append(self.client.create_server( |
671 | 7de6773d | Stavros Sachtouris | server_name, self.flavor_id, self.image_id, |
672 | e6ce9ae1 | Stavros Sachtouris | networks=[], |
673 | 7de6773d | Stavros Sachtouris | personality=self._personality(ssh_k_path, pub_k_path))) |
674 | 7de6773d | Stavros Sachtouris | except ClientError: |
675 | 7de6773d | Stavros Sachtouris | log.debug('Failed while creating server %s' % server_name) |
676 | 7de6773d | Stavros Sachtouris | raise |
677 | 7de6773d | Stavros Sachtouris | |
678 | 7de6773d | Stavros Sachtouris | if server_log_path: |
679 | 7de6773d | Stavros Sachtouris | print(' Store passwords in file %s' % server_log_path) |
680 | 7de6773d | Stavros Sachtouris | with open(abspath(server_log_path), 'w+') as f: |
681 | 7de6773d | Stavros Sachtouris | from json import dump |
682 | 7de6773d | Stavros Sachtouris | dump(servers, f, indent=2) |
683 | 7de6773d | Stavros Sachtouris | |
684 | 7de6773d | Stavros Sachtouris | print(' Wait for %s servers to built' % self.size) |
685 | 7de6773d | Stavros Sachtouris | for server in servers: |
686 | 7de6773d | Stavros Sachtouris | new_status = self.client.wait_server( |
687 | 7de6773d | Stavros Sachtouris | server['id'], |
688 | 7de6773d | Stavros Sachtouris | wait_cb=create_pb(' Creating %s...' % server['name'])) |
689 | 7de6773d | Stavros Sachtouris | print(' Status for server %s is %s' % ( |
690 | 7de6773d | Stavros Sachtouris | server['name'], new_status or 'not changed yet')) |
691 | 7de6773d | Stavros Sachtouris | return servers |
692 | 7de6773d | Stavros Sachtouris | |
693 | 7de6773d | Stavros Sachtouris | |
694 | 7de6773d | Stavros Sachtouris | def main(opts): |
695 | 7de6773d | Stavros Sachtouris | |
696 | 7de6773d | Stavros Sachtouris | print('1. Credentials and Endpoints') |
697 | 7de6773d | Stavros Sachtouris | auth, token = init_astakos() |
698 | 7de6773d | Stavros Sachtouris | endpoints, user_id = endpoints_and_user_id(auth) |
699 | 7de6773d | Stavros Sachtouris | |
700 | 7de6773d | Stavros Sachtouris | print('2. Upload the image file') |
701 | 7de6773d | Stavros Sachtouris | pithos = init_pithos(endpoints['pithos'], token, user_id) |
702 | 7de6773d | Stavros Sachtouris | |
703 | 7de6773d | Stavros Sachtouris | upload_image(pithos, opts.container, opts.imagefile) |
704 | 7de6773d | Stavros Sachtouris | |
705 | 7de6773d | Stavros Sachtouris | print('3. Register the image') |
706 | 7de6773d | Stavros Sachtouris | plankton = init_plankton(endpoints['plankton'], token) |
707 | 7de6773d | Stavros Sachtouris | |
708 | 7de6773d | Stavros Sachtouris | image = register_image( |
709 | 278fae3f | Stavros Sachtouris | plankton, 'my image', user_id, opts.container, opts.imagefile, |
710 | 278fae3f | Stavros Sachtouris | properties=dict( |
711 | 278fae3f | Stavros Sachtouris | osfamily=opts.osfamily, root_partition=opts.rootpartition)) |
712 | 7de6773d | Stavros Sachtouris | |
713 | 7de6773d | Stavros Sachtouris | print('4. Create virtual cluster') |
714 | 7de6773d | Stavros Sachtouris | cluster = Cluster( |
715 | e6ce9ae1 | Stavros Sachtouris | cyclades=init_cyclades(endpoints['cyclades'], token), |
716 | 7de6773d | Stavros Sachtouris | prefix=opts.prefix, |
717 | 7de6773d | Stavros Sachtouris | flavor_id=opts.flavorid, |
718 | 7de6773d | Stavros Sachtouris | image_id=image['id'], |
719 | 7de6773d | Stavros Sachtouris | size=opts.clustersize) |
720 | 7de6773d | Stavros Sachtouris | if opts.delete_stale: |
721 | 7de6773d | Stavros Sachtouris | cluster.clean_up() |
722 | 7de6773d | Stavros Sachtouris | servers = cluster.create( |
723 | 7de6773d | Stavros Sachtouris | opts.sshkeypath, opts.pubkeypath, opts.serverlogpath) |
724 | 7de6773d | Stavros Sachtouris | |
725 | 7de6773d | Stavros Sachtouris | # Group servers |
726 | 7de6773d | Stavros Sachtouris | cluster_servers = cluster.list() |
727 | 7de6773d | Stavros Sachtouris | |
728 | 7de6773d | Stavros Sachtouris | active = [s for s in cluster_servers if s['status'] == 'ACTIVE'] |
729 | 7de6773d | Stavros Sachtouris | print('%s cluster servers are ACTIVE' % len(active)) |
730 | 7de6773d | Stavros Sachtouris | |
731 | 7de6773d | Stavros Sachtouris | attached = [s for s in cluster_servers if s['attachments']] |
732 | 7de6773d | Stavros Sachtouris | print('%s cluster servers are attached to networks' % len(attached)) |
733 | 7de6773d | Stavros Sachtouris | |
734 | 7de6773d | Stavros Sachtouris | build = [s for s in cluster_servers if s['status'] == 'BUILD'] |
735 | 7de6773d | Stavros Sachtouris | print('%s cluster servers are being built' % len(build)) |
736 | 7de6773d | Stavros Sachtouris | |
737 | 7de6773d | Stavros Sachtouris | error = [s for s in cluster_servers if s['status'] in ('ERROR')] |
738 | 7de6773d | Stavros Sachtouris | print('%s cluster servers failed (ERROR satus)' % len(error)) |
739 | 7de6773d | Stavros Sachtouris | |
740 | 7de6773d | Stavros Sachtouris | |
741 | 7de6773d | Stavros Sachtouris | if __name__ == '__main__': |
742 | 7de6773d | Stavros Sachtouris | |
743 | 7de6773d | Stavros Sachtouris | # Add some interaction candy |
744 | 7de6773d | Stavros Sachtouris | from optparse import OptionParser |
745 | 7de6773d | Stavros Sachtouris | |
746 | 7de6773d | Stavros Sachtouris | kw = {} |
747 | 7de6773d | Stavros Sachtouris | kw['usage'] = '%prog [options]' |
748 | 7de6773d | Stavros Sachtouris | kw['description'] = '%prog deploys a compute cluster on Synnefo w. kamaki' |
749 | 7de6773d | Stavros Sachtouris | |
750 | 7de6773d | Stavros Sachtouris | parser = OptionParser(**kw) |
751 | 7de6773d | Stavros Sachtouris | parser.disable_interspersed_args() |
752 | 7de6773d | Stavros Sachtouris | parser.add_option('--prefix', |
753 | 7de6773d | Stavros Sachtouris | action='store', type='string', dest='prefix', |
754 | 7de6773d | Stavros Sachtouris | help='The prefix to use for naming cluster nodes', |
755 | e6ce9ae1 | Stavros Sachtouris | default='node') |
756 | 7de6773d | Stavros Sachtouris | parser.add_option('--clustersize', |
757 | 7de6773d | Stavros Sachtouris | action='store', type='string', dest='clustersize', |
758 | 7de6773d | Stavros Sachtouris | help='Number of virtual cluster nodes to create ', |
759 | 7de6773d | Stavros Sachtouris | default=2) |
760 | 7de6773d | Stavros Sachtouris | parser.add_option('--flavor-id', |
761 | 7de6773d | Stavros Sachtouris | action='store', type='int', dest='flavorid', |
762 | 7de6773d | Stavros Sachtouris | metavar='FLAVOR ID', |
763 | 7de6773d | Stavros Sachtouris | help='Choose flavor id for the virtual hardware ' |
764 | 7de6773d | Stavros Sachtouris | 'of cluster nodes', |
765 | 7de6773d | Stavros Sachtouris | default=42) |
766 | 7de6773d | Stavros Sachtouris | parser.add_option('--image-file', |
767 | 7de6773d | Stavros Sachtouris | action='store', type='string', dest='imagefile', |
768 | 7de6773d | Stavros Sachtouris | metavar='IMAGE FILE PATH', |
769 | 7de6773d | Stavros Sachtouris | help='The image file to upload and register ', |
770 | 7de6773d | Stavros Sachtouris | default='my_image.diskdump') |
771 | 7de6773d | Stavros Sachtouris | parser.add_option('--delete-stale', |
772 | 7de6773d | Stavros Sachtouris | action='store_true', dest='delete_stale', |
773 | 7de6773d | Stavros Sachtouris | help='Delete stale servers from previous runs, whose ' |
774 | 7de6773d | Stavros Sachtouris | 'name starts with the specified prefix, see ' |
775 | 7de6773d | Stavros Sachtouris | '--prefix', |
776 | 7de6773d | Stavros Sachtouris | default=False) |
777 | 7de6773d | Stavros Sachtouris | parser.add_option('--container', |
778 | 7de6773d | Stavros Sachtouris | action='store', type='string', dest='container', |
779 | 7de6773d | Stavros Sachtouris | metavar='PITHOS+ CONTAINER', |
780 | 7de6773d | Stavros Sachtouris | help='The Pithos+ container to store image file', |
781 | 7de6773d | Stavros Sachtouris | default='images') |
782 | 7de6773d | Stavros Sachtouris | parser.add_option('--ssh-key-path', |
783 | 7de6773d | Stavros Sachtouris | action='store', type='string', dest='sshkeypath', |
784 | 7de6773d | Stavros Sachtouris | metavar='PATH OF SSH KEYS', |
785 | 7de6773d | Stavros Sachtouris | help='The ssh keys to inject to server (e.g., id_rsa) ', |
786 | 7de6773d | Stavros Sachtouris | default='') |
787 | 7de6773d | Stavros Sachtouris | parser.add_option('--pub-key-path', |
788 | 7de6773d | Stavros Sachtouris | action='store', type='string', dest='pubkeypath', |
789 | 7de6773d | Stavros Sachtouris | metavar='PATH OF PUBLIC KEYS', |
790 | 7de6773d | Stavros Sachtouris | help='The public keys to inject to server', |
791 | 7de6773d | Stavros Sachtouris | default='') |
792 | 7de6773d | Stavros Sachtouris | parser.add_option('--server-log-path', |
793 | 7de6773d | Stavros Sachtouris | action='store', type='string', dest='serverlogpath', |
794 | 7de6773d | Stavros Sachtouris | metavar='FILE TO LOG THE VIRTUAL SERVERS', |
795 | 7de6773d | Stavros Sachtouris | help='Where to store information on created servers ' |
796 | 7de6773d | Stavros Sachtouris | 'including superuser passwords', |
797 | 7de6773d | Stavros Sachtouris | default='') |
798 | 278fae3f | Stavros Sachtouris | parser.add_option('--image-osfamily', |
799 | 278fae3f | Stavros Sachtouris | action='store', type='string', dest='osfamily', |
800 | 278fae3f | Stavros Sachtouris | metavar='OS FAMILY', |
801 | 278fae3f | Stavros Sachtouris | help='linux, windows, etc.', |
802 | 278fae3f | Stavros Sachtouris | default='linux') |
803 | 278fae3f | Stavros Sachtouris | parser.add_option('--image-root-partition', |
804 | 278fae3f | Stavros Sachtouris | action='store', type='string', dest='rootpartition', |
805 | 278fae3f | Stavros Sachtouris | metavar='IMAGE ROOT PARTITION', |
806 | 278fae3f | Stavros Sachtouris | help='The partition where the root home is ', |
807 | 278fae3f | Stavros Sachtouris | default='1') |
808 | 7de6773d | Stavros Sachtouris | |
809 | 7de6773d | Stavros Sachtouris | opts, args = parser.parse_args(argv[1:]) |
810 | 7de6773d | Stavros Sachtouris | |
811 | 7de6773d | Stavros Sachtouris | main(opts) |